Image processing method and device, electronic equipment, storage medium and program product

Abstract

The present disclosure provides an image processing method, device, electronic equipment, storage medium and program product, the method comprising: displaying a first image; wherein the first image contains at least one object; in response to a trigger operation for a target object in the at least one object, generating at least two second images including the target object; wherein the orientation of the target object in different second images is different. In this way, the richness of generating different second images with reference to the target object can be improved.

Description

Technical Field

[0001] This disclosure relates to the field of image processing, and more particularly to an image processing method, apparatus, electronic device, storage medium, and program product. Background Technology

[0002] In related technologies, a target object in a first image can be referenced to generate a second image containing the target object. However, the image effect of the second image containing the target object generated in related technologies is relatively simple. It does not refer to the target object to generate multiple different second images, resulting in low richness of the second image generated by referencing the target object. Summary of the Invention

[0003] To overcome the problems in related technologies, this disclosure provides an image processing method, apparatus, electronic device, storage medium, and program product to improve the richness of a second image generated after referencing a target object.

[0004] According to a first aspect of the present disclosure, an image processing method is provided, comprising:

[0005] Display a first image; wherein the first image contains at least one object;

[0006] In response to a triggering operation targeting a target object in at least one object, at least two second images including the target object are generated; wherein the target object is oriented differently in the different second images.

[0007] In some embodiments, generating at least two second images including a target object includes: displaying a preset interface and determining a layout area of ​​the target object in the preset interface based on touch operations on the preset interface; generating at least two second images based on the layout area; wherein the size and / or position of the target object in the second images is determined based on the layout area. This allows for precise control over the generation effect of the target object in the second images, adapting to a layout area configured for the target object.

[0008] In some embodiments, displaying a preset interface includes: displaying a preset interface of a first size; generating at least two second images based on a layout area includes: generating at least two second images of a second size based on the layout area; wherein the ratios between the edges of the preset interface of the first size and the corresponding edges of the second image of the second size are the same. This improves the visual consistency between the layout area of ​​the configured target object in the preset interface and the position area of ​​the target object in the actually generated second image.

[0009] In some embodiments, generating at least two second images based on a layout region includes: determining a target region in an initial image corresponding to the layout region; processing the target region in the initial image based on a first noise parameter corresponding to the image features of the target object in the first image, and processing other regions in the initial image besides the target region based on a preset second noise parameter to obtain a processed initial image; and performing denoising processing on the processed initial image to obtain at least two second images in which the target object is located within the target region. In this way, while achieving precise control over the generation position of the target object in the second images, the similarity between the target object in the generated second images and the target object in the first image can also be improved, thereby enhancing the image generation effect.

[0010] In some embodiments, at least two sample images including an object are obtained; wherein the same object has different orientations in different sample images; an initial model is trained based on the at least two sample images until preset training conditions are met to obtain an image generation model; wherein the image generation model is used to generate at least two images including the same object, and the same object has different orientations in different images generated by the image generation model; generating at least two second images including a target object includes: inputting the target object into the image generation model to obtain at least two second images. This can improve the efficiency of generating multiple second images for the same target object.

[0011] In some embodiments, training an initial model based on at least two sample images until preset training conditions are met to obtain an image generation model includes: for each sample image, labeling the sample image based on first descriptive information for the sample image and the orientation of objects in the sample image to obtain a labeled sample image; wherein the first descriptive information is used to describe at least one of the following: image content corresponding to objects in the sample image, background content of the sample image, and overall image content of the sample image; training the initial model based on all labeled sample images until preset training conditions are met to obtain an image generation model; inputting a target object into the image generation model to obtain at least two second images, including: inputting the target object and second descriptive information for image generation into the image generation model to obtain at least two second images. This improves the accuracy of the trained image generation model in generating second images with specific image content.

[0012] In some embodiments, acquiring at least two sample images including an object includes: acquiring at least two candidate images including the object from a preset video; and acquiring at least two first sample images from the at least two candidate images based on performance parameters of each candidate image; wherein the performance parameters are used to indicate the image quality of the candidate images. This improves the efficiency of acquiring sample images and the image quality of the acquired sample images.

[0013] In some embodiments, obtaining at least two first sample images from at least two candidate images based on the performance parameters of each candidate image includes: grouping the at least two candidate images into at least one set of candidate images based on the type of the object in each candidate image; determining a benchmark image that meets preset performance conditions from each set of candidate images based on the performance parameters of each candidate image in each set of candidate images, and determining the similarity between the benchmark image in each set of candidate images and every candidate image other than the benchmark image; and determining the benchmark image in each set of candidate images, as well as candidate images with a similarity greater than a preset similarity threshold, as first sample images. This can improve the image quality of the obtained first sample images and increase the efficiency of determining high-quality first sample images.

[0014] In some embodiments, the method further includes: segmenting the object and image content other than the object in the first sample image to obtain a second sample image containing the object; and training the initial model based on at least two sample images, including: training the initial model based on the first sample image and the second sample image. This enhances the adaptability of the initial model to different background content and different objects.

[0015] According to a second aspect of the present disclosure, an image processing apparatus is provided, the apparatus comprising:

[0016] The display module is configured to display a first image; wherein the first image contains at least one object;

[0017] The generation module is configured to generate at least two second images including the target object in response to a triggering operation targeting a target object in at least one object; wherein the target object has a different orientation in the different second images.

[0018] In some embodiments, the display module is configured to display a preset interface and determine the layout area of ​​the target object in the preset interface based on touch operations on the preset interface; the generation module is configured to generate at least two second images based on the layout area; wherein the size and / or position of the target object in the second image is determined based on the layout area.

[0019] In some embodiments, the display module is configured to display a preset interface of a first size; the generation module is configured to generate at least two second images of a second size based on a layout area; wherein the ratio between each side of the preset interface of the first size and the corresponding side of the second image of the second size is the same.

[0020] In some embodiments, the apparatus further includes: a determining module configured to determine a target region in an initial image corresponding to a layout region; the generating module further includes: a processing module configured to process the target region in the initial image based on a first noise parameter corresponding to an image feature of a target object in the first image, and to process other regions in the initial image besides the target region based on a preset second noise parameter to obtain a processed initial image; and to perform denoising processing on the processed initial image to obtain at least two second images in which the target object is located within the target region.

[0021] In some embodiments, the apparatus further includes: an acquisition module configured to acquire at least two sample images including an object; wherein the same object has different orientations in different sample images; a training module configured to train an initial model based on the at least two sample images until a preset training condition is met to obtain an image generation model; wherein the image generation model is used to generate at least two images including the same object, and the same object has different orientations in different images generated by the image generation model; and a generation module configured to input the target object into the image generation model to obtain at least two second images.

[0022] In some embodiments, the apparatus further includes: a labeling module configured to label each sample image based on first descriptive information for the sample image and the orientation of an object in the sample image, to obtain a labeled sample image; wherein the first descriptive information is used to describe at least one of the following: image content corresponding to an object in the sample image, background content of the sample image, and overall image content of the sample image; a training module configured to train an initial model based on all labeled sample images until a preset training condition is met to obtain an image generation model; and a generation module configured to input a target object and second descriptive information for image generation into the image generation model to obtain at least two second images.

[0023] In some embodiments, the acquisition module is configured to acquire at least two candidate images including an object from a preset video; and acquire at least two first sample images from the at least two candidate images based on the performance parameters of each candidate image; wherein the performance parameters are used to indicate the image quality of the candidate images.

[0024] In some embodiments, the apparatus further includes: a grouping module configured to group at least two candidate images based on the type of the object in each candidate image to obtain at least one set of candidate images; and a determining module configured to determine a benchmark image that meets a preset performance condition from each set of candidate images based on the performance parameters of each candidate image in each set of candidate images, and to determine the similarity between the benchmark image in each set of candidate images and each candidate image other than the benchmark image; and to determine the benchmark image in each set of candidate images, as well as the candidate images with a similarity greater than a preset similarity threshold, as first sample images.

[0025] In some embodiments, the apparatus further includes: a segmentation module configured to segment objects and image content other than objects in a first sample image to obtain a second sample image containing objects; and a training module configured to train an initial model based on the first sample image and the second sample image.

[0026] According to a third aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing computer programs or instructions; wherein the processor executes the computer programs or instructions to implement the steps of any of the image processing methods in the first aspect described above.

[0027] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium is provided, comprising: when a computer program or instructions in the storage medium are executed by a processor, implementing the steps of any of the image processing methods in the first aspect described above.

[0028] According to a fifth aspect of the present disclosure, a computer program product is provided, including a computer program or instructions, which, when executed by a processor, implement the steps of any of the image processing methods in the first aspect described above.

[0029] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects:

[0030] Compared to related technologies that generate only a single second image containing the target object, and where the target object has the same orientation in both the first and second images, in this embodiment, in response to a triggering operation targeting the target object in at least one object in the first image, the target object in the first image is referenced to generate multiple second images containing the target object, and the target object has a different orientation in each of the different second images. This improves the richness of the orientation of the object in the second images generated after referencing the target object.

[0031] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0032] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0033] Figure 1 This is a schematic diagram of an image processing method according to an exemplary embodiment. Figure 1 .

[0034] Figure 2 This is a schematic diagram of an image processing method according to an exemplary embodiment. Figure 2 .

[0035] Figure 3 This is an algorithm framework diagram illustrated according to an exemplary embodiment.

[0036] Figure 4 This is a schematic diagram illustrating a process for training an initial model according to an exemplary embodiment.

[0037] Figure 5 This is a structural block diagram of an image processing apparatus according to an exemplary embodiment.

[0038] Figure 6 This is a structural block diagram of an electronic device according to an exemplary embodiment.

[0039] Figure 7 This is a block diagram of an apparatus according to an exemplary embodiment.

[0040] Reference numerals: Second description information 311; Target object 312; Layout region 313; First encoding module 321; First encoding submodule 322; Second encoding submodule 323; Random replacement module 324; Multilayer perceptron 325; Linear mapping module 326; Nonlinear module 327; Noise processing module 328; Image generation module 329; Conversion module 3291; Text feature 331; First image feature 332; Second image feature 333; Third fusion feature 334; Orientation feature 335; Fourth fusion feature 336; Third image feature 337; Fourth image feature 338; Cropped image 340; Noise region 350; Initial image 361; Processed initial image 362; Random noise 370; Target region 380; Second image 390. Detailed Implementation

[0041] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0042] It should be noted that the image processing method provided in this disclosure can be applied to electronic devices with image processing capabilities. Here, the electronic device can include: a mobile electronic device or a fixed electronic device. The mobile electronic device can include: a mobile phone, tablet computer, laptop computer, in-vehicle electronic device, etc. The fixed electronic device can include: a desktop computer, smart TV, etc. In some embodiments, the operating system of the electronic device can include: an Input Output System (IOS) operating system, an Android operating system, etc.

[0043] It should be noted that electronic devices may include, but are not limited to, mobile communication electronic devices, portable entertainment devices, wearable devices, home appliances, augmented reality devices, virtual reality devices, and special-purpose devices. Among these, mobile communication electronic devices may include, but are not limited to, mobile phones, tablets, and smartwatches; portable entertainment devices may include, but are not limited to, digital cameras; wearable devices may include, but are not limited to, smart bracelets and smart glasses; home appliances may include, but are not limited to, televisions and video recorders; augmented reality (AR) devices may include, but are not limited to, AR glasses; virtual reality (VR) devices may include, but are not limited to, VR glasses; and special-purpose devices may include, but are not limited to, professional cameras (such as SLR cameras and point-and-shoot cameras).

[0044] It should be noted that the execution entity of the embodiments disclosed herein may be a central processing unit (CPU) in an electronic device in terms of hardware, and may be a related background service or application in the electronic device in terms of software, without limitation.

[0045] To better understand the technical solutions in the embodiments of this disclosure, the following provides an exemplary description of image processing methods in related technologies: In related technologies, the rapid development of diffusion-based image generation models has led to the development of many large-scale synthetic models. These models have been trained on massive datasets containing billions of image-text pairs, demonstrating impressive artistry, realism, and semantic alignment. However, relying solely on descriptive information (i.e., prompts used to guide image generation) is insufficient to fully capture user intent. Therefore, combining descriptive information with a first image to generate a second image has become a promising research direction.

[0046] The following provides an exemplary description of a method for combining descriptive information with a first image to generate a second image:

[0047] In related techniques, pre-trained text-to-image generation models can be utilized (e.g., text-to-image models built with stable diffusion techniques), and the image generation model can be fine-tuned using a very small number (3-5) of initial images. However, this process requires specialized network training, such as training the network in the word embedding space or specific layers of UNet, or training the network by adding side branches. However, this approach is inefficient for practical applications. Another technique is to retrain the text-to-image model on a large-scale personalized image dataset using a specially designed network architecture or training strategy. Nevertheless, this typically compromises fidelity and generalization ability compared to test-time fine-tuning methods.

[0048] Furthermore, some techniques can only generate a second image containing a specific reference object, such as a portrait or an animal. While some recently proposed algorithms can achieve open-domain customized image generation, they suffer from the limitation of only handling single concepts (i.e., a single reference object). However, few techniques fully consider comprehensive capabilities for single reference objects, multiple reference objects, no need for test-time fine-tuning, and open-domain zero-shot capability. Moreover, from the perspective of whether to introduce test-time fine-tuning, text-based image generation models based on prompts mainly fall into two frameworks. The first framework is a test-time fine-tuning framework, requiring several first images containing specific reference objects, and then directly fine-tuning the token embeddings of the first images to adapt to the learned visual concept. The second framework is one that uses the first images to fine-tune the generation model itself, including fine-tuning the entire UNet network and fine-tuning only the K and V layers of cross-attention. Additionally, some techniques have proposed personalized generation for multiple reference objects, adding regularization penalties to limit the confusion of attention to multiple reference objects. For generating second images based on multiple reference objects, multiple trained image generation models can be directly connected through conceptual neurons, such as training a separate Low-Rank Adaptation (LoRA) model for each reference object and then fusing them. Second images containing multiple reference objects are generated by learning the residuals of token embeddings and controlling attention. Due to the well-known time-consuming nature of test-time fine-tuning methods, another research approach involves constructing large amounts of domain-specific data or training with open-domain image data to avoid additional fine-tuning. InstructPix2Pix can perform various editing tasks according to human instructions, including object swapping, style transfer, and environment modification, simply by connecting the latent image of the first image during model noise injection. The Unified Multi-Modal Latent Diffusion (UMM-Diffusion) proposes a novel unified multimodal latent diffusion model that takes a joint text and image containing specified reference objects as input sequences and generates a second image with the reference objects. Its limitations include the inability to support multiple reference objects. The Text Compatible ImagePrompt Adapter for Text-to-Image Diffusion Models (IP-Adapter) uses a lightweight image cue adaptation method and employs a decoupled cross-attention strategy for existing text-to-image diffusion models.In addition, there are technologies that propose reference object-driven text-to-image generation models that can generate a second image on the spot and in a customized manner from a visual reference object.

[0049] As can be seen from the above, there are many methods in related technologies that use diffusion models to generate personalized second images. However, none of them consider the orientation of the reference object in the generated second image during the image generation process. As a result, the image generation process often only uses the frontal view of the reference object as a reference. The generated second image is also limited to a single orientation and does not expand to more diverse orientations.

[0050] Based on this Figure 1 A flowchart illustrating an image processing method according to an exemplary embodiment. Figure 1 ,like Figure 1 As shown, the method includes steps 101 to 102, wherein:

[0051] Step 101: Display a first image; wherein the first image contains at least one object.

[0052] It should be noted that the first image can serve as a reference image during the generation of a new image. For example, a new second image can be generated by referencing a target object contained in the first image.

[0053] In some embodiments, the objects contained in the third image can be segmented to obtain at least one first image containing a single object. Each first image can be displayed. Here, the first image may not contain any content other than the object. Alternatively, the first image may be an image containing at least one object and background content other than the at least one object. That is, the first image may not be an image segmented from the third image.

[0054] It should be noted that the method of acquiring the first image or the third image in this embodiment is not limited.

[0055] For example, the first image or the third image may be an image pre-stored in the electronic device, and can be obtained directly from the storage space of the electronic device. Alternatively, the first image or the third image may not be an image pre-stored in the electronic device, and may be an image obtained from outside the electronic device. For example, the first image or the third image may be an image obtained from another device or server that has established a communication connection with the electronic device.

[0056] For example, the first image or the third image may be a pre-acquired image; or, the first image or the third image may be an image acquired temporarily. For instance, the first image or the third image may be an image temporarily captured in response to a photo-taking command.

[0057] In some embodiments, a first image or a third image containing at least one object may be determined from at least two alternative images, and the first image or the third image may be displayed.

[0058] In some embodiments, in response to determining a first image or a third image from at least two candidate images, other candidate images besides the first image or the third image can be adjusted to a non-display state. In this case, only the first image or the third image can be kept in the display state to reduce visual interference caused by the candidate images to the first image or the third image.

[0059] In some embodiments, all candidate images contain objects. The selected candidate image can be determined as either a first image or a third image based on a first selection operation on the candidate images. Here, a first image or a third image that meets the user's needs can be selected from at least two candidate images through manual operation by the user.

[0060] In other embodiments, a first image or a third image containing at least one object can be determined from at least two candidate images based on the number of objects contained in the candidate images and / or the types of objects contained in the candidate images. Here, the determination of the first image or the third image can be automated based on the number of objects contained in the candidate images and / or the types of objects contained in the candidate images, thereby determining the degree of automation in determining the first image or the third image.

[0061] For example, candidate images containing a number of objects greater than or equal to a first threshold can be determined as the first image or the third image.

[0062] It should be noted that when the first or third image contains a large number of objects, the first or third image can provide a rich selection of objects for the user to identify the target object, thus improving the flexibility of selecting the target object.

[0063] For example, the number of objects included in a candidate image within a first range can be determined as either a first image or a third image. The value of the first range is greater than or equal to a first threshold, and the value of the first range is less than or equal to a second threshold.

[0064] It should be noted that if the first or third image contains too many objects, the image content of each object in the first or third image may be incomplete. In this case, even if the target object is selected from the objects in the first or third image, the reference effect of the target object may be poor when generating the second image. Therefore, in this embodiment of the disclosure, the first range can be flexibly determined according to user needs, and the candidate images containing the number of objects within the first range can be determined as the first or third image, so that the user can select the target object that can provide a better reference effect from a large number of objects.

[0065] For example, an alternative image containing an object of the target type can be determined as a first image or a third image.

[0066] In some embodiments, the target object type input by the first input operation can be obtained in response to the first input operation. Alternatively, at least two first selection controls can be displayed, wherein different selection controls are used to indicate different object types. The object type indicated by the selected first selection control can be determined as the target object type based on a second selection operation targeting the at least two first selection controls. Here, the target object type can be determined manually by the user, adapting to the user's needs by determining the candidate image containing the target object type as the first image.

[0067] Step 102, in response to a triggering operation for a target object in at least one object, generates at least two second images including the target object; wherein the target object has a different orientation in the different second images.

[0068] It should be noted that the orientation of the target object differs in images captured from different viewpoints. In other words, there is often a correspondence between the orientation of the target object in an image and the viewpoint from which the image was captured; the orientation of the target object in an image indicates the viewpoint used to capture the target object. However, in this disclosure, at least two second images can be generated without capturing the target object from a real-world viewpoint, and the target object's orientation differs in each second image. That is, second images corresponding to multiple viewpoints can be generated without capturing the target object from a real-world viewpoint.

[0069] In some embodiments, when the first image includes a single object, a single first image may be displayed. The triggering operation may be a determination operation that identifies an object in the first image as the target object. When the first image includes at least two objects, the triggering operation may be a third selection operation that selects the target object from the at least two objects included in the first image.

[0070] In one embodiment, where the first image comprises a single object and does not include background content other than the object, a single first image may be displayed. At least two second images, each including a target object, may be generated based on a triggering operation targeting a target object in the first image. Here, the triggering operation may be a determination operation that identifies a single object in the first image as the target object.

[0071] In some embodiments, where the first image includes a single object and does not include background content other than the object, at least two first images may be displayed. At least two second images including the target object may be generated based on a triggering operation targeting a target image among the at least two first images. The target object may be an object within the target image. The triggering operation can be understood as the operation of selecting a target image from the at least two first images and identifying an object within that target image as the target object.

[0072] In some embodiments, in response to a triggering operation targeting a target object in at least one object, at least two second images containing the target object are generated based on at least two preset orientations; wherein the target object has a different orientation in the different second images, and the orientation of the target object in the second image is determined based on a single preset orientation. Exemplarily, the at least two preset orientations may include at least two of the following: 0°, 45°, 90°, 135°, 180°, 225°, 270°, 315°, a top-down angle, and a bottom-up angle.

[0073] In some embodiments, in response to a triggering operation targeting a target object in at least one object, at least two second images containing the target object can be generated based on at least two preset orientations and second descriptive information for image generation; wherein the image content of the second images matches the second descriptive information; the target object has different orientations in the different second images, and the orientation of the target object in the second images is determined based on a single preset orientation.

[0074] In some embodiments, at least two orientations input by the second input operation can be obtained in response to the second input operation. Alternatively, at least two second selection controls can be displayed, wherein different second selection controls are used to indicate different candidate viewpoints. The candidate viewpoint indicated by the selected second selection control can be determined as the orientation based on a fourth selection operation for at least two second selection controls. Here, the user can manually determine the orientation to generate a second image that visually represents the orientation captured in the image, tailored to the user's needs.

[0075] In some embodiments, candidate description information may be displayed, and if a determination operation on the candidate description information is detected, the candidate description information is determined as the second description information. A second image containing the target object is generated based on at least two preset orientations and the second description information.

[0076] In one embodiment, in response to a third input operation, second descriptive information input by the third input operation can be obtained. Alternatively, at least two candidate second descriptive information can be displayed, and the second descriptive information can be selected from the at least two candidate descriptive information based on a fifth selection operation. At least two second images containing the target object can be generated based on the second descriptive information and at least two preset orientations.

[0077] In some embodiments, the second descriptive information may be text content pre-stored within the electronic device. Alternatively, the second descriptive information may be text content received by the electronic device.

[0078] In some embodiments, the second descriptive information may include text content describing the display style of the target object. The display style may be used to indicate at least one of the following: the shape, pose, color, and line art type of the target object.

[0079] In some embodiments, when there are at least two target objects, the second descriptive information may include text content describing the relative positional relationship between the at least two target objects.

[0080] In some embodiments, the second descriptive information may include text content indicating the image style. For example, if the second descriptive information is "warm style," then the image content of the second image generated based on the second descriptive information may present a warm style. Alternatively, if the second descriptive information is "cool style," then the image content of the second image generated based on the second descriptive information may present a cool style.

[0081] In some embodiments, the second descriptive information may include text content indicating the content to be generated in the image. For example, the second descriptive information may be "a tree," in which case the second image generated based on the second descriptive information may include a tree and a target reference object. For example, the second descriptive information may be text content indicating the scene in which the image is generated. For instance, the second descriptive information may be "on a cloudy day, in front of a movie theater," in which case the second image generated based on the second descriptive information may present a visual effect of a target object located in front of a movie theater on a cloudy day.

[0082] Here, the specific type of the second descriptive information is not limited. The electronic device can parse any of the acquired second descriptive information to generate a second image whose image content matches the second descriptive information.

[0083] In some embodiments, generating at least two second images including a target object based on at least two preset orientations and second descriptive information includes: using a pre-trained image generation model to generate at least two second images including a target object based on at least two preset orientations and second descriptive information.

[0084] It should be noted that the image generation model can be a pre-trained model. This model can output at least two second images based on a single input of second descriptive information and a target object, with the target object having different orientations in the different second images. For example, the orientation of the target object in the second image can be determined based on a preset orientation, and the orientation of the target object in different second images can be determined based on different preset orientations. The orientation of the target object in the second image being determined based on a preset orientation can mean that the similarity between the target object's orientation and the preset orientation is greater than a first similarity threshold. Here, it is not necessary to input orientation-related information into the image generation model. Instead, the second descriptive information and the target object can be input into the image generation model in a single step, triggering the model to perform multiple image generation operations based on the input second descriptive information and the target object, resulting in different second images corresponding to different preset orientations.

[0085] Alternatively, the image generation model can be used to output a second image that corresponds to the preset orientation and includes the target object, based on the input second descriptive information, a preset orientation, and a target object. That is, orientation-related information needs to be input into the image generation model to guide it in generating a second image corresponding to the preset orientation.

[0086] In some embodiments, generating at least two second images including a target object based on at least two preset orientations and second description information includes: for each preset orientation, generating a second image corresponding to the preset orientation based on the orientation features of the preset orientation, the text features of the second description information, and the image features of the target object.

[0087] Here, the preset orientation, the second descriptive information, and the target object can be input into the image generation model. The image generation model extracts the orientation features of the preset orientation, the text features of the second descriptive information, and the image features of the target object. Based on the orientation features of the preset orientation, the text features of the second descriptive information, and the image features of the target object, the image generation model generates a second image corresponding to the preset orientation.

[0088] It should be noted that if the first image does not contain any content other than the target object, it can be directly input into the image generation model to obtain the image features of the target object. If the first image contains content other than the target object, the target object in the first image can be segmented to obtain a segmented image containing the target object; this segmented image can then be input into the image generation model to obtain the image features of the target object.

[0089] It should be noted that, when no preset orientation needs to be input into the image generation model, the image generation model can pre-store orientation features extracted for each preset orientation. When a preset orientation needs to be input into the image generation model, the image generation model can temporarily extract the orientation features of the input preset orientation. Furthermore, in the process of generating at least two second images based on the same second descriptive information and target object, the image generation model can extract the text features of the second descriptive information and the image features of the target object once, and perform multiple image generation operations based on the extracted text features and image features.

[0090] In some embodiments, the second descriptive information can be encoded based on the first encoding module to obtain text features. For example, the first encoding module can be a Contrastive Language-Image Pre-training (CLIP) text encoder.

[0091] In some embodiments, the image features of the displayed target object include: a first type of image features and a second type of image features. The first type of image features indicates the image content corresponding to the target object, and the second type of image features indicates the image style corresponding to the target object.

[0092] For example, if the target object is a dog, then features can be extracted from the image content of the dog, such as its coat color, pattern, and facial features, to obtain the first type of image features.

[0093] For example, the image style may include, but is not limited to, at least one of the following: realistic style, night scene style, monochrome style, and anime / manga style.

[0094] In some embodiments, image features of the displayed target object can be extracted based on the second encoding module. The second encoding module may include a first encoding submodule and a second encoding submodule. The first encoding submodule can be used to extract a first type of image features. The second encoding submodule can be used to extract a second type of image features. Exemplarily, the first encoding submodule can be a visual feature encoder. The second encoding submodule can be a style feature encoder. For example, the second encoding submodule can be a CLIP image encoder.

[0095] In some embodiments, the first type of image features and the second type of image features can be fused to obtain a first fused feature; for each orientation, a second image under that orientation can be generated based on the first fused feature, text features and orientation features of that orientation.

[0096] In some embodiments, the first fusion feature can be compressed to obtain compressed features; the compressed features and text features can be fused by random replacement to obtain a second fusion feature. A second image containing the target object can be generated based on the second fusion feature and the orientation feature.

[0097] It should be noted that the image features of the target object displayed can also be understood as the image features extracted from any segmented image containing the target object in this disclosure.

[0098] In some embodiments, a panoramic image or an ultra-wide-angle image can be generated based on at least two second images. Alternatively, a 3D model can be generated based on at least two second images. Thus, multiple second images from different perspectives can be combined to generate a single multi-view image or 3D model that can be widely applied.

[0099] In some embodiments, the methods in this disclosure can be applied to preset scenarios. Preset scenarios may include, but are not limited to, at least one of the following: 3D modeling scenarios, video enhancement scenarios, AR / VR scenarios, monitoring scenarios, robot vision scenarios, and entertainment scenarios. For example, in a 3D modeling scenario, different second images of a target object under different orientations can be acquired and 3D modeling performed to generate a 3D model of the object. For example, in a video enhancement scenario, different second images of a target object under different orientations can be acquired to enhance and optimize the video, improving the visual experience. For example, in an AR / VR scenario, different second images of a target object under different orientations can be acquired in real time to meet the needs of augmented reality and virtual reality applications. This can help virtual reality applications become more realistic, allowing users to experience a more immersive experience. For example, characters and scenes in virtual reality games can be generated using at least two second images to achieve more natural movements and realistic environments. For example, in a monitoring scenario, acquiring different second images of a target object under different orientations can achieve more comprehensive and accurate security monitoring. For example, in a robot vision scenario, acquiring different second images of a target object under different orientations can help robots process environmental perception more accurately and efficiently. For example, robots can acquire environmental information from multiple angles and then generate 3D models or panoramic images to better perceive the environment. Exemplarily, in entertainment scenarios, multi-view images can be applied to the entertainment and cultural industries. For instance, images captured by multiple cameras from different perspectives can be used to generate panoramic images or 3D models, thereby providing a richer entertainment experience. In this embodiment, multi-view images can be generated based on at least two second images to enhance the entertainment experience. In summary, multi-view image generation technology has broad application prospects and can achieve more accurate and efficient data acquisition and processing in many fields, thereby improving user experience and overall effectiveness.

[0100] Compared to related technologies that generate only a single second image containing the target object, and where the target object has the same orientation in both the first and second images, in this embodiment, in response to a triggering operation targeting the target object in at least one object in the first image, the target object in the first image is referenced to generate multiple second images containing the target object, and the target object has a different orientation in each of the different second images. This improves the richness of the orientation of the object in the second images generated after referencing the target object.

[0101] Figure 2 This is a flowchart illustrating an image processing method according to an exemplary embodiment. Figure 2 ,like Figure 2 As shown, the method includes steps 201 to 207, wherein:

[0102] In step 201, the third image is acquired.

[0103] In step 202, semantic segmentation is performed on the objects in the third image to obtain at least one first image containing the objects.

[0104] In step 203, second descriptive information for image generation is obtained, and a first image containing the target object is determined from at least one first image containing the object.

[0105] In step 204, a second image including the target object is generated based on the second description information, the preset orientation, and the first image containing the target object. Here, step 204 may be performed for each of at least two preset orientations.

[0106] In step 205, it is determined whether to regenerate the second image;

[0107] If it is determined that the second image needs to be regenerated, then based on the second description information, the first image containing the target object, and the preset orientation corresponding to the second image to be regenerated, the second image including the target object is regenerated (i.e., step 204 can be re-executed) to obtain a new second image corresponding to the preset orientation. For example, the second description information, the first image containing the target object, and the preset orientation corresponding to the second image can be re-inputted into the image generation model to obtain a second image corresponding to the preset orientation. Alternatively, if it is not necessary to regenerate the second image, then step 206 is executed.

[0108] It should be noted that the system can automatically determine whether to regenerate the second image. For example, if the similarity between the target object in the second image and the target object in the first image is greater than or equal to a preset second similarity threshold, it can be determined that the second image does not need to be regenerated. If the similarity between the target object in the second image and the target object in the first image is less than the second similarity threshold, it can be determined that the second image needs to be regenerated.

[0109] It should be noted that it is possible to manually determine whether to regenerate the second image. For example, the second image can be displayed, and in response to detecting a first touch operation on the second image, it can be determined that the second image needs to be regenerated. In response to not detecting a first touch operation on the second image, it can be determined that the second image does not need to be regenerated. In this case, the first touch operation on the second image can be an operation for regenerating the second image.

[0110] Here, you can first automatically determine whether to regenerate the second image, and then manually determine whether to generate the second image. It's understandable that if the second image A is regenerated, the regenerated second image B will replace the second image A.

[0111] In step 206, super-resolution processing is performed on at least one second image corresponding to each preset orientation to obtain a processed image. It should be noted that the second image can be displayed, and in response to detecting a second touch operation on the second image, super-resolution processing is performed on the second image targeted by the second touch operation to improve the clarity of the second image.

[0112] In step 207, the processed image is output.

[0113] In some embodiments, generating at least two second images including the target object includes:

[0114] Display a preset interface and determine the layout area of ​​the target object in the preset interface based on the touch operation on the preset interface;

[0115] Based on the layout region, at least two second images are generated; wherein the size and / or position of the target object in the second image are determined based on the layout region.

[0116] It should be noted that, in order to distinguish it from the first and second touch operations mentioned above, the touch operation on the preset interface will be referred to as the third touch operation below.

[0117] In some embodiments, the layout area of ​​the target object in a preset interface can be determined based on the touch point of the third touch operation. For example, a rectangular or circular area of ​​a preset size centered on the touch point of the third touch operation can be defined as the layout area of ​​the target object in the preset interface. In this case, the third touch operation can be understood as an operation of selecting a layout area by tapping. Alternatively, the layout area of ​​the target object in the preset interface can be determined based on the touch area corresponding to the third touch operation. In this case, the touch operation can be an operation of selecting a region within the preset interface.

[0118] In some embodiments, the size and / or position of the target object in the second image is determined based on the layout area, which can be understood as: the size of the target object in the second image is determined based on the size of the layout area in the preset interface, and / or the position of the target object in the second image is determined based on the position of the layout area in the preset interface.

[0119] In this embodiment, the layout area of ​​the target object in the displayed preset interface can be determined, thereby intuitively displaying the layout area configured for the target object. Furthermore, during the process of generating a second image including the target object based on the layout area, the generation position of the target object in the second image can be precisely controlled, improving the image quality of the generated second image.

[0120] In some embodiments, a preset interface is displayed, including:

[0121] Displays the default interface of the first size;

[0122] Based on the layout region, generate at least two second images, including:

[0123] Based on the layout region, at least two second images of a second size are generated; wherein the ratio between each side of the preset interface of the first size and the corresponding side of the second image of the second size is the same.

[0124] In some embodiments, the first size and the second size may be different. At least two second images of the second size can be generated based on the ratio between the first and second sizes and the layout area. Here, the position of the target object in the second image and the position of the layout area in the preset interface can be matched. For example, if the layout area is located at the center of the preset interface, the target object can also be located at the center of the second image.

[0125] It should be noted that the preset interface and the second image can have the same shape. For example, both the preset interface and the second image can be rectangular. When both the preset interface and the second image are rectangular, the fact that the ratios between the sides of the preset interface of the first size and the corresponding sides of the second image of the second size are the same can be understood as: the ratio between the length of the preset interface and the length of the second image is the same as the ratio between the width of the preset interface and the width of the second image.

[0126] In one embodiment, the first size is smaller than the second size. This allows for the display of a smaller preset interface, thereby reducing the display resources required when displaying the preset interface and minimizing interference with other content displayed on the screen.

[0127] In this embodiment of the disclosure, since the ratios between the edges of the preset interface of the first size and the corresponding edges of the second image of the second size are the same, the position and / or size of the target object in the generated second image can be precisely controlled during the process of generating the second image based on the layout area of ​​the target object in the preset interface. This ensures that at least one of the position and size of the target object in the second image matches at least one of the position and size of the layout area in the preset interface, thereby improving the visual consistency between the configured layout area of ​​the target object in the preset interface and the position area of ​​the target object in the actually generated second image.

[0128] In some embodiments, at least two second images are generated based on the layout region, including:

[0129] Determine the target region in the initial image that corresponds to the layout region;

[0130] Based on the first noise parameter corresponding to the image features of the target object in the first image, the target region in the initial image is processed, and based on the preset second noise parameter, other regions in the initial image except the target region are processed to obtain the processed initial image.

[0131] The processed initial image is denoised to obtain at least two second images in which the target object is located within the target area.

[0132] It should be noted that the initial image can be a blank image, and the second image can be obtained by processing the initial image.

[0133] In some embodiments, the region parameters of the target region in the initial image can be determined based on the region parameters of the layout region in a preset interface. It should be noted that the target region can be the region used to generate the target object. Region parameters can include at least one of the following: shape parameters, size parameters, and position parameters.

[0134] In some embodiments, the preset second noise parameter can be a random noise parameter. Noise can be added to the entire region of the initial image based on the random noise parameter, and then the target region in the initial image can be re-noised based on a first noise parameter that incorporates image features of the displayed target object. In this case, the noise addition result of the target region will be replaced by the noise addition result after noise addition using the random noise parameter, and then by the noise addition result of the target region in the initial image based on the target noise parameter.

[0135] In some embodiments, the target region in the initial image may not be processed based on random noise parameters, but may be processed directly based on first noise parameters corresponding to the image features of the target object in the first image.

[0136] In this embodiment, image features of the target object can be fused into the noise added to the target region of the initial image. This allows for the retention of the target object's image features in the denoised initial image during denoising, and enables precise control over the target object's location within the target region in the generated second image. Thus, while achieving precise control over the target object's position in the second image, the similarity between the target object in the generated second image and the target object in the first image can be increased, thereby improving the image generation effect.

[0137] Figure 3 This is an algorithmic framework diagram of an image processing method illustrated according to some exemplary embodiments. The following describes the method through some exemplary embodiments. Figure 3 The algorithm framework shown is explained below:

[0138] In some embodiments, see Figure 3 The system can acquire second descriptive information 311 and extract text features 331 from it. The first encoding module 321 can extract features from the second descriptive information 311 to obtain text features 331. The second descriptive information 311 can be used to indicate the content of the image to be generated. For example, the second descriptive information 311 can be used to indicate the scene and content details of the new image generation. Here, the extracted text features 331 of the second descriptive information 311 can guide the image generation model to generate a second image with the required fidelity and accuracy.

[0139] Furthermore, a target object 312 can be acquired, and a first type of image feature can be extracted from the overall target object 312 to obtain a first image feature 332; a second type of image feature can be extracted from the overall target object 312 to obtain a second image feature 333. The first type of image feature indicates the image content corresponding to the target object, and the second type of image feature indicates the image style corresponding to the target object. The first type of image feature can be extracted from the overall target object 312 through a first encoding submodule 322, and the second type of image feature can be extracted from the overall target object 312 through a second encoding submodule 323. The first image feature 332 can be fused with text feature 331 to obtain a first fused feature. A second image 390 corresponding to the orientation can be generated based on the first fused feature and the orientation. Here, by fusing the first image feature 332 with text feature 331, a text encoding method that fuses image features is implemented, ensuring that the text content of the second descriptive information does not introduce excessive interference to the image content corresponding to the target object, reducing problems such as deformation distortion and reduced similarity. Simultaneously, the injection of the text features of the second descriptive information is based on the target object for expansion.

[0140] In some embodiments, see Figure 3 After obtaining the first fusion feature, the first image feature 332 and the second image feature 333 can be fused to obtain the second fusion feature. The first and second fusion features can then be fused to obtain the third fusion feature 334. Based on the third fusion feature and a preset orientation, a second image 390 corresponding to the preset orientation can be generated. Here, by fusing image and text features of multiple dimensions, the similarity between the image style and the target object in the generated second image, as well as the similarity between the target object in the second image and the target object in the first image, can be improved.

[0141] In some embodiments, see Figure 3 The first image feature 332 and the second image feature 333 can be fused using a multilayer perceptron (MLP) 325, and global features can be extracted through max-pooling to obtain the second fused feature. The feature scale of the second fused feature can be compressed to match the feature scale of the text feature 331. Based on the concat-replace module 324, the second fused feature and the text feature 331 (or the first fused feature) are fused using a random replacement method to obtain the third fused feature 334. In this way, a third fused feature 334 with uniform feature scale, fewer feature numbers, and fusion of image and text features from multiple dimensions can be obtained. A second image with high similarity to the original image can be generated based on this third fused feature 334.

[0142] In some embodiments, see Figure 3 The linear mapping module 326 can process at least two preset orientations separately to obtain orientation features 335 corresponding to each preset orientation. For example, the linear mapping module 325 can linearly map each orientation based on a linear mapping function a(x), converting it into a one-dimensional vector. The linear mapping module 326 can be composed of a trainable fully connected layer network. This one-dimensional vector can be represented as a feature of length 256. The orientation features 335 and the third fusion feature 334 of this disclosure can be fused to obtain a fourth fusion feature 336. A second image corresponding to a preset orientation can be generated based on the fourth fusion feature 336. Alternatively, a second image corresponding to an orientation can also be generated based on the fourth fusion feature 336 and any of the second fusion features of this disclosure.

[0143] Compared to related technologies that rely solely on text input to generate personalized images in an open domain, this embodiment introduces preset orientation information during the image generation model's modeling process. Orientation features are treated as a crucial input condition, similar to text input conditions. This allows for precise control over the generated image's appearance based on the preset orientation, improving the consistency and fidelity between the target object's orientation and the preset orientation in the generated second image. Furthermore, orientation features, text features, and image features can be fused, enabling the trained image generation model to consider the target object's image content, second descriptive information, and information related to the preset orientation.

[0144] In some embodiments, see Figure 3 The orientation feature 335 can be nonlinearly processed using the nonlinear module 327 to obtain the processed orientation feature. The processed orientation feature can then be fused with the third fusion feature 334 to obtain the fourth fusion feature 336. The nonlinear module 327 can be configured with a nonlinear activation function, which can be used to transform the orientation feature and increase its nonlinearity, represented as the a'(x) function. The processed orientation feature and the third fusion feature 334 can be dot-producted to obtain the dot-product result. This dot-product result can then be further concatenated with the third fusion feature 334 to obtain the fourth fusion feature 336. In this way, a strongly correlated orientation feature can be incorporated into the third fusion feature 334.

[0145] It should be noted that the initial model may include an image generation module 329. During the training of the initial model, the fourth fusion feature 336 obtained based on the sample images can be input into the image generation module 329 for learning. The location where the fourth fusion feature 336 is injected can be the cross-attention layer of the initial model, replacing the original single text feature injection method. The image generation module 329 can be a UNet framework. The image generation module 329 may include a Transformer module 3291. The Transformer module 3291 may include a self-attention layer, a learning adapter layer, and a cross-attention layer. The learning adapter layer can be an image attention layer, and any of the second fusion features disclosed herein can be injected into the image attention layer for learning or to guide the generation of the second image. During the training of the initial model, the key and value layers of the cross-attention layer and the learning adapter layer can be selectively activated, while the remaining layers are frozen. This method is used to make the model focus more on the learning adapter layer. In addition, to prevent model learning from crashing, during a training batch, the image features of objects with different preset orientations and their corresponding preset orientations can be iteratively trained multiple times to ensure that the model can better recognize the preset orientations and achieve modeling of text, image, and perspective information.

[0146] In some embodiments, see Figure 3 The layout area 313 of the target object in the preset interface can be determined. Based on the layout area 313, the target area 380 corresponding to the layout area 313 in the initial image 361 can be determined. Random noise 370 can be added to the entire area of ​​the initial image 361, and the target object can be cropped to a random size to obtain the cropped image 340. The first type of image features of the cropped image 340 can be extracted based on the first encoding submodule 322 to obtain the third image feature 337; the second type of image features of the cropped image 340 can be extracted based on the second encoding submodule 232 to obtain the fourth image feature 338; the third image feature 337 and the fourth image feature 338 can be noise-added based on the noise processing module 328 to obtain the noise area 350. The random noise in the target area 380 in the initial image 361 can be replaced with the noise in the noise area 350 to obtain the processed initial image 362. The multidimensional noise region 350 and the processed initial image 362 can be input into the image generation module 329, and denoising processing can be performed based on the fourth fusion feature and the second fusion feature in the input image generation module 329 to obtain the second image.

[0147] It should be noted that, by default, the initial noise in Unet is all random noise. Denoising random noise introduces a deviation into the entire denoising process, leading to a decrease in the similarity of the final result. However, in this embodiment, by adjusting the way initial noise is added to the initial image, image features are used to initialize the noise, and target region information is introduced. This allows the location of the raw image to be further incorporated into the raw image process, achieving precise control over the position of the target object in the generated second image.

[0148] In this embodiment, the image can be randomly cropped to a random size, and first and second type image features of the cropped image can be extracted. Although the size of the cropped image is small, a small image still contains some content; therefore, the first and second type image features of the cropped image can still indicate some information about the target object. By adding noise to the first and second type image features of the cropped image, initial image noise (i.e., image noise within the noise region 350) can be obtained. Furthermore, by randomly cropping the original image and adding noise to the image features of the small cropped image region, interference caused by using the original image for noise initialization can be reduced.

[0149] The noise features within the aforementioned noise region 350 and the noise features in the processed initial image 362 can be merged into a multi-dimensional noise feature group by concatenating features along the channel dimension, with the two noise features overlapping. This multi-dimensional noise feature group replaces the original random noise and serves as the initial input noise for the Unet model in subsequent denoising processes. This allows the generation of a second image with high similarity to the original image and where the target object is generated within the target region during the denoising process.

[0150] Understandably, in Figure 3 The algorithm framework shown mainly comprises four processes: second description information processing, target object processing, layout region processing, and image generation. The first three processes (second description information processing, target object processing, and layout region processing) can be executed once for the same target object and the same second description information, without needing to be executed multiple times. The fourth process (image generation) requires multiple iterations to generate a second image corresponding to each preset orientation. Here, the image generation process can be executed multiple times according to the number of preset orientations. For example, if the number of preset orientations is N, then N second images can be generated, with the orientation of the target object in different second images determined based on different preset orientations.

[0151] In some embodiments, at least two sample images of an object are acquired; wherein the same object is oriented differently in the different sample images.

[0152] The initial model is trained based on the at least two sample images until the preset training conditions are met, thereby obtaining an image generation model; wherein, the image generation model is used to generate at least two images including the same object, and the same object has different orientations in the different images generated by the image generation model;

[0153] Generate at least two second images that include the target object, including:

[0154] The target object is input into the image generation model to obtain at least two second images.

[0155] In some embodiments, at least two sample images can be acquired for the same object; wherein the same object is in a preset orientation in the sample images, and the preset orientation of the same object is different in different sample images. Here, the difference between the preset orientations of the same object in any two sample images can be greater than a preset difference threshold. For example, the difference threshold can be 45°.

[0156] In some embodiments, at least two preset orientations can be determined from a preset orientation range based on a preset interval value. For each object, a sample image including the object can be acquired based on at least two preset orientations. Exemplarily, the second number can be 10, the preset interval value can be 45°, and the orientation range can be [0°, 360°]. The second number of preset orientations can include: 0°, 45°, 90°, 135°, 180°, 225°, 270°, 315°, a top-down angle, and a bottom-up angle. Here, according to the 360-degree horizontal and vertical directions, with the front as 0 degrees, rotating clockwise every 45 degrees, one orientation is obtained at a time, thus obtaining 10 preset orientations. For the same object, at least 10 sample images can be acquired, with the same object in different preset orientations in different sample images.

[0157] In this embodiment of the disclosure, an image generation model that can generate multiple images with different orientations for the same object can be pre-trained. In the application stage, the image generation model can be directly used to generate at least two second images containing the target object, which can improve the efficiency of generating multiple second images for the same target object.

[0158] In some embodiments, an initial model is trained based on at least two sample images until preset training conditions are met to obtain an image generation model, including:

[0159] For each sample image, a label is generated based on the first descriptive information of the sample image and the orientation of the object in the sample image to obtain a labeled sample image; wherein, the first descriptive information is used to describe at least one of the following: the image content corresponding to the object in the sample image, the background content of the sample image, and the overall image content of the sample image;

[0160] The initial model is trained based on all labeled sample images until the preset training conditions are met, thus obtaining the image generation model.

[0161] The target object is input into the image generation model to obtain at least two second images, including:

[0162] The target object and the second descriptive information used for image generation are input into the image generation model to obtain at least two second images.

[0163] In some embodiments, the first description information may include at least one of the following descriptive sub-information: a first descriptive sub-information for describing the image content corresponding to an object in the sample image; a second descriptive sub-information for describing the background content of the sample image; and a third descriptive sub-information for describing the overall image content of the sample image. For each sample image, at least one of the first, second, and third descriptive information, as well as the orientation of the object in the sample image, can be used to label the sample image, resulting in a labeled sample image.

[0164] In some embodiments, the labels of annotated sample images may include: a first label, determined based on a first descriptor; a second label, determined based on a second descriptor; and a third label, used to indicate a preset orientation of the object in the sample image. For example, the Blip2 model can be used to label sample images. To incorporate the preset orientation information into the model training process, the preset orientation can be labeled with text labels to obtain the third label. Here, the third label can be understood as third descriptive information describing the preset orientation. The labels of annotated sample images may also include a fourth label, which can be the preset orientation itself. For example, if the preset orientation of the object is 45°, then the fourth label can be 45°.

[0165] In some embodiments, the first descriptive information may further include fourth descriptive information, which describes the position and / or size of the object in the sample image. The label of the annotated sample image may further include a fifth label, which is determined based on the fourth descriptive information. It should be noted that by annotating the position labels of the object's location, an image generation model capable of controlling the object's generated position can be trained based on the annotated sample image.

[0166] In this embodiment of the disclosure, the sample image can be labeled using at least one of the first descriptor information, the second descriptor information, and the third descriptor information. This allows for a precise description of the image content of the sample image from multiple dimensions, improving the correlation between the labeled sample image and the image content. Consequently, after training the initial model based on the labeled sample image, the accuracy of the trained image generation model in generating a second image with specific image content can be improved.

[0167] In some embodiments, acquiring at least two sample images including an object includes:

[0168] Obtain at least two candidate images containing the object from a preset video;

[0169] Based on the performance parameters of each candidate image, at least two first sample images are obtained from at least two candidate images; wherein the performance parameters are used to indicate the image quality of the candidate images.

[0170] In some embodiments, candidate images whose performance parameters are greater than a preset threshold among at least two candidate images can be determined as first sample images. It should be noted that the performance parameters can indicate at least one of the following: the completeness of objects in the candidate images, the sharpness of the candidate images, and the degree of occlusion of objects in the candidate images. The performance parameters can be positively correlated with the image quality of the candidate images. Here, based on the performance parameters of each of the at least two candidate images, blurry images, low-quality images, and images with severe subject occlusion can be filtered out from the at least two candidate images, thereby improving the quality of the determined sample images.

[0171] In some embodiments, a candidate image whose performance parameter is greater than a preset parameter threshold and which contains an object in a preset orientation can be determined as the first sample image.

[0172] In this embodiment, on the one hand, candidate images can be directly extracted from a preset video, and a first sample image can be determined based on the acquired candidate images, thereby reducing the difficulty of acquiring sample images from multiple perspectives and improving the efficiency of acquiring sample images. On the other hand, based on the performance parameters of each candidate image, candidate images with excessively low performance parameters among at least two candidate images can be filtered out, that is, candidate images with low image quality among at least two candidate images can be filtered out to obtain a first sample image with good image quality, thereby improving the training effect of training the initial model based on the first sample image, so as to train an image generation model with good image generation effect.

[0173] In some embodiments, based on the performance parameters of each candidate image, at least two first sample images are obtained from at least two candidate images, including:

[0174] Based on the type of object in each candidate image, at least two candidate images are grouped to obtain at least one set of candidate images;

[0175] Based on the performance parameters of each candidate image in each group of candidate images, a benchmark image that meets the preset performance conditions is determined from each group of candidate images, and the similarity between the benchmark image in each group of candidate images and each candidate image other than the benchmark image is determined.

[0176] The baseline image and the candidate images with a similarity greater than a preset similarity threshold in each group of candidate images are determined as the first sample images.

[0177] In some embodiments, the preset performance conditions may include at least one of the following: the sharpness of the candidate image is greater than a preset sharpness threshold, the integrity of the object in the candidate image is greater than a preset integrity threshold, and the occlusion degree of the object in the candidate image is less than a preset occlusion degree threshold. Thus, the candidate image with a clear subject and minimal occlusion in each group of candidate images can be used as the reference image.

[0178] In some embodiments, candidate images with a similarity greater than a preset similarity threshold and containing objects in a preset orientation can be determined as first sample images. Candidate images with a similarity to a reference image less than a preset similarity threshold and / or candidate images containing objects not in a preset orientation can be deleted from each group of candidate images. For example, the preset similarity threshold can be 0.6.

[0179] In some embodiments, duplicate candidate images in each group of candidate images may be removed before determining the first sample image in each group of candidate images. It should be noted that the similarity between duplicate candidate images may be greater than a preset third similarity threshold, and the fourth similarity threshold may be greater than the preset similarity threshold in this disclosure. For example, the third similarity threshold may be 0.99. This can reduce the likelihood of duplicate candidate images being identified as the first sample image, and can reduce the possibility of poor generalization of the trained image generation model due to single training based on the same first sample image.

[0180] In some embodiments, it can be determined whether the orientation of an object in a sample image is a preset orientation. Sample images containing objects that are not in the preset orientation can be deleted. In this way, the orientation of reference objects in the retained sample images can all be the preset orientation.

[0181] In some embodiments, a first sample image from each group of candidate images can be displayed, and the first sample image affected by the fourth touch operation can be deleted based on the fourth touch operation on the first sample image. Alternatively, suitable sample images can be further filtered through manual screening.

[0182] In some embodiments, after determining the first sample image in each group of candidate images, the orientation of the object in the first sample image can be compared with a preset orientation to obtain a comparison result. Based on the comparison result, the position and / or orientation of the object in the first sample image can be adjusted to obtain an adjusted first sample image.

[0183] It should be noted that candidate images obtained from real-world pre-set videos have inherent limitations, while the orientation of objects in the constructed sample images is relatively strict. Therefore, some candidate images may not meet the requirements of the constructed sample images. For example, a candidate image corresponding to a certain pre-set orientation may be missing, or there may be a deviation between the orientation of objects in the candidate images and the pre-set orientation. Based on this, the candidate images can be adjusted to construct sample images that meet the requirements.

[0184] For example, a stable diffusion inpaint model can be used to modify candidate images, including repositioning and / or orientation shifting of objects within the candidate images. Simultaneously, the stable diffusion model can be overlaid with an Ip-Adapter technique to create raw images of the objects, filling in missing elements under a preset orientation, thereby adjusting the position and / or orientation of objects in the candidate images. Through these operations, the completeness of the sample image set constructed based on the candidate images can be improved.

[0185] In this embodiment of the disclosure, candidate images can be classified based on the objects contained within them, so as to identify a high-quality reference image from each group of candidate images, and to quickly filter low-quality candidate images based on the similarity between the reference image and other images. This improves the image quality of the acquired first sample image and increases the efficiency of identifying high-quality first sample images.

[0186] In some embodiments, the method further includes:

[0187] The object and the image content other than the object in the first sample image are segmented to obtain a second sample image containing the object;

[0188] The initial model is trained based on at least two sample images, including:

[0189] The initial model is trained based on the first and second sample images.

[0190] It should be noted that segmenting the object and the image content other than the object in the first sample image can be understood as performing image cutout processing on the object in the first sample image to obtain the second sample image containing the object.

[0191] It should be noted that, for the same object, a preset number of first sample images and a preset number of second sample images containing that object can be obtained. The preset number can be the number of preset orientations. For example, if there are 10 preset orientations, then for each preset orientation, a first sample image and a second sample image corresponding to that preset orientation can be obtained; wherein, the object in the sample image corresponding to the preset orientation is located in that preset orientation.

[0192] In this embodiment of the disclosure, for model training, a first sample image containing an object can be matted out, and the matted second sample image and the original first sample image can be combined to train the initial model, thereby increasing the richness of the sample images used for training the initial model. Furthermore, since the sample image set contains both the first and second sample images, on the one hand, the first sample image can be used to train the initial model to enhance its adaptability to different background content, making it easier for the trained image generation model to generate second images of other background scenes. On the other hand, the second sample image can be used to train the initial model to reduce the influence of background content in the modeling process, enhancing the adaptability of the trained image generation model to different objects, making it easier for the trained image generation model to generate second images containing any object.

[0193] Figure 4 This is a flowchart illustrating a method for training an initial model to obtain an image generation model according to an exemplary embodiment. See also Figure 4 The method includes the following steps 401 to 407:

[0194] In step 401, a preset video is acquired.

[0195] In step 402, at least two candidate images containing objects are obtained from the preset video.

[0196] In step 403, based on the performance parameters of each candidate image, at least two candidate images that do not meet the performance requirements are filtered to obtain filtered candidate images.

[0197] In step 404, based on the relationship between the orientation of the object in the filtered candidate image and the preset orientation, the position and / or orientation of the object in the filtered candidate image are adjusted to obtain the first sample image.

[0198] In step 405, the objects in the first sample image are segmented to obtain a second sample image containing the objects.

[0199] In step 406, for each of the first and second sample images, the sample image is labeled based on the first description information of the sample image and the orientation of the object in the sample image, to obtain the labeled sample image.

[0200] In step 407, the initial model is trained based on all labeled sample images until the preset training conditions are met, thus obtaining the image generation model.

[0201] Figure 5 This is a structural block diagram of an image processing apparatus according to an exemplary embodiment, see below. Figure 5 The device includes:

[0202] Display module 501 is configured to display a first image; wherein the first image contains at least one object;

[0203] The generation module 502 is configured to generate at least two second images including the target object in response to a triggering operation for a target object in at least one object; wherein the target object has a different orientation in the different second images.

[0204] In some embodiments, the display module 501 is configured to display a preset interface and determine the layout area of ​​the target object in the preset interface based on touch operations on the preset interface; the generation module 502 is configured to generate at least two second images based on the layout area; wherein the size and / or position of the target object in the second image is determined based on the layout area.

[0205] In some embodiments, the display module 501 is configured to display a preset interface of a first size; the generation module 502 is configured to generate at least two second images of a second size based on a layout area; wherein the ratio between each side of the preset interface of the first size and the corresponding side of the second image of the second size is the same.

[0206] In some embodiments, the apparatus further includes: a determining module configured to determine a target region in the initial image corresponding to the layout region; the generating module 502 further includes: a processing module configured to process the target region in the initial image based on a first noise parameter corresponding to the image features of the target object in the first image, and to process other regions in the initial image besides the target region based on a preset second noise parameter to obtain a processed initial image; and to perform denoising processing on the processed initial image to obtain at least two second images in which the target object is located within the target region.

[0207] In some embodiments, the apparatus further includes: an acquisition module configured to acquire at least two sample images including an object; wherein the same object has different orientations in different sample images; a training module configured to train an initial model based on the at least two sample images until a preset training condition is met to obtain an image generation model; wherein the image generation model is used to generate at least two images including the same object, and the same object has different orientations in different images generated by the image generation model; and a generation module 502 configured to input the target object into the image generation model to obtain at least two second images.

[0208] In some embodiments, the apparatus further includes: a labeling module configured to label each sample image based on first descriptive information for the sample image and the orientation of an object in the sample image, to obtain a labeled sample image; wherein the first descriptive information is used to describe at least one of the following: image content corresponding to an object in the sample image, background content of the sample image, and overall image content of the sample image; a training module configured to train an initial model based on all labeled sample images until a preset training condition is met to obtain an image generation model; and a generation module configured to input a target object and second descriptive information for image generation into the image generation model to obtain at least two second images.

[0209] In some embodiments, the acquisition module is configured to acquire at least two candidate images including an object from a preset video; and acquire at least two first sample images from the at least two candidate images based on the performance parameters of each candidate image; wherein the performance parameters are used to indicate the image quality of the candidate images.

[0210] In some embodiments, the apparatus further includes: a grouping module configured to group at least two candidate images based on the type of the object in each candidate image to obtain at least one set of candidate images; and a determining module configured to determine a benchmark image that meets a preset performance condition from each set of candidate images based on the performance parameters of each candidate image in each set of candidate images, and to determine the similarity between the benchmark image in each set of candidate images and each candidate image other than the benchmark image; and to determine the benchmark image in each set of candidate images, as well as the candidate images with a similarity greater than a preset similarity threshold, as first sample images.

[0211] In some embodiments, the apparatus further includes: a segmentation module configured to segment objects and image content other than objects in a first sample image to obtain a second sample image containing objects; and a training module configured to train an initial model based on the first sample image and the second sample image.

[0212] The image processing method provided in this embodiment can be applied to electronic devices. Figure 5 This is a structural block diagram illustrating an electronic device 500 according to an exemplary embodiment. For example, the electronic device 500 may be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, etc.

[0213] Reference Figure 6 The electronic device 600 may include one or more of the following components: processing component 602, memory 604, power supply component 606, multimedia component 608, audio component 610, input / output (I / O) interface 612, sensor component 614, and communication component 616.

[0214] Processing component 602 typically controls the overall operation of electronic device 600, such as operations associated with at least one of display, telephone call, data communication, camera operation, and recording operation. Processing component 602 may include one or more processors 620 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 602 may include one or more modules to facilitate interaction between processing component 602 and other components. For example, processing component 602 may include a multimedia module to facilitate interaction between multimedia component 608 and processing component 602.

[0215] Memory 604 is configured to store various types of data to support the operation of electronic device 600. Examples of such data include at least one of the following: instructions for any application or method operating on electronic device 600, contact data, phonebook data, messages, pictures, and videos. Memory 604 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0216] Power supply component 606 provides power to various components of electronic device 600. Power supply component 606 may include at least one of the following: a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to electronic device 600.

[0217] Multimedia component 608 includes a screen that provides an output interface between electronic device 600 and user. In some embodiments, the screen may include a Liquid Crystal Display (LCD) and a Touch Panel (TP). If the screen includes a Touch Panel, the screen may be implemented as a touchscreen to receive input signals from the user. The Touch Panel includes one or more touch sensors to sense touches, swipes, and gestures on the Touch Panel. The touch sensors may sense not only the boundaries of touch or swipe actions but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 608 includes a front-facing camera and / or a rear-facing camera. When device 600 is in an operating mode, such as shooting mode or video mode, the front-facing camera and / or rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0218] Audio component 610 is configured to output and / or input audio signals. For example, audio component 610 includes a microphone (MIC) configured to receive external audio signals when electronic device 600 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 604 or transmitted via communication component 616. In some embodiments, audio component 610 also includes a speaker for outputting audio signals.

[0219] I / O interface 612 provides an interface between processing component 602 and peripheral interface modules, such as keyboards, click wheels, and buttons. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0220] Sensor assembly 614 includes one or more sensors for providing state assessments of various aspects of electronic device 600. For example, sensor assembly 614 can detect the on / off state of electronic device 600, the relative positioning of components such as the display and keypad of electronic device 600, changes in position of electronic device 600 or one of its components, the presence or absence of user contact with electronic device 600, orientation or acceleration / deceleration of electronic device 600, and temperature changes of electronic device 600. Sensor assembly 614 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 614 may also include an optical sensor, such as a Complementary Metal Oxide Semiconductor (CMOS) or Charge Coupled Device (CCD) image sensor, for use in imaging applications.

[0221] In some embodiments, the sensor assembly 614 may also include, but is not limited to, at least one of the following: an accelerometer, a gyroscope, a magnetic sensor, a pressure sensor, and a temperature sensor.

[0222] Communication component 616 is configured to facilitate wired or wireless communication between electronic device 600 and other devices. Electronic device 600 can access wireless networks based on communication standards, such as Wi-Fi, 4G, 6G, or combinations thereof. In one exemplary embodiment, communication component 616 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 616 also includes a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Ultra Wide Band (UWB), Bluetooth (BT), and other technologies.

[0223] In an exemplary embodiment, the electronic device 600 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components.

[0224] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 604 including executable instructions or a computer program, which can be executed by a processor 620 of an electronic device 600 to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, floppy disk, and optical data storage device, etc.

[0225] A non-transitory computer-readable storage medium, when the instructions in the storage medium are executed by a processor of a mobile electronic device, enables the mobile electronic device to perform any of the image processing methods described above in the embodiments of this disclosure. For example, the image processing method includes:

[0226] Display a first image; wherein the first image contains at least one object;

[0227] In response to a triggering operation targeting a target object in at least one object, at least two second images including the target object are generated; wherein the target object is oriented differently in the different second images.

[0228] This disclosure provides a computer program product comprising a computer program or executable instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer program or executable instructions from the computer-readable storage medium and executes the computer program or executable instructions, causing the computer device to perform any of the image processing methods described above in this disclosure.

[0229] Figure 7 This is a block diagram illustrating an apparatus 700 for image generation according to an exemplary embodiment. For example, apparatus 700 may be provided as a server. (Refer to...) Figure 7 The apparatus 700 includes a processing component 722, which further includes one or more processors, and memory resources represented by memory 732 for storing instructions, such as application programs, that can be executed by the processing component 722. The application programs stored in memory 732 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 722 is configured to execute instructions to perform the aforementioned image processing method:

[0230] Display a first image; wherein the first image contains at least one object;

[0231] In response to a triggering operation targeting a target object in at least one object, at least two second images including the target object are generated; wherein the target object is oriented differently in the different second images.

[0232] The device 700 may also include a power supply component 726 configured to perform power management of the device 700, a wired or wireless network interface 750 configured to connect the device 700 to a network, and an input / output (I / O) interface 758. The device 700 can operate an operating system stored in memory 732, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, or similar.

[0233] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the foregoing claims.

[0234] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. An image processing method, characterized in that, include: Display a first image; wherein the first image contains at least one object; In response to a triggering operation targeting a target object in the at least one object, at least two second images including the target object are generated; wherein the target object has a different orientation in the different second images.

2. The image processing method according to claim 1, characterized in that, The generation of at least two second images including the target object includes: Display a preset interface, and determine the layout area of ​​the target object in the preset interface based on the touch operation on the preset interface; Based on the layout region, the at least two second images are generated; wherein the size and / or position of the target object in the second image is determined based on the layout region.

3. The image processing method according to claim 2, characterized in that, The preset display interface includes: Displays the default interface of the first size; The generation of the at least two second images based on the layout region includes: Based on the layout area, at least two second images of a second size are generated; wherein the ratio between each side of the preset interface of the first size and the corresponding side of the second image of the second size is the same.

4. The image processing method according to claim 2, characterized in that, The generation of the at least two second images based on the layout region includes: Determine the target region in the initial image that corresponds to the layout region; Based on a first noise parameter corresponding to the image features of the target object in the first image, the target region in the initial image is processed, and based on a preset second noise parameter, other regions in the initial image besides the target region are processed to obtain a processed initial image; The processed initial image is denoised to obtain at least two second images in which the target object is located within the target region.

5. The image processing method according to claim 1, characterized in that, The method further includes: Obtain at least two sample images containing an object; wherein the same object is oriented differently in the different sample images. The initial model is trained based on the at least two sample images until the preset training conditions are met, thereby obtaining an image generation model; wherein, the image generation model is used to generate at least two images including the same object, and the same object has different orientations in the different images generated by the image generation model; The generation of at least two second images including the target object includes: The target object is input into the image generation model to obtain the at least two second images.

6. The image processing method according to claim 5, characterized in that, The step of training the initial model based on the at least two sample images until preset training conditions are met to obtain the image generation model includes: For each of the sample images, the sample image is labeled based on the first description information of the sample image and the orientation of the object in the sample image to obtain a labeled sample image; wherein, the first description information is used to describe at least one of the following: the image content corresponding to the object in the sample image, the background content of the sample image, and the overall image content of the sample image. The initial model is trained based on all the labeled sample images until the preset training conditions are met, thus obtaining the image generation model. The step of inputting the target object into the image generation model to obtain the at least two second images includes: The target object and the second descriptive information used for image generation are input into the image generation model to obtain the at least two second images.

7. The image processing method according to claim 5, characterized in that, The acquisition of at least two sample images of the object includes: Obtain at least two candidate images containing the object from a preset video; Based on the performance parameters of each candidate image, at least two first sample images are obtained from the at least two candidate images; wherein the performance parameters are used to indicate the image quality of the candidate images.

8. The image processing method according to claim 7, characterized in that, The step of obtaining at least two first sample images from the at least two candidate images based on the performance parameters of each candidate image includes: Based on the type of the object in each candidate image, the at least two candidate images are grouped to obtain at least one group of candidate images; Based on the performance parameters of each candidate image in each group of candidate images, a benchmark image that meets the preset performance conditions is determined from each group of candidate images, and the similarity between the benchmark image in each group of candidate images and each candidate image other than the benchmark image is determined. The baseline image and the candidate images with a similarity greater than a preset similarity threshold in each group of candidate images are determined as the first sample images.

9. The image processing method according to claim 7, characterized in that, The method further includes: The object and other image content in the first sample image are segmented to obtain a second sample image containing the object; The training of the initial model based on the at least two sample images includes: The initial model is trained based on the first sample image and the second sample image.

10. An image processing apparatus, characterized in that, The device includes: A display module is configured to display a first image; wherein the first image contains at least one object; The generation module is configured to generate at least two second images including the target object in response to a triggering operation for a target object in the at least one object; wherein the target object has a different orientation in the different second images.

11. The image processing apparatus according to claim 10, characterized in that, The device also includes a determining module; The display module is configured to display a preset interface; The determining module is configured to determine the layout area of ​​the target object in the preset interface based on the touch operation on the preset interface; The generation module is configured to generate at least two second images including the target object based on the layout region; wherein the size and / or position of the target object in the second image is determined based on the layout region.

12. The image processing method according to claim 11, characterized in that, The display module is configured to display a preset interface of a first size; The generation module is configured to generate at least two second images of a second size based on the layout area; wherein the ratio between each side of the preset interface of the first size and the corresponding side of the second image of the second size is the same.

13. An electronic device, characterized in that, include: processor; Memory used to store computer programs or instructions; The processor executes the computer program or instructions to implement the steps of the method according to any one of claims 1 to 9.

14. A non-transitory computer-readable storage medium storing a computer program or instructions, characterized in that, When the computer program or instructions in the storage medium are executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

15. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 9.

Images