Picture album generation method

Abstract

The invention discloses a picture album generation method, and the method comprises the steps: obtaining a scene picture, and extracting story description information from the scene picture; generating a target story text corresponding to the story description information, and determining a story plot corresponding to the target story text; and according to the prompt words in each story plot, generating a figure story picture corresponding to each story plot, and according to the figure story picture corresponding to each story plot, constructing a picture album. In the application, the story text is generated by acquiring the story description information in the scene picture, and the figure story picture of each story line is generated based on the cue word of each story line in the story text, so that the picture album is constructed based on the figure story picture corresponding to each story line; in other words, the picture album can be generated in a one-key mode through the scene picture, the generation time of the picture album is shortened, and the generation efficiency of the picture album is improved.

Description

Technical Field

[0001] The present application belongs to the field of data processing technology, and in particular, relates to a method for generating a picture album. Background Art

[0002] Picture album is a special form of visual art. It combines real photos of people with comic strips, telling stories through a series of continuous character comic strips and plot texts, and has a strong sense of reality and vividness.

[0003] In the exemplary technology, when making a picture album, first write a story text, refer to the situational content of the story text, and use traditional photography technology to take real-life modeling pictures of the same posture and position, and use a computer to merge the real people and background pictures to produce a series of real-life comic strips with storylines, and then synthesize the picture album based on each real-life comic strip.

[0004] However, it takes a long time to write the story text, and it also takes a long time to shoot real people and background pictures through traditional photography technology, resulting in low efficiency in the production of the album. Summary of the invention

[0005] The embodiment of the present application provides a method for generating a picture album to solve the problem of low efficiency in producing the picture album.

[0006] In a first aspect, an embodiment of the present application provides a method for generating a picture album, comprising:

[0007] Obtaining a scene image, and extracting story description information from the scene image;

[0008] Generate a target story text corresponding to the story description information, and determine a story plot corresponding to the target story text;

[0009] According to the prompt words in each of the story plots, a character story picture corresponding to each of the story plots is generated, and a picture album is constructed according to the character story pictures corresponding to each of the story plots.

[0010] In one embodiment, generating a target story text corresponding to the story description information includes:

[0011] Obtaining a story generation instruction, and parsing the story generation instruction to obtain a target parameter, wherein the target parameter includes at least one of a story type and a text style type;

[0012] A target story text matching the target parameters is generated according to the story description information.

[0013] In one embodiment, generating a target story text corresponding to the story description information includes:

[0014] Generate a plurality of first story texts according to the story description information;

[0015] Determining a rating score for each of the first story texts;

[0016] A first target score is determined among the rating scores, and a first story text corresponding to the first target score is determined as the target story text, wherein the first target score is used to indicate a rating score that is greater than a first preset score.

[0017] In one embodiment, generating a target story text corresponding to the story description information includes:

[0018] Generate a second story text corresponding to the story description information, and display the second story text;

[0019] In response to an editing operation on the second story text, the second story text is modified according to the editing operation to obtain the target story text.

[0020] In one embodiment, generating a character story picture corresponding to each story plot according to a prompt word in each story plot includes:

[0021] Obtaining prompt words for the story plot and generating story illustrations corresponding to the story plot;

[0022] Determining layout parameters of a target object in the story illustration, the target object including a person and / or an object in the story illustration, the layout parameters being used to indicate a circumscribed rectangular frame of the target object;

[0023] extracting composition features in the story illustration according to the layout parameters;

[0024] According to the composition features corresponding to the storyline, the prompt words, the story illustrations and the diffusion model, the preset noise image corresponding to the storyline is denoised to obtain a character story image corresponding to the storyline.

[0025] In one embodiment, the diffusion model is provided with a composition adapter, and the composition adapter is used to process the composition features corresponding to the storyline.

[0026] In one embodiment, the diffusion model is provided with an encoder, and the encoder is used to process the prompt word and the story illustration so as to unify the graphic content of the prompt word and the story illustration.

[0027] In one embodiment, the diffusion model is provided with a sub-space transformation self-attention noise estimation network, and the sub-space transformation self-attention noise estimation network denoises the preset noise map based on the composition features, the prompt words and the story illustrations.

[0028] In one embodiment, before denoising the preset noise image corresponding to the storyline according to the composition features corresponding to the storyline, the prompt words, the story illustrations and the diffusion model, the method further includes:

[0029] Obtain multiple training samples, and train the preset model according to each training sample to obtain an intermediate model;

[0030] A face training image is obtained, and the intermediate model is trained according to the face training image to obtain a diffusion model, wherein the face training image includes a face image of a target person specified by a user.

[0031] In one embodiment, obtaining a face training image includes:

[0032] Acquire a first picture uploaded by the user, where the first picture includes a face of a target person specified by the user at a first angle;

[0033] Displaying a plurality of second pictures, each of the second pictures includes a face of the target person at a second angle, the second angle being different from the first angle;

[0034] In response to detecting a selection operation, a face training image is constructed based on the second image selected by the selection operation and the first image.

[0035] In one embodiment, denoising a preset noise image corresponding to the storyline according to the composition features corresponding to the storyline, the prompt words, the story illustrations, and the diffusion model includes:

[0036] De-noising a preset noise image corresponding to the storyline according to the composition features corresponding to the storyline, the prompt words, the story illustrations, and the diffusion model to obtain a plurality of intermediate images corresponding to the storyline;

[0037] Extracting a first image feature and a text feature of each of the intermediate images, and determining a first similarity between the first image feature and the text feature of the intermediate image, and determining a second similarity between the first image feature of the intermediate image and a second image feature in the template face image;

[0038] Determine a quality score corresponding to the intermediate picture according to the first similarity, the second similarity, the first image feature, and the text feature corresponding to the intermediate picture;

[0039] A second target score is determined among the quality scores, and a character story picture corresponding to the storyline is determined based on the intermediate picture corresponding to the second target score, wherein the second target score is a quality score greater than a second preset score.

[0040] In one embodiment, determining the character story picture corresponding to the story plot according to the intermediate picture corresponding to the second target score includes:

[0041] The face in the middle picture corresponding to the second target score is merged with the face in the template face image to obtain a character story picture corresponding to the storyline.

[0042] In one embodiment, after constructing the picture album according to the character story pictures corresponding to each of the story plots, the method further includes:

[0043] Generating speech corresponding to the target story text;

[0044] A story video is generated based on the voice and the character story pictures in the album.

[0045] In one embodiment, after constructing the picture album according to the character story pictures corresponding to each of the story plots, the method further includes:

[0046] Displaying the album;

[0047] In response to a sharing operation on the album, determining a target terminal corresponding to the analysis operation;

[0048] The picture album is sent to the target terminal.

[0049] In one embodiment, after displaying the album, the method further includes:

[0050] Obtaining comment information on the album;

[0051] Determining the elements to be modified in the album according to the comment information;

[0052] The elements to be modified in the album are modified based on the comment information.

[0053] In a second aspect, an embodiment of the present application provides a picture album generating device, including:

[0054] An acquisition module, used to acquire a scene image and extract story description information from the scene image;

[0055] A first generating module, used to generate a target story text corresponding to the story description information, and determine a story plot corresponding to the target story text;

[0056] The second generating module is used to generate a character story picture corresponding to each story plot according to the prompt words in each story plot, and to construct a picture album according to the character story pictures corresponding to each story plot.

[0057] In a third aspect, an embodiment of the present application provides a picture album generating device, the picture album generating device comprising: a processor and a memory storing computer program instructions;

[0058] When the processor executes the computer program instructions, the picture album generating method as described in one aspect is implemented.

[0059] In a fourth aspect, an embodiment of the present application provides a computer storage medium, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the method for generating a picture album as described in the first aspect is implemented.

[0060] In a fifth aspect, an embodiment of the present application provides a computer program product. When instructions in the computer program product are executed by a processor of an electronic device, the electronic device executes the album generation method as in the first aspect.

[0061] The picture album generation method provided in the embodiment of the present application generates a story text by acquiring the story description information in the scene picture, and generates a character story picture of each story plot based on the prompt words of each story plot in the story text, thereby constructing a picture album based on the character story pictures corresponding to each story plot, that is, the picture album can be generated by one click through the scene pictures, which reduces the generation time of the picture album and improves the generation efficiency of the picture album. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] In order to more clearly illustrate the technical solution of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0063] Figure 1 It is a schematic diagram of an application scenario in which the method for generating a picture album provided in an embodiment of the present application can be implemented;

[0064] Figure 2 It is a flowchart of the first embodiment of the method for generating a picture album of the present application;

[0065] Figure 3 It is a flowchart of the second embodiment of the method for generating a picture album of the present application;

[0066] Figure 4 It is a flowchart of the third embodiment of the method for generating a picture album of the present application;

[0067] Figure 5It is a schematic diagram of the process of obtaining the target story text in this application;

[0068] Figure 6 It is a flowchart of the fourth embodiment of the method for generating a picture album of the present application;

[0069] Figure 7 It is a schematic diagram of the process of obtaining the compositional features in this application;

[0070] Figure 8 is a schematic diagram of the structure of the patterning adapter in this application;

[0071] Fig. 9 It is a schematic diagram of the workflow of multimodal joint encoding in this application;

[0072] Fig.10 This is a schematic diagram of the workflow of the sub-space transformation self-attention layer in this application;

[0073] Fig.11 It is a flowchart of the fifth embodiment of the method for generating a picture album of the present application;

[0074] Fig.12 It is a flowchart of the sixth embodiment of the method for generating a picture album of the present application;

[0075] Fig.13 This is a schematic diagram of the acquisition process of the training model of this application;

[0076] Fig.14 It is a flowchart of the seventh embodiment of the method for generating a picture album of the present application;

[0077] Fig.15 It is a brief flow chart of the album generation in this application;

[0078] Fig.16 It is a functional module schematic diagram of the picture album generating device involved in this application;

[0079] Fig.17 It is a structural schematic diagram of the picture album generating device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0080] The features and exemplary embodiments of various aspects of the present application will be described in detail below. In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain the present application, rather than to limit the present application. For those skilled in the art, the present application can be implemented without the need for some of these specific details. The following description of the embodiments is only to provide a better understanding of the present application by illustrating the examples of the present application.

[0081] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "include..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0082] Picture album is a special form of visual art. It combines real photos of people with comic strips, telling stories through a series of continuous character comic strips and plot texts, and has a strong sense of reality and vividness.

[0083] In the exemplary technology, when making a picture album, first write a story text, refer to the situational content of the story text, and use traditional photography technology to take real-life modeling pictures of the same posture and position, and use a computer to merge the real people and background pictures to produce a series of real-life comic strips with storylines, and then synthesize the picture album based on each real-life comic strip.

[0084] The inventor of the present application discovered that it takes a long time to write the story text, and it also takes a long time to photograph real people and background images using traditional photography techniques, resulting in low efficiency in producing the album.

[0085] The inventor of the present application therefore came up with the idea of ​​generating a story text by acquiring the story description information in the scene picture, and generating a character story picture of each story plot based on the prompt words of each story plot in the story text, thereby constructing a picture album based on the character story pictures corresponding to each story plot. In other words, a picture album can be generated with one click through the scene pictures, which reduces the generation time of the picture album and improves the generation efficiency of the picture album.

[0086] The embodiment of the present application provides a method for generating a picture album. The following first illustrates an application scenario diagram to which the method for generating a picture album provided by the embodiment of the present application can be applied.

[0087] like Figure 1As shown, the user uploads scene pictures in the album generation device 200 based on the mobile terminal 100, and the album generation device 200 extracts story description information from the scene pictures and generates a story text corresponding to the story description information. The album generation device 200 generates character story pictures corresponding to each story plot based on the prompt words of each story plot in the story text, and the album generation device 200 assembles the various character story pictures to obtain a picture album, and outputs the picture album.

[0088] The following combination Figure 1 , the picture album generation method provided in the embodiment of the present application is introduced.

[0089] Figure 2 FIG. 1 is a flow chart showing a first embodiment of the method for generating a picture album provided by the present application. Figure 2 As shown, the album generation method includes:

[0090] Step S201, obtaining a scene picture, and extracting story description information from the scene picture.

[0091] In this embodiment, the execution subject is a picture album generating device. For the convenience of description, the following uses the term "device" to refer to the picture album generating device. The device can be a server or any terminal device with a picture album generating function.

[0092] The user can select a picture from the cloud disk or local album and upload the picture as a scene picture to the device to generate a picture album.

[0093] After the device obtains the scene picture, it extracts the story description information from the scene picture. Exemplarily, an image understanding large model may be set in the device, and the story description information can be obtained by extracting key information from the picture through the image understanding large model. Key information is, for example, scene and task. The image understanding large model includes a mixture of encoder-decoder framework, which combines understanding and generation capabilities. The mixture of encoder-decoder framework refers to combining multiple encoder and decoder models together, each model processing a specific task or data type. The image understanding large model is trained through Captioning and Filtering technology. Training in this way can improve the data cleaning and data enhancement capabilities of the image understanding large model, so that the image understanding large model can effectively extract key information of the scene picture as story description information.

[0094] Step S202, generating a target story text corresponding to the story description information, and determining a story plot corresponding to the target story text.

[0095] After obtaining the story description information, the device generates a target story text corresponding to the story description information. Exemplarily, the device can use a large language model to generate the target story text corresponding to the story description information. The device inputs the story description information into the large language model, and the large language model can output the story text as the target story text.

[0096] After obtaining the target story text, the device determines each story plot in the target story text. Exemplarily, the device divides the target story text into a minimum plot as a unit to obtain multiple story plots. The minimum plot is, for example, a paragraph in the target story text, or the minimum plot is the text content between two adjacent periods in the target story text.

[0097] Step S203, generating a character story picture corresponding to each story plot according to the prompt words in each story plot, and constructing a picture album according to the character story pictures corresponding to each story plot.

[0098] The device generates a corresponding character story picture for each storyline. Exemplarily, the device searches for prompt words from the storyline, and the prompt words are divided into global prompt words and local prompt words. Global prompt words are, for example, descriptive terms for scenes in the storyline, such as "sunny" and "gloomy environment" can be used as global prompt words. Local prompt words can be characters, objects, character expressions, character positions, character actions, object shapes, distances between objects and characters, and placements of objects in the storyline.

[0099] The device extracts prompt words from the storyline, and based on the prompt words, a character story picture corresponding to the storyline can be generated. For example, the storyline describes that character A is dining on a ship, and the prompt words are "character A", "ship", and "dining", then a character story picture of "a ship sailing in the ocean, character A on the ship sits at a table on the deck of the ship, there is food on the table, and the action of character A is to pick up the food" will be generated.

[0100] In this way, the character story pictures corresponding to each story plot can be generated, and the device generates and constructs a picture album based on the character story pictures of each story plot. Specifically, the character story pictures are assembled according to the position of the story plot in the target story text to obtain the picture album.

[0101] In this embodiment, a story text is generated by acquiring story description information in a scene picture, and a character story picture of each story plot is generated based on the prompt words of each story plot in the story text, so that a picture album is constructed based on the character story pictures corresponding to each story plot. That is, a picture album can be generated with one click through scene pictures, without the need for manual writing of the story text, and without the need for manual photography of the scenes in the story text to obtain pictures, which simplifies the picture album generation process, reduces the picture album generation time, and improves the picture album generation efficiency.

[0102] Reference Figure 3 , Figure 3 The flowchart of the second embodiment of the method for generating a picture album of the present application is shown. Based on the first embodiment, step S202 includes:

[0103] Step S301, obtain a story generation instruction, and parse the story generation instruction to obtain a target parameter, where the target parameter includes at least one of a story type and a text style type.

[0104] In this embodiment, the device can generate a story text based on the personalized needs of the user. Exemplarily, the user inputs an auxiliary instruction to the device. The auxiliary instruction contains the personalized parameters of the user, and the device parses the auxiliary instruction to obtain the personalized parameters as target parameters, and the target parameters include at least one of the story type and the text style type. The story type includes but is not limited to fairy tales, inspirational stories, touching stories, etc.; the text style type includes but is not limited to tragedy, comedy, lyricism, freshness, etc.

[0105] Step S302: Generate a target story text matching the target parameters according to the story description information.

[0106] After obtaining the target parameters, the device generates a target story text based on the target parameters and the story description information, that is, the target story text contains the story plot of the story description information, and the target story text is the text style type desired by the user, or the target story is the story type desired by the user. Specifically, the story description information and the target parameters are input into the large language model to obtain the target story text output by the large language model.

[0107] In this embodiment, the device generates target story text information that meets the user's preferences based on the user's personalized needs for the story text, thereby improving the user experience.

[0108] Reference Figure 4 , Figure 4 The schematic diagram of the third embodiment of the picture album generation method provided by the present application is shown. Figure 2 or Figure 3 In the illustrated embodiment, step S202 includes:

[0109] Step S401, generating a plurality of first story texts according to story description information.

[0110] In this embodiment, the device inputs the story description information into the large language model, and the large language model outputs a plurality of story texts, each of which is defined as a first story text.

[0111] In addition, when the user has personalized needs, the target parameters and story description information are input into the large language model to obtain multiple first story texts output by the large language model.

[0112] Step S402, determining the scoring value of each first story text.

[0113] Since the large language model cannot understand language ambiguity and cannot understand and judge the emotions of the generated story text, the large language model will generate multiple first story texts, and then use automatic evaluation indicators to score each generated first story text, so as to screen out the most suitable story text. The automatic evaluation indicator does not require a reference answer, and can determine whether the story text meets people's preferences from dimensions such as the repetitiveness, coherence, and logic of the story plot. The automatic evaluation indicator is a scoring score. The higher the scoring score of the story text, the closer the story text is to people's preferences.

[0114] In this regard, the device determines the score of each first story text. Exemplarily, a model can be used to determine the score of the first story text, that is, the first story text can be input into the model to obtain the score of the first story text, and the model is trained by training story texts marked with scores. The score of the first story text: S 故事 (l) = sigmoid(Wv+b), where W and b are model parameters, S 故事 (I) is the scoring value.

[0115] Step S403, determining a first target score among various scoring scores, and determining a first story text corresponding to the first target score as a target story text, wherein the first target score is used to indicate a scoring score greater than a first preset score.

[0116] After determining the score of each first story text, the device determines a first target score among the score values, the first target score being a score value greater than the first preset score value, that is, the score value greater than the first preset score value is determined as the first target score value. When there are multiple score values ​​greater than the first preset score value, the largest score value is determined as the first target score value.

[0117] After determining the first target score, the device may use the first story text corresponding to the first target score as the target story text.

[0118] In this embodiment, the device obtains multiple first story texts for each storyline and determines a rating score for each first story text, thereby selecting a story text that meets people's emotions based on the rating score.

[0119] In one embodiment, after the device generates a second story text corresponding to the story description information, the device displays the second story text. The user can edit the plot, characters and other contents of the second story text according to personal preferences, and the editing includes but is not limited to adding, deleting, and modifying. When the device detects an editing operation on the story text, it modifies the second story text based on the editing operation to obtain a target story text.

[0120] It should be noted that the second story text may be the first story text corresponding to the first target score, and the first story text may be generated based on the user's story generation instruction and story description parameters. Figure 5 , the target story text is generated as follows:

[0121] 1. Select scene pictures from the cloud;

[0122] 2. The image understanding model extracts story description information from the scene image;

[0123] 3. The large language model generates multiple first story texts based on the story description information and the story generation instructions;

[0124] 4. Evaluate the multiple first story texts to obtain scoring scores, and obtain the best story text based on the scoring scores, where the best story text is the first story text corresponding to the scoring score greater than the first preset score;

[0125] 5. Edit the best story text to obtain the target story text.

[0126] In this embodiment, the user can edit the story text to obtain a target story text that meets the user's preferences, thereby improving the user experience.

[0127] Reference Figure 6 , Figure 6 The flowchart of the fourth embodiment of the method for generating a picture album provided by the present application is shown, and step S203 includes:

[0128] Step S601, obtaining prompt words of the story plot and generating story illustrations corresponding to the story plot.

[0129] In this embodiment, the device inputs the target story text into the large language model after the target story text, and the large language model outputs K story plots, the global prompt word P corresponding to each story plot, and the target story text is input into the large language model. i , and the story illustrations x corresponding to each storylinei .

[0130] Each prompt word [P1, P2, ..., P k ]=G[F T2P , T, K], where G is a large language model, F T2P To generate auxiliary instructions for the storyline, the auxiliary instructions are used to describe the requirements of the differential storyline and are input by the user; T is the target story text, and K is the number of prompt words.

[0131] Step S602, determining layout parameters of a target object in the story illustration, where the target object includes a person and / or an object in the story illustration, and the layout parameters are used to indicate a circumscribed rectangular frame of the target object.

[0132] In order to improve the effect of story illustrations and reduce the complexity of generating character story images, the large language model can be used to add corresponding layout parameters to each story illustration. The layout parameters are the circumscribed rectangular frame of the target object in the story illustration. The target object refers to the character and / or object in the story illustration, and the circumscribed rectangular frame of the target object refers to the box containing the target object. The device inputs the target story text into the story illustration obtained by the large language model, which contains the circumscribed rectangular frame of the target object. The layout parameters can guide the generation of character story images as control signals.

[0133] Assume σ i is the layout parameter of the i-th story segment, F P2L To generate auxiliary instructions for layout parameters, F P2L Describes the requirements for the layout parameters of the generated story illustrations. The layout parameters of each storyline are: [σ1, σ2, …, σ k ]=G[F P2L , [P1, P2, …, P k ]].

[0134] The global prompt word P i According to the target number k described by the sentence i Disassemble into k i local prompt words representing different target objects That is k i It represents the number of local prompt words of the i-th story illustration and the number of bounding rectangles of the target object, and each local prompt word corresponds to a target object. and They represent the local prompt word and the bounding rectangle of the target object respectively. Therefore, the layout parameters of the target object in the story illustration are as follows:

[0135]

[0136] Step S603, extracting composition features in the story illustration according to the layout parameters.

[0137] Dense composition features such as the contour key points of the target object represent the overall composition distribution of the image. Composition features as control conditions will help generate high-quality image content.

[0138] In this regard, the device extracts the composition features of the outline key points of the target object based on the layout parameters. For example, assuming that the jth target object of the i-th story link is I ij , the local prompt word of the target object is First, the local prompt word is input into the diffusion model to generate a single target object image, and then the general detection model is used to obtain the parameters of the external rectangular box of the target object in the image. Then will As the input of the segmentation algorithm, the segmentation mask m is obtained ij Finally, the pixel difference network is used to extract the contour key points E of the target object ij Therefore, the set of contour key points of the target object in the i-th story link can be obtained as E i , E i This is the composition feature. Figure 7 , the steps to obtain the composition features are:

[0139] 1. The local cue word is input into the diffusion model to generate a target, which is a picture of the target object;

[0140] 2. Get the bounding box (circumscribed rectangular box) of the image of the target object;

[0141] 3. Segmentation target, that is, segmentation of the picture;

[0142] 4. Extract contour points of the segmented image.

[0143] Step S604, denoising the preset noise image corresponding to the storyline according to the composition features, prompt words, story illustrations and diffusion model corresponding to the storyline, to obtain the character story image corresponding to the storyline.

[0144] In this embodiment, a diffusion model is provided in the device. The diffusion model is a probabilistic generation model that performs well in the field of image synthesis. It performs two-stage diffusion calculations in the latent space. The first stage of the forward diffusion process is to gradually add Gaussian noise to the input data x0 through a T-step Markov chain to obtain the obfuscated data x T , let q(x0) be the original distribution of the data, and its prior distribution is

[0145]

[0146] Where t represents the number of steps, N represents Gaussian distribution, I is the variance of Gaussian noise, β t is a hyperparameter.

[0147] The second stage of back diffusion is to use the noise estimation model ε θ Fit the inverse denoising process to reconstruct the sample. Assume that ε is the sample x t The real noise, ε θ (x t , t) is the predicted noise, C is the introduced category or image and other conditions used to guide the generation of the target image, then the noise estimation model ε θ The mean square error loss function is:

[0148] The device obtains a preset noise map corresponding to each storyline. The preset noise map can be randomly generated.

[0149]

[0150] The composition features of the storyline, prompt words, story illustrations and preset noise maps are input into the diffusion model, and the diffusion model can output character story pictures corresponding to the storyline. That is, the composition features of the storyline, prompt words and story illustrations guide the diffusion model to denoise the preset noise map, thereby obtaining character story pictures that match the storyline.

[0151] In this embodiment, the device obtains prompt words of the storyline, generates story illustrations of the storyline, determines the layout parameters of the characters and / or objects in the story illustrations, extracts composition features based on the layout parameters, and thus denoises the preset noise map based on the composition features of the storyline, the prompt words, and the story illustration-guided diffusion model, thereby obtaining a character story picture that matches the storyline.

[0152] In one embodiment, the diffusion model is provided with a composition adapter, which is used to process the composition features corresponding to the storyline. Specifically, the composition features such as the outline key points of the target object are input as input parameters to the composition adapter. The composition adapter can align the internal knowledge and external information of the diffusion model, and the composition adapter can provide more refined structural control to generate high-quality story illustrations. Figure 8 As shown in Figure 1, the composition adapter includes multiple units, each of which consists of a deformable convolutional layer and two residual blocks. The introduced deformable convolutional layer can flexibly extract target features of any shape and better express the area of ​​interest of the image. A total of N scales of composition features F are extracted. E The N sub-unit features of the noise estimation network are concatenated and input, and the composition features of N scales are extracted as the output of each unit of the composition adapter.

[0153] In this embodiment, a composition adapter is provided in the diffusion model, and the obtained contour key points of the target object are used as additional conditions through the composition adapter to guide and control the diffusion model to generate a stable and fine picture.

[0154] In one embodiment, the diffusion model is provided with an encoder, which is used to process the prompt words and story illustrations to unify the text and image content of the prompt words and story illustrations. Specifically, the encoder can be a multimodal joint encoder, which can encode text and images into a unified multimodal potential control, effectively eliminating ambiguity between modalities and ensuring consistency between text and images.

[0155] Reference Fig. 9 , Fig. 9 This is the workflow diagram of the multimodal joint encoder. The text prompt words include global prompt words and local prompt words. The bounding rectangle of the target object and the prompt words are recorded as y, and the single target picture set is recorded as x. s , the single target picture refers to the picture of the target object extracted from the story illustration, and the position of the target picture in the prompt word is recorded as O. Assume y is 'Rose on the boat', x s Then it is the character Rose and the boat, position O=0,2.

[0156] The multimodal joint encoder takes as input the parameters (y, x s , O) is encoded into the joint latent space to obtain the joint encoding sequence h u Specific examples: Fig. 9 As shown, first, for the image x s Extract features and input them into the multi-layer perceptron MLP to be trained to obtain pseudo-word features, and then replace the word features of the corresponding text encoding position O to obtain the recombined encoding sequence w r The recombinant coding sequence w r After CLIP (Contrastive Language-Image Pre-training, multimodal pre-training neural network) encoding, the reorganized hidden layer sequence h is obtained. r .

[0157] Finally, the reorganized hidden layer sequence h r The sequence of non-position O is replaced by the original hidden layer sequence h y The value of , thus obtaining the joint coding sequence h u =ITUE(y,x s , O).

[0158] Furthermore, the diffusion model can set the composition adapter and encoder, so the diffusion model needs to train the parameters of the multimodal joint encoder and the composition adapter, so the optimization objective function of the diffusion model is:

[0159] In this embodiment, the encoder in the diffusion model can encode text and images into a unified multimodal latent

[0160]

[0161] Controls effectively eliminate ambiguity between modes and ensure consistency between text and images.

[0162] In one example, the diffusion model is provided with a noise estimation network, which denoises a preset noise map based on composition features, cue words, and story illustrations.

[0163] Compared with traditional denoising networks, visual self-attention networks have stronger scalability and flexibility and more modeling advantages. In order to reduce the computational complexity of the diffusion model and extract representative global information, the original attention mechanism is customized through the subspace transformation self-attention network, which is replaced by the subspace transformation self-attention layer. The entire network consists of N sub-units for fitting the noise distribution, that is, the noise estimation network is the subspace transformation self-attention noise estimation network.

[0164] Reference Fig.10 , since the computational burden of the feature space dimension is several times that of the channel dimension, this embodiment adopts transformation self-attention to strengthen feature learning, thereby reducing the computational burden and reverse sampling time; and uses three overlapping convolutional layers to perform subspace mapping (Subspace Projection, SP) on the attention model parameters Q, K, and V respectively, and then characterizes the self-attention in the feature subspace. Assuming that the length of feature F is A, the dimension is B, the subspace transformation parameter is r, and the learnable scale parameter is α, the specific calculation of the subspace transformation self-attention ST is as follows:

[0165]

[0166] In order to compare the computational complexity of different self-attention calculation methods, let r = 0.5, feature length dimension A = 64 × 64, feature dimension B = 32, and count the total computational complexity of multiplication and addition of different self-attention structures. As shown in Table 1, the computational complexity of subspace transformation self-attention is the smallest, proving that this method can effectively reduce computational overhead.

[0167] Table 1 The computational cost of different self-attention calculation methods

[0168] structure Calculation amount (multiplication + addition) Self-Attention <![CDATA[2.13×10 9 ]]> Transforming Self-Attention <![CDATA[16.65×10 6 ]]> Subspace Transformation Self-Attention <![CDATA[5.11×10 6 ]]>

[0169] In this embodiment, the diffusion model can reduce the computational complexity of the diffusion model by setting a sub-space transformation self-attention noise estimation network.

[0170] Reference Fig.11, Fig.11 The flowchart of the fifth embodiment of the method for generating a picture album provided by the present application is shown. Figure 6 In the embodiment shown, before step S604, the following steps are further included:

[0171] Step S1101, obtain multiple training samples, and train the preset model according to each training sample to obtain an intermediate model.

[0172] In this embodiment, the device needs to train the diffusion model. The diffusion model usually uses CLLP to extract image features, but CLLP uses weakly aligned data for training, and the extracted features have a wide range and poor expression in terms of facial features. In order to extract high-fidelity facial information, the original person image in the face library is used as the target image, and the face model is used as a feature encoder to encode the target image to obtain a training sample for the training of the diffusion model.

[0173] After obtaining each training sample, the device trains the preset model based on each training sample to obtain an intermediate model.

[0174] Step S1102, obtaining a face training image, and training the intermediate model according to the face training image to obtain a diffusion model, wherein the face training image includes a face image of a target person specified by a user.

[0175] In order to further improve the similarity of faces and improve the facial details of people, fine-tuning technology is used in the training stage to fuse the face training images into the intermediate model, so that the images generated by the diffusion model have the facial contour features of the person specified by the user. The face training images include the face images of the target person specified by the user.

[0176] To this end, the device obtains a face training image, and trains an intermediate model based on the face training image to obtain a diffusion model.

[0177] In this embodiment, the preset model is trained with training samples to obtain an intermediate model, and then the intermediate model is trained with face training images to obtain a diffusion model, so that the image generated by the diffusion model can have the facial contour features of the person specified by the user.

[0178] Reference Fig.12 , Fig.12 The flowchart of the sixth embodiment of the method for generating a picture album provided by the present application is shown. Fig.11 In the illustrated embodiment, step S1102 includes:

[0179] Step S1201, obtaining a first picture uploaded by a user, where the first picture includes a face of a target person specified by the user at a first angle.

[0180] In this embodiment, the user stores a large amount of image data in the cloud. The user can select a head portrait of a person for which a digital image avatar needs to be created from the image data in the cloud. The picture containing the head portrait of the person is defined as the first picture, and the first picture is the face of the target person specified by the user at a first angle. The first angle can be a frontal face, that is, the first picture is a clear frontal face image of the target person. The user uploads the first picture to the device, that is, the device obtains the first picture uploaded by the user.

[0181] Step S1202, displaying a plurality of second pictures, each of which includes the face of the target person at a second angle, where the second angle is different from the first angle.

[0182] After obtaining the first image, the device detects the first image to determine whether the first image meets the quality requirements. Exemplarily, the device analyzes the face, face angle, and face attributes through a face detection model and a face attribute model, and if the analysis result indicates that the first image meets the quality requirements, multiple second images are displayed, each of which is a second angle of the face of the target person, and the second angle is different from the first angle.

[0183] Exemplarily, after obtaining the first picture, the device obtains all pictures containing the face of the person in the first picture from the database, and crops the face area of ​​the obtained picture to obtain a picture of the person containing the face area as the second picture for display. The user can select a picture from each second picture as a secondary picture to supplement the outline of the target person from other angles.

[0184] Step S1203, in response to detecting a selection operation, constructing a face training image based on the second image selected by the selection operation and the first image.

[0185] After detecting the selection operation, the device constructs a face training image based on the second image selected by the selection operation and the first image. The second image selected by the selection operation can be multiple, and the more second images selected, the better the effect of generating a digital image of the person.

[0186] In addition, the selected second image also needs to be detected by a face detection model and a face attribute model to obtain a second image that meets the instruction requirements.

[0187] The first picture is taken as the main picture, and the selected second picture is taken as the slave picture. The main picture and the slave picture are processed by the face refined rotation method based on the image rotation model based on orientation judgment and the face detection and key point model. Then, the human body analysis model and the portrait skin beautification model are used to obtain high-quality face images. The face attribute model and the text annotation model are combined with the label post-processing method to generate refined labels for the face images, thereby obtaining face training images. The face training images are the task digital image avatars.

[0188] Refer to 13, Fig.13 This is a schematic diagram of the process of obtaining face training images. The specific steps are as follows:

[0189] 1. The user uploads the main picture, and the main picture is tested for faces, face angles, and face attributes to determine whether the main picture meets the quality requirements;

[0190] 2. When the main image meets the quality requirements, the user uploads the secondary image, and performs face recognition, face queue, and image quality analysis on the secondary image to determine whether the secondary image meets the quality requirements;

[0191] 3. If the slave image meets the quality requirements, the slave image and the main image are processed to obtain a digital image of the character based on human body analysis, portrait skin beautification, image quality analysis, facial key points, text annotation, and image rotation.

[0192] In this embodiment, the device obtains a first picture uploaded by a user, and displays multiple second pictures based on the first picture, thereby constructing a face training image corresponding to the person specified by the user based on the first picture and the selected second pictures.

[0193] Reference Fig.14 , Fig.14 The flowchart of the seventh embodiment of the method for generating a picture album provided by the present application is shown. Figure 6 or Fig.11 In the illustrated embodiment, step S604 includes:

[0194] Step S1401, denoising a preset noise image corresponding to the storyline according to the composition features, prompt words, story illustrations and diffusion model corresponding to the storyline, to obtain a plurality of intermediate images corresponding to the storyline.

[0195] In this embodiment, in order to screen out high-quality illustrations, each storyline has multiple story illustrations, that is, each storyline generates a story illustration candidate set, and the story illustration candidate set has multiple illustrations of the storyline, and the device selects high-quality illustrations from the story illustration candidate set.

[0196] Exemplarily, the device inputs composition features, prompt words, story illustrations and preset noise maps corresponding to the storyline into a diffusion model, and the diffusion model outputs multiple intermediate pictures corresponding to the storyline, and the multiple intermediate pictures are a pre-selected set of story illustrations.

[0197] Step S1402, extracting the first image feature and the text feature of each intermediate image, and determining the first similarity between the first image feature and the text feature of the intermediate image, and determining the second similarity between the first image feature of the intermediate image and the second image feature in the template face image.

[0198] After obtaining each intermediate image, it is necessary to evaluate the quality of each intermediate image. Specifically, extract the first image feature F of each intermediate image. i And the text feature F t The device calculates the cosine similarity between the first image feature and the text feature of the intermediate image as the first similarity.

[0199] Assume that the prompt word pi of the i-th story link has M candidate images x m , then the first similarity is:

[0200] S 图，文 (p i , x m )=cos(F t (p i ), F i (x m ))

[0201] After determining the first similarity, the device obtains the second image feature F associated with the template face image n , calculate the first image feature F of each intermediate image i With the second image feature F n The second similarity between the two. The prompt word p of the i-th link i The mth candidate image x m The second similarity is:

[0202] S 人脸 (p i , x m )=cos(F m , F n )

[0203] It should be noted that the template face image is obtained from the training samples, that is, the face image in each training sample is evaluated using the face quality assessment model, and the face with the best quality is selected as the template face image.

[0204] Step S1403, determining a quality score corresponding to the middle picture according to the first similarity, the second similarity, the first image feature, and the text feature corresponding to the middle picture.

[0205] After determining the second similarity, the device determines a quality score corresponding to the middle picture based on the first similarity, the second similarity, the first image feature, and the text feature corresponding to the middle picture.

[0206] For example, suppose the prompt word p in the i-th story link is i There are M candidate images x m ,Enc t and Enci Represent the text features and image features extracted by the human preference classifier encoder respectively. The text feature of the middle picture is based on the value corresponding to the first image feature:

[0207] S 图 (p i , x m )=Enc t (p i )·Enc i (x m )×100

[0208] The quality score is: S 插图 =a1S 图 +a2S 图，文 +a3S 人脸 , and a1, a2, and a3 are the corresponding weights, which can be set manually.

[0209] Step S1404, determining a second target score among various quality scores, and determining a character story picture corresponding to the plot according to the intermediate picture corresponding to the second target score, wherein the second target score is a quality score greater than a second preset score.

[0210] After obtaining the quality score of each intermediate image, the device determines a second target score among the quality scores. The second target score is a quality score greater than the second preset score. When there are multiple quality scores greater than the second preset score, the largest quality score is used as the second target score.

[0211] After determining the second target score, the device determines the character story picture corresponding to the storyline based on the intermediate picture corresponding to the second target score.

[0212] In one example, the device directly uses the middle picture corresponding to the second target score as the character story picture corresponding to the storyline.

[0213] In another example, the face in the middle picture corresponding to the second target score is fused with the face in the template face image to obtain a character story picture corresponding to the storyline.

[0214] Reference Fig.15 , Fig.15The following is a workflow diagram of the diffusion model trained in this embodiment. Specifically, the target contour key point set is input into the composition adapter; the global prompt words, local prompt words, and single target picture set (a set of pictures of the target object obtained in the story illustration) are input into the multimodal joint encoder to unify the pictures and texts in the multimodal space; the subspace transformation self-attention noise estimation network is trained with training samples and face training images, and then the noise map is input; the subspace transformation self-attention noise estimation network, through the input of the composition adapter and the output of the multimodal latent space, denoises the noise map to obtain a pre-selected set of story illustrations. The intermediate picture corresponding to the second target score is selected from the pre-selected set of story illustrations, and the intermediate picture is face-fused with the template face image to obtain a candidate set of story illustrations, which includes the character story pictures corresponding to each story link.

[0215] In this embodiment, each intermediate picture in the pre-selected set of story illustrations for each story link is automatically scored and sorted in multiple dimensions, so as to screen out pictures with higher quality scores as character story pictures of the story plot.

[0216] In one embodiment, in order to increase the interest of the album, Huacheng can be synthesized into speech, so that the album can be generated into a video. Exemplarily, after the device constructs the album based on the character story pictures corresponding to each story plot, it generates speech corresponding to the target story text, thereby generating a story video based on the cloudy and rainy and the character story pictures in the album.

[0217] In this embodiment, it supports the synthesis of short videos of digital character stories, provides a variety of speech synthesis styles and sound effects, and supports the adjustment of speech speed and reading style to adapt to the preferences and personalized needs of different users.

[0218] In one embodiment, the album can be shared, for example, the album or story video can be shared to a social platform, such as a mobile cloud disk circle, a third-party software, etc., to enhance the interaction and communication between the user and friends, relatives, or other members of the social circle, and timely analyze their own stories and moods. Specifically, after the album is constructed based on the character story pictures corresponding to each story plot, the album is displayed, and when a sharing operation for the album is detected, the target terminal corresponding to the sharing operation is determined, and the album is sent to the target terminal.

[0219] Furthermore, in order to provide users with better personalized services and optimize the model effect, a user evaluation feedback option is set on the display page of the story album. User ratings are collected from multiple dimensions such as character similarity, picture and text consistency, story satisfaction, and illustration satisfaction. The ratings are scored using a Likert scale of 1-5, with higher scores representing greater satisfaction, as shown in the following table. Subsequently, the user evaluations can be converted into training data, and the model can be optimized using reinforcement learning of human preferences to make the generated stories more in line with user preferences.

[0220] The device obtains comment information on the picture album, determines the elements to be modified in the picture album according to the comment information, and modifies the elements to be modified in the picture album based on the comment information.

[0221] For example, a user evaluation feedback option is set on the display page of the album, and user ratings are collected from multiple dimensions such as character similarity, consistency between pictures and texts, story satisfaction and illustration satisfaction. The ratings are scored on a scale of 1-5, with higher scores representing greater satisfaction, as shown in Table 2 below.

[0222] Subsequently, the user evaluation can be converted into training data, and the model can be optimized by using reinforcement learning of human preferences to make the generated story more in line with user preferences. Among them, the elements to be modified are, for example, digital characters, story content, story illustrations, etc. If the digital character is not similar to the person himself, the element to be modified is the character; if the story content is inconsistent with the illustration, the character story picture is used as the element to be modified; if the story's coherence, logic or plot satisfaction is low, the element to be modified is the story text; if the coordination or quality of the illustrations in the album is low, the element to be modified is the character story picture.

[0223] Table 2 User evaluation table

[0224]

[0225] In this embodiment, the user's cloud disk data is combined with the generation model. The user only needs to upload a picture and specify the album characters. Combined with technologies such as speech synthesis, a digital character story album with rich plots and emotional warmth can be automatically generated, providing cloud disk users with intelligent and interesting applications with the characteristics of easy operation, low cost and strong innovation.

[0226] In addition, according to the characteristics of story picture albums that need to generate fine composition and ensure strict consistency between the plot and the picture, a story illustration generator is designed, which consists of modules such as a multimodal composition conditional diffusion model, a subspace transformation self-attention noise estimation network, and facial feature retention. It maps text and images into a unified space, generates story illustrations with fine composition, and effectively reduces computational overhead to achieve high-fidelity digital character effects.

[0227] The cloud-based digital character story album generation method proposed in this embodiment can automatically generate high-quality story texts and story illustrations, minimize user interaction, evaluate the quality of the album from multiple dimensions, and provide functions such as personalized settings and social sharing.

[0228] like Fig.16 As shown, the embodiment of the present application further provides a picture album generating device 1600, and the picture album generating device 1600 includes:

[0229] An acquisition module 1610 is used to acquire a scene image and extract story description information from the scene image;

[0230] The first generating module 1620 is used to generate a target story text corresponding to the story description information and determine the story plot corresponding to the target story text;

[0231] The second generating module 1630 is used to generate a character story picture corresponding to each story plot according to the prompt words in each story plot, and to construct a picture album according to the character story pictures corresponding to each story plot.

[0232] In one embodiment, the album generating device 1600 is specifically used for:

[0233] Obtaining a story generation instruction, and parsing the story generation instruction to obtain a target parameter, where the target parameter includes at least one of a story type and a text style type;

[0234] According to the story description information, a target story text matching the target parameters is generated.

[0235] In one embodiment, the album generating device 1600 is specifically used for:

[0236] Generate multiple first story texts according to the story description information;

[0237] Determine the scoring score for each first story text;

[0238] A first target score is determined among the various scoring scores, and a first story text corresponding to the first target score is determined as the target story text, wherein the first target score is used to indicate a scoring score greater than a first preset score.

[0239] In one embodiment, the album generating device 1600 is specifically used for:

[0240] Generate a second story text corresponding to the story description information, and display the second story text;

[0241] In response to the editing operation on the second story text, the second story text is modified according to the editing operation to obtain a target story text.

[0242] In one embodiment, the album generating device 1600 is specifically used for:

[0243] Obtain the prompt words of the story plot and generate story illustrations corresponding to the story plot;

[0244] Determine layout parameters of a target object in the story illustration, the target object includes a person and / or an object in the story illustration, and the layout parameters are used to indicate a bounding rectangular frame of the target object;

[0245] Extracting compositional features in story illustrations based on layout parameters;

[0246] According to the composition features, prompt words, story illustrations and diffusion model corresponding to the storyline, the preset noise image corresponding to the storyline is denoised to obtain the character story image corresponding to the storyline.

[0247] In one embodiment, the diffusion model is provided with a composition adapter, and the composition adapter is used to process the composition features corresponding to the storyline.

[0248] In one embodiment, the diffusion model is provided with an encoder, and the encoder is used to process the prompt words and the story illustrations so as to unify the graphic content of the prompt words and the story illustrations.

[0249] In one embodiment, the diffusion model is provided with a sub-space transformation self-attention noise estimation network, which denoises a preset noise image based on composition features, prompt words, and story illustrations.

[0250] In one embodiment, the album generating device 1600 is specifically used for:

[0251] Obtain multiple training samples, and train the preset model according to each training sample to obtain an intermediate model;

[0252] A face training image is obtained, and an intermediate model is trained according to the face training image to obtain a diffusion model. The face training image includes a face image of a target person specified by a user.

[0253] In one embodiment, the album generating device 1600 is specifically used for:

[0254] Acquire a first picture uploaded by a user, where the first picture includes a face of a target person specified by the user at a first angle;

[0255] displaying a plurality of second images, each of the second images including a face of the target person at a second angle, the second angle being different from the first angle;

[0256] In response to detecting a selection operation, a face training image is constructed based on the second image selected by the selection operation and the first image.

[0257] In one embodiment, the album generating device 1600 is specifically used for:

[0258] According to the composition features, prompt words, story illustrations and diffusion model corresponding to the storyline, the preset noise image corresponding to the storyline is denoised to obtain multiple intermediate images corresponding to the storyline;

[0259] Extracting a first image feature and a text feature of each intermediate image, and determining a first similarity between the first image feature and the text feature of the intermediate image, and determining a second similarity between the first image feature of the intermediate image and a second image feature in the template face image;

[0260] Determine a quality score corresponding to the middle picture according to the first similarity, the second similarity, the first image feature, and the text feature corresponding to the middle picture;

[0261] A second target score is determined among the quality scores, and a character story picture corresponding to the plot is determined based on the intermediate picture corresponding to the second target score. The second target score is a quality score greater than a second preset score.

[0262] In one embodiment, the album generating device 1600 is specifically used for:

[0263] The face in the middle picture corresponding to the second target score is fused with the face in the template face image to obtain a character story picture corresponding to the story plot.

[0264] In one embodiment, the album generating device 1600 is specifically used for:

[0265] Generate speech corresponding to the target story text;

[0266] Generate a story video based on the voice and pictures of the characters and stories in the album.

[0267] In one embodiment, the album generating device 1600 is specifically used for:

[0268] Display album;

[0269] In response to a sharing operation on the album, determining a target terminal corresponding to the analysis operation;

[0270] Send the album to the target terminal.

[0271] In one embodiment, the album generating device 1600 is specifically used for:

[0272] Get the comment information for the album;

[0273] Determine the elements to be modified in the album based on the comment information;

[0274] Modify the elements to be modified in the album based on the comment information.

[0275] It should be noted that the picture album generating device is a device corresponding to the above-mentioned picture album generating method. All implementation methods in the above-mentioned method embodiment are applicable to the embodiment and can achieve the same technical effect.

[0276] Fig.17 A schematic diagram of the hardware structure of the picture album generating device provided in an embodiment of the present application is shown.

[0277] The picture album generating device may include a processor 1701 and a memory 1702 storing computer program instructions.

[0278] Specifically, the processor 1701 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0279] Memory 1702 may include a large capacity memory for data or instructions. By way of example and not limitation, memory 1702 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive or a combination of two or more of these. In appropriate cases, memory 1702 may include removable or non-removable (or fixed) media. In appropriate cases, memory 1702 may be inside or outside the integrated gateway disaster recovery device. In a specific embodiment, memory 1702 is a non-volatile solid-state memory.

[0280] In certain embodiments, the memory 1702 may include a read-only memory (ROM), a random access memory (RAM), a magnetic disk storage medium device, an optical storage medium device, a flash memory device, an electrical, optical or other physical / tangible memory storage device. Thus, in general, the memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., a memory device) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to an aspect of the present disclosure.

[0281] The processor 1701 reads and executes the computer program instructions stored in the memory 1702 to implement any one of the picture album generation methods in the above embodiments.

[0282] In one example, the album generation device may further include a communication interface 1703 and a bus 1710. Fig.17 As shown, the processor 1701, the memory 1702, and the communication interface 1703 are connected via a bus 1710 and communicate with each other.

[0283] The communication interface 1703 is mainly used to implement communication between various modules, devices, units and / or equipment in the embodiments of the present application.

[0284] Bus 1710 includes hardware, software or both, and the parts of online data flow billing equipment are coupled to each other. For example, but not limitation, bus may include accelerated graphics port (AGP) or other graphics bus, enhanced industrial standard architecture (EISA) bus, front-end bus (FSB), hypertransport (HT) interconnection, industrial standard architecture (ISA) bus, infinite bandwidth interconnection, low pin count (LPC) bus, memory bus, micro channel architecture (MCA) bus, peripheral component interconnection (PCI) bus, PCI-Express (PCI-X) bus, serial advanced technology attachment (SATA) bus, video electronics standard association local (VLB) bus or other suitable bus or two or more of these combinations. In appropriate cases, bus 1010 may include one or more buses. Although the present application embodiment describes and shows a specific bus, the present application considers any suitable bus or interconnection.

[0285] In addition, in combination with the album generation method in the above embodiment, the embodiment of the present application can provide a computer storage medium for implementation. The computer storage medium stores computer program instructions; when the computer program instructions are executed by a processor, any one of the album generation methods in the above embodiment is implemented.

[0286] The embodiment of the present application also provides a computer program product, including a computer program, and when the computer program is processed and executed, any one of the picture album generation methods in the above embodiments is implemented.

[0287] It should be clear that the present application is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present application is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present application.

[0288] The functional blocks shown in the above block diagram can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present application are programs or code segments that are used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link by a data signal carried in a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (RF) links, etc. The code segment can be downloaded via a computer network such as the Internet, an intranet, etc.

[0289] It should also be noted that the exemplary embodiments mentioned in this application describe some methods or systems based on a series of steps or devices. However, this application is not limited to the order of the above steps, that is, the steps can be performed in the order mentioned in the embodiment, or in a different order from the embodiment, or several steps can be performed simultaneously.

[0290] Aspects of the present disclosure are described above with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable terminal device picture album generating device to produce a machine so that these instructions executed by the processor of the computer or other programmable terminal device picture album generating device enable the implementation of the function / action specified in one or more boxes of the flowchart and / or block diagram. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field programmable logic circuit. It can also be understood that each box in the block diagram and / or flowchart and the combination of boxes in the block diagram and / or flowchart can also be implemented by dedicated hardware that performs a specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions.

[0291] The above are only specific implementation methods of the present application. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, modules and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. It should be understood that the protection scope of the present application is not limited to this. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in this application, and these modifications or replacements should be included in the protection scope of this application.

Claims

1. A method for generating a picture album, characterized in that: The picture album generation method comprises: Obtaining a scene image, and extracting story description information from the scene image; Generate a target story text corresponding to the story description information, and determine a story plot corresponding to the target story text; According to the prompt words in each of the story plots, a character story picture corresponding to each of the story plots is generated, and a picture album is constructed according to the character story pictures corresponding to each of the story plots.

2. The method according to claim 1, characterized in that The generating a target story text corresponding to the story description information includes: Obtaining a story generation instruction, and parsing the story generation instruction to obtain a target parameter, wherein the target parameter includes at least one of a story type and a text style type; A target story text matching the target parameters is generated according to the story description information.

3. The method according to claim 1, characterized in that The generating a target story text corresponding to the story description information includes: Generate a plurality of first story texts according to the story description information; Determining a rating score for each of the first story texts; A first target score is determined among the rating scores, and a first story text corresponding to the first target score is determined as the target story text, wherein the first target score is used to indicate a rating score that is greater than a first preset score.

4. The method according to claim 1, characterized in that: The generating a target story text corresponding to the story description information includes: Generate a second story text corresponding to the story description information, and display the second story text; In response to an editing operation on the second story text, the second story text is modified according to the editing operation to obtain the target story text.

5. The method according to claim 1, characterized in that The step of generating a character story picture corresponding to each story plot according to the prompt words in each story plot comprises: Obtaining prompt words for the story plot and generating story illustrations corresponding to the story plot; Determining layout parameters of a target object in the story illustration, the target object including a person and / or an object in the story illustration, the layout parameters being used to indicate a circumscribed rectangular frame of the target object; extracting composition features in the story illustration according to the layout parameters; According to the composition features corresponding to the storyline, the prompt words, the story illustrations and the diffusion model, the preset noise image corresponding to the storyline is denoised to obtain a character story image corresponding to the storyline.

6. The method according to claim 5, characterized in that The diffusion model is provided with a composition adapter, and the composition adapter is used to process the composition features corresponding to the storyline.

7. The method according to claim 5, characterized in that The diffusion model is provided with an encoder, and the encoder is used to process the prompt words and the story illustrations so as to unify the graphic content of the prompt words and the story illustrations.

8. The method according to claim 5, characterized in that The diffusion model is provided with a sub-space transformation self-attention noise estimation network, and the sub-space transformation self-attention noise estimation network denoises the preset noise map based on the composition features, the prompt words and the story illustrations.

9. The method according to claim 5, characterized in that Before denoising the preset noise image corresponding to the storyline according to the composition features corresponding to the storyline, the prompt words, the story illustrations and the diffusion model, the method further includes: Obtain multiple training samples, and train the preset model according to each training sample to obtain an intermediate model; A face training image is obtained, and the intermediate model is trained according to the face training image to obtain a diffusion model, wherein the face training image includes a face image of a target person specified by a user.

10. The method for generating a picture album according to claim 9, characterized in that: The step of obtaining a face training image comprises: Acquire a first picture uploaded by the user, where the first picture includes a face of a target person specified by the user at a first angle; Displaying a plurality of second pictures, each of the second pictures includes a face of the target person at a second angle, the second angle being different from the first angle; In response to detecting a selection operation, a face training image is constructed based on the second image selected by the selection operation and the first image.

11. The method according to claim 5, characterized in that The step of denoising a preset noise image corresponding to the storyline according to the composition features corresponding to the storyline, the prompt words, the story illustrations and the diffusion model comprises: De-noising a preset noise image corresponding to the storyline according to the composition features corresponding to the storyline, the prompt words, the story illustrations, and the diffusion model to obtain a plurality of intermediate images corresponding to the storyline; Extracting a first image feature and a text feature of each of the intermediate images, and determining a first similarity between the first image feature and the text feature of the intermediate image, and determining a second similarity between the first image feature of the intermediate image and a second image feature in the template face image; Determine a quality score corresponding to the intermediate picture according to the first similarity, the second similarity, the first image feature, and the text feature corresponding to the intermediate picture; A second target score is determined among the quality scores, and a character story picture corresponding to the storyline is determined based on the intermediate picture corresponding to the second target score, wherein the second target score is a quality score greater than a second preset score.

12. The method according to claim 11, characterized in that The step of determining the character story picture corresponding to the story plot according to the intermediate picture corresponding to the second target score includes: The face in the middle picture corresponding to the second target score is merged with the face in the template face image to obtain a character story picture corresponding to the storyline.

13. The method according to claim 1, characterized in that After the picture album is constructed according to the character story pictures corresponding to each of the story plots, the method further includes: Generating speech corresponding to the target story text; A story video is generated based on the voice and the character story pictures in the album.

14. The method according to claim 1, characterized in that After the picture album is constructed according to the character story pictures corresponding to each of the story plots, the method further includes: Displaying the album; In response to a sharing operation on the album, determining a target terminal corresponding to the analysis operation; The picture album is sent to the target terminal.

15. The method according to claim 14, characterized in that After displaying the album, the method further includes: Obtaining comment information on the album; Determining the elements to be modified in the album according to the comment information; The elements to be modified in the album are modified based on the comment information.

Images