Picture book generation method, device, electronic device, storage medium and product

Abstract

The present disclosure provides a picture book generation method, device, electronic device, storage medium and product. The picture book generation method includes: extracting scene information and / or character information from first multimedia data; generating second multimedia data associated with the first multimedia data according to the extraction result; and creating a picture book according to the first multimedia data and the second multimedia data.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, and in particular to a picture book generation method, device, electronic device, storage medium and product. Background Art

[0002] Picture books are mainly images with a small amount of text. Picture books can show readers the story and information through the combination of text and images. Their vivid and vivid graphic presentation is favored by readers.

[0003] Currently, picture books are generally drawn by picture book creators; there is also a way to automatically generate picture books based on AI. At this stage, automatically generated picture books are generally pieced together based on images and texts provided by users, and the content of the picture books is relatively monotonous. Summary of the invention

[0004] The present disclosure provides a picture book generation method, device, electronic device, storage medium and product.

[0005] According to a first aspect of the present disclosure, a picture book generation method is provided, comprising:

[0006] Extracting scene information and / or character information from the first multimedia data;

[0007] generating second multimedia data associated with the first multimedia data according to the extraction result;

[0008] A picture book is created according to the first multimedia data and the second multimedia data.

[0009] According to a second aspect of the present disclosure, there is provided a picture book generating device, comprising:

[0010] An extraction module, used to extract scene information and / or character information from the first multimedia data;

[0011] a data generating module, configured to generate second multimedia data associated with the first multimedia data according to the extraction result;

[0012] A creation module is used to create a picture book according to the first multimedia data and the second multimedia data.

[0013] According to a third aspect of the present disclosure, there is provided an electronic device, including:

[0014] at least one processor; and

[0015] a memory communicatively connected to the at least one processor; wherein,

[0016] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute any one of the picture book generation methods described above.

[0017] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the picture book generation method according to any one of the above items.

[0018] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the picture book generation method according to any one of the above items.

[0019] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0021] Figure 1 A flowchart of a picture book generation method according to an exemplary embodiment of the present disclosure;

[0022] Figure 2 A schematic diagram of a user interface provided for a picture book generation method according to an exemplary embodiment of the present disclosure;

[0023] Figure 3 This is a diagram showing the effect of beautifying a character in a picture book generation method according to an exemplary embodiment of the present disclosure;

[0024] Figure 4 A schematic diagram of a module of a picture book generating device provided in an exemplary embodiment of the present disclosure;

[0025] Figure 5 The present invention is a block diagram of an electronic device for implementing the picture book generation method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0026] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0027] Figure 1A flowchart of a picture book generation method provided in an exemplary embodiment of the present disclosure, the picture book generation method comprising the following steps:

[0028] Step 101: extract scene information and / or character information from first multimedia data.

[0029] The first multimedia data includes at least one of the following data types: images, videos, audios, texts, and picture books, etc. The first multimedia data may be data shot by the user, or data selected by the user from recommended candidate multimedia data, or an initial picture book generated based on a theme selected by the user and a document provided.

[0030] The scene information includes at least one of the following: scene time, scene location, etc. The character information includes at least one of the following: gender, age, height, posture, expression, emotion, clothing, etc.

[0031] In the case that the first multimedia data includes an image, in step 101, image recognition is performed on the image, and scene information and / or character information is extracted from the image recognition result.

[0032] In the case that the first multimedia data includes audio, in step 101, speech recognition is performed on the audio, and scene information and / or character information is extracted from the speech recognition result.

[0033] In the case where the first multimedia data includes a video, in step 101, frames are extracted from the images contained in the video, image recognition is performed on the frame extraction result (the image obtained by extracting frames from the image), speech recognition is performed on the audio contained in the video, and scene information and / or character information is extracted from the image recognition result and the speech recognition result.

[0034] In the case that the first multimedia data includes text, in step 101, semantic recognition is performed on the text, and scene information and / or character information is extracted from the semantic recognition result.

[0035] In one embodiment, image recognition, speech recognition and semantic recognition are respectively implemented based on a pre-trained image recognition model, speech recognition model and semantic recognition model. The image recognition model, speech recognition model and semantic recognition model are trained separately.

[0036] In one embodiment, image recognition, speech recognition and semantic recognition are realized by a content extraction model (large model) including image recognition function, speech recognition function and semantic recognition function. In step 101, the first multimedia data is input into the content extraction model, so that the content extraction model extracts features related to scenes and characters from the first multimedia data, and outputs scene information and / or character information according to the features; wherein the content extraction model is obtained by training a neural network according to multimedia data samples labeled with scene information and character information.

[0037] In one embodiment, scene information extraction and character information extraction are performed separately. Character information extraction can be, but is not limited to, using open source image detection and segmentation models, such as yolo, SAM, paddleHub algorithms, etc. Of course, scene information extraction can also use open source image detection and segmentation models, such as yolo, SAM, paddleHub algorithms, etc.

[0038] Step 102: Generate second multimedia data associated with the first multimedia data according to the extraction result.

[0039] The second multimedia data may be generated based only on the scene information or character information extracted from the first multimedia data, or the second multimedia data may be generated based on the scene information and character information extracted from the first multimedia data.

[0040] The scene information and / or character information contained in the generated second multimedia data are associated with the scene information and / or character information of the first multimedia data, that is, the scene information and / or character information contained in the second multimedia data are not completely the same as the scene information and / or character information contained in the first multimedia data, but are similar, so as to ensure the coherence of the picture book subsequently generated based on the first multimedia data and the second multimedia data. For example, the first multimedia data shows a little girl picking up shells at the beach, and the second multimedia data shows the little girl playing in the water at the beach; or the first multimedia data shows a little boy climbing a mountain with his father, and the second multimedia data shows the little boy taking pictures at the top of the mountain.

[0041] The second multimedia data includes at least one of the following data types: image, video, audio, text, etc. The second multimedia data is data extended and expanded according to the first multimedia data, and the scene information and / or character information contained in the second multimedia data is associated with the scene information and / or character information contained in the first multimedia data.

[0042] It should be noted that the data type included in the second multimedia data may be the same as that of the first multimedia data, or may be different from that of the first multimedia data. For example, in step 102, an image (second multimedia data) may be generated based on scene information and / or character information extracted from a video (first multimedia data); an image (second multimedia data) may be generated based on scene information and / or character information extracted from an image (first multimedia data); an image (second multimedia data) may be generated based on scene information and / or character information extracted from an image and audio (first multimedia data), etc.

[0043] In one embodiment, before executing step 102, it is first determined whether the extraction result is rich enough; if the determination result is yes, a picture book is directly created based on the first multimedia data; if the determination result is no, step 102 is executed.

[0044] The judgment conditions for determining whether the extraction results are rich enough may include but are not limited to at least one of the following: whether the number of scenes contained in the scene information is greater than the quantity threshold, whether the number of characters contained in the character information is greater than the quantity threshold, and whether the number of changes in the character's posture / expression / emotion contained in the character information is greater than the quantity threshold.

[0045] In this embodiment, when it is determined that the data content of the first multimedia data is rich enough to create a picture book, the picture book is created directly based on the first multimedia data without generating the second multimedia data, which can reduce the amount of calculation for generating the picture book.

[0046] Step 103: Create a picture book according to the first multimedia data and the second multimedia data.

[0047] If the data types of the first multimedia data and the second multimedia data include images and texts, and the texts are related to the image content, in step 103 , the texts are typeset around the images corresponding to the texts to obtain a picture book.

[0048] If the data types of the first multimedia data and the second multimedia data include images and texts, but the texts are less in content, in step 103, all or part of the images are described by pictures to obtain story content corresponding to the images, and the story content and texts are typeset around the corresponding images to obtain a picture book.

[0049] If the data types of the first multimedia data and the second multimedia data only include images, in step 103, a picture story is generated for all or part of the images to obtain story content corresponding to the images, and the story content of the images is typeset around the images to obtain a picture book.

[0050] The specific implementation method of describing the picture can be found in the relevant technical description, which will not be repeated here.

[0051] In the disclosed embodiment, second multimedia data associated with the first multimedia data is generated based on scene information and / or character information extracted from the first multimedia data, and then a picture book is created based on the first multimedia data and the second multimedia data. This can make the content of the generated picture book richer and more interesting.

[0052] In one embodiment, in step 102, when the extraction result includes scene information, a second scene time is generated according to the first scene time included in the scene information, and the first multimedia data is adjusted according to the second scene time to obtain the second multimedia data, wherein the time difference between the first scene time and the second scene time is less than the first time difference threshold.

[0053] The first time difference threshold may be a fixed value or a random value.

[0054] The scene time represents the time when the scene in the multimedia data occurs. It should be noted that the scene time can be represented by a specific time, for example, the scene time is morning, afternoon, afternoon, dusk, etc.; the scene time can also be represented by other forms such as weather and climate, for example, the scene time is when it rains, snows, winds, etc.

[0055] The number of second scene times can be set according to actual conditions. The number of second scene times can be 1, corresponding to the generation of one second multimedia data; the number of second scene times can be multiple, corresponding to the generation of multiple second multimedia data. The number of second scene times can be negatively correlated with the amount of second multimedia data. The time spans of the second scene times can be the same or different, and the embodiments of the present disclosure are not particularly limited to this.

[0056] In this embodiment, the first multimedia data is interpolated according to the scene time to obtain the second multimedia data associated with the first multimedia data, thereby increasing the data basis for generating the picture book. In this embodiment, the scene location of the first multimedia data and the scene location of the second multimedia data can be changed or unchanged.

[0057] In one embodiment, the step of adjusting the first multimedia data according to the second scene time includes: determining an image attribute that matches the second scene time, and adjusting the image included in the first multimedia data according to the image parameter to obtain the image included in the second multimedia data. The image attribute includes at least one of the following: pixels, resolution, size, color, bit depth, hue, saturation, brightness, color channel, hierarchical composition of the image, etc.

[0058] For example, the scene of the image contained in the first multimedia data is the seaside at noon, and the scene of the image contained in the generated second multimedia data is the seaside in the afternoon. The image contained in the second multimedia data is darker in brightness and redder in tone than the image contained in the first multimedia data.

[0059] In one embodiment, when the image included in the first multimedia data includes a target element, the position of the target element in the scene is adjusted based on the time difference between the first scene time and the second scene time. The target element is an element whose position changes over time, and may include but is not limited to: the sun, the moon, a shadow, etc.

[0060] For example, the scene of the image contained in the first multimedia data is the seaside at noon, the sun is located in the middle of the image, and the shadow of the person is small; the scene of the image contained in the generated second multimedia data is the seaside in the afternoon, the sun is located on the right side of the image, and the shadow of the person is elongated to the left.

[0061] In one embodiment, in step 102, when the extraction result includes scene information, a second scene location is generated according to the first scene location included in the scene information, and the first multimedia data is adjusted according to the second scene location to obtain the second multimedia data. The distance difference between the first scene location and the second scene location is less than the distance difference threshold.

[0062] The difference threshold can be a fixed value or a random value.

[0063] For example, the first scene is located halfway up a mountain, and the second scene is located at the top of a mountain; the first scene is located at the seaside, and the second scene is located at a small house on the seaside.

[0064] The number of second scene locations can be set according to actual conditions. The number of second scene locations can be 1, corresponding to the generation of one second multimedia data; the number of second scene locations can be multiple, corresponding to the generation of multiple second multimedia data. The number of second scene locations can be negatively correlated with the amount of second multimedia data. The time spans of the second scene locations can be the same or different, and the embodiments of the present disclosure are not particularly limited to this.

[0065] In this embodiment, it is equivalent to interpolating the first multimedia data according to the scene location to obtain the second multimedia data associated with the first multimedia data, thereby increasing the data basis for generating the picture book.

[0066] In one embodiment, in step 102, when the extraction result includes scene information, a second scene time is generated according to the first scene time included in the scene information, a second scene location is generated according to the first scene location included in the scene information, and second multimedia data is generated according to the second scene time and the second scene location. The time difference between the first scene location and the second scene location is less than a first time difference threshold, and the distance difference between the first scene location and the second scene location is less than a distance difference threshold.

[0067] For example, the first multimedia data represents the middle of a mountain in the morning, and the second multimedia data represents the top of a mountain at noon.

[0068] In this embodiment, the second multimedia data generated based on the second scene time and the second scene location is closer to reality, and the picture book generated based on the first multimedia data and the second multimedia data is more logical.

[0069] In one embodiment, the step of adjusting the first multimedia data according to the second scene location to obtain the second multimedia data includes: determining image enhancement parameters that match the second scene location, and adjusting the image contained in the first multimedia data according to the image enhancement parameters to obtain the image contained in the second multimedia data.

[0070] The image enhancement parameters include at least one of the following: cropping size, rotation angle, enlargement size, reduction size, mirror image, etc.

[0071] In one embodiment, a correspondence between the distance difference between the second scene location and the first scene location and the image enhancement parameter is preset, and the image enhancement parameter matching the second scene location is determined based on the correspondence. The correspondence may be predefined.

[0072] The first scene location of the image contained in the first multimedia data and the second scene location of the image contained in the second multimedia data may be the same, that is, the distance difference threshold is 0, but the specific display scenes of the image contained in the first multimedia data and the image contained in the second multimedia data may be different, for example, the image contained in the second multimedia data is cropped from the image contained in the first multimedia data; or the image contained in the second multimedia data is a left perspective, and the image contained in the second multimedia data is a right perspective.

[0073] In one embodiment, when creating a picture book in step 103, each image contained in the first multimedia data and the second multimedia data is described by looking at the pictures to obtain the story content corresponding to each image, the images are sorted according to the scene time of each image, and the story content of each image is arranged around the sorted images to obtain a picture book.

[0074] In this embodiment, the images are first sorted according to the scene time of each image, and then the story content of each image is arranged around the sorted image, which can make the story content of the picture book more fluent.

[0075] In one embodiment, the picture book generation method further includes the following steps: in response to a request to replace the character information included in the first multimedia data, adjusting the picture book according to the new character information included in the replacement request.

[0076] The new character information reflects the character image, and may include but is not limited to at least one of the following parameters: gender, age, height, posture, expression, emotion, clothing, etc. In the generation of audio picture books, the character information may also include speech speed, intonation, accent, etc.

[0077] The new character information can be obtained from images taken by the user. The images taken by the user can be provided by the user, and the characters in the images taken by the user can be the user himself, small animals, or family members. The characters are extracted from the images taken by the user, and the new character information is determined based on the extraction results. If the user decides to adjust the picture book, a replacement request is generated, and then the picture book is adjusted according to the new character information contained in the replacement request, including replacing the characters in the images and videos contained in the picture book with the characters represented by the new character information, and replacing the story content of the picture book with the characters contained in the new character information.

[0078] The specific algorithm for character extraction can be, but is not limited to, using open source image detection and segmentation models, such as yolo, SAM, paddleHub algorithms, etc. The algorithm for generating new character information can use the Wenxinda model, NLP model, etc. to understand the character extraction results, generate corresponding prompt words, and then use open source Wenshengtu models, such as Stable Diffusion, to generate new character information.

[0079] The new character information may be obtained from the character description provided by the user, and the character description may be provided in the form of, but not limited to, voice or text description. The character description may be a detailed description or a keyword.

[0080] The new character information can also be determined based on the selected character information. Specifically, several candidate character information is provided for the user to select, and the candidate character information selected by the user is determined as the new character information. Figure 2 This embodiment provides a user interface, which displays candidate items for generating new character information, such as role settings and personality settings. In the generation of audio picture books, the speech speed setting of the new character information can also be set.

[0081] In this embodiment, personalized customization of the picture book is achieved, which enhances the interest of picture book generation.

[0082] In one embodiment, the picture book generation method further includes the following steps: in response to a request to replace the character information included in the second multimedia data, adjusting the picture book according to the new character information included in the replacement request.

[0083] In this embodiment, the specific implementation method of replacing the character information included in the second multimedia data is similar to the specific implementation method of replacing the character information included in the first multimedia data, and will not be repeated here.

[0084] In one embodiment, the picture book generation method further includes the following steps: in response to a request to replace the scene information included in the first multimedia data, adjusting the picture book according to the new scene information included in the replacement request.

[0085] The new scene information may be obtained from images taken by the user. The images taken by the user may be provided by the user. The scene is extracted from the images taken by the user, and the new scene information is determined based on the extraction result. If the user determines to adjust the picture book, a replacement request is generated, and then the picture book is adjusted based on the new scene information contained in the replacement request, including replacing the images and scenes in the video contained in the picture book with the scenes represented by the new scene information, and replacing the story content of the picture book with the scenes represented by the new scene information.

[0086] The new scene information may be obtained from the scene description provided by the user, and the scene description may be provided in the form of, but not limited to, voice or text description. The scene description may be a detailed description or a keyword.

[0087] The new scene information may also be determined based on the selected scene information. Specifically, several candidate scene information are provided for the user to select, and the candidate scene information selected by the user is determined as the new scene information.

[0088] In this embodiment, personalized customization of the picture book is achieved, which enhances the interest of picture book generation.

[0089] In one embodiment, the picture book generation method further includes the following steps: in response to a request to replace the scene information included in the second multimedia data, adjusting the picture book according to the new scene information included in the replacement request.

[0090] In this embodiment, the specific implementation manner of replacing the scene information included in the second multimedia data is similar to the specific implementation manner of replacing the scene information included in the first multimedia data, and will not be repeated here.

[0091] In one embodiment, adjusting the picture book according to the new character information included in the replacement request includes: beautifying the image of the character represented by the new character information, and replacing the character in the image included in the picture book with the beautified character.

[0092] Image beautification includes: converting characters into cartoon effects (see Figure 3 ) and / or beautify the person.

[0093] In one embodiment, image beautification uses open source algorithms, such as Stable Diffusion Wenshengtu, CycleGan model, etc., which can beautify the images uploaded by users. The cost is low. The user only needs to upload one picture, and there is no training cost.

[0094] In one embodiment, a Lora model is customized, and the user uploads a certain number of images, and the Lora model is self-trained to generate images with user characteristics and beautified. The advantage is that the degree of personalization is higher.

[0095] In this embodiment, by beautifying the image of the characters, the characters in the picture book can be made more lovely and more popular with readers.

[0096] In one embodiment, the user can select the beautification content by himself to increase the interaction with the user during the picture book generation process, thereby further increasing the interest of the picture book generation.

[0097] On the basis of being in accordance with the common sense in the art, the above-mentioned preferred conditions can be arbitrarily combined to obtain the preferred embodiments of the present disclosure.

[0098] Corresponding to the aforementioned picture book generation method embodiment, the present disclosure also provides an embodiment of a picture book generation device.

[0099] Figure 4 An exemplary embodiment of the present disclosure provides a picture book generating device, comprising:

[0100] An extraction module 41, configured to extract scene information and / or character information from the first multimedia data;

[0101] A data generating module 42, configured to generate second multimedia data associated with the first multimedia data according to the extraction result;

[0102] The creation module 43 is used to create a picture book according to the first multimedia data and the second multimedia data.

[0103] Optionally, the extraction module is specifically used for:

[0104] In the case where the first multimedia data includes an image, performing image recognition on the image, and extracting scene information and / or character information from the image recognition result;

[0105] In the case where the first multimedia data includes audio, performing speech recognition on the audio, and extracting scene information and / or character information from the speech recognition result;

[0106] In the case where the first multimedia data includes a video, frames are extracted from the images contained in the video, image recognition is performed on the frame extraction results, voice recognition is performed on the audio contained in the video, and scene information and / or character information is extracted from the image recognition results and the voice recognition results.

[0107] Optionally, the extraction module is specifically used for:

[0108] The first multimedia data is input into a content extraction model so that the content extraction model extracts features related to scenes and / or characters from the first multimedia data, and outputs the scene information and character information based on the features; wherein the content extraction model is obtained by training a neural network based on multimedia data samples labeled with scene information and / or character information.

[0109] Optionally, both the first multimedia data and the second multimedia data include images; and the data generating module is used to:

[0110] a time generating unit, configured to generate a second scene time according to a first scene time included in the scene information when the extraction result includes the scene information; wherein a time difference between the first scene time and the second scene time is less than a first time difference threshold;

[0111] A first adjustment unit, configured to adjust the first multimedia data according to the second scene time to obtain the second multimedia data;

[0112] and / or, a location generating unit, configured to generate a second scene location according to a first scene location included in the scene information when the extraction result includes scene information; wherein a distance difference between the first scene location and the second scene location is less than a distance difference threshold;

[0113] The second adjustment unit is used to adjust the first multimedia data according to the second scene location to obtain the second multimedia data.

[0114] Optionally, the first adjustment unit is specifically configured to:

[0115] determining an image attribute that matches the second scene time;

[0116] The image included in the first multimedia data is adjusted according to the image parameter to obtain the image included in the second multimedia data.

[0117] Optionally, the first adjustment unit is specifically configured to:

[0118] The position and posture of the target element in the image included in the first multimedia data is adjusted based on the time difference between the first scene time and the second scene time to obtain the second multimedia data.

[0119] Optionally, the first multimedia data and the second multimedia data both include images;

[0120] The creation module includes:

[0121] a story generating unit, configured to generate a story based on each image included in the first multimedia data and the second multimedia data, and obtain story contents corresponding to each image;

[0122] A sorting unit, used to sort the images according to the scene time of each image;

[0123] The layout unit is used to layout the story content of each image around the sorted images to obtain the picture book.

[0124] Optionally, it also includes: an adjustment module;

[0125] The adjustment module is used for responding to a first replacement request for character information included in the first multimedia data, and adjusting the picture book according to new character information included in the first replacement request;

[0126] and / or, the adjustment module is used to respond to a second replacement request for character information included in the second multimedia data, and adjust the picture book according to new character information included in the second replacement request;

[0127] And / or, the adjustment module is used to respond to a third replacement request for scene information included in the first multimedia data, and adjust the picture book according to new scene information included in the third replacement request;

[0128] And / or, the adjustment module is used to respond to a fourth replacement request for scene information included in the second multimedia data, and adjust the picture book according to new scene information included in the fourth replacement request.

[0129] Optionally, the new character information is obtained from an image taken by a user;

[0130] Alternatively, the new character information is obtained from the character description.

[0131] Optionally, the adjustment module is specifically used to:

[0132] beautifying the image of the character represented by the new character information;

[0133] The characters in the images contained in the picture book are replaced with the characters with beautified images.

[0134] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can refer to the partial description of the method embodiments. The device embodiments described above are only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the disclosed solution. A person of ordinary skill in the art can understand and implement it without paying any creative work.

[0135] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0136] Figure 5 A schematic block diagram of an example electronic device 500 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0137] like Figure 5 As shown, the device 500 includes a computing unit 501, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 502 or a computer program loaded from a storage unit 508 into a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the device 500 can also be stored. The computing unit 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0138] A number of components in the device 500 are connected to the I / O interface 505, including: an input unit 506, such as a keyboard, a mouse, etc.; an output unit 507, such as various types of displays, speakers, etc.; a storage unit 508, such as a disk, an optical disk, etc.; and a communication unit 509, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 509 allows the device 500 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0139] The computing unit 501 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 501 performs the various methods and processes described above, such as the picture book generation method. For example, in some embodiments, the picture book generation method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 908. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 500 via the ROM 502 and / or the communication unit 509. When the computer program is loaded into the RAM 503 and executed by the computing unit 501, one or more steps of the picture book generation method described above may be performed. Alternatively, in other embodiments, the computing unit 501 may be configured to perform the picture book generation method in any other appropriate manner (e.g., by means of firmware).

[0140] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0141] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0142] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0143] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0144] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communications network). Examples of communications networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0145] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0146] The computer-readable storage medium provided by the embodiments of the present disclosure is a non-transitory computer-readable storage medium having computer instructions, wherein the computer instructions are used to enable the computer to execute the method provided by any of the above embodiments.

[0147] The computer program product provided by the embodiments of the present disclosure includes a computer program, and when the computer program is executed by a processor, the method provided by any of the above embodiments is implemented.

[0148] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0149] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A picture book generation method, comprising: Extracting scene information and / or character information from the first multimedia data; generating second multimedia data associated with the first multimedia data according to the extraction result; The first multimedia data and the second multimedia data both include images; Generating second multimedia data associated with the first multimedia data according to the extraction result includes: In the case where the extraction result includes scene information, generating a second scene time according to a first scene time included in the scene information; wherein a time difference between the first scene time and the second scene time is less than a first time difference threshold; adjusting the first multimedia data according to the second scene time to obtain the second multimedia data; and / or, when the extraction result includes scene information, generating a second scene location according to a first scene location included in the scene information; wherein a distance difference between the first scene location and the second scene location is less than a distance difference threshold; adjusting the first multimedia data according to the second scene location to obtain the second multimedia data; A picture book is created according to the first multimedia data and the second multimedia data.

2. The picture book generation method according to claim 1, extracting scene information and / or character information from the first multimedia data, comprising: In the case where the first multimedia data includes an image, performing image recognition on the image, and extracting scene information and / or character information from the image recognition result; In the case where the first multimedia data includes audio, performing speech recognition on the audio, and extracting scene information and / or character information from the speech recognition result; In the case where the first multimedia data includes a video, frames are extracted from the images contained in the video, image recognition is performed on the frame extraction results, voice recognition is performed on the audio contained in the video, and scene information and / or character information is extracted from the image recognition results and the voice recognition results.

3. The picture book generation method according to claim 1, extracting scene information and character information from the first multimedia data, comprising: The first multimedia data is input into a content extraction model so that the content extraction model extracts features related to scenes and / or characters from the first multimedia data, and outputs the scene information and character information based on the features; wherein the content extraction model is obtained by training a neural network based on multimedia data samples labeled with scene information and / or character information.

4. The picture book generation method according to claim 1, adjusting the first multimedia data according to the second scene time to obtain the second multimedia data, comprising: determining an image attribute that matches the second scene time; The image included in the first multimedia data is adjusted according to the image parameter to obtain the image included in the second multimedia data.

5. The picture book generation method according to claim 1, adjusting the first multimedia data according to the second scene time to obtain the second multimedia data, comprising: The position and posture of the target element in the image included in the first multimedia data is adjusted based on the time difference between the first scene time and the second scene time to obtain the second multimedia data.

6. The picture book generation method according to claim 1, wherein the first multimedia data and the second multimedia data both contain images; Creating a picture book according to the first multimedia data and the second multimedia data includes: Performing picture description on each image included in the first multimedia data and the second multimedia data to obtain story content corresponding to each image; sorting the images according to the scene time of each image; The story content of each image is arranged around the sorted images to obtain the picture book.

7. The method for generating a picture book according to any one of claims 1 to 6, further comprising: In response to a first replacement request for character information included in the first multimedia data, adjusting the picture book according to new character information included in the first replacement request; and / or, in response to a second replacement request for character information included in the second multimedia data, adjusting the picture book according to new character information included in the second replacement request; and / or, in response to a third replacement request for scene information included in the first multimedia data, adjusting the picture book according to new scene information included in the third replacement request; And / or, in response to a fourth replacement request for scene information included in the second multimedia data, adjusting the picture book according to new scene information included in the fourth replacement request.

8. The picture book generation method according to claim 7, wherein the new character information is obtained from an image taken by a user; Alternatively, the new character information is obtained from the character description.

9. The picture book generation method according to claim 7, adjusting the picture book according to the new character information included in the replacement request, comprising: beautifying the image of the character represented by the new character information; The characters in the images contained in the picture book are replaced with the characters whose images are beautified.

10. A picture book generating device, comprising: An extraction module, used to extract scene information and / or character information from the first multimedia data; a data generation module, configured to generate second multimedia data associated with the first multimedia data according to the extraction result; the first multimedia data and the second multimedia data both contain images; the data generation module is configured to: a time generating unit, configured to generate a second scene time according to a first scene time included in the scene information when the extraction result includes the scene information; wherein a time difference between the first scene time and the second scene time is less than a first time difference threshold; A first adjustment unit, configured to adjust the first multimedia data according to the second scene time to obtain the second multimedia data; and / or, a location generating unit, configured to generate a second scene location according to a first scene location included in the scene information when the extraction result includes scene information; wherein a distance difference between the first scene location and the second scene location is less than a distance difference threshold; A second adjustment unit, configured to adjust the first multimedia data according to the second scene location to obtain the second multimedia data; A creation module is used to create a picture book according to the first multimedia data and the second multimedia data.

11. The picture book generating device according to claim 10, wherein the extraction module is specifically used for: In the case where the first multimedia data includes an image, performing image recognition on the image, and extracting scene information and / or character information from the image recognition result; In the case where the first multimedia data includes audio, performing speech recognition on the audio, and extracting scene information and / or character information from the speech recognition result; In the case where the first multimedia data includes a video, frames are extracted from the images contained in the video, image recognition is performed on the frame extraction results, voice recognition is performed on the audio contained in the video, and scene information and / or character information is extracted from the image recognition results and the voice recognition results.

12. The picture book generating device according to claim 10, wherein the extraction module is specifically used for: The first multimedia data is input into a content extraction model, so that the content extraction model extracts features related to scenes and / or characters from the first multimedia data, and outputs the scene information and character information according to the features; wherein, The content extraction model is obtained by training a neural network based on multimedia data samples labeled with scene information and / or character information.

13. The picture book generating device according to claim 10, wherein the first adjustment unit is specifically configured to: determining an image attribute that matches the second scene time; The image included in the first multimedia data is adjusted according to the image parameter to obtain the image included in the second multimedia data.

14. The picture book generating device according to claim 10, wherein the first adjusting unit is specifically configured to: The position and posture of the target element in the image included in the first multimedia data is adjusted based on the time difference between the first scene time and the second scene time to obtain the second multimedia data.

15. The picture book generating device according to claim 10, wherein the first multimedia data and the second multimedia data both contain images; The creation module includes: A story generating unit, configured to perform picture description on each image included in the first multimedia data and the second multimedia data to obtain story content corresponding to each image; A sorting unit, used for sorting the images according to the scene time of each image; The layout unit is used to layout the story content of each image around the sorted images to obtain the picture book.

16. The picture book generating device according to any one of claims 10 to 15, further comprising: Adjustment module; The adjustment module is used for responding to a first replacement request for character information included in the first multimedia data, and adjusting the picture book according to new character information included in the first replacement request; and / or, the adjustment module is used to respond to a second replacement request for character information included in the second multimedia data, and adjust the picture book according to new character information included in the second replacement request; And / or, the adjustment module is used to respond to a third replacement request for scene information included in the first multimedia data, and adjust the picture book according to new scene information included in the third replacement request; And / or, the adjustment module is used to respond to a fourth replacement request for scene information included in the second multimedia data, and adjust the picture book according to new scene information included in the fourth replacement request.

17. The picture book generating device according to claim 16, wherein the new character information is obtained from an image taken by a user; Alternatively, the new character information is obtained from the character description.

18. The picture book generating device according to claim 16, wherein the adjustment module is specifically used for: beautifying the image of the character represented by the new character information; The characters in the images contained in the picture book are replaced with the characters whose images are beautified.

19. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the picture book generation method according to any one of claims 1 to 9.

20. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to enable the computer to execute the picture book generation method according to any one of claims 1-9.

21. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method for generating a picture book according to any one of claims 1 to 9 is implemented.

Images