Content generation system, method, program, and storage medium

Abstract

A content generation system generates content composed of one or more frames of dynamic images or static images including a character, the content generation system including: a display control section that displays a user interface including a first two-dimensional operation region and a preview image of a frame being edited by a user on a display section; an expression decision section that decides an expression of the character on the preview image in accordance with a position of a pointer on the first two-dimensional operation region; and a content generation section that generates content composed of one or more frames including the frame edited by the user.

Description

Technical Field

[0001] This invention relates to content generation systems, methods, programs, and storage media. Background Technology

[0002] Previously, methods for controlling virtual characters (or "avatars") using multimodal models were known (see Patent Document 1).

[0003] Existing technical documents Patent documents Patent Document 1: Japanese Patent Publication No. 2022-534708 Summary of the Invention The problem that the invention aims to solve However, there are currently no proposals regarding the generation of content consisting of frames of characters containing expressions that the user expects.

[0004] The problem to be solved by the present invention is to easily generate content consisting of frames of characters containing expressions desired by the user.

[0005] Technical solutions for solving the problem [1] One method involves a content generation system that generates content consisting of one or more frames containing dynamic or static images of characters. The content generation system has the following features: The display control unit displays the user interface containing the first two-dimensional operation area and a preview image of the frame edited by the user on the display unit; An expression determination unit determines the expression of the character in the preview image based on the position of a pointer on the first two-dimensional operation area; and The content generation unit generates content consisting of one or more frames, including the frame edited by the user.

[0006] [2] One approach involves a content generation system, in which the content generation system described in [1] is... The user interface also includes a gaze determination unit, which includes a second two-dimensional operation area, and determines the gaze of the character in the preview image based on the position of the pointer on the second two-dimensional operation area.

[0007] [3] A content generation system involving a method, in the content generation system described in [1] or [2], The content generation system also includes an action determination unit, which determines the action of the character on the preview image based on the user's operation of the user interface.

[0008] [4] One approach involves a content generation system, in which the content generation system described in [3] is... The user interface includes a one-dimensional operation area. The motion determination unit determines the playback range of the motion based on the position of the slider on the one-dimensional operation area.

[0009] [5] A content generation system involving a method, in the content generation system described according to any one of [1] to [4], The content generation system also includes an action determination unit, which determines the action of the character in the preview image based on the position of the pointer on the first two-dimensional operation area.

[0010] [6] A content generation system involving a method, in the content generation system described according to any one of [1] to [5], The content generation system also includes a character determination unit, which determines the characters to be displayed on the preview image.

[0011] [7] A content generation system involving a method, in the content generation system described according to any one of [1] to [6], The content generation system also includes a background image determination unit, which determines the background image of the preview image.

[0012] [8] A content generation system relating to a method, in the content generation system described according to any one of [1] to [7], The content generation system also includes a text information determination unit, which determines at least one of the following based on the user's operation of the user interface: the text to be displayed on the preview image, the type of the speech bubble displaying the text, the color of the text, and the size of the text.

[0013] [9] A content generation system relating to a method, in the content generation system described according to any one of [1] to [8], The first two-dimensional operation area is a user interface used to set the character's emotions.

[0014]

[10] A content generation system relating to a method, in the content generation system described according to any one of [1] to [9], The content generation system also includes a sound generation unit, which generates sound data for a character by inputting the image data of the character selected by the user into a sound model. The sound model is a model that uses the character image and the sound data associated with the character image as learning data.

[0015]

[11] A content generation system relating to one method, in the content generation system described according to any one of [1] to

[10] , The content generation system also includes a speech data generation unit, which generates speech data of a character that responds to the user's speech data by inputting the user's speech data into a language model. The language model is a model that has been learned by using the character's image data and / or voice data as part of the learning data.

[0016]

[12] A content generation system relating to one method, in the content generation system described according to any one of [1] to

[11] , The content generation system also includes a character image generation unit, which generates character images by inputting image data from the user into a learned image processing model.

[0017]

[13] A content generation system relating to one method, in the content generation system described according to any one of [1] to

[12] , The content generation system also includes a background image generation unit, which generates a background image by inputting the image data input by the user into a learned image processing model.

[0018]

[14] A content generation system relating to a method, in the content generation system described according to any one of [1] to

[13] , The image data generated by the character image generation unit and the background image generation unit are three-dimensional image data and are configured in the metaverse space.

[0019]

[15] A content generation system relating to one method, in the content generation system described according to any one of [1] to

[14] , The content generation system also includes an NFT issuance department, which issues NFTs related to the content generated by the content generation department and / or the image data of characters within the content.

[0020]

[16] One approach involves generating content consisting of one or more frames containing moving or still images of characters, the approach having: The display control steps involve displaying a user interface containing the first two-dimensional operation area and a preview image of a frame edited by the user on the display unit; The expression determination step involves determining the character's expression in the preview image based on the position of the pointer on the first two-dimensional operation area; and The content generation step generates content consisting of one or more frames, including the frame edited by the user.

[0021]

[17] One method relates to a program for causing a computer to execute a method for generating content consisting of one or more frames containing moving or still images of characters, the method having: The display control steps involve displaying a user interface containing the first two-dimensional operation area and a preview image of a frame edited by the user on the display unit; The expression determination step involves determining the character's expression in the preview image based on the position of the pointer on the first two-dimensional operation area; and The content generation step generates content consisting of one or more frames, including the frame edited by the user.

[0022]

[18] One method relates to a storage medium that is a computer-readable storage medium storing the program described in

[17] .

[0023]

[19] One approach relates to a content generation system that generates content consisting of one or more frames containing dynamic or static images of characters, the content generation system comprising: The display control unit displays the user interface and a preview image of the frame edited by the user on the display unit; The character determination unit determines the character to be displayed on the preview image based on the user interface operations performed by the user. An expression determination unit determines the character's expression based on the user interface operations performed by the user; A gaze determination unit determines the character's gaze based on the user interface operations performed by the user. An action determination unit determines the character's actions based on the user interface operations performed by the user. The background determination unit determines the background image of the preview image based on the user interface operations performed by the user. A text information determination unit, based on user interface operations performed by the user, determines at least one of the following: the text to be displayed on the preview image, the type of speech bubble displaying the text, the color of the text, and the size of the text; and The content generation unit generates content consisting of one or more frames, including the frame edited by the user. Attached Figure Description

[0024] Figure 1 This is a diagram illustrating an example of the system structure of the content generation system 1 in this embodiment.

[0025] Figure 2 This is a block diagram illustrating an example of the hardware structure of the content generation system 1 in this embodiment.

[0026] Figure 3 This is a block diagram illustrating an example of the functional structure of the content generation system 1 in this embodiment.

[0027] Figure 4 This is a flowchart illustrating an example of the operation of the content generation system 1 in this embodiment.

[0028] Figure 5 This is an example of a content creation screen in this embodiment.

[0029] Figure 6 This is an example of a content creation screen in this embodiment, and it is also an image showing the state after the character is determined.

[0030] Figure 7 This is an example of a content creation screen in this embodiment, and it shows the state after the character's expression is determined.

[0031] Figure 8 This is an example of a content creation screen in this embodiment, and it shows the state after the character's gaze is determined.

[0032] Figure 9 This is an example of a content creation screen in this embodiment, and it is also an image showing the state of a character's actions.

[0033] Figure 10 This is an example of a content creation screen in this embodiment, and it is also an image showing the state after the character's actions are determined.

[0034] Figure 11 This is an example of a content creation screen in this embodiment, and it shows the state after the character configuration is determined.

[0035] Figure 12 This is an example of a content creation screen in this embodiment, and it shows the state after the background image is determined.

[0036] Figure 13 This is an example of a content creation screen in this embodiment, and it is also an image showing the state after the text information is determined.

[0037] Figure 14 This diagram shows an example of the facial expression setting unit in this embodiment.

[0038] Figure 15 This diagram shows an example of the line-of-sight setting unit in this embodiment.

[0039] Figure 16 This diagram illustrates an example of the motion setting unit in this embodiment.

[0040] Figure 17 This diagram shows another example of the facial expression setting unit in this embodiment.

[0041] Figure 18 This is a diagram showing another example of the line-of-sight setting unit in this embodiment. Detailed Implementation

[0042] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. It should be noted that in each drawing, structural elements with equivalent functions are labeled with the same reference numerals, and detailed descriptions of structural elements with the same reference numerals are not repeated.

[0043] (Overview of the content generation system) The content generation system of this embodiment is used to allow users to generate content consisting of one or more frames, such as comics and animations. The following is a detailed description of the content generation system of this embodiment.

[0044] (Structure of the content generation system) Figure 1 This is a diagram illustrating the general structure of the content generation system 1 in this embodiment. (Example) Figure 1 As shown, the content generation system 1 of this embodiment includes a user terminal 2 (terminal device) and a server 3 for user use.

[0045] User terminal 2 and server 3 are connected to each other via a network 4, such as the Internet, in a manner that enables communication. Network 4 can be either wired or wireless, and the type and form of the network are not limited. At least a portion of user terminal 2 and server 3 is implemented using a computer (information processing device). User terminal 2 is, for example, a personal computer, smartphone, tablet, or other terminal device. Multiple user terminals 2 exist depending on the number of users utilizing content generation system 1.

[0046] (Hardware structure) Next, the hardware structure of the content generation system 1 in this embodiment will be described. Figure 2 This is a block diagram illustrating an example of the hardware structure of the user terminal 2 and server 3 included in the content generation system 1 of this embodiment.

[0047] In user terminal 2, CPU 201 is a processing device that controls the overall operation of user terminal 2. ROM 202 is a non-volatile memory that stores the control program and various data executed by CPU 201. RAM 203 is a volatile memory used in the loading area and working area of ​​the program executed by CPU 201. Storage device 204 is a storage unit for storing various information; it can be a storage device built into the main body of user terminal 2 or a storage device that can be removable from the storage medium. Input device 205 is a device for user input of information for user terminal 2, such as a keyboard, mouse, touch panel, microphone, etc. Display 206 is a display device for displaying various information (user interface, etc.). Imaging element 207 is a photoelectric conversion element for capturing images of the subject. Communication I / F (interface) 208 is an interface for connecting to network 4. Bus 209 is a bus line that connects the above structural elements to each other.

[0048] In server 3, CPU 301 is the processing device that controls the overall operation of server 3. ROM 302 is a non-volatile memory that stores the control program and various data executed by CPU 301. RAM 303 is a volatile memory used in the loading area and working area of ​​the program executed by CPU 301. Storage device 304 is a storage unit for storing various information; it can be built into the main body of server 3 or the storage medium can be removable. Communication I / F (interface) 305 is an interface for connecting to network 4. Bus 306 is a bus line that connects the above-mentioned components to each other.

[0049] (Functional Structure) Next, the functional structure of the content generation system 1 in this embodiment will be explained. Figure 3 This is a diagram illustrating an example of the functional structure of the content generation system 1 in this embodiment.

[0050] First, the functional structure of user terminal 2 will be explained. For example... Figure 3 As shown, the user terminal 2 has a communication unit 21, a control unit 22 that controls the overall operation of the user terminal 2, an input unit 23 for the user to input various information, an output unit 24 for outputting various information, a shooting unit 25 for capturing images of the subject, and a storage unit 26 for storing various information.

[0051] Communication unit 21 is the communication interface between user terminal 2 and network 4. Communication unit 21 sends and receives information between user terminal 2 and server 3 via network 4.

[0052] The control unit 22 includes a display control unit 22a, a frame determination unit 22b, a character determination unit 22c, an expression determination unit 22d, a gaze determination unit 22e, an action determination unit 22f, a character configuration determination unit 22g, a background image determination unit 22h, and a text information determination unit 22i.

[0053] The display control unit 22a displays the content creation screen (operation screen) of the content generation application (program) of the content generation system 1 on the output unit 24 (display unit) described later.

[0054] Figure 5 This diagram illustrates an example of a display control unit 22a creating a screen 50 (operation screen) on the output unit 24 (display unit) to display content. For example... Figure 5 As shown, the content creation screen 50 displays multiple GUIs (Graphical User Interfaces) for editing the frames that constitute the content. Specifically, the content creation screen 50 includes: a tab 51 for switching editing items; a frame overview area 52 that displays thumbnail images of the frames that constitute the content and allows selection of frames to be edited; a preview display area 53 that displays a preview image of the frame being edited; a background image area 53a that is at least a portion of the preview display area and displays a background image; a character selection area 54 for selecting characters; a character expression setting unit 55; a character gaze setting unit 56; a character motion setting unit 57; and a cursor C for the user to perform selection operations for each GUI.

[0055] return Figure 3 The frame determination unit 22b determines the frame edited by the user from one or more frames constituting the content. Specifically, the frame determination unit 22b selects the frame displayed in the frame overview area 52 (see reference) based on the user's selection using cursor C. Figure 5 The manipulation of the thumbnail image of the frame determines the frame that the user can edit. Figure 5 This indicates that among the thumbnail images of frames 1 to 4 displayed in the frame overview area, the user selects frame 1 and the frame determination unit 22b determines frame 1 as the frame to be edited.

[0056] Furthermore, frames can be added to the frame overview area 52 by selecting the "+" button 52a displayed within the frame overview area 52. Additionally, frames can be deleted from the frame overview area 52 by selecting the "-" button 52b. Figure 5 In the example shown, frame 1 is used. Additionally, by selecting the play button 52c, the content generation unit 32c (described later) generates one or more frames displayed in the frame overview area 52. Figure 5 The example shown consists of frames 1 through 4, and the generated content is displayed in the preview area. For example, in Figure 5In the example shown, frames 1 to 4, which are thumbnailed in the frame overview area 52, are displayed in the preview display area 53 in a slideshow manner, continuously switching at specified time intervals.

[0057] return Figure 3 The character determination unit 22c determines the character to be displayed in the frame being edited by the user. Specifically, the character determination unit 22c determines the character to be displayed in the character selection area 54 (see reference). Figure 5 The character selected by the user using cursor C is displayed in the frame edited by the user. The image of the character determined by character determination unit 22c is overlaid on the preview image of the frame displayed in preview display area 53. Figure 5 As shown in the example, when the "Do Not Display" button is selected from the character selection area 54, the character is not displayed in the preview display area 53. However, when any character is selected from the character selection area 54, the character selection unit 22c determines the selected character. Figure 6 As shown in the example, the image of the character A selected by the user is overlaid on the preview image of the frame displayed in the preview display area 53. The character A displayed in the preview display area 53 can be moved or zoomed in / out using the cursor C. For example, when using a mouse as the input device 205 of the user terminal 2, the cursor C can be manipulated with the mouse to drag the character A in the preview display area 53, thereby moving the character A to any position. Alternatively, the character A can be zoomed in or out by using the mouse wheel during the dragging operation.

[0058] Furthermore, the image of character A displayed in the preview display area 53 can be any image, whether it is a dynamic image or a static image. However, if the motion determination unit, described later, determines that "doing no action" is the character's action (action), the image of the character displayed in the preview display area 53 becomes a static image.

[0059] The expression determination unit 22d determines the "expression" of character A displayed in the preview display area 53 based on the operation performed by the user on the expression setting unit 55.

[0060] Figure 14 This is an example illustration representing the facial expression setting section 55. (Example) Figure 14 As shown, the expression setting unit 55 includes a two-dimensional operation area 55a (first two-dimensional operation area) and a pointer 55b that can move on the two-dimensional operation area 55a. The expression setting unit 55 can set the expression (joy, anger, sorrow, happiness) of character A displayed in the preview display area 53 according to the position of the pointer 55b on the two-dimensional operation area 55a. Specifically, on the two-dimensional operation area 55a of the expression setting unit 55, the more the pointer 55b is moved towards the "joy" side (…), the more… Figure 14Moving the pointer 55b in the +X direction (in the preview display area 53) makes the character A's expression more likely to express "happiness." Conversely, in the two-dimensional operation area 55a of the expression setting unit 55, moving the pointer 55b more towards the "sorrow" side (in the +X direction) makes the expression more likely to express "happiness." Figure 14 Moving the pointer 55b in the X direction (or the X-axis direction) displays a more "sad" expression towards the character A in the preview display area 53. Additionally, in the two-dimensional operation area 55a of the expression setting unit 55, moving the pointer 55b towards the "happy" side (or the "joyful" side) further enhances the expression. Figure 14 Moving the pointer 55b in the +Y direction (in the Y direction) displays a more joyful expression in the character A's facial expression in the preview display area 53. Conversely, in the two-dimensional operation area 55a of the expression setting unit 55, moving the pointer 55b towards the "anger" side (in the +Y direction)... Figure 14 The expression of character A displayed in the preview display area 53 changes more towards a more expressive "anger" as the pointer 55b on the two-dimensional operation area 55a of the expression setting unit 55 is moved (in the -Y direction). Thus, the expression determination unit 22d determines the "expression" of character A displayed in the preview display area 53 based on the position of the pointer 55b on the two-dimensional operation area 55a of the expression setting unit 55. Therefore, the user of the content generation system 1 can set the "expression" of characters in the content through intuitive operation.

[0061] Figure 7 This indicates the state of character A's "expression" after the user uses cursor C to manipulate the expression setting unit 55. For example... Figure 7 As shown, because the pointer 55b on the two-dimensional operation area 55a of the expression setting unit 55 moves towards the "joy" side, the expression of character A displayed in the preview display area 53 changes to an expression of "joy".

[0062] return Figure 3 The gaze determination unit 22e determines the gaze of the character A displayed in the preview display area 53 based on the operation performed by the user on the gaze setting unit 56.

[0063] Figure 15 This is a diagram illustrating an example of the line-of-sight setting unit 56. (See diagram below.) Figure 15 As shown, the gaze setting unit 56 includes a two-dimensional operation area 56a (a second two-dimensional operation area) and a pointer 56b that can move on the two-dimensional operation area 56a. The gaze setting unit 56 can set the gaze (gaze direction) of the character A displayed in the preview display area 53 according to the position of the pointer 56b on the two-dimensional operation area 56a. Specifically, on the two-dimensional operation area 56a of the gaze setting unit 56, the more the pointer 56b is moved to the "right" side (…), the more… Figure 15 Moving the pointer 56b in the +X direction (direction) causes the character A's gaze to shift more to the right (the character's eyes move to the left of the screen). Conversely, in the two-dimensional operation area 56a of the gaze setting unit 56, the more the pointer 56b is moved to the "left" (direction), the more the character A's gaze shifts to the right (the character's eyes move towards the left of the screen). Figure 15Moving the pointer 56b in the X direction (or the X-axis direction) causes the character A's gaze to shift more to the left (the character's eyes move to the right of the screen). Additionally, in the two-dimensional operation area 56a of the gaze setting unit 56, the more the pointer 56b is moved "up" (towards the left), the more the character A's gaze shifts. Figure 15 Moving the pointer 56b in the +Y direction (direction) upwards the character A's gaze in the preview display area 53, the more upwards the pointer appears. Conversely, in the two-dimensional operation area 56a of the gaze setting unit 56, the more the pointer 56b is moved "downwards" (in the +Y direction), the higher the character A's gaze appears. Figure 14 As the character A moves (in the -Y direction), the line of sight of the character A displayed in the preview display area 53 moves downwards. Thus, the line-of-sight determination unit 22e determines the "line of sight" of the character A displayed in the preview display area 53 based on the position of the pointer 56b on the two-dimensional operation area 56a of the line-of-sight setting unit 56. Therefore, the user of the content generation system 1 can set the "line of sight" of characters in the content through intuitive operation.

[0064] Figure 8 This indicates the state of character A's "viewpoint" after the user uses cursor C to manipulate the viewpoint setting unit 56. For example... Figure 8 As shown, because the pointer 56b on the two-dimensional operation area 56a of the gaze setting unit 56 moves to the "right" side, the gaze of the character A displayed in the preview display area 53 changes to the right (towards the right side of the screen).

[0065] return Figure 3 The action decision unit 22f determines the action (action) of the character A displayed in the preview display area 53 based on the operation performed by the user on the action setting unit 57.

[0066] Figure 16 This is a diagram showing an example of the motion setting unit 57. (See diagram for example.) Figure 16 As shown, the motion setting unit 57 includes a drop-down menu 571 for selecting the motion category of a character and a playback range setting unit 572 for setting the playback range of the selected motion category. Furthermore, the playback range setting unit 572 includes a first axis 572a, a first slider 572b movable on the first axis 572a, a second axis 572c, and a second slider 572d movable on the second axis 572c.

[0067] In the motion setting unit 57, if the cursor C ( Figure 16 (Not shown in the image) Selecting drop-down menu 571 displays a list of possible action categories. Then, by selecting the desired action category from the displayed action categories, the action determination unit 22f determines the character's action.

[0068] In addition, the axes (first axis 572a and second axis 572c) of the playback range setting unit 572 represent the time axis of the preset action. By moving the sliders (first slider 572b and second slider 572d) on this axis, the action can be set within the desired playback range.

[0069] Figure 9 This is a diagram showing the status of the list displaying action categories after selecting the drop-down menu 571 of the action setting section 57 with the cursor C. For example, when selecting from... Figure 9 If "waving" is selected from the list of action categories shown, the action determination unit 22f determines the character's action as "waving," as follows. Figure 10 As shown, character A, displayed in preview display area 53, dynamically changes in a manner that represents an action (animation) such as "waving".

[0070] return Figure 3 The character configuration decision unit 22g determines the character configuration settings made by the user in the character configuration setting unit 58 (see reference). Figure 11 The operation determines the configuration of character A displayed in preview display area 53. For example, as... Figure 11 As shown, the character configuration setting unit 58 includes settings for "Camera," "Character Position," "Rotation," and "Zoom In / Out." In the "Camera" settings, one can set the character's "full body," "upper body," and "head," which correspond to the camera's field of view. In the "Character Position" settings, one can set the character's display position on the preview display area 53. In the "Rotation" settings, the character A on the preview display area 53 can be rotated in the yaw or pitch direction at predetermined angular intervals (e.g., 90-degree intervals). Furthermore, in the "Zoom In / Out" settings, by moving the slider on the axis, one can zoom in or out on the character A displayed on the preview display area 53.

[0071] return Figure 3 The background image determination unit 22h determines the background image based on the background image setting unit 59 (see reference 59) made by the user. Figure 12 The operation determines the background image to be displayed in the preview display area 53.

[0072] Figure 12 This diagram illustrates how a background image is set through user operation of the background image setting unit 59, and how the state of the background image is determined by the background image determination unit 22h. (Example) Figure 12 As shown, the background image setting unit 59 includes, for example, settings for "background", "background position", "background size" and "lighting".

[0073] In the "Background" setting, in addition to selecting from multiple pre-prepared background images (2D or 3D images, static or animated images), one can also select "Monochrome" (no background image). Furthermore, by selecting the "Add" button, any background image chosen by the user can be added. Moreover, by selecting the "Auto Generate" button, a background image can be automatically generated using a learned image processing model. Specifically, depending on whether the "Auto Generate" button is selected, the background image generation unit (not shown) of the control unit 32 inputs the user-selected character, the character's expression, gaze, actions, and text information (described later) as input data into the learned image processing model, thereby generating a background image. This image processing model, for example, uses Generative Adversarial Networks (GANs) for learning. Alternatively, static images such as landscape photos prepared by the user can also be input into the image processing model to generate a background image. Thus, a background preferred by the user can be automatically generated. Here, when the background image is generated as three-dimensional data (three-dimensional image), the three-dimensional data of the background image can also be configured together with the three-dimensional data of the character generated by the character image generation unit 32a in the metaverse space.

[0074] In the "Background Position" setting, the position of the background image area 53a on the preview display area 53 can be set. Specifically, the position of the background image area 53a on the preview display area 53 can be set by moving the pointer on the two-dimensional operation area. In the "Background Size" setting, the size of the background image area 53a on the preview display area 53 can be set. Specifically, the size of the background image area 53a on the preview display area 53 can be enlarged or reduced by moving the slider on the axis. In the "Lighting" setting, "Daytime," "Evening," "Nighttime," etc., can be selected from the drop-down menu to set the background image depiction corresponding to the time of day.

[0075] return Figure 3 The text information decision unit 22i determines the text information based on the user's settings for the text information setting unit 60 (see reference 60). Figure 13 The operation determines the text information to be displayed in the preview display area 53.

[0076] Figure 13 This diagram illustrates how text information is set through user operation of the text information setting unit 60, and how the text information determination unit 22i determines the state of the text information. For example... Figure 13As shown, the text information setting unit 60 includes settings for "speech bubble," "font," "text," "text color," and "text size." In the "speech bubble" setting, in addition to selecting from multiple pre-prepared speech bubble images, users can also select "no display" (no text information) or "text only" (no speech bubble image). In the "font" setting, users can select the font type of the text displayed in the preview display area 53 from a drop-down menu. In the "text" setting, users can input text displayed in the preview display area 53 through input operations via the input unit 23 (described later). In the "text color" setting, the color of the text displayed in the preview display area 53 can be set. In the "text size" setting, the size of the text displayed in the preview display area 53 can be set. When text information is set by the text information setting unit 60, the text information determination unit 22i determines the text information, which is then displayed in the preview display area 53. Figure 13 In the example shown, a speech bubble image 61, composited with the text "Hello" entered by the user in the "Text" setting of the text information setting unit 60, is created by combining the speech bubble image selected by the user in the "Speech Bubble" setting of the text information setting unit 60 with the text "Hello" in the "Text" setting of the text information setting unit 60. This speech bubble image 61 can be moved in the preview display area 53 using user operations (e.g., dragging) with the cursor C. Furthermore, the size of the speech bubble image 61 can be changed (zoomed out) by operating (e.g., dragging) the zoom-in / zoom-out icon D that appears near the speech bubble image 61. If an area outside the preview display area 53 is selected using the cursor C, the position and size of the speech bubble image 61 in the preview display area 53 are determined, and the zoom-in / zoom-out icon D disappears.

[0077] return Figure 3 The input unit 23 is an element for providing user input information to the user terminal 2, such as a keyboard, mouse, touch panel, microphone, gesture input device, etc.

[0078] The output unit 24 is an interface for outputting various information (images, sounds) from the user terminal 2 to the user, such as an image display device (display unit) like a liquid crystal display, or a speaker. When the output unit 24 is configured as a display unit, a GUI for receiving operations from the user is displayed in the display unit via the display control unit 22a.

[0079] The imaging unit 25 captures images of the subject through the camera module including the imaging element in the user terminal 2. The images captured by the imaging unit 25 are output to the control unit 22, where various image processing is performed. They are then displayed on the display unit (output unit 24) via the display control unit 22a, or sent to the server 3 via the network 4 through the communication unit 21.

[0080] The storage unit 26 is, for example, a data storage device such as built-in memory or external memory (SD memory card, etc.). The storage unit 26 stores various data processed by the control unit 22, various information downloaded from the server 3 by the communication unit 21 via the network 4, images captured by the imaging unit 25, etc. Furthermore, the storage unit 26 may not necessarily be located within the user terminal 2; part or all of the storage unit 26 may be located in other devices that can communicate with the user terminal 2 via the network 4.

[0081] Next, the functional structure of server 3 will be explained. For example... Figure 3 As shown, server 3 has a communication unit 31, a control unit 32, and a storage unit 33.

[0082] Communication unit 31 is the communication interface between server 3 and network 4. Communication unit 31 sends and receives information between server 3 and user terminal 2 via network 4.

[0083] The control unit 32 controls the overall operation of the server 3. In addition, the control unit 32 includes a character image generation unit 32a, a sound generation unit 32b, a content generation unit 32c, a speech data generation unit 32d, and an NFT distribution unit 32e.

[0084] The character image generation unit 32a generates character image data (two-dimensional or three-dimensional image data) based on images received from the user terminal 2 (e.g., facial images, full-body images, or pre-prepared character images captured by the camera unit 25). For example, character images can also be generated using image generation AI as the character image generation unit 32a. Specifically, the character image generation unit 32a generates character images, for example, by inputting user-inputted image data via the user terminal 2 into an image processing model learned using generative adversarial networks (GANs). The generated character images can also be stored in the character image DB33a of the storage unit 33 (described later) or transmitted to the user terminal 2 via the network 4 through the communication unit 31.

[0085] The sound generation unit 32b generates sound data corresponding to the character image (two-dimensional or three-dimensional image) generated by the character image generation unit 32a. Specifically, the sound generation unit 32b inputs the character image generated by the character image generation unit 32a into the learned sound synthesis model and outputs sound data. Thus, it is possible to automatically generate sound data matching the impression of the character's appearance, atmosphere, and personality based on the character image. Furthermore, the sound model is, for example, a model that learns by using multiple character images and sound data associated with those character images (such as lines from animated characters) as learning data.

[0086] Furthermore, the sound generation unit 32b performs text-to-sound synthesis, generating sound based on the sound data of the text determined by the text information determination unit 22i (which has already been generated based on the sound data generated by the sound generation unit 32b) and the sound data of the text generated by the speech data generation unit 32d (described later). The sound generated by the text-to-sound synthesis by the sound generation unit 32b is output as a sound file.

[0087] The content generation unit 32c generates content (comics, animations, etc.) consisting of one or more frames edited by a user through various interfaces displayed on the display unit (output unit 24) of the user terminal 2. For example, the content generation unit 32c responds to... Figure 5 The play button 52c in the frame overview area 52 of the content creation screen 50 is selected to generate content data. Here, the content data is the data of one or more frames edited by the user packaged into a single file, such as MP4, AVI, or other animated image files; HEIF (High Efficiency Image File Format) or AVIF (AV1 Image File Format), which can store multiple images; and, if an audio file is generated by the audio generation unit 32b, the audio file is included. The generated content data is stored in the content DB33e of the storage unit 33 (described later) and transmitted to the user terminal 2 via the network 4 through the communication unit 31.

[0088] The speech data generation unit 32d generates speech data of a character that responds to the user's speech data (voice data or text data) input via the input unit 23 of the user terminal 2. This enables the character generated by the character image generation unit 32a to function as a chatbot. Specifically, the speech data generation unit 32d converts the user's speech data input via the input unit 23 of the user terminal 2 into text data and inputs it into a language model (large-scale language model), outputting the speech data of the character that responds to the user's speech, thereby generating speech data. Here, the language model can also be a model learned by using the character's image data and / or voice data as part of the learning data. Thus, by inputting the character's image and voice data along with the user's speech data into the language model, speech data matching the character's appearance, atmosphere, and personality can be generated. The character's speech data generated by the speech data generation unit 32d is transmitted to the user terminal 2 via the network 4 through the communication unit 31, and output as sound from the output unit 24 (speaker) of the user terminal 2.

[0089] The NFT Issuance Department 32e issues NFTs (Non-Fungible Tokens) related to the data of the content generated by the Content Generation Department 32c or the image data of characters within that content (data of character images (two-dimensional or three-dimensional images) generated by the Character Image Generation Department 32a). That is, the NFT Issuance Department 32e NFTs the content generated by the Content Generation Department 32c and / or the image data of characters within that content. Specifically, for example, when the NFT Issuance Department 32e receives an instruction to NFT the data of the content or the data of the character image based on an operation of an interface (output unit 24) displayed on the user terminal 2 (not shown), it generates an NFTID associated with the data of the content or the data of the character image. Then, the NFT Issuance Department 32e generates code containing metadata including the generated NFTID and sends it to a blockchain network (not shown). Thus, it is possible to NFT the data of the content and the data of the character images within the content. Furthermore, users can buy and sell the NFT-enhanced data of the content or the data of the character images on an NFT exchange or exchange it with other NFT-enhanced data.

[0090] The storage unit 33 is, for example, a data storage device such as built-in memory or external memory (SD memory card, etc.). The storage unit 33 stores various data processed by the control unit 32, various information received by the communication unit 31 from the user terminal 2 via the network 4, and various databases (DBs). These databases include character images DB33a, background images DB33b, speech bubble images DB33c, and content DB33d.

[0091] The character image DB33a stores pre-prepared image data of multiple characters (including 2D or 3D model data). Additionally, the character image DB33a stores character images generated by the character image generation unit 32a of the control unit 32. The character images stored in the character image DB33a can be associated with multiple action modes. Upon request from the user terminal 2, the character images stored in the character image DB33a are transmitted to the user terminal 2 via the network 4 through the communication unit 31, and displayed on the display unit (output unit 24) through the display control unit 22a of the user terminal 2.

[0092] Background image DB33b stores image data of multiple pre-prepared background images. The background images stored in background image DB33b are sent to user terminal 2 via network 4 through communication unit 31 according to the request of user terminal 2, and are displayed on display unit (output unit 24) through display control unit 22a of user terminal 2.

[0093] The speech bubble image DB33c stores image data of multiple pre-prepared speech bubble images. The speech bubble images stored in the speech bubble image DB33c are sent to the user terminal 2 via the communication unit 31 and the network 4 according to the request of the user terminal 2, and are displayed on the display unit (output unit 24) by the display control unit 22a of the user terminal 2.

[0094] The content DB33d stores the data of the content generated by the content generation unit 32c. The data of the content stored in the content DB33d is sent to the user terminal 2 via the communication unit 31 and the network 4 according to the request of the user terminal 2, and is displayed on the display unit (output unit 24) by the display control unit 22a of the user terminal 2.

[0095] Furthermore, the storage unit 33 does not necessarily need to be located within the server 3; part or all of the storage unit 33 may also be located within other devices that can communicate with the server 3 via the network 4.

[0096] (An example of an action) Next, an example of the actions of content generation system 1 will be described. Figure 4 This is a flowchart illustrating the action of a user of the content generation system 1 using the user interface displayed on the display unit (output unit 24) of the user terminal 2 to generate content consisting of one or more frames.

[0097] First, on the user terminal 2, the display control unit 22a displays the user interface on the display unit (output unit 24) (step S1).

[0098] Next, on the user terminal 2, based on the user operation on the user interface displayed on the display unit (output unit 24), the frame determination unit 22b determines the frame to be edited (step S2).

[0099] Next, on the user terminal 2, the role is determined by the role determination unit 22c based on the user operation on the user interface displayed on the display unit (output unit 24) (step S3).

[0100] Next, on the user terminal 2, the expression determination unit 22d determines the character's expression based on the user operation on the user interface (expression setting unit 55) displayed on the display unit (output unit 24) (step S4).

[0101] Next, on the user terminal 2, the gaze determination unit 22e determines the character's gaze based on the user operation on the user interface (gaze setting unit 56) displayed on the display unit (output unit 24) (step S5).

[0102] Next, on the user terminal 2, the action determination unit 22f determines the character's action based on the user operation on the user interface (action setting unit 57) displayed on the display unit (output unit 24) (step S6).

[0103] Next, on the user terminal 2, based on the user operation on the user interface (role configuration setting unit 58) displayed on the display unit (output unit 24), the role configuration determination unit 22g determines the role configuration (step S7).

[0104] Next, on the user terminal 2, the background image is determined by the background image determination unit 22h based on the user operation on the user interface (background image setting unit 59) displayed on the display unit (output unit 24) (step S8).

[0105] Next, on the user terminal 2, the text information is determined by the text information determination unit 22i based on the user operation on the user interface (text information setting unit 60) displayed on the display unit (output unit 24) (step S9).

[0106] Next, on server 3, the content generation unit 32c generates content containing one or more frames (step S10).

[0107] As explained above, according to this embodiment, the content generation system 1 generates content consisting of one or more frames containing dynamic or static images of characters. The content generation system 1 has a display control unit 22a that displays a user interface (expression setting unit 55) containing a two-dimensional operation area 55a and a preview image (preview display area 53) of a frame edited by the user on a display unit (output unit 24), an expression determination unit 22d that determines the expression of the character in the preview image based on the position of the pointer 55b on the two-dimensional operation area 55a, and a content generation unit 32c that generates content consisting of one or more frames containing frames edited by the user. Therefore, it is possible to easily generate content consisting of frames of characters containing expressions desired by the user.

[0108] Furthermore, in this embodiment, as an example, an example of setting expressions corresponding to the four emotions of "joy," "anger," "sorrow," and "happiness" through the expression setting unit 55 has been described. However, it is also possible to set expressions corresponding to more than four emotions. For example, expressions corresponding to the eight emotions (Plutchik model) of "anger," "fear," "expectation," "surprise," "joy," "sadness," "trust," and "disgust" can also be set.

[0109] In addition, in this embodiment, as an example, an example of setting the character's actions using the drop-down menu 571 has been described. However, it is also possible to pre-associate actions corresponding to expressions that can be set by the expression setting unit 55, and automatically set actions corresponding to expressions set by the expression setting unit 55.

[0110] Furthermore, in this embodiment, as an example, an example was described where an expression setting unit 55 and an action setting unit 57 were set separately, and the "expression" and "action" were set through each setting unit. However, the "expression" and "action" can also be set through a single setting unit. For example, an emotion setting unit with a two-dimensional operation area capable of setting multiple emotions can be set as a user interface, and the "expression" and "action" of the character corresponding to the emotion set by the emotion setting unit can be pre-associated, and the "expression" and "action" corresponding to the emotion set by the emotion setting unit can be set automatically. Here, the types of emotions set by the emotion setting unit can be, for example, the four emotions of "joy," "anger," "sorrow," and "happiness," or the eight emotions of "anger," "fear," "expectation," "surprise," "joy," "sadness," "trust," and "disgust" (Pluchk model), or even more types of emotions.

[0111] Furthermore, in this embodiment, an example of a rectangular two-dimensional operation area 55a being described for the expression setting unit 55 has been provided, but the shape of the two-dimensional operation area is not limited to this. For example, as Figure 17 As shown, the expression setting unit 55 may also have a circular two-dimensional operation area 55c. By moving the dot-shaped pointer 55d within the circular two-dimensional operation area 55c, various expressions such as joy, anger, sorrow, and happiness can be set. Alternatively, the shape of the two-dimensional operation area may be set to a shape other than a rectangle or a circle. By moving the pointer within the shape, various expressions can be set.

[0112] Furthermore, in this embodiment, an example of a rectangular two-dimensional operation area 56a being described for the gaze setting unit 56 has been provided, but the shape of the two-dimensional operation area is not limited to this. For example, as Figure 18 As shown, the line-of-sight setting unit 56 may have a circular two-dimensional operation area 56c. By moving a dot-shaped pointer 56d within the circular two-dimensional operation area 56c, the line of sight in each direction can be set. Alternatively, the shape of the two-dimensional operation area may be set to a shape other than a rectangle or a circle. By moving the pointer within the shape, the line of sight in each direction can be set.

[0113] Any part or all of the functional units described in this specification can also be implemented by a program. The programs mentioned in this specification can be distributed by recording non-temporarily on a computer-readable recording medium, or by communication lines such as the Internet (including wireless communication), or by being installed on any terminal.

[0114] Based on the foregoing description, those skilled in the art may conceive of additional effects and various modifications of the present invention, but the invention is not limited to the embodiments described above. Various additions, modifications, and partial deletions can be made without departing from the scope of the patent claims and the conceptual ideas and spirit of the invention derived from their equivalents.

[0115] For example, a device described as a single apparatus (or component, hereinafter the same) in this specification (including an apparatus depicted as a single apparatus in the accompanying drawings) may also be implemented by multiple apparatuses. Conversely, an apparatus described as multiple apparatuses in this specification (including an apparatus depicted as multiple apparatuses in the accompanying drawings) may also be implemented by a single apparatus. Alternatively, a part or all of a unit or function included in one apparatus (e.g., a server) may be included in another apparatus (e.g., a user terminal).

[0116] Furthermore, not all matters described in this specification are necessary conditions. In particular, matters described in this specification but not in the scope of the patent claims may be considered as arbitrary additional matters.

[0117] Furthermore, it should be noted that the applicant is only aware of the publicly known inventions described in the "Prior Art Documents" section of this specification, and the present invention is not necessarily intended to solve the problems of those publicly known inventions. The problem to be solved by the present invention should be determined in consideration of the entire specification. For example, in this specification, where there is a description that aims to achieve a given effect through a specific structure, a problem contrary to that given effect can also be solved. However, this does not mean that such a specific structure is necessarily a necessary condition.

Claims

1. A content generation system for generating content consisting of one or more frames containing a moving image or a static image of a character, the content generation system comprising: The display control unit displays the user interface containing the first two-dimensional operation area and a preview image of the frame edited by the user on the display unit; An expression determination unit determines the expression of the character in the preview image based on the position of a pointer on the first two-dimensional operation area; and The content generation unit generates content consisting of one or more frames, including the frame edited by the user.

2. The content generation system according to claim 1, wherein, The user interface also includes a gaze determination unit, which includes a second two-dimensional operation area, and determines the gaze of the character in the preview image based on the position of the pointer on the second two-dimensional operation area.

3. The content generation system according to claim 1, wherein, The content generation system also includes an action determination unit, which determines the action of the character on the preview image based on the user's operation of the user interface.

4. The content generation system according to claim 3, wherein, The user interface includes a one-dimensional operation area. The motion determination unit determines the playback range of the motion based on the position of the slider on the one-dimensional operation area.

5. The content generation system according to claim 1, wherein, The content generation system also includes an action determination unit, which determines the action of the character in the preview image based on the position of the pointer on the first two-dimensional operation area.

6. The content generation system according to claim 1, wherein, The content generation system also includes a character determination unit, which determines the characters to be displayed on the preview image.

7. The content generation system according to claim 1, wherein, The content generation system also includes a background image determination unit, which determines the background image of the preview image.

8. The content generation system according to claim 1, wherein, The content generation system also includes a text information determination unit, which determines at least one of the following based on the user interface operation performed by the user: the text to be displayed on the preview image, the type of speech bubble displaying the text, the color of the text, and the size of the text.

9. The content generation system according to claim 1, wherein, The first two-dimensional operation area is a user interface used to set the character's emotions.

10. The content generation system according to claim 1, wherein, The content generation system also includes a sound generation unit, which generates sound data for a character by inputting the image data of the character selected by the user into a sound model. The sound model is a model that uses the character image and the sound data associated with the character image as learning data.

11. The content generation system according to claim 1, wherein, The content generation system also includes a speech data generation unit, which generates speech data of a character that responds to the user's speech data by inputting the user's speech data into a language model. The language model is a model that has been learned by using the character's image data and / or voice data as part of the learning data.

12. The content generation system according to claim 1, wherein, The content generation system also includes a character image generation unit, which generates character images by inputting image data from the user into a learned image processing model.

13. The content generation system according to claim 12, wherein, The content generation system also includes a background image generation unit, which generates a background image by inputting the image data input by the user into a learned image processing model.

14. The content generation system according to claim 13, wherein, The image data generated by the character image generation unit and the background image generation unit are three-dimensional image data and are configured in the metaverse space.

15. The content generation system according to claim 12, wherein, The content generation system also includes an NFT issuance department, which issues NFTs related to the content generated by the content generation department or the image data of characters within the content.

16. A method for generating content consisting of one or more frames containing moving or still images of characters, comprising: The display control steps involve displaying a user interface containing the first two-dimensional operation area and a preview image of a frame edited by the user on the display unit; The expression determination step determines the expression of the character in the preview image based on the position of the pointer on the first two-dimensional operation area; as well as The content generation step generates content consisting of one or more frames, including the frame edited by the user.

17. A program for causing a computer to perform a method of generating content consisting of one or more frames containing moving or still images of characters, the method comprising: The display control steps involve displaying a user interface containing the first two-dimensional operation area and a preview image of a frame edited by the user on the display unit; The expression determination step determines the expression of the character in the preview image based on the position of the pointer on the first two-dimensional operation area; as well as The content generation step generates content consisting of one or more frames, including the frame edited by the user.

18. A storage medium that stores the program of claim 17 in a computer-readable format.

19. A content generation system for generating content consisting of one or more frames containing a moving image or a still image of a character, the content generation system comprising: The display control unit displays the user interface and a preview image of the frame edited by the user on the display unit; The character determination unit determines the character to be displayed on the preview image based on the user interface operations performed by the user. An expression determination unit determines the character's expression based on the user interface operations performed by the user; A gaze determination unit determines the character's gaze based on the user interface operations performed by the user. An action determination unit determines the character's actions based on the user interface operations performed by the user. The background determination unit determines the background image of the preview image based on the user interface operations performed by the user. The text information determination unit determines at least one of the following based on the user interface operation performed by the user: the text to be displayed on the preview image, the type of speech bubble displaying the text, the color of the text, and the size of the text. as well as The content generation unit generates content consisting of one or more frames, including the frame edited by the user.

Images