Information processing systems, information processing methods, and programs
The information processing system addresses the limitation of avatar generation by using a machine-learned model to process user input, enabling detailed modifications of avatar appearances and accessories, thereby increasing the freedom and quality of avatar creation.
Patent Information
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- ANOTHERBALL PTE LTD
- Filing Date
- 2024-11-12
- Publication Date
- 2026-05-22
Smart Images
Figure 2026084728000001_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to an information processing system, an information processing method, and a program.
Background Art
[0002] Patent Documents 1 and 2 disclose an information processing system for generating an avatar.
Prior Art Documents
Patent Documents
[0003]
Patent Document 1
Patent Document 2
Summary of the Invention
Problems to be Solved by the Invention
[0004] Although the above-mentioned documents disclose a technique for generating items for an avatar, there is room for improvement in the degree of freedom of avatar generation.
Means for Solving the Problems
[0005] According to one aspect of the present invention, there is provided an information processing system including a processor configured to execute each of the following steps by reading a program. In an input information acquisition step, first input information regarding the appearance of an avatar and second input information indicating the content of changes to a predetermined range of the avatar are acquired. In a first image generation step, image generation information including the first input information and the second input information is input to a first image generation model, and the first image generation model outputs a first image in which a predetermined range reflecting the content of the change is drawn. Here, the first image generation model is a learning model that is machine-learned to be able to input the first input information and the second input information and output the first image.
[0006] This configuration allows for the generation of images with modified areas based on the user's requests, relative to the base avatar. Therefore, it increases the degree of freedom in avatar creation. [Brief explanation of the drawing]
[0007] [Figure 1] This is a diagram showing the configuration of Information Processing System 1. [Figure 2] This is a block diagram showing the hardware configuration of the information processing device 10. [Figure 3] This is a block diagram showing the hardware configuration of user terminal 20. [Figure 4] This is a block diagram showing the functions realized by the information processing device 10 (control unit 11) and the user terminal 20 (control unit 21). [Figure 5] This figure shows an example of an avatar image entered as the first input information. [Figure 6] This figure shows an example of the first image with decorative items attached. [Figure 7] This figure shows an example of a second image generated based on the avatar image in Figure 5. [Figure 8] This is an activity diagram showing an example of the flow of information processing (avatar image creation process) performed by information processing system 1. [Modes for carrying out the invention]
[0008] Embodiments of the present invention will be described below with reference to the drawings. The various features shown in the embodiments below can be combined with each other.
[0009] Incidentally, the program for implementing the software appearing in one embodiment may be provided as a non-transitory computer-readable medium, or it may be provided as a downloadable medium from an external server, or it may be provided so that the program is launched on an external computer and its functions are realized on a client terminal (so-called cloud computing).
[0010] Furthermore, in various information processing according to one embodiment, an input and an output corresponding to the input can be realized. Here, as long as an output is obtained as a result of the input, the form of the information referenced in such information processing (hereinafter referred to as "reference information") is not limited. The reference information may be, for example, rule-based information such as a database, a lookup table, or a predetermined function (including a decision formula such as a regression equation constructed by a statistical method), or a pre-trained model that has learned the correlation between input and output in advance, or a large-scale language model that can output a desired result by inputting a prompt.
[0011] Furthermore, in one embodiment, "part" may include, for example, hardware resources implemented by a circuit in a broad sense, and the information processing of software that can be specifically realized by these hardware resources. Also, in one embodiment, various types of information are handled, and this information can be represented, for example, by the physical values of signal values representing voltage and current, the high or low values of signal values as a set of binary bits composed of 0s or 1s, or by quantum superposition (so-called qubits), and communication and calculations can be performed on a circuit in a broad sense.
[0012] Furthermore, a circuit in a broad sense is a circuit realized by combining at least a suitable combination of circuits, circuits, processors, and memory. The processor may be a general-purpose processor or a dedicated circuit. In other words, it includes application-specific integrated circuits (ASICs), programmable logic devices (for example, simple programmable logic devices (SPLDs), complex programmable logic devices (CPLDs), and field programmable gate arrays (FPGAs)), etc.
[0013] 1. Hardware Configuration This section describes the hardware configuration.
[0014] <Information Processing System 1> Figure 1 is a configuration diagram representing information processing system 1. Information processing system 1 comprises a communication line 2, an information processing device 10, and a plurality of user terminals 20. The information processing device 10 and the user terminals 20 are configured to communicate with each other via the communication line 2. The connection between the information processing device 10 and the user terminals 20 may be wired or wireless.
[0015] In one embodiment of the information processing system 1, the information processing system 1 consists of one or more devices or components. These components will be described below. For example, the information processing system 1 may consist only of an information processing device 10, or only of a user terminal 20.
[0016] The information processing system 1 provides a service (hereinafter referred to as the "avatar creation service") for creating and registering an avatar used by a user in a virtual space. An "avatar" is a character image (virtual character) mainly used as the user's alter ego in network communication. The avatar is displayed on the user terminal 20 or the like, together with, for example, a virtual space on the network, a background, etc., and operates in accordance with the actions, speech, facial expression changes, etc. of the distributor. When the avatar is a 2D character, the avatar data constituting the avatar includes a set of a plurality of images for expressing actions. The avatar data also includes numerical values for specifying parameters such as the positions of feature points and the amount of movement (change amount).
[0017] In addition to the avatar creation service, the information processing system 1 may provide a video distribution service using the created and registered avatar. The "distribution service" is a service that provides a platform for a user to distribute a video or view a video distributed by another user using an avatar.
[0018] <Information processing apparatus 10> FIG. 2 is a block diagram showing the hardware configuration of the information processing apparatus 10. As shown in FIG. 2, the information processing apparatus 10 includes a control unit 11, a storage unit 12, a communication unit 13, and a communication bus 14. The control unit 11, the storage unit 12, and the communication unit 13 are electrically connected to each other inside the information processing apparatus 10 via the communication bus 14.
[0019] <Control unit 11> The control unit 11 performs processing and control of the overall operation related to the information processing device 10. The control unit 11 is, for example, a Central Processing Unit (CPU). The control unit 11 realizes various functions related to the information processing device 10 by reading predetermined programs stored in the memory unit 12. That is, information processing by software stored in the memory unit 12 can be concretely realized by the control unit 11, which is an example of hardware, and can be executed as each functional unit included in the control unit 11. These will be described in more detail in the next section. Note that the control unit 11 is not limited to being a single unit, and the information processing device 10 may have multiple control units 11 for each function. Furthermore, the information processing device 10 may have a configuration that combines these.
[0020] <Storage section 12> The storage unit 12 stores various types of information as defined above. This can be done, for example, as a storage device such as a solid-state drive (SSD) that stores various programs related to the information processing device 10 executed by the control unit 11, or as a memory such as random access memory (RAM) that stores temporarily necessary information (arguments, arrays, etc.) related to program calculations. The storage unit 12 stores various programs, variables, etc. related to the information processing device 10 executed by the control unit 11.
[0021] <Communications Department 13> The communication unit 13 preferably uses wired communication methods such as USB, IEEE1394, Thunderbolt®, and wired LAN network communication, but may also include wireless LAN network communication, mobile communication such as LTE / 5G, and Bluetooth® communication as needed. In other words, it is more preferable to implement it as a collection of these multiple communication methods. That is, the information processing device 10 may communicate various information from the outside via the communication unit 13 and the network.
[0022] The information processing device 10 may be on-premise or in a cloud-based configuration. In the case of a cloud-based information processing device 10, for example, the above-mentioned functions and processing may be provided in the form of SaaS (Software as a Service) or cloud computing.
[0023] <User terminal 20> The user terminal 20 is an information processing terminal used by a user who utilizes the information processing functions provided by the information processing device 10. Figure 3 is a block diagram showing the hardware configuration of the user terminal 20. As shown in Figure 3, the user terminal 20 comprises a control unit 21, a storage unit 22, a communication unit 23, an input unit 24, an output unit 25, and a communication bus 26. The control unit 21, storage unit 22, communication unit 23, input unit 24, and output unit 25 are electrically connected within the user terminal 20 via the communication bus 26. The descriptions of the control unit 21, storage unit 22, and communication unit 23 are the same as the descriptions of each part of the information processing device 10 and are therefore omitted.
[0024] <Input section 24> The input unit 24 receives action input from the user. The action input is transmitted as a command signal to the control unit 21 via the communication bus 26. The control unit 21 can perform predetermined controls or calculations based on the transmitted command signal as needed. The input unit 24 may be included in the casing of the user terminal 20 or it may be external. For example, the input unit 24 may be implemented as a touch panel integrated with the output unit 25. When the input unit 24 is implemented as a touch panel, the user can input tap actions, swipe actions, etc. to the input unit 24. Instead of a touch panel, the input unit 24 can be a switch button, mouse, trackpad, QWERTY keyboard, etc.
[0025] <Output section 25> The output unit 25 displays a graphical user interface (GUI) screen that allows the user to take action. The output unit 25 may be included in the casing of the user terminal 20 or it may be an external component. Specifically, the output unit 25 can be implemented as a display device such as a CRT display, liquid crystal display, organic EL display, or plasma display. It is preferable that these display devices be used in accordance with the type of user terminal 20.
[0026] 2. Functional Configuration This section describes the functional configuration of this embodiment. Information processing by software stored in the memory unit 12 is specifically realized by the control unit 11, which is an example of hardware, and can be executed as each functional unit included in the control unit 11 (the processor provided by the information processing system 1).
[0027] Figure 4 is a block diagram showing the functions realized by the information processing device 10 (control unit 11) and the user terminal 20 (control unit 21).
[0028] As shown in Figure 4A, the information processing device 10 (control unit 11) includes a reception unit 111, an input information acquisition unit 112, a first image generation unit 113, a second image generation unit 114, a setting unit 115, and an artificial intelligence unit 120. As shown in Figure 4B, the user terminal 20 (control unit 21) includes a display unit 211 and an operation acquisition unit 212.
[0029] <Reception Desk 111> The reception unit 111 is configured to receive input information for creating an avatar desired by the user. Specifically, the reception unit 111 receives input of first input information and second input information from the user terminal 20.
[0030] The first input information is information about the appearance of the avatar. The first input information includes, for example, user-perceivable information such as images and text, as well as metadata associated with the image data. The first input information may include only one of the following: images, text, and metadata, or any combination thereof.
[0031] If an image is input as the first input information, for example, the reception unit 111 accepts the upload of an image from the user terminal 20 or the specification of an address indicating the storage location of the image on the network. The image input as the first input information may include, for example, an avatar image that includes the appearance of the base avatar.
[0032] If text is entered as the first input information, for example, the reception unit 111 receives the text input from the user terminal 20. The text entered as the first input information includes sentences or keywords that describe the appearance of the base avatar. The information described in the text includes, for example, the type of character (person, animal, robot, etc.), build, hairstyle, clothing, skin color, etc.
[0033] Metadata included in the first input information may include, for example, the properties of the image data. These image data properties may include, for example, the image size, quality, and generation conditions.
[0034] The "avatar image" included in the first input information is an image depicting the entire avatar or a part of the avatar (e.g., the upper body). Furthermore, "entire avatar" is not necessarily limited to "the entire body of the avatar"; if the user is using an avatar that only shows the upper body, "entire avatar" may mean "the upper body of the avatar." The avatar image may be 2D image data or 3D image data, but is typically 2D image data. Figure 5 shows an example of an avatar image input as the first input information.
[0035] "Generation conditions" are a type of property of an image generated by an image generation model (e.g., an image generation AI), and include information such as the direction of lighting, brushstrokes, style (e.g., line drawing, oil painting, etc.), and composition in the image. Typically, generation conditions include variables used by the image generation AI during image generation. These variables, also called generation IDs or seed values, are codes used by the image generation AI to reproduce images with the same impression (style).
[0036] The image generation model that generates the image included in the first input information may be a different model from the first image generation model that generates the first image and the second image generation model that generates the second image, as described later, or it may be the same model as at least one of them. However, from the viewpoint of improving the reproducibility of the art style depending on the generation conditions, they may all be the same model.
[0037] The first input information should ideally include an avatar image with the avatar drawn on it, and image generation variables indicating the conditions for generating the avatar image. With this configuration, the first image can be generated using the generation conditions of the base avatar image, thus improving the consistency between the base avatar image and the generated first image.
[0038] The second input information is information indicating the changes to a predetermined range of the avatar specified in the first input information. In other words, the reception unit 111 receives the changes from the user as the second input information. The "predetermined range" may be the entire avatar. Alternatively, the "predetermined range" may be a predetermined part (a part of the body) of the avatar. In other words, the second input information may indicate changes to a predetermined part of the avatar. With this configuration, it is possible to generate an image in which only the predetermined part of the avatar has been changed according to the user's request.
[0039] The "specified area" includes, for example, body parts (e.g., head, neck, arms, chest, waist, legs, etc.), ranges encompassing multiple parts (e.g., upper body, right half, left half, lower body, etc.), and constituent elements of parts (e.g., hair, eyes, eyebrows, nose, mouth, ears, fingers, nails, etc.). The specified area may also be specified as a range, for example, "from the right shoulder down."
[0040] "Changes" include modifications to the appearance of a specified body part (e.g., length, thickness, color, etc.). Specifically, examples of appearance-related changes include "shortening the hair" and "lengthening the legs."
[0041] Furthermore, the "changes" may include changes to accessories on a predetermined part (for example, attaching, removing, or replacing accessories). In other words, the second input information may include information indicating accessories to be attached to a predetermined part within a predetermined range. With such a configuration, it becomes possible to generate an image of an avatar with arbitrary accessories attached. Specifically, examples of changes related to accessories include "a large bracelet on the wrist."
[0042] Here, "accessories" refers to items worn by the avatar, such as clothing, hats, shoes, gloves, accessories (e.g., earrings, necklaces, bracelets, hair accessories, etc.), weapons (e.g., staffs, swords, guns, etc.), armor (e.g., shields, helmets, etc.), and other tools.
[0043] The reception unit 111 may present the user with a selection of decorative items available to them (e.g., images, names, etc.) and accept the user's selection of decorative items. The decorative items available to the user may be determined, for example, by the user's purchase history of items (decorative items) in the services provided by the information processing system 1, the user's attributes (rank, amount charged, etc.), etc.
[0044] The second input information may further include information regarding at least one of the following: the position where the ornament is placed, and the appearance of the ornament. With such a configuration, it becomes possible to adjust the position or appearance of the ornament to suit the user's preference. The "appearance of the ornament" may include the size, color, orientation, and presence or absence of accessories (decorative items) of the ornament.
[0045] The second input information may further include information indicating the attributes of the accessory. With this configuration, the image generation model described later can be switched according to the attributes of the accessory, thereby improving the quality of the first image in which the accessory is attached. Here, "attributes of the accessory" include the category of the accessory (e.g., clothing, hat, shoes, gloves, accessories, weapons, armor, other tools, etc.), the attachment position of the accessory (the part on the avatar to which the accessory is attached), the era of the accessory, the worldview of the accessory, etc.
[0046] The second input information includes metadata in addition to user-perceivable information such as images, audio, and text. The second input information may include information in multiple formats from images, audio, and text.
[0047] If an image is input as the second input information, for example, the reception unit 111 accepts the upload of an image from the user terminal 20 or the specification of an address indicating the storage location of the image on the network. The image input as the second input information may include, for example, an image of an accessory as information indicating an accessory to be attached to the avatar. With this configuration, it is possible to generate a first image with the accessory attached (an image reflecting the changes, as described later) using an existing image of the accessory. Note that the image of the accessory may be an image of the accessory alone, or an image of the accessory attached to the avatar (or a part of the avatar).
[0048] Furthermore, the image input as the second input information may include an existing image that depicts the predetermined range to be modified (i.e., an image corresponding to the first image generated in the past). With this configuration, a new first image can be generated by referring to the first image generated in the past, making it easier to obtain a first image that is closer to the user's image. The existing first image used as the second input information can be selected, for example, from first images registered in an image database.
[0049] If text is entered as the second input information, for example, the reception unit 111 receives the text input from the user terminal 20. The text entered as the second input information includes sentences or keywords that describe the changes to the avatar. The information described in the text includes, for example, the predetermined range to be changed (the entire avatar or parts), the content of the changes (for example, changes to appearance, addition of accessories, etc.).
[0050] Metadata included in the second input information may include, for example, the properties of the image data. These properties may include, for example, the image size, quality, and generation conditions. The image generation conditions in the second input information are the same as those in the first input information.
[0051] The second input information should preferably include an image representing the changes (for example, an image of an ornament, an existing first image, etc.) and an image generation variable indicating the conditions for generating that image. With this configuration, the first image can be generated using the image generation conditions entered as the changes, making it easier to obtain a first image that closely matches the user's image.
[0052] The reception unit 111 may accept at least a portion of the second input information through user operation via an image. For example, the reception unit 111 may display the avatar image included in the first input information on the user terminal 20 and accept input from the user terminal 20 to select the part of the avatar image that the user wishes to change (a predetermined range). Specifically, the reception unit 111 may accept as a predetermined range the part of the avatar (for example, an "arm") that the user has performed an enclosing operation, pressing operation, etc. on the avatar image displayed on the user terminal 20.
[0053] Alternatively, for example, the reception unit 111 may display input reception objects such as numerical input fields and sliders on the user terminal 20 as parameters representing changes to the size, color, texture, etc., of avatar parts, accessories, etc., within a predetermined range, and accept numerical input from the user terminal 20 using the input reception objects.
[0054] As an example of a combination of first and second input information, the first input information may include an avatar image containing the avatar's appearance, and the second input information may include text describing the changes. With this configuration, the first image can be generated by giving textual instructions to the avatar image, allowing a predetermined range of an existing avatar to be modified to match the user's image.
[0055] <Input Information Acquisition Unit 112> The input information acquisition unit 112 is configured to acquire first input information regarding the appearance of the avatar and second input information indicating changes to a predetermined range of the avatar, which are received by the reception unit 111.
[0056] The input information acquisition unit 112 may convert the changes entered by the user as second input information (e.g., text, numbers, etc.) into tokens for input to the first image generation model. With this configuration, the user can input changes to the avatar through various operations.
[0057] A token is a processing unit (character set) of input data in an image generation AI (such as the first image generation model described later). The input information acquisition unit 112 divides the text contained in the second input information into units such as words, characters, or subwords, and tokenizes them.
[0058] Furthermore, the input information acquisition unit 112 may convert the information regarding the appearance of the avatar entered by the user as first input information (e.g., text) into tokens for input to the first image generation model.
[0059] The input information acquisition unit 112 may acquire an existing image depicting an ornament as second input information, based on information indicating the ornament included in the second input information (for example, the name of the ornament). The existing image is stored, for example, in the image database of the storage unit 12. The existing image may be an image of an avatar with an ornament attached to a predetermined part, an image of an avatar with an ornament attached that has been cropped to a predetermined area, or an image of only the ornament. The input information acquisition unit 112 may also search for reference images of the ornament from the network based on the information indicating the ornament and acquire them as second input information.
[0060] <First image generation unit 113> The first image generation unit 113 is configured to input image generation information, including the first input information and the second input information, acquired by the input information acquisition unit 112, into the first image generation model of the artificial intelligence unit 120, and to cause the first image generation model to output a first image that draws a predetermined range reflecting the changes. The first image generation unit 113 also displays the generated first image on the user terminal 20. Note that the image generation information may include information other than the first input information and the second input information (for example, supplementary information created by the first image generation unit 113).
[0061] Here, if the predetermined range included in the second input information is the entire avatar, the first image will be an image depicting the entire avatar. On the other hand, if the predetermined range included in the second input information is a part of the avatar, the first image will be an image depicting a part of the avatar (the part designated as the predetermined range).
[0062] Furthermore, if the second input information includes information indicating an ornament, the first image generation unit 113 uses the first image generation model to generate a first image in which the ornament is attached to a predetermined part specified by the second input information.
[0063] Figure 6 shows an example of a first image with an accessory attached. Figure 6A is an example of a first image (a front view of a right hand holding a cane) generated when, for example, second input information is provided, specifying a particular body part as "right hand" and the accessory as "cane". In other words, when the change to the avatar indicated by the first input information, "give the right hand a cane," is input into the first image generation model as second input information, the first image in Figure 6A is generated.
[0064] The first image generation model is a machine learning model that takes first input information and second input information (image generation information) as inputs and outputs a first image. The first image generation model may be a dedicated model that has been machine-learned using combinations of image generation information and the first image corresponding to said image generation information as training data, or it may be a general-purpose image generation AI that has not undergone machine learning using the above training data, or it may be a combination of a dedicated model and an image generation AI (for example, a machine learning model in which the image generation AI has been fine-tuned using the above training data).
[0065] An image generation AI is a machine learning (deep learning) trained model that takes prompts containing information and / or instructions about the image to be generated as input, and outputs an image generated based on the prompts. Prompts can include images, text, metadata, etc. As a first image generation model, the image generation AI, for example, encodes the prompt (or tokens replacing the prompt) into a feature vector and generates a face image by inputting the feature vector into an image generator.
[0066] If the first image generation model is an image generation AI, the first image generation unit 113 inputs a prompt to the first image generation model that includes image generation information acquired by the input information acquisition unit 112 and an instruction to generate a first image based on the image generation information, causing the first image generation model to output a first image. With this configuration, the first image can be generated using a general-purpose image generation AI, thereby reducing the construction cost of the information processing system 1.
[0067] The prompts input to the first image generation model may include those contained in the image generation information (for example, text entered by the user), those generated each time by the first image generation unit 113 based on the image generation information, or those that are pre-set (for example, "Generate an image based on the conditions specified by the user").
[0068] If the changes entered by the user have been converted into tokens by the input information acquisition unit 112 (i.e., the second input information includes tokens), the first image generation unit 113 inputs the tokens into the first image generation model as at least part of the second input information, along with the first input information and prompts.
[0069] The first image generation model may include multiple sub-image generation models. In this case, the first image generation unit 113 may input the first input information and the second input information to the sub-image generation model selected from among the multiple sub-image generation models according to the attributes of the ornament.
[0070] A sub-image generation model is, for example, a learning model specifically designed for generating images of decorative items. Multiple sub-image generation models may be image generation AIs of different types (e.g., different learning methods, network configurations, parameters, etc.), or they may be dedicated models trained on different training data. Furthermore, multiple sub-image generation models may include a mix of image generation AIs and dedicated models.
[0071] The first image generation unit 113 selects a sub-image generation model based on reference information (e.g., a table, function, or decision expression) that describes the correspondence between the attributes of the ornament and the sub-image generation model. For example, the first image generation unit 113 may select a sub-image generation model based on the name of the ornament.
[0072] The first image generation unit 113 may generate multiple derived images as the first image, each depicting a predetermined range at a different angle or orientation. In other words, the first image may include multiple derived images. With such a configuration, a dynamic avatar can be generated using the generated first image.
[0073] In 2D character avatars, one image suitable for the avatar's movement is selected from multiple derived images and displayed as the avatar (or part of the avatar). The parameters of the derived images (number of derived images, angle or posture of each derived image, etc.) are determined based on settings related to the avatar's movement.
[0074] The first image generation unit 113 may, for example, cause the image generation AI, which acts as the first image generation model, to generate multiple derived images simultaneously by prompting it to generate multiple derived images with different angles or orientations. Alternatively, the first image generation unit 113 may repeatedly cause the image generation AI to generate derived images one by one, for example, by changing the angle or orientation included in the prompts input to the image generation AI, which acts as the first image generation model. Furthermore, the first image generation unit 113 may generate multiple derived images using a dedicated model that has been machine-trained to generate multiple derived images based on image generation information as input.
[0075] The image in Figure 6B is, for example, a derived image of a predetermined range (right hand holding a cane) from the left side (90 degrees counterclockwise) of the image in Figure 6A. The image in Figure 6C is, for example, a derived image of a predetermined range of the image in Figure 6A viewed from the right side (270 degrees counterclockwise). The derived image in Figure 6D is, for example, a derived image of a predetermined range of the image in Figure 6A viewed from the rear side (180 degrees counterclockwise). The derived image in Figure 6E is, for example, a derived image of a predetermined range of the image in Figure 6A with a changed hand position.
[0076] The first image generation unit 113 may register the first image in the image database upon receiving a save instruction from the user for the first image. The first image is registered in the image database in association with a predetermined range.
[0077] If multiple derived images are generated as the first image, these derived images may be registered in the image database in a hierarchical structure. For example, the multiple derived images may be registered in a hierarchical structure consisting of a representative image and images related to this representative image.
[0078] The first image generation unit 113 may, upon receiving a user's instruction to save the first image, replace the first image with an existing image that has drawn a predetermined area and is registered in the image database, and then register it. With this configuration, only the first image that the user likes can be kept in the image database, thus reducing storage capacity. For example, if a first image (existing image) with "arm" as the predetermined area is already registered in an image database linked to a user's account, when the user instructs to save a new first image with "arm" as the predetermined area, the registered first image will be deleted and the new first image will be registered.
[0079] Furthermore, if multiple registration slots for the first image are provided for a predetermined range for each user account, the first image generation unit 113 may accept a designation from the user for the registration slot to replace the first image. A registration slot is a storage area for registering images such as the first image, and a predetermined number of slots are allocated to each user account.
[0080] Furthermore, in the image database, for a predetermined range where the first image is not registered (the first image has not been generated), an image obtained by cropping a portion of the avatar image input as the first input information may be provisionally registered (as an initial value).
[0081] The first image generation unit 113 may, for example, generate a single first image as a representative image, present the representative image to the user, and accept a save instruction from the user for the representative image. Furthermore, upon receiving a save instruction for the representative image, the first image generation unit 113 may generate multiple derived images of the representative image. The representative image is, for example, an image of a predetermined range viewed from the front. As a result, derived images are generated only when the user saves the first image, thus reducing the processing load on the information processing system 1.
[0082] The first image generation unit 113 may receive a request from the user to regenerate the first image after generating the first image. When the first image generation unit 113 receives a regeneration request from the user, the reception unit 111 accepts the re-input of the first and second input information (partial modification of input content, replacement of image, etc.). The acquisition of image generation information by the input information acquisition unit 112 and the generation of the first image by the first image generation unit 113 are then repeated.
[0083] Note that the first image does not necessarily have to be registered in the image database. Alternatively, instead of the first image generated by the first image generation unit 113, an image within a predetermined range (including derived images) created based on the second image generated by the second image generation unit 114 (described later) (for example, extracted from the second image) may be registered in the image database.
[0084] <Second image generation unit 114> The second image generation unit 114 is configured to generate a second image in which an avatar is drawn that reflects the changes included in the second input information, within a predetermined range included in the second input information. With this configuration, an avatar that can be used immediately by the user can be generated. The second image generation unit 114 also displays the generated second image on the user terminal 20. The second image generation unit 114 generates the second image using, for example, the following first generation pattern or second generation pattern.
[0085] In the first generation pattern, the second image generation unit 114 inputs the first input information (information regarding the appearance of the avatar) and the first image generated by the first image generation unit 113 to the second image generation model, causing the second image generation model to output the second image. In particular, the second image generation unit 114 may input the avatar image included in the first input information and the first image generated by the first image generation unit 113 to the second image generation model, causing the second image generation model to output the second image. With such a configuration, the first image and the original avatar image are integrated to generate a new avatar, thereby improving the quality of the resulting avatar.
[0086] The second image generation unit 114 may input multiple first images to the second image generation model. This makes it possible to generate an avatar image in which multiple parts (predetermined ranges) of the original avatar image have been modified.
[0087] The second image generation model for the first generation pattern is a machine learning model that takes the first input information (avatar image) and the first image as input and outputs the second image. The second image generation model may be a dedicated model that has been machine learning-trained using the combination of the first input information and the first image and the second image corresponding to these combinations as training data, or it may be a general-purpose image generation AI similar to the first image generation model, or it may be a combination of a dedicated model and an image generation AI (for example, a machine learning model in which the image generation AI has been fine-tuned using the above training data).
[0088] When using an image generation AI as the second image generation model, the image generation AI used as the first image generation model and the image generation AI used as the second image generation model may be the same or they may be of different types.
[0089] If the second image generation model of the first generation pattern is an image generation AI, the second image generation unit 114 inputs a prompt to the second image generation model that includes the first input information (avatar image) acquired by the input information acquisition unit 112, the first image generated by the first image generation unit 113, and an instruction to generate a second image based on the first input information and the first image (for example, an instruction to generate an image by combining the first image with the avatar image), causing the second image generation model to output the second image. With this configuration, the second image can be generated using a general-purpose image generation AI, thus reducing the construction cost of the information processing system 1.
[0090] In the second generation pattern, the second image generation unit 114 inputs image generation information, including the first and second input information, to the second image generation model of the artificial intelligence unit 120, causing the second image generation model to output a second image. Note that the image generation information input to the second image generation model does not necessarily have to match the image generation information input to the first image generation model.
[0091] The second image generation model for the second generation pattern is a machine learning model that takes image generation information (first input information and second input information) as input and outputs a second image. The second image generation model may be a dedicated model that has been machine learning-trained using the combination of the first input information, the second input information, and the second image corresponding to these combinations as training data, or it may be a general-purpose image generation AI similar to the first image generation model, or it may be a combination of a dedicated model and an image generation AI (for example, a machine learning model in which the image generation AI has been fine-tuned using the above training data).
[0092] If the second image generation model of the second generation pattern is an image generation AI, the second image generation unit 114 inputs a prompt to the second image generation model that includes the first input information and the second input information acquired by the input information acquisition unit 112, and an instruction to generate a second image based on the first input information and the second input information, causing the second image generation model to output a second image.
[0093] Figure 7 shows an example of a second image generated based on the avatar image in Figure 5. The second image in Figure 7 is an image of the avatar in which a change (second input information), such as "holding a cane in the right hand," has been made to the avatar image in Figure 5 (first input information).
[0094] In both the first and second generation patterns, the second image generation unit 114 may generate multiple derived avatar images as the second image, each depicting the avatar at a different angle or pose. The procedure for generating multiple derived avatar images can be the same as the procedure for generating multiple derived images in the first image generation unit 113.
[0095] The second image generation unit 114 may generate the second image upon receiving a save instruction from the user for the first image. With this configuration, a modified avatar image is generated only if the user likes the first image, thus reducing the processing load on the information processing system 1.
[0096] The second image generation unit 114 may register the second image in the image database upon receiving a save instruction from the user for the second image. The second image may be registered in the image database, for example, linked to the first image.
[0097] The second image generation unit 114 may accept a user's save instruction for the second image, along with the designation of a registration slot for registering the avatar image. If an avatar image is already registered in the designated registration slot, the registered avatar image is deleted, and the second image is registered as the new avatar image.
[0098] The second image registered in the image database is used, for example, together with the first image, as avatar data to construct the avatar when the user uses the avatar (when the avatar is displayed on the user terminal 20).
[0099] <Settings section 115> The setting unit 115 is configured to set the user's image generation rights according to the user's behavioral history or attributes. The image generation rights include at least one of the following: the number of first images that can be registered for one predetermined range (e.g., the number of registration slots), the number of first images that can be generated in a predetermined period for the user (e.g., one day, one month, etc.), the range that can be selected as the predetermined range, and the range that can be set as the change content. With such a configuration, the first image generation rights can be set according to the user's behavioral history or attributes, so for example, benefits can be given to the user according to their usage level.
[0100] "User activity history or attributes" include, for example, the number, frequency, or duration of user logins to the streaming service, the number or duration of viewing of streams by specific users (streamers) within the streaming service, the user's rank within the streaming service, and the amount of charges incurred within the streaming service.
[0101] User ranks include, for example, event ranks determined by the number of times an event is attended, and membership ranks determined by contract plans such as subscriptions.
[0102] The charge amount is the amount charged to a user for providing benefits to other users or for purchasing items by the user. "Providing benefits to other users" includes, for example, tipping, giving items or currency within the service as gifts, etc. "Items" include avatar decorations, effects that can be used during streaming or viewing, etc. Furthermore, the "charge amount" may be the total amount of charges a user has made to date, or the total amount of charges made to a specific user (streamer).
[0103] The settings unit 115 may set the user's image generation rights according to the user's billing amount. With this configuration, the first image generation rights can be set according to the user's billing amount, so that benefits can be given to users who bill a large amount.
[0104] As an example of the "selectable range as a predetermined range" in image generation permissions, the setting unit 115 may set it so that only the "left arm" is selectable for users whose billing amount is less than a predetermined threshold, and so that the "right arm" and "head" are also selectable for users whose billing amount is above the threshold. Also, as an example of the "settable range as change content" in image generation permissions, the setting unit 115 may set it so that only the "sword" is selectable as an accessory for users whose billing amount is less than a threshold, and so that accessories other than the "sword" are also selectable for users whose billing amount is above the threshold.
[0105] <Artificial Intelligence Department 120> The artificial intelligence unit 120 is configured to receive input from each functional unit and return the instructed output. The artificial intelligence used by each functional unit of the information processing device 10 may be common to all units, or it may be prepared individually for each functional unit.
[0106] The artificial intelligence unit 120 may include learning models such as generative AI, transformers, and recurrent neural networks (RNNs). Specific machine learning algorithms for constructing these learning models include nearest neighbors, naive Bayes, decision trees, support vector machines, and deep learning using neural networks.
[0107] The artificial intelligence unit 120 may have a trained model constructed by a learning method such as supervised learning, unsupervised learning, or self-supervised learning. In supervised learning, machine learning is performed using training data (machine learning data). Training data consists of pairs of input data and output data (correct answer data) for training. Furthermore, the language model may not only be one trained for a specific task, but also a general-purpose model that can be used universally for a wide range of tasks. The trained model included in the artificial intelligence unit 120 can undergo additional training as transfer learning or fine-tuning.
[0108] <Display section 211> The display unit 211 of the user terminal 20 is configured to display the screen indicated by the screen data transmitted from the information processing device 10 on the output unit 25.
[0109] <Operation acquisition unit 212> The operation acquisition unit 212 of the user terminal 20 is configured to accept operations from the user of the user terminal 20.
[0110] 3. Information Processing Methods This section describes the information processing method of the information processing device 10. In this information processing method, each part of the information processing device 10 is executed by a computer as a step.
[0111] Specifically, this information processing method comprises a reception step, an input information acquisition step, a first image generation step, and a second image generation step. In the reception step, the system receives input of the changes from the user. In the input information acquisition step, it acquires first input information relating to the appearance of the avatar and second input information indicating the changes to a predetermined range of the avatar. In the first image generation step, image generation information including the first and second input information is input to the first image generation model, causing the first image generation model to output a first image depicting the predetermined range in which the changes are reflected. In the second image generation step, a second image is generated depicting the avatar with the changes reflected to the predetermined range.
[0112] Figure 8 is an activity diagram showing an example of the flow of information processing (avatar image creation process) performed by information processing system 1. The information processing will be explained below in accordance with each activity in this activity diagram.
[0113] The avatar image creation process begins with the user inputting information. The user inputs image generation information (first input information and second input information) on the user terminal 20 (Activity A110). The information processing device 10 acquires the image generation information input on the user terminal 20 (Activity A120).
[0114] After acquiring the image generation information, the information processing device 10 generates a first image based on the image generation information (Activity A130). Subsequently, the information processing device 10 outputs the generated first image to the user terminal 20 (Activity A140). As a result, the first image is displayed on the user terminal 20 (Activity A150).
[0115] The user confirms the first image displayed on the user terminal 20 and enters a save or regeneration instruction on the user terminal 20 (Activity A160). If a regeneration instruction is entered (if a save instruction is not entered), the generated first image is discarded, and the process is repeated from Activity A110 (input of image generation information).
[0116] On the other hand, if a save instruction is entered, the information processing device 10 generates multiple derived images other than the image displayed on the user terminal 20 (Activity A170). Subsequently, the information processing device 10 generates a second image based on the first input information and the first image, or based on the first input information and the second input information (Activity A180). Note that the generation of derived images may be performed after the generation of the second image, or simultaneously with the generation of the second image.
[0117] After generating the derived image and the second image, the information processing device 10 registers the first image and the second image in the image database (activity A190). Note that the registration of the first image may be performed before the generation of the second image.
[0118] 4. Effect The operation of this embodiment can be summarized as follows: It is possible to generate an image with the necessary ranges modified according to the user's requests, based on a base avatar. Therefore, the degree of freedom in avatar creation can be increased.
[0119] Although embodiments of the present invention have been described above, the present invention is not limited thereto and can be modified as appropriate without departing from the technical spirit of the invention.
[0120] 5. Others In the above embodiment, the information processing device 10 performed various storage and control functions, but multiple external devices may be used instead of the information processing device 10. That is, various information and programs may be stored in a distributed manner across multiple external devices using blockchain technology or the like. For example, the artificial intelligence unit 120 may reside in a device outside the information processing device 10.
[0121] The embodiments of this model are not limited to the information processing system 1, but may also be an information processing method or a program. The information processing method comprises each step executed by the information processing system 1. The program causes a computer to execute each step of the information processing system 1.
[0122] At least one of the devices included in the information processing system 1 may be located outside the country in which the functions of the information processing system 1 are performed.
[0123] The information processing system 1 may consist only of the information processing device 10. In other words, the information processing system 1 does not necessarily have to include a user terminal 20.
[0124] The information processing system 1 does not necessarily have to include a second image generation unit 114. For example, the information processing system 1 may only have the function of generating a first image.
[0125] The product may be provided in any of the following embodiments.
[0126] (1) An information processing system comprising a processor configured to perform the following steps by reading a program, wherein in the input information acquisition step, first input information relating to the appearance of an avatar and second input information indicating changes to a predetermined range of the avatar are acquired; in the first image generation step, image generation information including the first input information and the second input information is input to a first image generation model, and the first image generation model is made to output a first image in which the predetermined range reflecting the changes is drawn, wherein the first image generation model is a machine learning model that is capable of taking the first input information and the second input information as input and outputting the first image.
[0127] This configuration allows for the generation of images with modified areas based on the user's requests, relative to the base avatar. Therefore, it increases the degree of freedom in avatar creation.
[0128] (2) An information processing system as described in (1) above, wherein the second input information indicates the changes made to a predetermined part of the avatar.
[0129] With this configuration, it is possible to generate images in which only specific parts of the avatar are modified according to the user's requests.
[0130] (3) An information processing system according to (1) or (2) above, wherein in the first image generation step, a prompt including the image generation information and an instruction to generate the first image based on the image generation information is input to the first image generation model, causing the first image generation model to output the first image.
[0131] With this configuration, the first image can be generated using a general-purpose image generation AI, thereby reducing the cost of building the information processing system.
[0132] (4) An information processing system according to any one of (1) to (3) above, wherein the first input information includes an avatar image including the appearance of the avatar, and the second input information includes the text of the changes.
[0133] With this configuration, the first image can be generated by giving text instructions to the avatar image, allowing a predetermined range of the existing avatar to be modified to match the user's image.
[0134] (5) An information processing system according to any one of (1) to (4) above, wherein the processor is configured to further perform the following steps: in the reception step, it receives input of the changes from a user; in the input information acquisition step, it converts the changes entered by the user into tokens for input to the first image generation model; and in the first image generation step, it inputs the tokens to the first image generation model as at least part of the second input information.
[0135] With this configuration, users can input changes to their avatar through various operations.
[0136] (6) An information processing system according to any one of (1) to (5) above, wherein the second input information includes information indicating an ornament to be attached to a predetermined part included in the predetermined range, and the first image generation step generates the first image in which the ornament is attached to the predetermined part.
[0137] This configuration makes it possible to generate images in which an avatar is fitted with any desired accessories.
[0138] (7) An information processing system as described in (6) above, wherein the second input information further includes information relating to at least one of the position in which the ornament is placed and the appearance of the ornament.
[0139] With this configuration, it becomes possible to adjust the position or appearance of decorative items to suit the user's preferences.
[0140] (8) An information processing system as described in (6) or (7) above, wherein the first input information includes an avatar image in which the avatar is drawn and an image generation variable indicating the conditions for generating the avatar image.
[0141] With this configuration, the first image can be generated using the generation conditions of the original avatar image, thus increasing the sense of unity between the original avatar image and the generated first image.
[0142] (9) An information processing system according to any one of (6) to (8) above, wherein the second input information includes information indicating the attributes of the ornament, the first image generation model includes a plurality of sub-image generation models, and in the first image generation step, the first input information and the second input information are input to a sub-image generation model selected from the plurality of sub-image generation models according to the attributes of the ornament.
[0143] With this configuration, the image generation model is switched according to the attributes of the ornament, which improves the quality of the first image in which the ornament is attached.
[0144] (10) An information processing system according to any one of (6) to (9) above, wherein the second input information includes an image of the ornament as information indicating the ornament.
[0145] With this configuration, it is possible to generate a first image of the ornament being worn using an existing image of the ornament.
[0146] (11) An information processing system according to any one of (1) to (10) above, wherein the second input information includes an existing first image in which the predetermined range has been drawn.
[0147] With this configuration, a new first image can be generated by referencing a previously generated first image, making it easier to obtain a first image that closely matches the user's image.
[0148] (12) An information processing system according to any one of (1) to (11) above, wherein in the first image generation step, a plurality of derived images are generated as the first image, each of which depicts the predetermined range at a different angle or orientation.
[0149] With this configuration, a dynamic avatar can be generated using the first image that is produced.
[0150] (13) An information processing system according to any one of (1) to (12) above, wherein in the first image generation step, upon receiving a user instruction to save the first image, the first image is replaced with the first image in which the predetermined range is drawn and registered in the database, and the first image is registered.
[0151] With this configuration, only the first image that the user likes can be kept in the database.
[0152] (14) An information processing system according to any one of (1) to (13) above, wherein the processor is configured to further perform the following steps, and in the second image generation step, an information processing system that generates a second image in which the avatar reflecting the changes is drawn within the predetermined range.
[0153] This configuration allows users to generate avatars that can be used immediately.
[0154] (15) An information processing system as described in (14) above, wherein the first input information includes an avatar image including the appearance of the avatar, and in the second image generation step, the avatar image and the first image are input to a second image generation model, and the second image generation model is caused to output a second image, wherein the second image generation model is a machine learning model that is capable of taking the avatar image and the first image as input and outputting a second image.
[0155] With this configuration, the first image and the original avatar image are combined to generate a new avatar, thus improving the quality of the resulting avatar.
[0156] (16) An information processing system as described in (14) or (15) above, wherein in the first image generation step, upon receiving a save instruction from a user for the first image, the first image is registered in a database, and in the second image generation step, upon receiving the save instruction, the second image is generated.
[0157] With this configuration, a modified avatar image is generated only if the user likes the first image, thus reducing the processing load on the information processing system.
[0158] (17) An information processing system according to any one of (1) to (16) above, wherein the processor is configured to perform the following steps, the setting step of setting the user's image generation authority according to the user's behavior history or attributes, wherein the image generation authority includes at least one of the number of first images that can be registered for one predetermined range, the number of first images that can be generated by the user during a predetermined period, a range that can be selected as the predetermined range, and a range that can be set as the change content.
[0159] With this configuration, the right to generate the first image can be set according to the user's behavioral history or attributes, so for example, users can be given benefits based on their usage.
[0160] (18) Information processing system as described in (17) above, wherein the processor is configured to perform the following steps, in which setting step, the user's image generation rights are set according to the user's charge amount, where the charge amount is the amount charged to the user for providing benefits to other users or for the user's purchase of items.
[0161] With this configuration, the right to generate the first image can be set according to the amount of money the user spends, allowing for the granting of benefits to users who spend more.
[0162] (19) An information processing method comprising each step performed by the information processing system described in any one of (1) to (18) above.
[0163] (20) A program that causes a computer to perform each step of the information processing system described in any one of (1) to (18) above. Of course, this is not always the case. [Explanation of symbols]
[0164] 1: Information Processing System 2: Communication lines 10: Information Processing Device 11: Control Unit 12: Storage section 13: Communications Department 14: Communications bus 20: User terminal 21: Control Unit 22: Storage section 23: Communications Department 24: Input section 25: Output section 26: Communications bus 111: Reception Department 112: Input Information Acquisition Unit 113: First image generation unit 114: Second image generation unit 115: Settings section 120: Artificial Intelligence Department 211:Display section 212: Operation acquisition section
Claims
1. An information processing system, A processor configured to perform the following steps by reading a program, In the input information acquisition step, first input information relating to the appearance of the avatar and second input information indicating the changes made to a predetermined range of the avatar are acquired. In the first image generation step, image generation information including the first input information and the second input information is input to the first image generation model, and the first image generation model is instructed to output a first image in which the predetermined range reflecting the changes is drawn. Here, the first image generation model is an information processing system that is a machine learning model capable of taking the first input information and the second input information as inputs and outputting a first image.
2. In the information processing system described in claim 1, The second input information is an information processing system that indicates the changes made to a predetermined part of the avatar.
3. In the information processing system described in claim 1, An information processing system that, in the first image generation step, inputs a prompt to the first image generation model, which includes the image generation information and an instruction to generate the first image based on the image generation information, causing the first image generation model to output the first image.
4. In the information processing system described in claim 1, An information processing system in which the first input information includes an avatar image including the appearance of the avatar, and the second input information includes text of the changes.
5. In the information processing system described in claim 1, The aforementioned processor is configured to perform the following steps: In the registration step, the user inputs the aforementioned changes. In the input information acquisition step, the changes entered by the user are converted into tokens for input into the first image generation model. An information processing system that, in the first image generation step, inputs the token as at least part of the second input information into the first image generation model.
6. In the information processing system described in claim 1, The second input information includes information indicating an ornament to be attached to a predetermined part included in the predetermined range, An information processing system that, in the first image generation step, generates the first image in which the ornament is attached to the predetermined part.
7. In the information processing system described in claim 6, An information processing system wherein the second input information further includes information relating to at least one of the position in which the ornament is placed and the appearance of the ornament.
8. In the information processing system described in claim 6, An information processing system in which the first input information includes an avatar image in which the avatar is drawn and an image generation variable indicating the conditions for generating the avatar image.
9. In the information processing system described in claim 6, The second input information includes information indicating the attributes of the ornament, The first image generation model includes a plurality of sub-image generation models, An information processing system that, in the first image generation step, inputs the first input information and the second input information to a sub-image generation model selected from among the plurality of sub-image generation models according to the attributes of the ornament.
10. In the information processing system described in claim 6, An information processing system in which the second input information includes an image of the ornament as information indicating the ornament.
11. In the information processing system described in claim 1, An information processing system in which the second input information includes an existing image in which the predetermined range has been drawn.
12. In the information processing system described in claim 1, An information processing system that, in the first image generation step, generates a plurality of derived images as the first image, each of which depicts the predetermined range at a different angle or orientation.
13. In the information processing system described in claim 1, An information processing system that, in the first image generation step, receives a user instruction to save the first image, replaces the first image with an existing image that has been registered in the database and depicts the predetermined range, and then registers it.
14. In the information processing system described in claim 1, The aforementioned processor is configured to perform the following steps: An information processing system that, in the second image generation step, generates a second image in which the avatar reflecting the changes within the predetermined range is drawn.
15. In the information processing system described in claim 14, The first input information includes an avatar image, which includes the appearance of the avatar. In the second image generation step, the avatar image and the first image are input to the second image generation model, and the second image generation model is made to output the second image. Here, the second image generation model is an information processing system that is a machine learning model capable of taking the avatar image and the first image as inputs and outputting the second image.
16. In the information processing system described in claim 14, In the first image generation step, upon receiving a user instruction to save the first image, the first image is registered in the database. The second image generation step involves an information processing system that, upon receiving the save instruction, generates the second image.
17. In the information processing system described in claim 1, The aforementioned processor is configured to perform the following steps: In the setup step, the user's image generation permissions are set according to the user's behavioral history or attributes. Herein, the image generation authority includes at least one of the number of first images that can be registered for one predetermined range, the number of first images that can be generated by the user during a predetermined period, the range that can be selected as the predetermined range, and the range that can be set as the change content, in an information processing system.
18. In the information processing system described in claim 17, The aforementioned processor is configured to perform the following steps: In the above setup step, the user's image generation permission is set according to the user's billing amount. Herein, the amount charged is the amount charged to the user for providing benefits to other users or for the user's purchase of items, in an information processing system.
19. Information processing method, An information processing method comprising each step performed by the information processing system according to any one of claims 1 to 18.
20. It is a program, A program for causing a computer to perform each step of the information processing system described in any one of claims 1 to 18.