Program, information processing method, and server

Abstract

The present invention improves convenience and the like related to a chat.　A program to be executed by a server for executing processing related to a chat including a first user of a first terminal and a second user of a second terminal causes the server to execute: receiving, by a communication unit of the server from the first terminal, text information based on an input to the first terminal on which a chat room including the first user and the second user is displayed; generating an image set including one or more images on the basis of the text information; transmitting, to the first terminal, first information related to the image set by the communication unit; receiving, by the communication unit from the first terminal, second information related to one or more images selected from the image set based on the input to the first terminal on which the image included in the image set is displayed in the chat room on the basis of the first information; and transmitting, by the communication unit to the second terminal, third information based on the second information.

Description

Program, information processing method, server

[0001] The present disclosure relates to a program, an information processing method, a server, and the like.

[0002] When users communicate using a chat application or the like, they may use stamp images or the like that can effectively express the user's emotions, etc. For example, Patent Literature 1 discloses application software that conveys a message from the user to a recipient user via the image or character of an electronic stamp, in addition to the emotion of the image or character of the electronic stamp.

[0003] Japanese Patent Application Laid-Open No. 2019-028728

[0004] According to a first aspect of the present invention, a program executed by a server that performs processing related to a chat including a first user of a first terminal and a second user of a second terminal receives text information from the first terminal based on input to the first terminal on which a chat room including the first user and the second user is displayed, generates an image set including one or more images based on the text information, transmits first information related to the image set to the first terminal by the communication unit, receives second information from the first terminal by the communication unit related to one or more images selected from the image set based on input to the first terminal on which images included in the image set are displayed in the chat room based on the first information, and transmits third information based on the second information to the second terminal by the communication unit. According to a second aspect of the present invention, an information processing method of a server that performs processing related to a chat including a first user of a first terminal and a second user of a second terminal includes receiving text information from the first terminal based on input to the first terminal on which a chat room including the first user and the second user is displayed, by a communication unit of the server, generating an image set including one or more images based on the text information, transmitting first information related to the image set to the first terminal by the communication unit, receiving second information from the first terminal by the communication unit related to one or more images selected from the image set based on input to the first terminal on which images included in the image set are displayed in the chat room based on the first information, and transmitting third information based on the second information to the second terminal by the communication unit.According to a third aspect of the present invention, a server that performs processing related to a chat including a first user of a first terminal and a second user of a second terminal comprises a communication unit and a control unit, wherein the communication unit receives text information from the first terminal based on input to the first terminal on which a chat room including the first user and the second user is displayed, the control unit generates an image set including one or more images based on the text information, the communication unit transmits first information related to the image set to the first terminal, receives second information from the first terminal on which images included in the image set are displayed in the chat room based on the first information, and transmits third information based on the second information to the second terminal.

[0005] 1 is a diagram showing an example of the system configuration of a communication system according to the first embodiment. FIG. 1 is a diagram showing an example of a function implemented by a control unit of a server according to the first embodiment. FIG. 2 is a diagram showing an example of information stored in a storage unit of a server according to the first embodiment. FIG. 2 is a diagram showing an example of account registration data according to the first embodiment. FIG. 3 is a diagram showing an example of stamp image additional learning model management data according to the first embodiment. FIG. 4 is a diagram showing an example of stamp image generation prompt management data according to the first embodiment. FIG. 5 is a diagram showing an example of a stamp management database according to the first embodiment. FIG. 6 is a diagram showing an example of a function implemented by a control unit of a terminal according to the first embodiment. FIG. 7 is a diagram showing an example of information stored in a storage unit of a terminal according to the first embodiment. FIG. 8 is a diagram showing an example of a screen displayed on a display unit of a terminal according to the first embodiment. FIG. 9 is a diagram showing an example of a screen displayed on a display unit of a terminal according to the first embodiment. FIG. 10 is a flowchart showing an example of a process flow executed by each device according to the first embodiment. FIG. 11 is a diagram showing an example of a screen displayed on a display unit of a terminal according to the first modification. FIG. 12 is a flowchart showing an example of a process flow executed by each device according to the first modification. FIG. 13 is a diagram showing an example of a screen displayed on a display unit of a terminal according to the first modification. FIG. 10 is a diagram showing an example of a screen displayed on the display unit of the terminal according to the second embodiment. FIG. 11 is a diagram showing an example of a screen displayed on the display unit of the terminal according to the second embodiment. FIG. 12 is a diagram showing an example of a screen displayed on the display unit of the terminal according to the second embodiment. A flowchart showing an example of the flow of processing performed by each device according to the second embodiment. FIG. 13 is a diagram showing an example of a screen displayed on the display unit of the terminal according to the second modification. FIG. 14 is a diagram showing an example of a screen displayed on the display unit of the terminal according to the second modification. A flowchart showing an example of the flow of processing performed by each device according to the second modification. FIG. 15 is a diagram showing an example of information stored in a memory unit of a server according to the third embodiment. FIG. 16 is a diagram showing an example of a screen displayed on the display unit of the terminal according to the third embodiment. FIG. 17 is a diagram showing an example of a screen displayed on the display unit of the terminal according to the third embodiment. A flowchart showing an example of the flow of processing performed by each device according to the third embodiment. FIG. 18 is a diagram showing an example of information stored in a memory unit of a server according to the fourth embodiment.A diagram showing an example of a screen displayed on the display unit of a terminal according to a fourth embodiment. A diagram showing an example of a screen displayed on the display unit of a terminal according to a fourth embodiment. A flowchart showing an example of the flow of processes executed by each device according to the fourth embodiment. A diagram showing an example of a screen displayed on the display unit of a terminal according to a fourth modified example. A diagram showing an example of a screen displayed on the display unit of a terminal according to a fourth modified example.

[0006] Compliance with Legal Matters It should be noted that the disclosure described herein is subject to compliance with the laws of the country of implementation necessary for the implementation of this disclosure, such as secrecy of communications.

[0007] <Embodiments> In this specification, for ease of understanding, there are places where the phrase "for example" is used, but please note that not only those places but also the entire embodiment described below are not limited to the contents of that description.

[0008] An embodiment for implementing a program etc. according to the present disclosure will be described with reference to the drawings.

[0009] The production of a terminal of the claimed invention (terminal of the claimed invention) may include, for example, the concept that a state is created in which the functions of the claimed invention can be realized (a state in which the claimed invention can be executed) on a terminal owned (possessed) by a user by receiving (or receiving and storing) a program (for example, an application program) described in this specification on the terminal.

[0010] Furthermore, the production of the system of the invention claimed in the present application (the system of the invention of the present application) may include, for example, the concept that a state in which the functions of the invention of the system claimed in the present application can be realized (a state in which the invention claimed in the present application can be executed) is created by receiving a program described in this specification (for example, an application program) transmitted from a server included in the system of the present application at a terminal included in the system of the present application (or by storing the received program in the terminal).

[0011] Furthermore, in this specification, a system can be, for example, a configuration including multiple devices. The multiple devices may be a combination of devices of the same type, a combination of devices of different types, or a combination of devices of the same type and devices of different types. Note that a system can also be, for example, considered to be a configuration in which multiple devices work together to perform some kind of processing.

[0012] Furthermore, a system relating to a client (client device) and a server can be considered to be, for example, at least one of the following: (1) Terminal and Server (2) Server (3) Terminal

[0013] (1) is, for example, a system including at least one terminal and at least one server. One example of this is a client-server system.

[0014] The server is configured by the following devices, for example, and may be a single device or a combination of multiple devices.

[0015] Specifically, the server is configured to have, for example, at least one processor (for example, CPU: Central Processing Unit, GPU: Graphics Processing Unit, APU: Accelerated Processing Unit, DSP: Digital Signal Processor (for example, ASIC: Application Specific Integrated Circuit, FPGA: Field Programmable Gate Array), etc.), computer device (processor + memory), control device, arithmetic device, processing device, etc., and may be configured to have multiple of the same type of device (for example, CPU + CPU, homogeneous multi-core processor, etc.), or multiple of different types of device (for example, CPU + DSP, heterogeneous multi-core processor, etc.), or may be a combination of multiple devices (for example, processor + computer device, processor + arithmetic device, multiple devices made heterogeneous, etc.). The processor may also be a virtual processor.

[0016] Furthermore, when a server performs some processing, if the server is configured with a single device, the processing described in the embodiments is performed by the single device. Furthermore, if the server is configured with multiple devices, some processing may be performed by one device and other processing may be performed by the other device. For example, if the server is configured with a processor and an arithmetic device, the processor may perform a first processing and the arithmetic device may perform a second processing. Furthermore, if the server is configured with multiple devices, the devices may be physically located apart from each other.

[0017] Furthermore, the server functions may be provided in the form of PaaS, IaaS, or SaaS in cloud computing, for example.

[0018] The control unit of the system can be at least one of the control unit of the terminal and the control unit of the server. That is, for example, the control unit of the system can be any of (1A) only the control unit of the terminal, (1B) only the control unit of the server, or (1C) both the control unit of the terminal and the control unit of the server.

[0019] Furthermore, the control and processing performed by the system's control unit (hereinafter collectively referred to as "control, etc.") may be (1A) performed only by the terminal's control unit, (1B) performed only by the server's control unit, or (1C) performed by both the terminal's control unit and the server's control unit. Also, in (1C), for example, some of the control, etc. performed by the system's control unit may be performed by the terminal's control unit, and the remaining control, etc. may be performed by the server's control unit. In this case, the allocation (allocation) of control, etc. may be equal, or may be allocated in different proportions.

[0020] Furthermore, when referring to the communication unit of a server, if the server is configured with a single device, it may refer to the communication unit itself that is included in the single device. Furthermore, if the server is configured with multiple devices, the communication unit of the server may be configured to include each communication unit that each device includes. For example, if the server includes a first device and a second device, and the first device has a first communication unit and the second device has a second communication unit, the communication unit of the server may be conceptually understood to include both the first communication unit and the second communication unit.

[0021] (2) can be, for example, a system consisting of multiple servers (hereinafter referred to as a "server system") In this case, the configuration of each server can be similarly applied to the configuration described above.

[0022] The control etc. performed by the server system may be performed by only one server (2A) among multiple servers, or by only other servers (2B), or by one server and other servers (2C). Also, in (2C), for example, some of the control etc. performed by the server system may be performed by one server, and the remaining control etc. may be performed by other servers. In this case, the allocation (allocation) of the control etc. may be equal, or may be allocated in different proportions.

[0023] (3) can be, for example, a system composed of multiple terminals. This system can be, for example, the following systems: - A system in which terminals have server functions (distributed system). This can be realized, for example, by using blockchain technology. - A system in which terminals communicate wirelessly with each other. This can be realized, for example, by communicating using a P2P (peer-to-peer) method using short-range wireless communication technology such as Bluetooth (registered trademark).

[0024] The above is not limited to the control unit, but also applies to each functional unit such as an input / output unit, a communication unit, a storage unit, and a clock unit, which may be components of the system.

[0025] In the following embodiment, a system including a terminal and a server (a client-server system) is exemplified. Note that the server system described in (2) above can also be applied as the server.

[0026] Furthermore, instead of a system including a terminal and a server, it is also possible to apply a system that does not include a server, such as the system described in (3) above. In this case, the embodiment can be configured based on the blockchain technology described above. Specifically, for example, data stored and managed in the server described in the following embodiment is stored on the blockchain. Then, the terminal generates a transaction to the blockchain, and when the transaction is approved on the blockchain, the data stored on the blockchain is updated.

[0027] It should be noted that even when the term "terminal" is used, this does not mean that the terminal is limited to a client device in a client-server environment. In other words, the term "terminal" may also include the concept of a device that is not in a client-server environment.

[0028] Furthermore, in this specification, the expression "through a communication I / F" is used as appropriate. This may, for example, indicate that a device transmits and receives various information and data through a communication I / F (through a communication unit) based on the control of a control unit (such as a processor).

[0029] Furthermore, in this specification, when the terms "related to" or "related to" are used, for example, "B related to A" or "B related to A" may mean "B" that has some kind of relationship with "A." Specific examples of this will be described later.

[0030] Furthermore, in this specification, when a device processes two or more items, such as "transmitting A and B" or "receiving A and B," this may include both performing "A" and "B" in sync (hereinafter referred to as "simultaneous") and performing "A" and "B" at different times (hereinafter referred to as "non-simultaneous"). For example, when referring to transmitting first information and second information, this may include both transmitting the first information and the second information in sync and transmitting the first information and the second information at different times. Note that, taking into account lag (time lag), "simultaneous" may also include "almost simultaneously."

[0031] Note that, although "A" and "B" are performed at different times, this only needs to be performed for "A" and "B," and the purposes thereof do not necessarily have to be the same. For example, when transmitting first information and second information as described above, it is sufficient to transmit the first information and the second information, and this may include cases where the first information and the second information are transmitted for the same purpose, as well as cases where the first information and the second information are transmitted for different purposes.

[0032] In the following embodiments, a messaging service is exemplified as an example of a service that allows users to chat (hereinafter referred to as a "chat service"). An application that realizes the chat service is referred to as a "chat application," and an application that realizes the messaging service is referred to as a "messaging application." For example, the chat application may enable users to chat in a chat room.

[0033] Note that messaging services (MS) (including instant messaging services (IMS)) can also be considered a form of social networking services (SNS). Therefore, messaging services and social networking services may or may not be distinguished. In other words, messaging services may be included in social networking services.

[0034] In the following embodiments, an instant messaging service (IMS) that transmits and receives simple messages between multiple devices (e.g., terminals) via a server will be described as an example of a messaging service. The instant messaging application allows users to chat in a chat room.

[0035] A chat room (for example, a talk room) can be a UI (User Interface) or GUI (Graphical User Interface) that allows each user to view content transmitted and received between the terminals of multiple users.

[0036] In a messaging service, for example, a user can register any account as a "friend" and exchange messages via a chat room.

[0037] Note that "friend" may refer to, for example, a relationship between a first user and a second user when the first user's account and the second user's account in a messaging application are unilaterally or bilaterally associated. For example, when the first user's account and the second user's account are mutually associated, the first user and the second user may be said to be "friends." Also, for example, when the first user's account is associated with the second user's account, the first user and the second user may be said to be "friends" even if the second user's account is not associated with the first user's account.

[0038] In addition, talk rooms can include one-on-one user talk rooms (hereinafter referred to as "one-on-one talk rooms"), group talk rooms including multiple users (hereinafter referred to as "group talk rooms"), and talk rooms with OA providers (for example, providers affiliated with messaging service providers) (hereinafter referred to as "OA talk rooms").

[0039] Furthermore, an account of a messaging application that is an account of a business operator and not a general user may be referred to as an "OA (Official Account)," and a user of this official account may be referred to as an "official user." This may also be referred to as an "official account user" or an "official account business operator." In contrast, an account of a messaging application that is an account of a user that is not an OA business operator may be referred to as a "general account," and a user of a general account may be referred to as a "general user." This may also be referred to as a "general account user." In other words, messaging application accounts may include both general accounts and official accounts.

[0040] Furthermore, for example, an office automation company may be able to use a terminal similar to that of a user with a general account to send and receive messages to and from other devices via the server.

[0041] Furthermore, a message (message information) may be, for example, information used in a messaging service to define a sender and a destination, and may be information consisting of identification information (message ID) for identifying a message and message content. Furthermore, message content may be, for example, the contents of a message excluding the message ID. The message content may be one or more pieces of content.

[0042] Furthermore, information set as identification information for identifying a message is referred to as a "message ID." Message content included in messages with the same message ID can be considered to be identified by this message ID. Therefore, identification information for identifying a message can be considered to be essentially synonymous with identification information for identifying message content. Alternatively, individual identification information (message content ID) may or may not be set for each message content.

[0043] The content may include, for example, text content in text format, image content in image format (including at least one of still images and moving images), sound content in sound format (including voice), etc. In addition to these, the content may also include operation content such as buttons and icons for user operation, and link content such as URIs (including URLs, etc.).

[0044] The text may include, for example, at least one of national characters represented by character codes, extended characters, machine-dependent characters, numbers, symbols, graphics, and codes. Note that the text does not have to include at least one of the above characters, extended characters, machine-dependent characters, numbers, symbols, graphics, and codes, and may include other text.

[0045] The image may include at least one of various types of image information such as an icon, a button, a stamp, a pictogram, and a banner image.

[0046] Unlike the above definition, it is possible to define that a superordinate concept of a message is content, or that content and message are synonymous, but it is not necessary to define them in this way.

[0047] <Embodiment> As an example, when creating stamps to be used as a means for smoothly advancing communication in a messaging application, the user who is the creator needs to perform tedious work such as preparing and processing photographs, illustrations, etc. Also, as an example, it is preferable to create multiple stamps based on a common character, concept, etc., and there may be a high barrier to entry for, for example, easily processing and publishing photographs of a user's pet or illustrations drawn by the user as stamps.

[0048] This embodiment is, for example, an embodiment relating to a service that allows a stamp to be easily created based on an image provided by a user.

[0049] Moreover, this embodiment is an embodiment relating to a service that allows a user to easily create and send a stamp based on text input by the user, for example.

[0050] Here, the stamp refers to, for example and without limitation, an icon image or the like that can be used to add emotion or expression to a text message in a chat service, and may be called a "sticker." Note that a stamp may be a type of sticker.

[0051] In the following, for example, a collection of stamps consisting of any number of stamps and having some kind of relationship, such as a common character or concept, will be referred to as a "stamp set." Also, for example, an image associated with an individual stamp will be referred to as a "stamp image." In contrast, for example, a collection of stamp images in a stamp set will be referred to as a "stamp set image." Note that, for example, the following will focus on examples in which the stamp images are still images, but the stamp images may also be moving images or animations.

[0052] In the following, as an example, one or more image sets provided by a user for generating a stamp image will be referred to as "base images." A base image may also be referred to as a "base image set." In the following, characters, subjects, etc. commonly included in the base images will be referred to as "objects." An object may also be referred to as a concept common to the base images. An identifier for identifying an object will be referred to as an "instance prompt." An identifier for identifying general properties common to objects or a classification (class or category) of a learning subject will be referred to as a "class prompt." Specific examples of instance prompts and class prompts will be described later. In the following, as an example, an example in which the base image is a still image will be illustrated, but the base image may also be a video or animation.

[0053] Furthermore, an object may be considered as a type of element included in an image (image element), and in the examples described below, an object may be considered as an "element." However, elements are not limited to objects, and other examples will be described later.

[0054] First Embodiment The first embodiment is an embodiment in which a stamp set image is created based on a base image. The content described in the first embodiment can be similarly applied to any of the other embodiments and other modified examples.

[0055] In the following embodiments, a service for realizing support for creating stamp set images or stamp sets will be referred to as a "stamp set creation support service." An application for realizing the stamp set creation support service will be referred to as a "stamp set creation application." With the stamp set creation application, a user can easily create stamp set images or stamp sets by registering a base image.

[0056] As a form for realizing the stamp set creation support service (stamp set creation application), any of the following forms can be applied, for example: (A1) A form in which the stamp set creation support service function is provided as one function of a messaging application (A2) A form in which the messaging service function is provided as one function of a stamp set creation application (B1) A form in which an application (integrated application) having the messaging service function and the stamp set creation support service function is configured (C1) A form in which the stamp set creation application is configured as an application separate from the messaging application (C2) A form in which the messaging application is configured as an application separate from the stamp set creation application (D) A form in which the stamp set creation application is configured as a standalone application

[0057] In the embodiments (A1), (A2), and (B1), for example, the messaging service provider and the stamp set creation support service provider can be the same provider. In this case, one method is to use a common account for the user's account in the messaging application and the stamp set creation application. In this case, another method is to automatically associate (link) the user's account in the messaging application with the user's account in the stamp set creation application.

[0058] In the embodiments (C1) to (C2), for example, the messaging service provider may be a different provider from the stamp set creation support service provider. Also, in the embodiments (C1) to (C2), a process for associating (linking) the user's account in the messaging application with the user's account in the stamp set creation application may be performed.

[0059] In the following, as an example, a description will be given of the case where the server 10 is operated and managed by a messaging service provider in accordance with the form (A1).

[0060] In the following description, the user of terminal 20A communicating with server 10 will be referred to as "user A.A.", the user of terminal 20B as "user B.B.", the user of terminal 20C as "user C.C.", and so on.

[0061] 1-1 is a diagram illustrating an example of the system configuration of a communication system 1 according to this embodiment. In the communication system 1, for example, a server 10 is connected to one or more terminals 20 (terminal 20A, terminal 20B, terminal 20C, ...) via a network 30.

[0062] The server 10 has a function of providing, for example, a messaging service or a stamp set creation support service to a terminal 20 owned by a user via the network 30. The server 10 can also be expressed as a messaging service server, a stamp set creation support server, an image generation server, etc. In this embodiment, for example, the user of the server 10 is a company that provides a messaging service or a stamp set creation support service (a provider of the messaging service or stamp set creation support service). Note that the number of servers 10 and the number of terminals 20 connected to the network 30 are not limited to the above. In this embodiment, the "stamp set creation support service" is a service provided by a provider (server 10) such as a company that provides a stamp set creation support service, and may be provided to a user (user's terminal 20), for example.

[0063] The terminal 20 (terminal 20A, terminal 20B, terminal 20C, etc.) may be any information processing terminal capable of implementing the functions described in each embodiment. Examples of the terminal 20 include smartphones, mobile phones (feature phones), computers (including, but not limited to, desktops, laptops, tablets, etc.), media computer platforms (including, but not limited to, cable and satellite set-top boxes, digital video recorders, etc.), handheld computer devices (including, but not limited to, personal digital assistants (PDAs), email clients, etc.), wearable devices (glasses-type devices, watch-type devices, etc.), virtual reality (VR) terminals, smart speakers (voice recognition devices), or other types of computers or communication platforms. The terminal 20 may also be referred to as an information processing terminal.

[0064] For example, the configurations of terminals 20A, 20B, and 20C may be the same. Furthermore, if necessary, the terminal used by user X may be referred to as terminal 20X, and user information in a predetermined service associated with user X or terminal 20X may or may not be referred to as user information X. Note that user information is user information associated with an account used by the user in a predetermined service. User information includes, by way of example and without limitation, information associated with a user, such as the user's name, user icon image, user age, user gender, user address, user hobbies and interests, and user identifier, which is input by the user or assigned by the predetermined service, and may be any one or a combination of these.

[0065] The network 30 serves to connect the devices constituting the communication system 1. That is, the network 30 refers to a communication network that provides connection paths so that the above-mentioned various devices can be connected and then transmit and receive data.

[0066] One or more portions of network 30 may or may not be a wired or wireless network. Network 30 may include, by way of example, an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a wide area network (WAN), a wireless WAN (WWAN), a metropolitan area network (MAN), a portion of the Internet, a portion of the Public Switched Telephone Network (PSTN), a cellular network, integrated service digital networks (ISDN), wireless LAN, long term evolution (LTE), code division multiple access (CDMA), Bluetooth, satellite communications, etc., or a combination of two or more thereof. Network 30 may include one or more networks 30.

[0067] The server 10 (not limited to, but an example of a server, information processing device, or information management device) has a function of providing predetermined services (in this embodiment, a messaging service or a stamp set creation support service) to the terminal 20, etc. The server 10 may be any information processing device capable of implementing the functions described in each embodiment. Examples of the server 10 include a server device, a computer (e.g., a desktop, laptop, tablet, etc.), a media computer platform (e.g., a cable or satellite set-top box, digital video recorder, etc.), a handheld computer device (e.g., a PDA, email client, etc.), or other types of computers or communication platforms. The server 10 may also be referred to as an information processing device. When there is no need to distinguish between the server 10 and the terminal 20, the server 10 and the terminal 20 may or may not each be referred to as an information processing device.

[0068] [Hardware (HW) Configuration of Each Device] The HW configuration of each device included in the communication system 1 will be described.

[0069] (1) HW Configuration of Terminal FIG. 1-1 shows an example of the HW configuration of the terminal 20. The terminal 20 includes, for example, a control unit 21 (CPU: central processing unit), a storage unit 28, a communication I / F 22 (interface), an input / output unit 23, a clock unit 29A, and a position calculation information detection unit 29B. The HW components of the terminal 20 are connected to each other, for example, via a bus B. Note that it is not essential for the HW configuration of the terminal 20 to include all components. For example, the terminal 20 may or may not be configured so that individual components or multiple components can be removed.

[0070] The communication I / F 22 transmits and receives various data via the network 30. The communication may be performed either wired or wirelessly, and any communication protocol may be used as long as mutual communication is possible. The communication I / F 22 has a function of communicating with various devices, such as the server 10, via the network 30. The communication I / F 22 transmits various data to various devices, such as the server 10, in accordance with instructions from the control unit 21. The communication I / F 22 also receives various data transmitted from various devices, such as the server 10, and transmits it to the control unit 21. The communication I / F 22 may also be simply referred to as a communication unit. When the communication I / F 22 is configured as a physically structured circuit, it may also be referred to as a communication circuit.

[0071] The input / output unit 23 includes a device for inputting various operations to the terminal 20, a device for outputting processing results processed by the terminal 20, etc. The input / output unit 23 may be an integrated unit having an input unit and an output unit, or may be separate units having an input unit and an output unit, or may not be so.

[0072] The input unit is realized by any one or a combination of all types of devices that can accept input from a user and transmit information related to the input to the control unit 21. Examples of the input unit include hardware keys such as a touch panel, a touch display, and a keyboard, a pointing device such as a mouse, a camera (for inputting operations via moving images), and a microphone (for inputting operations via voice).

[0073] The output unit is realized by any one or a combination of all types of devices that can output the processing results processed by the control unit 21. Examples of the output unit include a touch panel, a touch display, a speaker (audio output), a lens (for example, 3D (three dimensions) output or hologram output), a printer, etc.

[0074] Although this is merely an example, the input / output unit 23 includes a display unit 24 , a sound input unit 25 , a sound output unit 26 , and an imaging unit 27 .

[0075] The display unit 24 is realized by any one of all types of devices or a combination thereof that can display according to the display data written to the frame buffer. Examples of the display unit 24 include a touch panel, a touch display, a monitor (e.g., a liquid crystal display or an organic electroluminescence display (OELD)), a head mounted display (HDM), a projection mapping, a hologram, and a device that can display images, text information, etc. in air (which may or may not be a vacuum). Note that these display units 24 may or may not be capable of displaying display data in 3D.

[0076] The sound input unit 25 is used to input sound data (including voice data; the same applies below). The sound input unit 25 includes a microphone and the like. The sound output unit 26 is used to output sound data. The sound output unit 26 includes a speaker and the like. The imaging unit 27 is used to acquire image data (including still image data and moving image data; the same applies below). The imaging unit 27 includes a camera and the like.

[0077] When the input / output unit 23 is a touch panel, the input / output unit 23 and the display unit 24 may be disposed opposite each other and have approximately the same size and shape.

[0078] The clock unit 29A is a built-in clock of the terminal 20 and outputs time information (timekeeping information). The clock unit 29A is configured, for example, with a clock that uses a crystal oscillator. The clock unit 29A can also be expressed, for example, as a timekeeping unit or a time information detection unit.

[0079] The clock unit 29A may or may not have a clock that conforms to the NITZ (Network Identity and Time Zone) standard or the like.

[0080] The position calculation information detection unit 29B is a functional unit that detects (measures) information (hereinafter referred to as "position calculation information") necessary for the control unit 21 to calculate (measure) the position of the own terminal 20. The position calculation information detection unit 29B can also be expressed as a position calculation sensor unit, for example.

[0081] The position calculation information detection unit 29B includes, for example, a satellite positioning sensor (satellite positioning unit) which is a sensor or unit for calculating the position of the terminal 20 using a satellite positioning system such as GPS (Global Positioning System), an inertial measurement sensor (inertial measurement unit (IMU (Inertial Measurement Unit))) which is a sensor or unit for calculating the position of the terminal 20 using an inertial navigation system, a UWB positioning sensor (UWB positioning unit) which is a sensor or unit for calculating the position of the terminal 20 using UWB (Ultra Wide Band), and the like.

[0082] The satellite positioning unit includes, for example, an RF receiving circuit that converts RF (Radio Frequency) signals, including positioning satellite signals transmitted from positioning satellites and received by an antenna (not shown), into digital signals, and a baseband processing circuit that performs correlation calculation processing on the digital signals output from the RF receiving circuit to capture the positioning satellite signals, and outputs information such as satellite orbit data and time data extracted from the positioning satellite signals as information for position calculation.

[0083] The inertial measurement unit has an inertial sensor that is a sensor that detects information necessary for calculating the position of the terminal 20 by inertial navigation calculation. The inertial sensor includes, for example, a three-axis acceleration sensor and a three-axis gyro sensor, and outputs the acceleration detected by the acceleration sensor and the angular velocity detected by the gyro sensor as information for position calculation.

[0084] The UWB positioning unit includes, for example, an ultra-wideband RF (Radio Frequency) receiving circuit that converts an ultra-wideband RF signal, including an ultra-wideband pulse signal for positioning transmitted from a positioning beacon and received by an antenna (not shown), into a digital signal, and a relative position calculation processing circuit that calculates the relative position between the terminal 20 and the positioning beacon based on the digital signal output from the ultra-wideband RF receiving circuit. Note that, for example, the UWB positioning unit may or may not cause the terminal 20 to function as a positioning beacon by transmitting an ultra-wideband RF signal including an ultra-wideband pulse signal for positioning from an antenna (not shown).

[0085] For example, the control unit 21 calculates the position of its own terminal 20 at regular intervals or specific intervals based on the position calculation information detected by the position calculation information detection unit 29B. The position of the terminal is referred to as the "terminal position," and the calculated terminal position is referred to as the "calculated terminal position." The control unit 21 may, but need not, associate the calculated terminal position with the date and time when the calculated terminal position is calculated and store the calculated terminal position in the storage unit 28 as calculated terminal position history data.

[0086] The control unit 21 has a circuit physically structured to execute the functions realized by the code or instructions contained in the program, and is realized by, for example, a data processing device built into hardware. Therefore, the control unit 21 may or may not be expressed as a control circuit.

[0087] The control unit 21 includes, for example, a central processing unit (CPU), a microprocessor, a processor core, a multiprocessor, an application-specific integrated circuit (ASIC), or a field programmable gate array (FPGA).

[0088] The storage unit 28 has a function of storing various programs and various data required for the operation of the terminal 20. The storage unit 28 includes, for example, various storage media such as a hard disk drive (HDD), a solid state drive (SSD), a flash memory, a random access memory (RAM), and a read only memory (ROM). The storage unit 28 may or may not be expressed as a memory.

[0089] Terminal 20 stores program P in storage unit 28, and by executing this program P, control unit 21 executes the processes of each unit included in control unit 21. In other words, program P stored in storage unit 28 causes terminal 20 to realize each function executed by control unit 21. Furthermore, this program P may or may not be expressed as a program module.

[0090] (2) HW Configuration of Server FIG. 1-1 shows an example of the HW configuration of the server 10. The server 10 includes, for example, a control unit 11 (CPU), a memory unit 15, a communication I / F 14 (interface), an input / output unit 12, a display unit 13, and a clock unit 19. The HW components of the server 10 are connected to each other, for example, via a bus B. Note that the HW of the server 10 does not necessarily have to include all of the components. For example, the HW of the server 10 may or may not be configured so that individual components or multiple components can be removed.

[0091] The control unit 11 has circuits that are physically structured to execute the functions realized by the codes or instructions contained in the program, and is realized, for example, by a data processing device built into hardware.

[0092] The control unit 11 is typically a central processing unit (CPU), but may also be a microprocessor, a processor core, a multiprocessor, an ASIC, an FPGA, or other devices. In the present disclosure, the control unit 11 is not limited to these.

[0093] The storage unit 15 has a function of storing various programs and various data required for the operation of the server 10. The storage unit 15 is realized by various storage media such as an HDD, an SSD, or a flash memory. However, in the present disclosure, the storage unit 15 is not limited to these. Furthermore, the storage unit 15 may or may not be expressed as a memory.

[0094] The communication I / F 14 transmits and receives various data via the network 30. The communication may be performed either wired or wirelessly, and any communication protocol may be used as long as mutual communication is possible. The communication I / F 14 has a function of communicating with various devices, such as the terminal 20, via the network 30. The communication I / F 14 transmits various data to various devices, such as the terminal 20, in accordance with instructions from the control unit 11. The communication I / F 14 also receives various data transmitted from various devices, such as the terminal 20, and transmits it to the control unit 11. The communication I / F 14 may also be simply referred to as a communication unit. When the communication I / F 14 is configured as a physically structured circuit, it may also be referred to as a communication circuit.

[0095] The input / output unit 12 includes a device for inputting various operations to the server 10, a device for outputting processing results processed by the server 10, etc. The input / output unit 12 may be an integrated input unit and an output unit, or may be separate input unit and output unit, or may not be the same.

[0096] The input unit is realized by any one of or a combination of all types of devices that can accept input from a user and transmit information related to the input to the control unit 11. The input unit is typically realized by hardware keys such as a keyboard or a pointing device such as a mouse. Note that the input unit may or may not include, for example, a touch panel, a camera (for operation input via moving images), or a microphone (for operation input via voice).

[0097] The output unit is realized by any one or a combination of all types of devices that can output the processing results processed by the control unit 11. Examples of the output unit include a touch panel, a touch display, a speaker (sound output), a lens (for example, 3D (three dimensions) output or hologram output), a printer, etc.

[0098] By way of example only, the input / output unit 12 includes a display unit 13, for example.

[0099] The display unit 13 is realized by a display or the like. The display is typically realized by a monitor (for example, a liquid crystal display or an organic electroluminescence display (OELD)). The display may or may not be a head-mounted display (HDM) or the like. These displays may or may not be capable of displaying display data in 3D. In the present disclosure, the display is not limited to these.

[0100] The clock unit 19 is a built-in clock of the server 10 and outputs time information (timekeeping information). The clock unit 19 is configured to include, for example, an RTC (Real Time Clock) as a hardware clock, a system clock, etc. The clock unit 19 can also be expressed as, for example, a timekeeping unit or a time information detection unit.

[0101] (3) Others The server 10 stores a program P in the storage unit 15, and by executing this program P, the control unit 11 executes the processes of each unit included in the control unit 11. In other words, the program P stored in the storage unit 15 causes the server 10 to realize each function executed by the control unit 11. This program P may or may not be expressed as a program module. The same applies to other devices.

[0102] In each embodiment of the present disclosure, the description will be given assuming that the CPU of the terminal 20 and / or the server 10 executes the program P to realize the present invention. The same applies to other devices.

[0103] The control unit 21 of the terminal 20 and / or the control unit 11 of the server 10 may or may not realize each process not only by a CPU having a control circuit, but also by a logic circuit (hardware) formed in an integrated circuit (IC (Integrated Circuit) chip, LSI (Large Scale Integration)), or a dedicated circuit. These circuits may be realized by one or more integrated circuits, and multiple processes shown in each embodiment may or may not be realized by a single integrated circuit. LSIs may also be referred to as VLSIs, super LSIs, ultra LSIs, etc., depending on the level of integration. Therefore, the control unit 21 may or may not be referred to as a control circuit. The same applies to other devices.

[0104] Furthermore, the program P (e.g., a software program, a computer program, or a program module) of each embodiment of the present disclosure may or may not be provided in a state stored in a computer-readable storage medium. The storage medium can store the program P in a "non-transitory tangible medium." The program P may or may not be intended to realize part of the functions of each embodiment of the present disclosure. Furthermore, the program P may or may not be a so-called difference file (difference program) that can realize the functions of each embodiment of the present disclosure in combination with a program P already recorded on a storage medium.

[0105] The storage medium may include one or more semiconductor-based or other integrated circuits (ICs) (such as, for example, field programmable gate arrays (FPGAs) or application-specific ICs (ASICs)), hard disk drives (HDDs), hybrid hard drives (HHDs), optical disks, optical disk drives (ODDs), magneto-optical disks, magneto-optical drives, floppy diskettes, floppy disk drives (FDDs), magnetic tapes, solid-state drives (SSDs), RAM drives, secure digital cards, or drives, any other suitable storage media, or any suitable combination of two or more thereof. The storage medium may be volatile, non-volatile, or a combination of volatile and non-volatile, where appropriate. Note that the storage medium is not limited to these examples and may be any device or medium capable of storing the program P. Furthermore, the storage medium may or may not be referred to as a memory.

[0106] The server 10 and / or the terminal 20 can realize the functions of the multiple functional units shown in each embodiment by reading the program P stored in the storage medium and executing the read program P. The same applies to other devices.

[0107] Furthermore, the program P of the present disclosure may or may not be provided to the server 10 and / or the terminal 20 via any transmission medium capable of transmitting a program (such as a communication network or broadcast waves). The server 10 and / or the terminal 20, for example, executes the program P downloaded via the Internet or the like to realize the functions of the multiple functional units shown in each embodiment. The same applies to other devices.

[0108] Furthermore, each embodiment of the present disclosure may be realized in the form of a data signal in which the program P is embodied by electronic transmission. At least a portion of the processing in the server 10 and / or the terminal 20 may, or may not, be realized by cloud computing consisting of one or more computers. At least a portion or all of the processing in the terminal 20 may, or may not, be configured to be performed by the server 10. In this case, at least a portion or all of the processing of each functional unit of the control unit 21 of the terminal 20 may, or may not, be configured to be performed by the server 10. At least a portion or all of the processing in the server 10 may, or may not, be configured to be performed by the terminal 20. In this case, at least a portion or all of the processing of each functional unit of the control unit 11 of the server 10 may, or may not, be configured to be performed by the terminal 20. Unless explicitly stated otherwise, a determination configuration in the embodiments of the present disclosure is not essential, and a predetermined process may, or may not, be executed when a determination condition is met, or a predetermined process may, or may not, be executed when a determination condition is not met.

[0109] The program of the present disclosure is implemented using, for example, a scripting language such as ActionScript or JavaScript (registered trademark), a compiler language such as Objective-C or Java (registered trademark), or a markup language such as HTML Living Standard.

[0110] [Functional Configuration of Each Device] (1) Functional Configuration of Server Fig. 1-2 is a diagram showing an example of functions realized by the control unit 11 of the server 10 in this embodiment. The control unit 11 includes, as functional units, an application management processing unit 111 and an image generation model unit 113, for example.

[0111] The application management processing unit 111 has a function of executing application management processing in accordance with an application management processing program 151 stored in the storage unit 15, for example.

[0112] The image generation model unit 113 has a function of generating and outputting an image based on, for example, input keywords or sentences (which may be called "prompts"). The image generation model unit 113 may be configured, for example, with a text-to-image model (trained neural network model) such as DALL-E, Stable Diffusion, or Imagen.

[0113] A prompt may be a command or text instruction that causes an image generation device to generate an image. A prompt may also be a factor that determines the content of the image generated by the image generation device. The prompt may or may not be entered by a user.

[0114] It should be noted that the image generation model unit 113 (which is not a limitation and is an example of an image generation device that generates an image based on a prompt) is not limited to being provided in the server 10. As a non-limiting example, the image generation device may be provided outside the server 10 (which is not a limitation and is an example of an information processing device). Furthermore, as a non-limiting example, the image generation device and the information processing device may communicate with each other to advance the processing in this embodiment.

[0115] 1-3 is a diagram showing an example of information stored in the storage unit 15 of the server 10 in this embodiment. The storage unit 15 stores, for example, an application management processing program 151 executed as application management processing, account registration data 153, stamp image additional learning model management data 155, stamp image generation prompt management data 157, and a stamp set management database 159.

[0116] The account registration data 153 is registration data related to the account of an application (in this embodiment, a messaging application), and an example of the data configuration is shown in Fig. 1-4. For example, the account registration data 153 stores a user name, an application ID, and other registration information in association with each other.

[0117] The user name is the name of the account of the terminal 20 that uses this application, and for example, the name that the user of the terminal 20 registers when using the application is stored.

[0118] The application ID is information used to identify an application account, or the account itself. This application ID is preferably a value unique to each account, and for example, a unique value (proper value) is set and stored for each account by the server 10. The application ID is information associated with the terminal 20 or the user of the terminal 20, and is an example of information related to the terminal or the user of the terminal.

[0119] Other registration information may include, for example, various types of information such as identification information for identifying terminal 20, the telephone number (terminal telephone number) of terminal 20, an email address (terminal email address), and authentication information such as passwords (login password, authentication password, etc.) used for various authentications in the application.

[0120] The identification information for identifying the terminal 20 may be, for example, a terminal ID (for example, an IMEI (International Mobile Equipment Identity)).

[0121] For example, the other registration information may be stored in association with user information.

[0122] Note that the application ID may or may not be replaced with a "user ID." Furthermore, if the application allows only one account to be registered per terminal 20, for example, the "identification information for identifying the terminal 20 = identification information for identifying the user of the terminal 20 = application ID" may be used.

[0123] Also, for example, it may or may not be possible to assign multiple terminal IDs to one application ID. In this case, it may or may not be possible to simultaneously launch an application on multiple terminals 20 using one application ID as an identification (login) target.

[0124] It is also possible to apply a method of managing accounts using information such as a terminal telephone number instead of various IDs such as an application ID. In this case, instead of storing information on IDs such as an application ID in the account registration data 153, information on terminal telephone numbers can be stored in the account registration data 153. It is also possible to have one-to-one correspondence between information on IDs such as application IDs and information on terminal telephone numbers, without replacing information on IDs such as application IDs with information on terminal telephone numbers, but this is not necessary.

[0125] In the following examples, for the sake of simplicity, it is assumed that one account is registered for one terminal 20. In this case, as described above, "identification information for identifying the terminal 20 = identification information for identifying the user of the terminal 20 = application ID," and therefore, the term "user of the account" used in the following description may or may not be substantially synonymous with "terminal of the account."

[0126] The stamp image additional learning model management data 155 is management data related to an image generation model (referred to as an "additional learning model") that has undergone additional learning of a base image in the image generation model unit 113, and an example of the data configuration is shown in Figure 1-5. For example, the stamp image additional learning model management data 155 stores an additional learning model ID, an additional learning model URI, an instance prompt, a class prompt, and other model information in association with each other.

[0127] The additional learning model ID is information used to identify an additional learning model. This additional learning model ID is preferably a unique value for each account, and for example, a unique value (specific value) is set and stored for each model by the server 10. Note that an instance prompt may be used as the additional learning model ID.

[0128] The additional learning model URI is, for example, information indicating a storage location of a model (data file) in which the structure and parameters of the additional learning model are described. For example, the image generation model unit 113 may be configured to read the model specified by the additional learning model URI and drive an additional learning model that can generate an additional learning model for a base image that has undergone additional learning.

[0129] The instance prompt is a character string that serves as a keyword for generating an image containing an object commonly included in the base images that have undergone additional training in the additional training model. The instance prompt may be referred to as an identifier for identifying an object, or as information for specifying the object of the image to be generated. The instance prompt may also be referred to as an "instance token." This instance prompt is preferably a unique value (specific value) for each base image (object, such as a character or concept, commonly included in the base images) for which additional training is performed. For example, a unique value (specific value) is set and stored for each base image by the server 10. Furthermore, the instance prompt is preferably set to a character string (for example, random character examples such as "sts," "usu," or "lvl") that does not match any previously trained prompt token (for example, a general word, etc.).

[0130] The class prompt is a character string for specifying the class (category) of an object in a base image that has undergone additional training in the additional training model. The class prompt may also be called a "class token." For example, the class prompt is set and stored as a character string corresponding to the category of the base image identified by an image identification unit (not shown) of the server 10.

[0131] For example, if a "rabbit character" appears as an object in the base image, the class prompt may be set to "rabbit." The instance prompt may be set to "cpc," which is an identification string that does not match common words. For example, if a "bear character" appears as an object in the base image, the class prompt may be set to "bear." The instance prompt may be set to "usu," which is an identification string.

[0132] It is also possible to avoid storing class prompts in association with instance prompts.

[0133] Other model information may include, for example, the model name associated with the instance prompt, the name and description of the object that can be generated by the model (e.g., "rabbit character"), information about all or part of the base image used for additional learning, and example images generated by applying the instance prompt to the additional learning model.

[0134] The stamp image generation prompt management data 157 is data for managing prompt templates for generating stamp set images, and an example of the data configuration of the stamp image generation prompt management data 157A is shown in Figure 1-6. For example, the stamp image generation prompt management data 157A stores a prompt template ID, a positive template prompt, and other information in association with each other.

[0135] The prompt template ID is information used to identify a prompt template used to generate a stamp image, and for example, a unique value (proper value) is set and stored for each template by the server 10.

[0136] The positive template prompt is, for example, a character string that serves as a template for a prompt, set by a user of the server 10. For example, the positive template prompt may include information indicating the replacement position between the instance prompt and the class prompt (for example, "<X>" in this figure). Furthermore, at least some prompts (tokens) may be set differently in positive template prompts with different prompt template IDs. In this way, different prompts may be generated from positive template prompts specified by different prompt template IDs, and as a result, different images may be generated.

[0137] For example, a negative template prompt may be further stored in association with the prompt template ID. The negative template prompt is, for example, a character string that serves as a template for a negative prompt, set by a user of the server 10. For example, the negative template prompt may include prompts such as "text, lower, error, NSFW, ...," which are elements that should not be included in order to generate an image suitable for a stamp image.

[0138] For example, when a stamp template ID, an instance prompt, and a class prompt are specified, the control unit 11 of the server 10 creates a prompt (which may be called a "positive prompt") that includes the instance prompt, the class prompt, and a positive template prompt. For example, the instance prompt and the class prompt may be applied to the "<X>" portion of the positive template prompt to generate a positive prompt. Also, a negative prompt that includes a negative template prompt may be created.

[0139] For example, in Figure 1-6, if the instance prompt is specified as "cpc" and the class prompt is specified as "rabbit," when the prompt template ID "T0001" is set, the prompt created will be "illustration of cpc rabbit, best quality, full shot, ...".

[0140] The incremental learning model may then generate images based on the prompts.

[0141] The incremental learning model may generate images based on positive and negative prompts.

[0142] Alternatively, unlike the present example, positive template prompts and negative template prompts common to each stamp template may be stored as common positive prompts and common negative prompts without being associated with the stamp template ID. Template prompts with different content and other registered information may be stored in association with the stamp template ID. It is also possible not to store common positive prompts or common negative prompts. In this case, for example, when a stamp template ID, an instance prompt, and a class prompt are specified, the control unit 11 of the server 10 creates, for example, a positive prompt including the instance prompt, the class prompt, the template prompt, and the common positive prompt, or a negative prompt including the common negative prompt.

[0143] The stamp set management database 159 is, for example, a database for managing information about stamp sets created by users, and an example of its data structure is shown in Fig. 1-7. The stamp set management database 159 stores stamp set management data as management data for each created stamp set.

[0144] In each stamp set management data, for example, a stamp set ID, a stamp set name, a creator application ID, other stamp set information, and stamp management data are stored in association with each other.

[0145] The stamp set ID is information for identifying the created stamp set, and for example, when a stamp set is registered, a unique value (proper value) is set and stored for each stamp set by the server 10.

[0146] The stamp set name is the name of the stamp set, and for example, the name set by the user when registering the stamp set is stored.

[0147] The creator application ID is information for identifying the user who created the stamp set, and for example, stores the application ID of the user who created the stamp set, which is registered in the account registration data 153. Note that if the user who created the stamp set is a user of the server 10 (the operator of the service), the creator application ID may store identification information indicating the operator.

[0148] Other stamp set information may be able to store various information, such as a description of the stamp set, the selling price of the stamp set, identification information of the additional learning model used to create the stamp set, and information about the seed value used when generating the image of the additional learning model.

[0149] The stamp management data is data for managing information about each stamp included in the stamp set. For example, the stamp management data stores a stamp ID, stamp text, and stamp image URI in association with each other.

[0150] The stamp ID is information for identifying a stamp in a stamp set, and is, for example, set and stored as a unique value (for example, a serial number) for each stamp by the server 10. For example, by specifying the stamp set ID and the stamp ID, an individual stamp (stamp image) can be specified.

[0151] Stamp text is a character string associated with a stamp image. Stamp text may be text that represents the meaning of the stamp image. Stamp text may be a character string that serves as a query key used when calling up the stamp image. Stamp text may also be text contained in the stamp image.

[0152] The stamp image URI is information indicating the storage location of the created stamp image. For example, when a stamp image is created, the server 10 stores the location where the image file of the stamp image is saved.

[0153] (2) Functional Configuration of the Terminal Fig. 1-8 is a diagram showing an example of functions realized by the control unit 21 of the terminal 20 in this embodiment. The control unit 21 includes, as a functional unit, an application processing unit 211 for executing application processing in accordance with an application processing program 281 stored in the storage unit 28, for example.

[0154] 1-9 is a diagram showing an example of data stored in the storage unit 28 of the terminal 20 in this embodiment. The storage unit 28 stores, for example, an application processing program 281 to be executed as application processing, and an application ID 283 corresponding to the terminal 20 or the account of the user of the terminal 20.

[0155] <Display Screen> Display screen examples will be described below. The transition of the display screens described below is merely an example of the transition of the display screens for realizing the method of the present disclosure. In the transition of the display screens exemplified below, the display of some display screens may be omitted, or other display screens may be added. Furthermore, the terms used in the display screens described below may differ from the terms used in the processes described thereafter (some terms may not be consistent).

[0156] In the following, the terminal 20 is illustrated as an example in which the terminal 20 is a smartphone equipped with a display unit 24 having a vertically long display. For example, the smartphone has a touch panel that functions as an input unit, which is disposed opposite the display, constituting a touch screen. When an element such as an icon, button, item, or input area is displayed on the display, and a part of the touch panel that faces the area where the element is displayed is operated by a user, a program associated with the element or a subroutine of the program may be executed.

[0157] Note that the flow of screen transitions may be explained by showing predetermined screens displayed on different terminals 20 within one drawing or across multiple drawings. In this case, for example, a predetermined screen may be displayed on a different terminal 20 based on (triggered by) at least one of the following: - Automatically displayed - Displayed when some kind of notification (including push notification, etc.; the same applies below) is sent to the terminal 20 and the user performs an operation in response to the notification - No notification is sent to the terminal 20, but the user performs a predetermined operation on an application - Displayed when some kind of notification is sent to the terminal 20 and the user performs a predetermined operation on the application that sent the notification

[0158] In the following, an example of an account that creates a stamp set will be user A. A's terminal 20A will be used as an example. In this example, the stamp set is created within a messaging application, and the name of the messaging application will be referred to as "Messaging App." Note that the stamp set may also be created within a stamp set creation application, for example, different from the messaging application. In this case, a similar screen configuration may also be used.

[0159] 1-10 to 1-12 are diagrams showing examples of screens displayed on the display unit 24 of the terminal 20A in this embodiment. In this example, a user A.A. of the terminal 20A creates a stamp set image using an image of his / her pet dog, a Pomeranian, that he / she captured in advance.

[0160] The left side of Figure 1-10 shows a home screen of a messaging application (not limited to, but an example of, a chat application), with the name of the messaging application displayed in the top center of the screen, and the icon image and user name of the messaging application of the user of this terminal 20A displayed in the top right of the screen.

[0161] Below that is an area showing the current position, page, etc. within the messaging application (hereinafter collectively referred to as the "in-app position display area"). In this example, the word "Home" is displayed because this is the home screen.

[0162] Below the in-app location display area, a friend / group list display area is configured to display a list of friends and groups in the messaging application. In this example, the friend / group list display area is configured to display a friend list for displaying a list of friends of the account of this terminal 20 and a group list for displaying a list of groups that include the account of this terminal 20 as a member.

[0163] Below the friend / group list display area is a service list display area that displays a list of services provided within the messaging application or that can be used in conjunction with the messaging application. In this example, the service list display area displays a "Stamp Set Creation" service for creating stamp sets and a "Dress Up" service for changing designs in chat rooms, etc.

[0164] For example, when the "Create stamp set" service button is tapped by the user, the display changes to the stamp set creation screen shown in the center of FIG. 1-10.

[0165] For example, this stamp set creation screen is configured to display, below the in-app location display area, a thumbnail image setting area for setting a thumbnail image of the stamp set to be created, a stamp set name setting area for setting a title (stamp set name), and a stamp set description setting area below that for setting a description of the stamp to be created.

[0166] Below the stamp set description setting area, an individual stamp editing area STR1 is displayed for creating and editing the individual stamps (stamp images) that make up the stamp set. In this image, individual stamps have not been created in the individual stamp editing area STR1, so the frames representing the eight stamps are displayed with dotted lines. For example, by tapping the frame representing an individual stamp, you can create and edit individual stamps within the stamp set.

[0167] Below the individual stamp editing area STR1, there are displayed an "Automatic Stamp Set Creation" button SGB1 for generating stamp images in bulk and automatically creating a stamp set, and a "Stamp Set Creation Complete" button SRB1 for registering the created stamp set. On this screen, for example, because the information required to register a stamp set has not been entered in the individual stamp editing area STR1 or the stamp set name setting area, the "Stamp Set Creation Complete" button SRB1 is displayed in a grayed-out state, and its function is disabled.

[0168] For example, when the "Automatic Stamp Set Generation" button SGB1 is tapped by the user, the display changes to the automatic stamp set generation method selection screen shown on the right side of FIG. 1-10.

[0169] This stamp set automatic generation method selection screen displays, for example, that the method of generating stamp images can be selected, for example, by using a trained additional learning model prepared by the user or administrator, or by registering a new base image and creating an additional learning model. This screen displays that the available additional learning models can be selected from a "rabbit character" (for example, an additional learning model with additional learning model ID "M0001") and a "bear character" (for example, an additional learning model with additional learning model ID "M0002").

[0170] The additional learning models selectable on the stamp set automatic generation method selection screen may be set as follows: (M1). Additional learning model created by the service operator (M2). Additional learning model created in the past by the user himself / herself (M3). Additional learning model created in the past by a user who belongs to the same group as the user (M4). Additional learning model created in the past by any user (M5). Any combination of (M1) to (M4).

[0171] Also, for example, it may be possible to make it impossible to select a trained additional learning model on the stamp set automatic generation method selection screen.

[0172] This screen also displays that the method of registering a base image can be selected from "selecting an image from an album" and "taking a photo." When "selecting an image from an album," for example, image data stored in the storage unit 28 may be designated as the base image. When "taking a photo," for example, image data captured by the imaging unit 27 may be designated as the base image.

[0173] For example, when the user taps the "Select Image from Album" button, the display changes to, for example, the base image selection screen shown on the left side of FIG. 1-11. This screen is configured to display an album image selection area below the in-app location display area, which allows the user to select an image to be used as a base image from images in the album. Also, below the album image selection area, a "Select Image" button SLB1 is displayed for selecting an image selected in the album image selection area (for example, an image that has been checked by tapping on it) as a base image.

[0174] For example, when nine images of Pomeranians are selected in the album image selection area and the "Select Image" button SLB1 is tapped, the control unit 11 of the server 10 sets an instance prompt and a class prompt based on the selected base image. Then, for example, the image generation model unit 113 executes a stamp image generation model additional learning process so that an image similar to the base image is generated in association with the instance prompt and the class prompt.

[0175] Then, for example, the display changes to the base image learning progress display screen shown in the center of Figure 1-11. On this screen, for example, a list of images selected in the album image selection area is displayed below the message "Learning base images." Also, below that, a progress bar indicating the progress of additional learning is displayed. For example, a "Stop learning" function button for stopping the additional learning process is displayed below the progress bar.

[0176] Additionally, a "Generate Stamp Set Image" button IGB1 for generating a stamp set image using the trained additional learning model is displayed at the bottom of the base image learning progress display screen. On this screen, since the additional learning process has not yet finished, the "Generate Stamp Set Image" button IGB1 is displayed in a grayed-out state, and its function is disabled.

[0177] For example, after time has passed and the additional learning process is completed, the display changes to the base image learning progress display screen shown on the right side of Figure 1-11. On this screen, the "Stop Learning" function button disappears and a message saying "Learning Completed" is displayed. In addition, the display mode of the "Create Stamp Set Image" button IGB1 changes, and the function is enabled.

[0178] For example, the base image learning progress display screen may be configured to return to the main menu or the like during the additional learning process. Also, for example, the base image learning progress display screen may be configured to return from the stamp set creation screen or the like.

[0179] Furthermore, when the additional learning process is completed, an additional learning completion notification indicating that the learning is completed may be sent to the user who registered the base image from the OA of the stamp set creation application, for example. Then, for example, the user may be able to return to the base image learning progress display screen based on a tap operation on the additional learning completion notification.

[0180] For example, when the "Generate Stamp Set Images" button IGB1 is tapped by the user, the display changes to, for example, the base image learning progress display screen shown on the left side of FIG. 1-12. On this screen, for example, a stamp number selection area NSR1 for selecting the number of stamp images to be generated is displayed in a raised manner, superimposed on the base image learning progress display screen after the additional learning process is completed. For example, the stamp number selection area NSR1 displays a row of selection buttons ranging from "8" to "40" as the number of selectable stamp images to be generated. Note that, for example, the stamp number selection area NSR1 may be configured to allow the user to input and specify the number of stamp images to be generated.

[0181] For example, when the user taps the button for selecting the number of stamps to be generated, "N (here, "N" = "8")," a stamp image generation process is executed based on a positive template prompt specified by any N prompt template IDs from the stamp image generation prompt management data 157A, and N stamp images are generated. Then, for example, the display changes to the generated stamp set image display screen shown in the center of FIG. 1-12.

[0182] On this screen, a specified number of generated stamp images are displayed side by side in a generated stamp image confirmation display area below the message "Stamp set image generated." Below the generated stamp image confirmation display area, a "Save trained AI model" button MSB1 for saving an additional learning model that has undergone additional learning processing based on the base image, a "Regenerate image" button RGB1 for regenerating the stamp image, and a "Confirm" button IAB1 for setting the stamp set image displayed in the generated stamp image confirmation display area as a stamp set are displayed side by side.

[0183] For example, when the "Save Trained AI Model" button MSB1 is tapped, a record may be added to the stamp image additional learning model management data 155, and the additional learning model may be stored in association with the instance prompt and class prompt set during learning. Note that when the additional learning process is completed, the additional learning model may be automatically saved in association with the stamp image additional learning model management data 155. By saving the additional learning model, the user can generate a stamp image based on a favorite base image by calling up the additional learning model, thereby saving the effort and time required for re-learning.

[0184] For example, when the "Regenerate Image" button RGB1 is tapped, the seed value may be reset in the additional learning model, and the stamp image generation process may be executed again. Also, for example, when the "Regenerate Image" button RGB1 is tapped, the stamp image generation process may be executed again based on a positive template prompt specified by a prompt template ID different from the one previously specified. This allows image generation to be repeated until the image desired by the user is generated.

[0185] As an example, it may be possible to individually select, by tapping or the like, a stamp image to be regenerated or a stamp image to be saved as is without being regenerated from among the multiple stamp images displayed in the generated stamp image confirmation display area.

[0186] Also, for example, when the "regenerate image" button RGB1 is tapped, the stamp number selection area NSR1 may be displayed again, allowing the number of stamp images included in the stamp set to be re-specified.

[0187] Note that the stamp image generation process may generate "M" images, which is more than the specified "N." For example, the user may be allowed to select "N" stamp images to be used as stamp set images from among the "M" stamp images displayed in the generated stamp image confirmation display area.

[0188] For example, when the user taps the "OK" button IAB1, the display changes to the stamp set creation screen shown on the right side of FIG. 1-12. On this screen, for example, the stamp set image confirmed in the generated stamp image confirmation display area is applied and displayed in the individual stamp editing area STR1. Also, for example, the first image of the generated stamp images is automatically set and displayed in the thumbnail image setting area.

[0189] In addition, the stamp set name setting area and stamp set description setting area may be configured to automatically input title and description templates based on the results of image recognition (image description analysis) of the base image and the generated stamp set image, for example.

[0190] Furthermore, by tapping an individual stamp image displayed in the individual stamp editing area STR1, it may be possible to correct or process the stamp image or input stamp text.

[0191] For example, when the "Stamp set creation complete" button SRB1 is tapped, stamp management data is added to the stamp set management database 159 and each piece of data is stored.

[0192] In addition, in the stamp management data, a part or all of the positive template prompt used to generate the stamp image may be reflected and stored as stamp text.

[0193] <Processing> Figure 1-13 is a flowchart showing an example of the flow of processing executed by each device in this embodiment. Starting from the left, this figure shows an example of processing executed by the control unit 21 of user A's terminal 20A, and an example of processing executed by the control unit 11 of the server 10. Note that the processing described below is merely an example of processing for realizing the method of the present disclosure, and is not limited to this. Furthermore, other steps may be added to the processing described below, or some steps may be omitted (deleted) from the processing described below.

[0194] First, the control unit 21 of the terminal 20A sends stamp set image generation request information for generating a stamp set image to the server 10 via the communication I / F 22 based on input to the input / output unit 23 of the terminal 20A (for example, user input (user operation input, sound input, etc.); same below) (A110).

[0195] When stamp set image generation request information is received from terminal 20A via communication I / F 14, control unit 11 of server 10 sends base image set transmission request information to terminal 20A via communication I / F 14, requesting the transmission of a base image including one or more additional learning images (S110).

[0196] When the base image set transmission request information is received from the server 10 via the communication I / F 22, the control unit 21 of the terminal 20A outputs the received base image set transmission request information. In the following, an example of information output will be exemplified in which the information is displayed on the display unit 24, but the control unit 21 of the terminal 20A may also, for example, output the received information as sound from the sound output unit 26.

[0197] Based on the user input in response to the output base image set transmission request information, the control unit 21 of the terminal 20A transmits base image set information including one or more pieces of image information to the server 10 via the communication I / F 22 (A120).

[0198] When the base image set information is received from each terminal 20 via the communication I / F 14, the control unit 11 of the server 10 executes an instance prompt setting process (S120). In the instance prompt setting process, the control unit 11 of the server 10 randomly generates a character string of a predetermined length (e.g., three characters) and compares it with a natural language dictionary (e.g., an English word dictionary) (not shown) stored in the storage unit 15. If the generated character string is not included in the natural language dictionary, the control unit 11 sets the generated character string as the instance prompt.

[0199] In the instance prompt setting process, the control unit 11 of the server 10 may select an instance prompt from preset instance prompt candidate character strings, for example.

[0200] In addition, in the instance prompt setting process, the control unit 11 of the server 10 may, for example, refer to the stamp image additional learning model management data 155 and set the instance prompt so that it does not overlap with an instance prompt that has already been stored.

[0201] Then, the control unit 11 of the server 10 executes a class prompt setting process (S130). In the class prompt setting process, the control unit 11 of the server 10 analyzes the base image received from the terminal 20 using an image identification unit (not shown), and sets a character string corresponding to the category of the base image as a class prompt. Note that the image recognition unit may be configured using an image classification model such as Vision Transformer or DeiT (Data-efficient image Transformers), for example.

[0202] When a class prompt is set, the control unit 11 of the server 10 generates a predetermined number of regularized images (for example, "the number of images in the base image x 10") using the set class prompt as a prompt by the image generation model unit 113. For example, if the class prompt is "dog," the regularized images will be images of dogs of any breed.

[0203] If class prompts are not used in the stamp image generation model additional learning process or the stamp image generation process, the class prompt setting process may not be executed.

[0204] For example, when an instance prompt or a class prompt for a base image is set, the control unit 11 of the server 10 executes a stamp image generation model additional learning process (S140).

[0205] In the stamp image generation model additional learning process, the control unit 11 of the server 10 generates, for example, a pre-learning model in which the structure and parameters are copied based on the image generation model unit 113. Then, the control unit 11 of the server 10 additionally learns a base image using, for example, DreamBooth, for the pre-learning model, using instance prompts, class prompts, and regularized images. DreamBooth may be implemented, for example, using the method described in "arXiv:2208.12242." This makes it possible to create an additional learning model that can generate images by reflecting objects such as characters and concepts common to the base image in the instance prompts.

[0206] In the stamp image generation model additional learning process, the control unit 11 of the server 10 may, for example, not use a regularized image, but perform fine tuning on the model before additional learning to additionally learn the base image.

[0207] Furthermore, in the stamp image generation model additional learning process, the control unit 11 of the server 10 may, for example, apply Low-Rank Adaptation (LoRA) to additionally learn the base image in DreamBooth or fine tuning. LoRA may be realized, for example, using the method described in "arXiv:2106.09685." This reduces the resources required for additional learning and shortens the learning time. Furthermore, when saving an additional learning model, if the entire model is fine-tuned, all parameters of the additional learning model must be saved, whereas with LoRA, only differential parameters from the pre-learning model need to be saved. This also has the advantage of allowing for efficient saving of additional learning models.

[0208] For example, when the stamp image generation model additional learning process is completed, the control unit 11 of the server 10 transmits additional learning completion information indicating that an additional learning model that reflects (learns) the base image has been created to the terminal 20A via the communication I / F 14 (S150). Note that the control unit 11 of the server 10 may add and store the additional learning model that has learned the base image to the stamp image additional learning model management data 155.

[0209] When the additional learning end information is received from the server 10 via the communication I / F 22, the control unit 21 of the terminal 20A outputs the received additional learning end information (for example, displays it on the display unit 24) (A130).

[0210] For example, based on user input, the control unit 21 of terminal 20A receives stamp set image number selection information, which is information regarding the number of stamp images included in the stamp set, and transmits the stamp set image number selection information to the server 10 via the communication I / F 22 (A140).

[0211] When the communication I / F 14 receives the stamp set image number selection information from the terminal 20A, the control unit 11 of the server 10 executes a stamp image generation prompt set setting process (S160). In the stamp image generation prompt set setting process, the control unit 11 of the server 10, for example, randomly selects prompt template IDs equal to the number of stamp images to be generated specified in the stamp set image number selection information. Note that the prompt template IDs may also be selected in ascending or descending order, for example. The control unit 11 of the server 10 then generates prompts (positive prompts) equal to the number of stamp images to be generated based on the positive template prompts associated with the prompt template IDs and the instance prompts and class prompts used in training the additional learning model.

[0212] In the stamp image generation prompt set setting process, the control unit 11 of the server 10 may select a prompt template ID from a set of predetermined prompt template IDs for each number of stamp images to be generated, which is specified in the stamp set image number selection information. This makes it possible to prevent the generation of similar prompts.

[0213] In the stamp image generation prompt set setting process, the control unit 11 of the server 10 may set a prompt set based on, for example, a positive template prompt linked to a prompt template ID and a randomly generated or selected class prompt. For example, a pre-set common noun (such as "apple" or "banana") may be randomly selected and applied to the "<X>" portion of the positive template prompt. In this case, the base image may include common elements such as common objects, or may not include common elements.

[0214] Then, the control unit 11 of the server 10 executes a stamp image generation process (S170). In the stamp image generation process, the control unit 11 of the server 10, for example, inputs a prompt to the additional learning model and acquires the generated image as a stamp image. This process is repeated the number of times corresponding to the number of stamp set images. Note that the inference process in the additional learning model may, for example, be executed by the image generation model unit 113. Also, the inference process in the additional learning model may, for example, be executed by the control unit 11 of the server 10.

[0215] Then, the control unit 11 of the server 10 transmits stamp set image information including the generated designated number of stamp images to the terminal 20A via the communication I / F 14 (S180).

[0216] When the stamp set image information is received from the server 10 via the communication I / F 22, the control unit 21 of the terminal 20A causes the display unit 24 to display the received stamp set image information (A150), for example.

[0217] Then, the control unit 21 of the terminal 20A determines whether to end the process (A190). If it is determined that the process should be continued (A190: NO), the control unit 21 of the terminal 20A returns the process to A110, for example. On the other hand, if it is determined that the process should be ended (A190: YES), the control unit 21 of the terminal 20A ends the process.

[0218] The control unit 11 of the server 10 then determines whether to terminate the process (S190). If it determines to continue the process (S190: NO), the control unit 11 of the server 10 waits for reception of stamp set image generation request information, and returns to S110. On the other hand, if it determines to terminate the process (S190: YES), the control unit 11 of the server 10 terminates the process.

[0219] As an example, if it is determined that a request for regeneration of a stamp set image has been made based on user input for the displayed stamp set image information, the control unit 21 of terminal 20A may be configured to send stamp set image regeneration request information requesting regeneration of the stamp set image to server 10 via communication I / F 22.

[0220] When it is determined that stamp set image regeneration request information has been received from terminal A via the communication I / F 14, the control unit 11 of the server 10 may, for example, re-execute the stamp image generation process and re-send the stamp set image information. When re-executing the stamp image generation process, the control unit 11 of the server 10 may change the seed value. Furthermore, the stamp image generation process may be executed using a prompt based on a different prompt template ID.

[0221] In S180, the control unit 11 of the server 10 may transmit the stamp set image information to a terminal 20 different from terminal 20A via the communication I / F 14. The terminal 20 different from terminal 20A may be a terminal 20 linked to user A.A's account, or may be a terminal 20 linked to an account of a user different from user A.A. The stamp set image information may also be transmitted to both terminal 20A and a terminal 20 different from terminal 20A. Furthermore, the destination of the stamp set image information including the generated stamp image may be set to one or more arbitrary terminals 20 based on user input at terminal 20A, for example and not by way of limitation.

[0222] Effect of First Example In this example, the server 10 (not limited to, an example of an information processing device) acquires base images (not limited to, an example of a first image set) for additional learning for the image generation model unit 113 (not limited to, an example of an image generation device that generates images based on prompts). The server 10 then causes the image generation device to learn the base images. The server 10 also sets a prompt set in a stamp image generation prompt set setting process. Then, the server 10 causes the additional learning model (not limited to, an example of an image generation device that generates images based on prompts) to generate stamp set images (not limited to, an example of a second image set) in the stamp image generation process. As an example of an effect of this example obtained by such a configuration, the information processing device can acquire the first image set and cause the image generation device to learn the first image set. The image generation device that has learned the first image set can then generate the second image set.

[0223] Furthermore, the server 10 can receive the first image set from the terminal 20A (not limited to, but an example of a first information processing device) and transmit the generated second image set to a terminal 20 (not limited to, but an example of a second information processing device) other than the first information processing device or the terminal 20A. Therefore, the information processing device can transmit the second image set to any information processing device.

[0224] In this embodiment, the server 10 acquires a base image (not limited to, but an example of a first image set) composed of images including a Pomeranian (not limited to, but an example of a first object). The server 10 then causes the image generation device to learn the base image based on an instance prompt or a class prompt (not limited to, but an example of first information) associated with the base image. In a stamp image generation prompt set setting process, the server 10 also sets a prompt set, which is a plurality of prompts based on the first information. Then, in a stamp image generation process, the server 10 causes an additional learning model (not limited to, but an example of an image generation device that generates images based on prompts) to generate stamp set images (not limited to, but an example of a second image set). Therefore, the stamp set images are generated as an image set composed of images based on a Pomeranian. As an example of an effect of this embodiment, the information processing device acquires a first image set composed of images including a first object, and causes the image generation device to generate a second image set composed of images based on the first object.

[0225] This embodiment also illustrates an example of a configuration in which the second image set is a stamp image set, which allows a user of the information processing device to easily create a stamp image set that is composed of images based on the first object.

[0226] <First Modification (1)> In the above embodiment, the server 10 unconditionally accepts the base image transmitted by the user of the terminal 20, but this is not limiting. For example, if the base image is not suitable for the additional learning process, a different base image may be selected. Also, for example, if a stamp image generated by an additional learning model trained on the base image is likely to cause copyright infringement, a different base image may be selected.

[0227] FIG. 1-14 is a diagram showing an example of a screen displayed on the display unit 24 of the terminal 20A in this modification.

[0228] The left side of Figure 1-14 shows an example of a base image selection screen similar to the left side of Figure 1-11. Here, for example, it is assumed that three photos are selected as base images. For example, when the "Select Image" button SLB1 is tapped, the display changes to the base image reset request screen shown in the center of Figure 1-14.

[0229] This screen is configured to display, for example, a list of images selected in the album image selection area below a warning message that reads "Please reselect a base image." Below that, a base image condition display area is displayed that displays predetermined conditions under which a base image can be accepted and whether the selected base image satisfies the predetermined conditions. In the base image condition display area, for example, if the predetermined conditions are not met, the text is displayed in a grayed-out display mode and a circle "OK" icon is attached. Furthermore, if the predetermined conditions are met, the text is displayed in black and a cross "NG" icon is attached.

[0230] On this screen, for example, three predetermined conditions are displayed in the base image condition display area. Also, it is displayed that one of the three predetermined conditions, the condition regarding the number of base images, is met.

[0231] Here, the predetermined condition may be set as follows, for example. For example, the predetermined condition may be set by the server 10 based on a setting input by the user of the server 10, or may be set automatically by the server 10. (C-1) When the number of base images required for additional learning is small. For example, if the number of base images is less than a set number (e.g., eight), a warning may be displayed urging the user to increase the number of base images because additional learning is likely to fail. Note that a specific number of base images, such as "Please select X more base images," may be displayed in the base image condition display area. For example, the process of determining the predetermined condition may determine whether the number of base images is less than or equal to the set number, and if the number is less than or equal to the set number, the predetermined condition may be satisfied.

[0232] (C-2) When different learning objects are included in the base image. For example, when a single base image contains multiple objects (different characters or subjects) or when different images in the base image contain different objects, additional learning is likely to fail. Therefore, a warning prompting the user to reselect the base image may be displayed. The base image condition display area may display an image containing multiple objects in the base image or multiple images containing different objects. For example, the process of determining the predetermined condition may be performed by identifying the base image using an image identification unit, and if dissimilar objects of different classes (e.g., "dog" and "cat") are detected from the base image, the predetermined condition may be satisfied.

[0233] (C-3) When a base image contains an image that constitutes copyright infringement. For example, if one or more images in the base image contain an illustration of a character or a subject that constitutes copyright infringement as an object, a warning may be displayed prompting the user to reselect the base image, since the image generated by the incremental learning model is likely to contain strong characteristics of the object that constitutes copyright infringement. Images in the base image that may constitute copyright infringement may be displayed in the base image condition display area. For example, the process of determining this predetermined condition involves constructing a copyrighted work determination filter that can determine whether or not a copyrighted work is contained in an arbitrary image using an image recognition model that has previously been trained with images of copyrighted works. Then, for example, if the base image is selected by the copyrighted work determination filter and determined to contain a copyrighted work, the predetermined condition may be satisfied. Similarly, predetermined conditions may be set so that images that violate public order and morals (e.g., pornographic images) are not accepted as base images.

[0234] (C-4) Any combination of (C-1) to (C-3).

[0235] For example, when the "Reselect Image" button SLB2 at the bottom of the screen for reselecting the base image is tapped, the display changes to the base image selection screen shown on the right side of Figure 1-14, allowing the user to reselect the base image.

[0236] 1-15 is a flowchart showing an example of the flow of processing executed by each device in this modified example. From the left, this figure shows an example of processing executed by the control unit 21 of user A's terminal 20A and an example of processing executed by the control unit 11 of server 10.

[0237] For example, when the base image set information is received from the terminal 20A via the communication I / F 14, the control unit 11 of the server 10 determines whether the base image satisfies a predetermined condition based on the received base image set information (S113). The method of determining whether the predetermined condition exists may be, for example, performed according to (C-1) to (C-4) above.

[0238] If it is determined that the images in the base image set satisfy the predetermined condition (S113: YES), the control unit 11 of the server 10 transmits base image set resetting request information to request reselection of a base image, etc. to the terminal 20 via the communication I / F 14 (S115). Then, the control unit 11 of the server 10 waits to receive base image set information, for example.

[0239] When it is determined that the images in the base image set do not satisfy the predetermined condition (S113: NO), the control unit 11 of the server 10 executes the processes from S120 onwards.

[0240] If it is determined that the base image set resetting request information has been received from the server 10 via the communication I / F 22 (A123: YES), the control unit 21 of the terminal 20A outputs the received base image set resetting request information (for example, displays it on the display unit 24) (A125).The control unit 21 of the terminal 20A then transmits base image set information including the reselected base image to the server 10 (A120).

[0241] When it is determined that the base image set resetting request information has not been received (A123: NO), the control unit 21 of the terminal 20A waits for reception of additional learning end information, for example.

[0242] Note that the determination of whether an image in the base image satisfies the predetermined conditions may be performed in part or in whole by the control unit 21 of the terminal 20A. For example, the predetermined condition (C-1) may be determined by the terminal 20, and the conditions (C-2) and (C-3) may be determined by the server 10.

[0243] By making such a determination, it is possible to prevent the acceptance of base images that are difficult to properly perform additional learning on, and to prevent the generation of images from the additional learning model that are inappropriate for presentation to an unspecified number of users.

[0244] This modification shows an example of a configuration in which, if the first element satisfies a set condition, the information processing device performs control to prevent the first image set from being learned. This makes it possible to prevent the image generating device from learning the first element included in the first image set based on the set condition. As a result, it is possible to prevent the image generating device from generating an image that includes the first element that satisfies the set condition.

[0245] Furthermore, this configuration can prevent the second image set from containing images that are subject to copyright infringement or that are contrary to public order and morals. It can also prevent the first image set from being improperly trained due to an inappropriate number of images in the first image set or the first image set containing multiple elements.

[0246] This modification also illustrates an example of a configuration in which, when the first element satisfies a set condition, the information processing device controls notification to the first information processing device, thereby notifying the user of the first information processing device that the set condition has been satisfied when the first element satisfies the set condition.

[0247] Furthermore, the information processing device can train the image generating device using the third image set by prompting the user of the first information processing device to send a third image set that is different from the first image set.

[0248] Furthermore, with this configuration, it is possible to notify the user of the first information processing device that the first image set contains images that are subject to copyright infringement or that contain elements that are contrary to public order and morals, and that additional learning of the first image set may not be performed properly.

[0249] <First Modification (2)> In the above embodiment, the class prompt is automatically set based on the base image, but this is not limiting. For example, in cases where it is difficult to distinguish the class prompt from the base image, the class prompt may be set based on user input.

[0250] FIG. 1-16 is a diagram showing an example of a screen displayed on the display unit 24 of the terminal 20A in this modified example. The left side of FIG. 1-16 is, for example, an example of a base image selection screen similar to the left side of FIG. 1-11. For example, when the "Select Image" button SLB1 is tapped, the display changes to, for example, the class prompt selection display screen shown in the center of FIG. 1-16.

[0251] This screen is configured to display, for example, a list of images selected in the album image selection area below the message "Selected base image." Also, below that, a message "Please select an appropriate word that is common to the base image" is displayed, and a class prompt selection area is displayed to allow the user to select a class prompt from class prompt candidates.

[0252] For example, in this screen, a sofa is reflected in the image of a Pomeranian, so that the class prompt candidates "dog," "animal," and "sofa" are displayed side by side in the class prompt selection area. For example, the class prompt candidates may be selected from character strings corresponding to the categories of base images that have the top "Y" scores (for example, "Y" = "3") among the base image analysis results in the image recognition unit.

[0253] For example, when the "dog" class prompt is selected by tapping in the class prompt selection area, the display changes to the base image learning progress display screen shown on the right side of Figure 1-16. In addition to the message "Learning base image," this screen also displays that the class prompt being used for learning is "dog."

[0254] If the class prompt does not correctly reflect the object in the base image (for example, if the class prompt for "Pomeranian image" is set to "cat"), a regularized image is generated based on the class prompt, and the regularized image is likely to be an image far removed from the features common to the base image. As a result, in the stamp image generation model additional training process, it may be difficult for the training to converge, and the training may not be performed properly. Furthermore, there is a possibility that the additional training model after training will not generate a stamp image that properly reflects the base image (an image different from the object common to the base image may be generated). By being able to set the class prompt appropriately, additional training can be performed smoothly and more natural stamp set images can be generated.

[0255] The class prompt selection display screen may be displayed when the difference in scores is small in the base image analysis results from the image recognition unit. Also, the class prompt may be input by the user.

[0256] <First Modification (3)> In the above embodiment, the processing related to the additional learning of the base image is executed in the server 10. However, this is not limited to this. For example, some or all of the processing related to the additional learning of the base image may be executed in the terminal 20.

[0257] For example, when the control unit 21 of the terminal 20 receives the base image, the control unit 21 may execute an instance prompt setting process. Then, the control unit 21 of the terminal 20 may transmit base image set information including the instance prompt to the server 10.

[0258] For example, when the control unit 21 of the terminal 20 receives a base image, the control unit 21 may execute an instance prompt setting process and a class prompt setting process. Then, the control unit 21 of the terminal 20 may transmit base image set information including the instance prompt and the class prompt to the server 10.

[0259] Also, for example, when the control unit 21 of the terminal 20 receives a base image, the control unit 21 may execute an instance prompt setting process, a class prompt setting process, and a stamp image generation model additional learning process. Then, the control unit 21 of the terminal 20 may transmit additional learning model information including the additional learning model, the instance prompt, and the class prompt to the server 10. Then, the control unit 11 of the server 10 may execute a stamp image generation process based on the received additional learning model information.

[0260] For example, some or all of the processes related to stamp image generation may be executed by the terminal 20 .

[0261] For example, when the control unit 11 of the server 10 executes the stamp image generation model additional learning process, the control unit 11 may transmit additional learning model information including the additional learning model, the instance prompt, and the class prompt to the terminal 20. Then, the control unit 21 of the terminal 20 may execute the stamp image generation prompt set setting process and the stamp image generation process based on the received additional learning model information.

[0262] Also, for example, when the control unit 11 of the server 10 executes the stamp image generation model additional learning process, it may transmit additional learning model information including the additional learning model, the instance prompt, and the class prompt to the terminal 20. Then, the control unit 11 of the server 10 may execute the stamp image generation prompt set setting process and transmit the set prompt set information to the terminal 20. The control unit 21 of the terminal 20 may execute the stamp image generation process based on the received additional learning model information and prompt set information.

[0263] Furthermore, when the control unit 21 of the terminal 20 executes the stamp image generation prompt set setting process, the control unit 21 may be configured to transmit the set prompt set information to the server 10. Then, the control unit 11 of the server 10 may execute the stamp set image generation process based on the received prompt set information.

[0264] By executing part or all of the processing of the server 10 on the terminal 20, the load on the server 10 can be reduced.

[0265] <First Modification (4)> In the above embodiment, it is assumed that the stamp set image can be generated from a messaging application, but this is not limiting. For example, it is also possible to generate the stamp set image by using the stamp set creation application described above.

[0266] In this case, the server 10A that provides the stamp set creation support service and the server 10B that provides the messaging service may be separate servers.

[0267] For example, the control unit 11A of the server 10A may include an image generation model unit 113. For example, the storage unit 15A of the server 10A may store stamp image additional learning model management data 155 and stamp image generation prompt management data 157.

[0268] For example, when a stamp set image is created using a stamp set creation application and the stamp set is registered, stamp set information regarding the stamp set including the stamp set image may be transmitted from server 10A to server 10B. Upon receiving the stamp set information, server 10B may, for example, add a record to stamp set management database 159 and register the stamp set.

[0269] The generated stamp set images may be used as an image set for use in a stamp set, or may be exported from a stamp set creation application, for example, so that they can be used for other purposes.

[0270] Alternatively, the stamp set creation application may be considered as an independent application that generates an image set based on a predetermined prompt set using an incremental learning model that has been incrementally trained on a base image.

[0271] <First Modification (5)> In the above embodiment, the additional learning model is configured to learn one base image (base image set), but this is not limited to this. For example, different instance prompts may be set for multiple base images, and additional learning may be performed.

[0272] As an example, an instance prompt "cpc" is set for a base image including a "rabbit character" object, and an instance prompt "usu" is set for a base image including a "bear character" object. Then, a class prompt is set for each object, and a stamp image generation prompt set setting process is executed. Then, as an example, for an additional learning model that has been additionally trained with two types of base images, the stamp image generation prompt set setting process is executed using the instance prompts "cpc and usu," and the stamp image generation process is executed, generating an image including a "rabbit character and a bear character" as a stamp image.

[0273] That is, a stamp set image including a plurality of objects may be generated by assigning different instance prompts to a plurality of base images including different objects and performing additional learning.

[0274] In addition, when multiple base images are additionally learned, for example, it may be possible to specify an instance prompt (instance token) assigned to each object included in the instance prompt based on a user input at the terminal 20. Furthermore, it may be possible to indirectly specify an instance token by selecting a base image associated with the instance token or a stamp image generated from the instance token.

[0275] <First Modification (6)> In the above embodiment, an example in which the stamp image is a still image has been illustrated, but this is not limiting. For example, the generated stamp image may be a moving image. When a moving image is generated as the stamp image, the image generation model unit 113 may be configured, for example, with a Text-to-Video model (trained neural network model) such as Imagen Video or Make-A-Video.

[0276] Furthermore, the base image is not limited to a still image. For example, if the base image is a moving image, the control unit 11 of the server 10 acquires the base image, executes a scene analysis process, and acquires a still image of an important scene in the moving image. Then, for example, the acquired still image may be used as a learning image (which may be a new base image) for the additional learning model to execute an instance prompt setting process, etc. Note that, if the image generation model unit 113 is a text-to-video model, an additional learning model may be created based on the base image, which is a moving image.

[0277] Second Example The second example is an example in which a stamp image is generated based on stamp text. The content described in the second example can be similarly applied to any of the other examples and other modified examples.

[0278] 2A is a diagram illustrating an example of functions implemented by the control unit 11 of the server 10 in this embodiment. The control unit 11 includes, as functional units, an application management processing unit 111, an image generation model unit 113, and a language model unit 115, for example.

[0279] For example, the language model unit 115 has a function of generating a prompt for generating a stamp image in the image generation model unit 113 based on an input keyword (such as a stamp text). For example, the language model unit 115 may be configured with a large-scale language model such as HyperCLOVA (registered trademark), GPT-4, or PaLM. Note that the language model unit 115 may be configured with a small-scale language model with a smaller number of parameters, specialized for learning related to prompt generation.

[0280] Note that the language model unit 115 (which is an example of a language generation device that generates a prompt based on the first prompt, and is not limited to this) is not limited to being provided in the server 10. As a non-limiting example, the language generation device may be provided outside the server 10 (which is an example of an information processing device, and is not limited to this). Furthermore, as a non-limiting example, the processing in this embodiment may proceed by communication between the language generation device and the information processing device.

[0281] 2B is a diagram showing an example of the data configuration of the stamp image generation prompt management data 157B in this embodiment. The stamp image generation prompt management data 157B stores, for example, a prompt template ID, a stamp text template, and other information in association with each other.

[0282] The prompt template ID and other information may be the same as those in the stamp image generation prompt management data 157A, for example.

[0283] The stamp text template is, for example, a character string that serves as a template for stamp text, or the stamp text itself that is set in the stamp, set by the user of the server 10. Note that different text may be set for stamp text of different prompt template IDs.

[0284] For example, when a stamp template ID, an instance prompt, and a class prompt are specified, the control unit 11 of the server 10 generates, for example, a prompt to be input to the image generation model unit 113 by using the language model unit 115. Hereinafter, for example, the prompt to be input to the language model unit 115 will be referred to as an "image generation prompt creation prompt," and the prompt to be input to the image generation model unit 113 will be referred to as an "image generation prompt."

[0285] For example, the control unit 11 of the server 10 first generates a prompt for creating a prompt for image generation using a prompt template for creating a prompt for image generation based on the stamp template ID, the instance prompt, and the class prompt.

[0286] The prompt template for creating a prompt for image generation is, for example, a template set by the user of the server 10, and may be, for example, the following template: "<Rule> Create a prompt that instructs the image generation model to generate an illustration. Create an English keyword list for the prompt from the sentence you will enter. Use only adjectives, verbs, and nouns in the keyword list. The first keyword is "*instance prompt* *class prompt*". <Output> *instance prompt* *class prompt*, keyword, keyword, (loop) <Input> *stamp text*"

[0287] For example, *instance prompt* and *class prompt* are replaced with the specified instance prompt and class prompt, and *stamp text* is replaced with the stamp text associated with the stamp template ID. Note that the replaced stamp text may be determined based on user input.

[0288] When the prompt for creating an image generation prompt is created, the control unit 11 of the server 10 generates an image generation prompt based on the prompt for creating an image generation prompt using, for example, the language model unit 115. If the instance prompt is specified as "cpc," the class prompt is specified as "rabbit," and the stamp text is "thank you," the generated image generation prompt will be, for example, "cpc rabbit, expressing thanks, best quality, ...."

[0289] For example, common positive prompts and common negative prompts may be stored, and a prompt template for creating an image generation prompt may be set to generate an image generation prompt incorporating the common positive prompt or the common negative prompt.

[0290] <Display Screen> FIGS. 2-3 to 2-5 are diagrams showing examples of screens displayed on the display unit 24 of the terminal 20A in this embodiment.

[0291] The left side of Figure 2-3 is, for example, a stamp set creation screen similar to the center of Figure 1-10. When, for example, the "Automatic Stamp Set Creation" button SGB2 is tapped on this screen, the display changes to, for example, the stamp set creation screen in the center of Figure 2-3.

[0292] On this screen, a stamp number selection area NSR2 for selecting the number of stamp images to be generated is configured to be displayed in a manner that rises from the bottom of the screen. Note that, for example, the stamp number selection area NSR2 may be configured so that the user can input and specify the number of stamp images to be generated.

[0293] For example, when the user taps the button for selecting the number of stamps to be generated (N (here, "N" = "8")) in the stamp number selection area NSR2, stamp text templates specified by any N prompt template IDs are selected from the stamp image generation prompt management data 157B. Then, for example, the display changes to the stamp set creation screen shown on the right side of FIG. 2-3.

[0294] On this screen, for example, the individual stamp editing area STR1 is configured to display the stamp text assigned in the stamp text template for the number of stamps specified in the stamp number selection area NSR2. Note that, for example, because no stamp images have been assigned to the stamps, the "Stamp Set Creation Complete" button SRB1 remains grayed out, with its function disabled. Also, for example, because stamp text has been assigned to the stamps, the "Automatic Stamp Set Generation" button SGB2 is displayed as a "Generate Stamp Set Image" button SGB3.

[0295] Unlike the present example, when the number-of-stamps selection button is tapped in the number-of-stamps selection area NSR2, a stamp text setting area may be displayed in which a list of stamp texts including some or all of the stamp text templates is displayed. Then, in the stamp text setting area, stamp text may be set in the individual stamp editing area STR1 based on an arbitrary stamp text template selected by the user.

[0296] For example, when the user taps a stamp with the stamp text "Bye-bye" set in the individual stamp editing area STR1, the display changes to the stamp set creation screen shown on the left side of FIG. 2-4.

[0297] On this screen, for example, a stamp editing area SER1 for specifying editing operations for individual stamps is configured to be displayed by rising from the bottom. In the stamp editing area SER1, for example, a "Use Album Image" button for specifying how to set the stamp image, a "Take Photo" button, and a "Draw Illustration" button are displayed below, along with a "Text Edit" button for setting (editing) the stamp text.

[0298] For example, when the "Edit Text" button is tapped by the user, the display changes to, for example, the stamp text editing screen shown in the center of Figure 2-4. On this screen, for example, in a text editing area TER1 for setting the character string of the stamp text to be set in the stamp, the character string "Leave it to me," which has been changed by the user from the character string "Bye-bye" set by the stamp text template, is input and displayed. Also, above the text editing area TER1, a stamp text placement setting area for setting the position at which the stamp image and stamp text are superimposed is displayed. In the stamp text placement setting area, for example, the position at which the stamp text and stamp image are superimposed may be set by dragging the stamp text "Leave it to me," for example. It is also possible to set the character size, color, font, etc. of the stamp text to be displayed in the stamp text placement setting area.

[0299] For example, if a word included in the NG word list related to a prompt that generates an image inappropriate for a stamp image is entered into the text editing area TER1, a message may be displayed requesting that the stamp text be re-entered, and a different string may be accepted again.

[0300] For example, when the stamp text is edited in the stamp editing area SER1 and the placement of the stamp text is set in the stamp text placement setting area, when the "OK" button SEB1 is tapped, for example, the edited content is reflected on the stamp text editing screen, and for example, the display changes to the stamp set creation screen on the right side of Figure 2-4.

[0301] This screen displays that the stamp text of the stamp that was set to "Bye-bye" in the individual stamp editing area STR1 has been changed to "Leave it to me."

[0302] For example, when the "Generate stamp set image" button SGB3 is tapped by the user, the display changes to, for example, the stamp set automatic generation method selection screen on the left side of Figure 2-5. This screen is, for example, the same as the screen on the right side of Figure 1-10. When the "Select image from album" button is tapped by the user on this screen, a base image is accepted, and a stamp image generation model additional learning process is executed based on the base image, and an additional learning model is generated. Once the additional learning model is generated, for example, an image generation prompt is generated based on the stamp text set for each stamp, the instance prompt set for the base image, and the class prompt. Then, a stamp image is generated by inputting the image generation prompt into the additional learning model. Once the number of stamp images selected in the stamp number selection area NSR2 have been generated, the display changes to, for example, the generated stamp set image display screen in the center of Figure 2-5.

[0303] On this screen, a specified number of stamp images generated by the additional learning model in response to a prompt based on the stamp text are displayed in a generated stamp image confirmation display area.

[0304] In addition, in the generated stamp image confirmation display area, the set stamp text may be displayed overlapping the generated stamp image.

[0305] For example, when the "OK" button IAB1 is tapped by the user, the display changes to the stamp set creation screen shown on the right side of FIG. 2-5. On this screen, for example, the stamp set image confirmed in the generated stamp image confirmation display area is applied and displayed in the individual stamp editing area STR1. For example, each stamp image is displayed with the set stamp text superimposed. Hereinafter, an image in which the stamp text is superimposed on a stamp image generated by the additional learning model (an image rendered by superimposing the stamp text on the generated image of the additional learning model) may be referred to as a "stamp image with text."

[0306] The rendering position and character color of the stamp text in the stamp image with text may be determined based on information set by the user in the stamp text placement setting area. Also, the character color of the stamp text may be determined by LayoutGAN or the like based on the stamp image generated by the additional learning model and the set stamp text, and the stamp text may be appropriately positioned on the stamp image with text.

[0307] 2-6 is a flowchart showing an example of the flow of processing executed by each device in this embodiment. From the left, this figure shows an example of processing executed by the control unit 21 of user A's terminal 20A and an example of processing executed by the control unit 11 of the server 10.

[0308] For example, when stamp set image number selection information is sent to the server 10 via the communication I / F 22, the control unit 21 of the terminal 20A receives stamp text for the number of stamp images based on user input, for example. The control unit 21 of the terminal 20A then transmits stamp text set information including the received stamp text to the server 10 via the communication I / F 22 (A210). The stamp text may be a single word or may consist of two or more words.

[0309] For example, when stamp set image number selection information is received from terminal 20A via communication I / F 14, control unit 11 of server 10 selects prompt template IDs, for example, randomly, for the number of stamp images to be generated as specified in the stamp set image number selection information. Then, control unit 11 of server 10 may transmit default stamp text set information including a stamp text template linked to the prompt template ID to terminal 20A via communication I / F 14. Upon receiving the default stamp text set information from server 10, control unit 21 of terminal 20A may display the received default stamp text set information on display unit 24. Then, control unit 21 of terminal 20A may accept stamp text based on a user input for the displayed default stamp text set information.

[0310] The control unit 11 of the server 10 may be configured not to transmit the default stamp text set information to the terminal 20. In other words, each stamp text may be determined by the server 10.

[0311] When the stamp text set information is received from the terminal 20A via the communication I / F 14, the control unit 11 of the server 10 executes a stamp image generation prompt generation process (S210).

[0312] In the stamp image generation prompt generation process, the control unit 11 of the server 10 generates, for example, a prompt for creating an image generation prompt for the number of stamp images to be generated based on the stamp text contained in the received stamp text set information and the instance prompt and class prompt used for learning the additional learning model.

[0313] Then, the control unit 11 of the server 10 inputs, for example, a prompt for creating an image generation prompt to the language model unit 115, and generates image generation prompts for the number of stamp images to be generated.

[0314] Then, the control unit 11 of the server 10 executes a stamp image generation process (S170). In the stamp image generation process, the control unit 11 of the server 10 inputs an image generation prompt to the additional learning model, and acquires the generated image as a stamp image. This process is repeated the number of times corresponding to the number of stamp set images.

[0315] Then, the control unit 11 of the server 10 executes a stamp text synthesis process (S220). In the stamp text synthesis process, the control unit 11 of the server 10 outputs a stamp image with text, which is a synthesis of the generated stamp image and the character string image of the stamp text used to generate the image generation prompt, for example.

[0316] Then, the control unit 11 of the server 10 transmits stamp set image information including the generated specified number of images to the terminal 20A via the communication I / F 14 (S180). The images included in the stamp set image information may be stamp images with text, or stamp images generated by applying an image generation prompt to an additional learning model. For example, the stamp set image information may include stamp images and stamp images with text. For example, the stamp set image information may include stamp text used to generate the image generation prompt, linked to the stamp images or the stamp images with text.

[0317] Effect of Second Example This example illustrates an example of a configuration in which an information processing device acquires stamp text template and stamp text set information (not limited to, but an example of first text information) and sets a prompt set based on an instance prompt or class prompt (not limited to, but an example of first information) and the first text information. This allows the acquired first text information to be reflected in the prompt set. As a result, the first text information can be reflected in the second image set generated by the image generation device.

[0318] Furthermore, by receiving the first image set and the first text information from the first information processing device, a second image set that includes the first object of the first image set and is generated based on the first text information can be transmitted to the first information processing device or the second information processing device, thereby allowing a user of the first information processing device or the second information processing device to easily obtain a second image set that includes the first object desired by the user and reflects the first text information intended by the user.

[0319] Furthermore, by setting a prompt for creating an image generation prompt (a non-limiting example of a first prompt) based on the first information and the first text information and causing the language model unit 115 (a non-limiting example of a language generation device) to generate an image generation prompt (a non-limiting example of a prompt set) based on the first prompt, the information processing device can cause the image generation device to generate images based on a prompt set that is more effective. Therefore, the generated second image set can be expected to be images of higher quality.

[0320] Furthermore, by combining character images based on the first text information with the images that make up the second image set, the user can easily understand what text each image in the second image set reflects.

[0321] Furthermore, this embodiment illustrates an example of a configuration in which the image generation model unit 113 (not limited to, but an example of an image generation device that generates an image based on a prompt) generates a stamp image (not limited to, but an example of an image) without accepting a prompt input from a user. That is, this embodiment illustrates the configuration of an image generation device that generates an image without accepting a prompt input from a user. This allows the image generation device to generate an image without accepting a prompt input from a user. As a result, it is possible to prevent the image generation device from accepting an inappropriate prompt from a user. This makes it possible to prevent the user from using the image generation device inappropriately, thereby improving the robustness of the image generation device.

[0322] <Second Modification (1)> For example, a stamp text template in the stamp image generation prompt management data 157B may store a set of stamp texts (stamp text sets) corresponding to the number of stamps, and may be configured so that stamp texts for generating different stamp images do not include words that would generate similar stamps. In other words, once a prompt template ID is specified and a stamp text template is determined, a stamp text set for generating a stamp set image may be determined. Then, a prompt for creating a prompt for each image generation may be generated based on the stamp text set, and then each image generation prompt may be generated.

[0323] This makes it possible to generate stamp set images with less overlapping types and a wide variety of images.

[0324] For example, the purpose of the stamp set generated for each stamp text template (for example, a "greeting stamp set" or an "apology stamp set") may be defined, and a stamp text set may be set according to the purpose. For example, a prompt template ID may be designated based on the result of a selection of the purpose by the user of the terminal 20.

[0325] <Second Modification (2)> In the above embodiment, the prompt for creating each image generation prompt is generated based on stamp text, etc. However, this is not limited to this. For example, a feature prompt that indicates the characteristics of the image to be generated may be set in the prompt for creating each image generation prompt.

[0326] FIG. 2-7 is a diagram showing an example of a screen displayed on the display unit 24 of each terminal 20 in this modified example. The left side of FIG. 2-7 is, for example, an example of a base image learning progress display screen similar to the right side of FIG. 1-11. For example, when the "Generate stamp set image" button IGB2 is tapped, the display changes to, for example, the base image learning progress display screen shown in the center of FIG. 2-7.

[0327] On this screen, for example, a stamp image style selection area TSR1 for selecting the style (art style) of the stamp image to be generated is displayed in a raised manner, superimposed on the base image learning progress display screen after the additional learning process is completed. In the stamp image style selection area TSR1, for example, selection buttons such as "Realistic," "SF," and "Manga" are displayed in a row as selectable stamp image styles.

[0328] For example, the user may be allowed to input and specify the style of the stamp image in the stamp image style selection area TSR1. In this case, for example, if the input style is a word included in a list of NG words that are inappropriate for a stamp image, a message indicating that the style cannot be accepted may be displayed on the terminal 20A.

[0329] For example, when the selection buttons for "Realistic" and "Cute" are selected and the "OK" button is tapped, for example, in the stamp image generation prompt generation process, a template such as "Please create a prompt that instructs the image generation model to generate an image. Please create a prompt so that the generated image reflects the style indicated in the *feature prompt*..." may be used in the prompt template for creating an image generation prompt, and the *feature prompt* portion may be replaced with one or more styles selected in the stamp image style selection area TSR1.

[0330] For example, a prompt for creating an image generation prompt is generated based on the above template, and a prompt for image generation is also generated. Then, a stamp image generation process is executed based on the image generation prompt, and N stamp images are generated. Then, for example, the display changes to the generated stamp set image display screen shown on the right side of Figure 2-7. On this screen, a stamp image reflecting the style selected in the stamp image style selection area TSR1 is generated and displayed in the generated stamp image confirmation display area.

[0331] For example, when the "regenerate image" button RGB2 is tapped, the stamp image style selection area TSR1 may be displayed again, allowing a different style to be selected.

[0332] Also, the feature prompt may be directly written as a positive prompt in the image generation prompt.

[0333] The feature prompt may be a background (e.g., a flower field, a night sky, etc.), an object the character carries (e.g., a bouquet, a hat, etc.), an object the character rides (e.g., a car, a rocket, etc.), etc. Also, information indicating the season or time (e.g., "spring," "summer," "morning," "night," etc.) may be selected as a feature prompt to define the overall impression of the stamp image.

[0334] By allowing the user to specify the style and the like in the prompt for creating the image generation prompt and the image generation prompt in this way, it is possible to help the user generate the stamp image they desire more easily. It is also possible to prevent the user from directly embedding a positive prompt in the image generation prompt, which may result in the generation of an image that is not suitable as a stamp image.

[0335] <Second Modification (3)> In the above embodiment, an example of generating a stamp set image consisting of multiple stamp images has been described, but the present invention is not limited to this. For example, the generated stamp image may be an image of a single stamp, or a single image itself.

[0336] 2-8 to 2-9 are diagrams showing examples of screens displayed on the display unit 24 of the terminal 20A in this modified example. For example, when the user taps the "+" icon for creating a single stamp in the individual stamp editing area STR1 on the stamp set creation screen on the left side of FIG. 2-8, the display changes to the stamp set creation screen in the center of FIG. 2-8.

[0337] On this screen, for example, a stamp editing area SER2 for specifying editing operations for individual stamps is configured to rise from the bottom. When the user taps the "Image Generation" button in the stamp editing area SER2, for example, the display changes to the stamp text editing screen shown on the right side of FIG. 2-8. On this screen, the character string "OK" entered by the user is displayed in the text editing area TER2. Then, when the "Confirm" button SEB2 is tapped, for example, the edited content on the stamp text editing screen is reflected, and for example, the display changes to the stamp set automatic generation method selection screen shown on the left side of FIG. 2-9.

[0338] When the user taps the "Select image from album" button on this screen, a base image is accepted, and a stamp image generation model additional learning process is executed based on the base image, generating an additional learning model. When the additional learning model is generated, for example, an image generation prompt is generated based on the stamp text set in the stamp, the instance prompt set in the base image, and the class prompt. Then, a stamp image is generated by inputting the image generation prompt into the additional learning model. Then, for example, the display changes to the generated stamp image display screen shown in the center of Figure 2-9.

[0339] On this screen, a stamp image (in this example, a stamp image with text) generated by the additional learning model in response to a prompt based on the input stamp text is displayed in the generated stamp image confirmation display area. For example, when the user taps the "OK" button IAB2, the display changes to the stamp set creation screen shown on the right side of Figure 2-9. On this screen, for example, the stamp image confirmed in the generated stamp image confirmation display area is applied and displayed in the individual stamp editing area STR1.

[0340] 2-10 is a flowchart showing an example of the flow of processing executed by each device in this modified example. From the left, this figure shows an example of processing executed by the control unit 21 of user A's terminal 20A and an example of processing executed by the control unit 11 of server 10.

[0341] First, the control unit 21 of the terminal 20A transmits, for example, stamp image generation request information for generating a stamp image to the server 10 via the communication I / F 22 based on a user input (A200).

[0342] For example, when the additional learning completion information is displayed (A130), the control unit 21 of the terminal 20A acquires stamp text that will be the basis for the stamp image to be generated based on the user input, and then transmits stamp text information including the acquired stamp text to the server 10 via the communication I / F 22 (A220).

[0343] If the stamp text information includes a word included in the NG word list that is inappropriate for a stamp image, the control unit 21 of the terminal 20A may display stamp text re-input request information on the display unit 24, which requests re-input of the stamp text. Then, the stamp text acquisition process may be executed again.

[0344] When the stamp text information is received from the terminal 20A via the communication I / F 14, the control unit 11 of the server 10 executes a stamp image generation prompt generation process based on the received stamp text (S210).Then, the control unit 11 of the server 10 executes a stamp image generation process based on the generated image generation prompt (S170).

[0345] For example, if the stamp text information includes a word included in a list of NG words that are inappropriate for a stamp image, the control unit 11 of the server 10 may transmit stamp text re-input request information to the terminal 20A, requesting re-input of the stamp text. Then, the control unit 21 of the terminal 20A may display the received stamp text re-input request information on the display unit 24 and execute step A220 again.

[0346] For example, when the control unit 11 of the server 10 executes the stamp text synthesis process (S220), it transmits stamp image information including the generated stamp image and the stamp image with text to the terminal 20A via the communication I / F 14 (S230).

[0347] When the stamp image information is received from the server 10 via the communication I / F 22, the control unit 21 of the terminal 20A causes the display unit 24 to display the received stamp image information (A230).

[0348] In this modification, the server 10 (not limited to, an example of an information processing device) acquires base images (not limited to, an example of a first image set) consisting of images including a Pomeranian (not limited to, an example of a first object) for additional learning for the image generation model unit 113 (not limited to, an example of an image generation device that generates images based on prompts). The server 10 then causes the image generation device to learn the first image set based on an instance prompt or a class prompt (not limited to, an example of first information) associated with the first image set. Furthermore, upon acquiring stamp text (not limited to, an example of first text information), the server 10 sets a stamp image generation prompt (not limited to, an example of a second prompt) based on the first information and the first text information. Then, in a stamp image generation process, the server 10 causes an additional learning model (not limited to, an example of an image generation device that generates images based on prompts) to generate a stamp image (not limited to, an example of a second image). Therefore, the configuration is shown in which an image based on a Pomeranian is generated as the second image. As an example of the effect of a modified example obtained by such a configuration, an information processing device can acquire a first image set consisting of images including a first object and first text information, and cause an image generation device to generate a second image that is based on the first object and reflects the first text information.

[0349] <Second Modification (4)> In the above embodiment, the stamp image generation prompt generation process is executed in the server 10. However, this is not limiting. For example, the stamp image generation prompt generation process may be executed in the terminal 20.

[0350] For example, the control unit 11 of the server 10 may transmit additional learning completion information, including the instance prompts and class prompts used in the stamp image generation model additional learning process, to the terminal 20A. Then, upon receiving stamp text for the number of stamp images, the control unit 21 of the terminal 20A may generate image generation prompt creation prompts for the number of stamp images to be generated based on the received stamp text and the instance prompts and class prompts. Then, the control unit 21 of the terminal 20A may generate image generation prompts, for example, by a language model unit (not shown), based on the generated image generation prompt creation prompts. Then, the control unit 21 of the terminal 20A may transmit image generation prompt set information, including the generated image generation prompts, to the server 10.

[0351] The control unit 21 of the terminal 20A may transmit image generation prompt creation prompt set information, including a prompt for creating an image generation prompt, to the server 10. The control unit 11 of the server 10 may then input an image generation prompt creation prompt based on the image generation prompt creation prompt set information to the language model unit 115, and generate an image generation prompt.

[0352] Furthermore, as described above, some or all of the learning of the base image and the image generation using the additional learning model may be performed on terminal 20A.

[0353] <Second Modification (5)> In the above embodiment, the control unit 11 of the server 10 outputs a stamp image with text by combining the generated stamp image with a character string image of the stamp text in the stamp text combining process, but this is not limited to this. As a non-limiting example, the control unit 11 of the server 10 may combine a character string image different from the stamp text with the stamp image in the stamp text combining process.

[0354] By way of example and not limitation, the control unit 11 of the server 10 may input the following prompt to the language model unit 115: "Please rephrase *stamp text* in different words." The resulting character string image of the output sentence may then be combined with the stamp image.

[0355] Third Embodiment The third embodiment is an example of a messaging service in which a stamp image is generated in real time based on input text and sent to a chat room. The content of the third embodiment is equally applicable to the other embodiments and other modifications.

[0356] In this embodiment, an example of the functions realized by the control unit 11 of the server 10 may be configured in the same manner as in FIG. 2-1, for example.

[0357] 3A is a diagram showing an example of information stored in the storage unit 15 of the server 10 in this embodiment. The storage unit 15 stores, for example, an application management processing program 151 and account registration data 153.

[0358] <Display Screen> Figures 3-2 to 3-3 are diagrams showing examples of screens displayed on the display unit 24 of the terminal 20A in this embodiment. In this example, it is assumed that images including rabbit characters, bear characters, etc. have already been learned by the image generation model unit 113. However, it is assumed that the learned rabbit characters and bear characters are not associated with instance prompts. The left side of Figure 3-2 shows, as an example, an example of a chat room screen between user A.A and user B.B, who is a friend of user A.A.

[0359] On this talk room screen, the current location display area indicates that this is a one-on-one talk room with "User B.B." Also, below that, a talk room content display area TR1 is displayed to display content sent and received within this talk room.

[0360] In the talk room content display area TR1, for example, the speech of user A.A. of the terminal 20A, which is the terminal itself, is displayed on the right side in a speech bubble, and the speech of other users is displayed on the left side in a speech bubble.

[0361] The talk room content display area TR1 displays a text message CT1 sent by user A.A about looking forward to camping, and a text message CT2 sent by user B.B about reserving a campsite.

[0362] Below the talk room content display area TR1, a content input area is displayed for inputting and selecting content to send to the talk room. In this screen, for example, the user selects the text message input area MIR1 in the content input area, and the character string "Thank you" is entered and displayed.

[0363] For example, when a user taps the stamp image generation button SCB1, which is indicated by a face icon and displayed to the right of the text message input field MIR1, a stamp image generation process is executed based on the character string entered in the text message input field MIR1. Then, for example, the display changes to the chat room screen shown on the right side of FIG. 3-2.

[0364] This screen is configured to display a stamp image generation progress indicator in the talk room content display area TR1, indicating that a stamp image is being generated. For example, by checking the stamp image generation progress indicator, the user of terminal 20A can understand that a stamp image is being generated and the progress of the generation process. For example, the stamp image generation process may be interrupted by tapping an image generation cancel button indicated by an "x" mark in the upper right corner of the stamp image generation progress indicator. This prevents the conversation from being hindered by the time required to generate the stamp image.

[0365] For example, when the stamp image generation process is completed, the display changes to the chat room screen shown on the left side of FIG. 3-3. This screen is configured to display, above the content input area, a generated stamp list information display area SSR1 for displaying a list of stamps based on the generated stamp image. For example, the generated stamp list information display area SSR1 is configured to display a predetermined number (e.g., five stamps) including stamp images generated by the image generation model unit 113 based on the character string "Thank you." Note that the character string "Thank you" entered in the text message input area MIR1 may be superimposed on each stamp displayed in the generated stamp list information display area SSR1.

[0366] For example, when a user taps a rabbit character stamp in the generated stamp list information display area SSR1, the display changes to the chat room screen shown on the right side of Figure 3-3. On this screen, the stamp content SCT1 selected in the generated stamp list information display area SSR1 is sent to and displayed in the chat room content display area TR1.

[0367] Although a one-to-one talk room has been illustrated in this example, stamps may be sent in a similar manner in a group talk room or an OA talk room.

[0368] 3-4 is a flowchart showing an example of the flow of processing executed by each device in this embodiment. From the left, this figure shows an example of processing executed by the control unit 21 of terminal 20A, an example of processing executed by the control unit 21 of terminal 20B, and an example of processing executed by the control unit 11 of server 10.

[0369] First, for example, the control unit 21 of terminal 20A, based on a user input, transmits chat request information to the server 10 via the communication I / F 22, requesting a chat with the user of terminal 20B via a talk room. Upon receiving the chat request information from terminal 20A via the communication I / F 14, the control unit 21 of the server 10 executes a chat start process for conducting a chat between terminal 20A and terminal 20B via the talk room (S310). In the chat start process, for example, the control unit 11 of the server 10 generates a talk room ID for identifying the talk room and transmits the generated talk room ID to each terminal 20 participating in the talk room via the communication I / F 14.

[0370] For example, when a talk room ID is received from the server 10 via the communication I / F 22, the control unit 21 of the terminal 20A and the control unit 21 of the terminal 20B execute a chat start process for chatting between the terminal 20A and the terminal 20B via the talk room (A310, B310). In the chat start process, the control unit 21 of each terminal 20 displays a talk room screen on the display unit 24, for example.

[0371] The chat request information may be transmitted from the terminal 20B to the server 10.

[0372] For example, when sending a message to a talk room is selected based on an input to the input / output unit 23 of the terminal 20B, the control unit 21 of the terminal 20B may transmit content transmission request information including the talk room ID and the text of the message to be sent to the server 10 via the communication I / F 22. Note that the control unit 21 of the terminal 20A may transmit the content transmission request information to the server 10.

[0373] When content transmission request information is received from the terminal 20 via the communication I / F 14, the control unit 11 of the server 10 may transmit content information (e.g., text content information, image content information, stamp content information, etc.) based on the content transmission request information to each terminal 20 in the talk room. When content information is received from the server 10 via the communication I / F 22, the control unit 21 of the terminal 20 may, for example, display the received content information in the talk room content display area.

[0374] For example, the control unit 21 of the terminal 20A accepts text content (text message) to be sent to the talk room based on input to the input / output unit 23 (A320). Then, for example, if generating a stamp based on the text content (character string) is selected based on the input to the input / output unit 23, the control unit 21 of the terminal 20A sends stamp text information including the input text content to the server 10 via the communication I / F 22 (A220). The text content in the stamp text information may be referred to as stamp text.

[0375] When the stamp text information is received from the terminal 20A via the communication I / F 14, the control unit 11 of the server 10 executes a stamp image generation prompt generation process (S330). In the stamp image generation prompt generation process, the control unit 11 of the server 10 generates a prompt for creating an image generation prompt based on the following prompt template for creating an image generation prompt, for example: "<Rule> Create a prompt that instructs the image generation model to generate an illustration. Create an English keyword list for the prompt from the sentence to be entered. Use only adjectives, verbs, and nouns in the keyword list. <Output> keyword, keyword, (loop) <Input> *stamp text*"

[0376] For example, *stamp text* is replaced with stamp text (for example, "Thank you") included in the stamp text information.

[0377] Then, the control unit 11 of the server 10 generates, for example, a prompt for generating an image based on the prompt for creating a prompt for generating an image, using the language model unit 115.

[0378] Then, the control unit 11 of the server 10 uses the generated image generation prompt to repeat the stamp image generation process (S170) and the stamp text synthesis process (S220) a predetermined number of times (a set number of times), for example, while changing the seed value. In the stamp image generation process, the server 10 may, for example, perform inference processing based on the image generation prompt using an image generation model unit 113 that has not undergone additional training. Note that the inference processing using an image generation model that has not undergone additional training may, for example, be performed by the control unit 11 of the server 10.

[0379] The control unit 11 of the server 10 may repeat the stamp image generation prompt generation process to generate different image generation prompts, and may repeat the stamp image generation process (S170) and the stamp text synthesis process (S220) a predetermined number of times. Also, the stamp text synthesis process may not be executed.

[0380] When a predetermined number of stamp images have been generated by repeating the process a predetermined number of times, the control unit 11 of the server 10 transmits generated stamp list information including the generated predetermined number of stamp images to the terminal 20A via the communication I / F 14 (S350). The generated stamp list information may also include stamp images with text. The generated stamp list information may also include information for identifying each stamp image (e.g., a temporary ID, etc.) and stamp text.

[0381] When the generated stamp list information is received from the server 10 via the communication I / F 22, the control unit 21 of the terminal 20A displays the received generated stamp list information on the display unit 24 (A330). For example, when a specific stamp included in the generated stamp list information is selected based on a user input to the generated stamp list information displayed on the display unit 24 (A340), the control unit 11 of the terminal 20A transmits generated stamp selection information including the temporary ID of the selected stamp to the server 10 via the communication I / F 22 (A350).

[0382] When the server 10 receives the stamp selection information from the terminal 20A via the communication I / F 14, the control unit 11 of the server 10 transmits stamp content information including the stamp image (or stamp image with text) specified in the received stamp selection information to each terminal 20 via the communication I / F 14 (S370). The stamp content information may include stamp text. Alternatively, the stamp content information may not include the stamp image or the stamp image with text. The stamp content information may include the image generation prompt generated in the stamp image generation prompt generation process.

[0383] When the stamp content information is received from the server 10 via the communication I / F 22, the control unit 21 of the terminal 20 causes the display unit 24 to display the received stamp content information (A370, B370).

[0384] Then, the control unit 21 of the terminal 20A determines whether or not to end the processing (A390). If it determines to continue the processing (A390: NO), the control unit 21 of the terminal 20A returns the processing to A310, for example. On the other hand, if it determines to end the processing (A390: YES), the control unit 21 of the terminal 20A ends the processing. The same may be done for the terminal 20B.

[0385] The control unit 11 of the server 10 then determines whether to terminate the process (S390). If it determines to continue the process (S390: NO), the control unit 11 of the server 10 returns the process to S310, for example. On the other hand, if it determines to terminate the process (S390: YES), the control unit 11 of the server 10 terminates the process.

[0386] Effect of Third Example In this example, the server 10, which executes processing related to a chat between user A.A (a non-limiting example of a first user) of terminal 20A (a non-limiting example of a first terminal) and user B.B (a non-limiting example of a second user) of terminal 20B (a non-limiting example of a second terminal), receives stamp text information (a non-limiting example of text information based on input to the first terminal displaying a chat room including the first user and the second user) from the first terminal, and generates a predetermined number of stamp images (a non-limiting example of an image set). The server 10 then transmits generated stamp list information (a non-limiting example of first information) to the first terminal. The server 10 then receives generated stamp selection information (a non-limiting example of second information) and transmits stamp content information (a non-limiting example of third information) based on the second information to the second terminal. This allows the first user to input text information into the chat room and view the image set generated based on the text information on the first terminal. The first user can then select a desired image from the image set on the first terminal, and transmit the third information based on the selected image to the second terminal of the second user, who is the other party in the chat room. This can improve and enhance the user experience when participating in a chat.

[0387] Furthermore, by generating an image set using an image generating device that generates images based on prompts based on text information, the first user can easily express the image he or she wants to generate using text information.

[0388] Furthermore, by having the language processing device generate the prompt, it is possible to obtain a prompt that enables the image generating device to generate a high-quality image.

[0389] <Third Modification (1)> In the above embodiment, the language model unit 115 generates the image generation prompt based on the prompt for creating the image generation prompt, but this is not limited to this. For example, the control unit 11 of the server 10 may generate an image generation prompt using stamp text (which may be referred to as a character string acquired by the terminal 20A) in the stamp text information from the terminal 20A as a positive prompt.

[0390] For example, if the stamp text is "Thank you," the prompt for image generation may be "Thanks" or "thanks."

[0391] The image generation prompt may be generated by combining a common positive prompt or a common negative prompt with stamp text.

[0392] Alternatively, a positive template prompt including information for indicating the replacement position of the stamp text may be set, and the stamp text may be applied to the positive template prompt to generate a prompt for generating an image.

[0393] That is, the control unit 11 of the server 10 may not include the language model unit 115 .

[0394] <Third Variation (2)> For example, if the language model unit 115 determines that the image generation prompt is an inappropriate prompt based on the prompt for creating the image generation prompt, the control unit 11 of the server 10 may execute the stamp image generation prompt generation process again to generate a different image generation prompt.

[0395] The determination of whether or not a prompt is inappropriate may be, for example, by determining whether or not a word included in the NG word list is included as a positive prompt in the generated image generation prompt. Alternatively, for example, a stamp image generated by the image generation prompt may be applied to a proper image determination filter or the like to determine whether or not an inappropriate image may be generated.

[0396] <Third Modification (3)> In the above embodiment, when an image generation prompt is generated in the stamp image generation prompt generation process, the stamp image generation process is executed. However, this is not limited to this. For example, the storage unit 15 of the server 10 may store a combination of a previously generated image generation prompt and a stamp image generated using that image generation prompt. Then, for example, when an image generation prompt similar to a previously generated image generation prompt is generated, the control unit 11 of the server 10 may not execute the stamp image generation process, but may use the previously generated stamp image as the generation result of the additional learning model.

[0397] The similarity of the image generation prompts may be determined based on, for example, the cosine similarity between the embedded representations of the prompts, and the image generation prompts may be determined to be similar if the cosine similarity is greater than or equal to a threshold value (e.g., 0.7).

[0398] This allows the stamp image generation process using an additional learning model to be omitted for similar prompts, thereby reducing resources and processing time, and by minimizing user wait times for operations, users can use the service more comfortably.

[0399] <Third Modification (4)> In the above embodiment, the stamp image generation prompt generation process and the stamp image generation process are executed in the server 10, but this is not limiting. For example, the control unit 21 of the terminal 20A may execute some or all of the stamp image generation prompt generation process, the stamp image generation process, and the stamp text synthesis process.

[0400] For example, when the control unit 21 of the terminal 20A receives text content, the control unit 21 may execute a stamp image generation prompt generation process based on the received text content. Then, when the image generation prompt is generated, stamp image generation request information including the image generation prompt may be transmitted to the server 10. For example, when the control unit 11 of the server 10 receives the stamp image generation request information from the terminal 20A, the control unit 11 may execute a stamp image generation process based on the received image generation prompt.

[0401] By executing some or all of the processing in the server 10 on the terminal 20A, the load on the server 10 can be reduced.

[0402] <Third Modification (5)> In the above embodiment, in the generated stamp selection information, one stamp is selected on the terminal 20A based on the stamp image included in the generated stamp list information, and stamp content information is transmitted to the terminals 20 participating in the talk room. However, this is not limited to this. For example, in the generated stamp selection information, multiple stamp images may be selected from the stamp images included in the generated stamp list information.

[0403] For example, when a first stamp image and a second stamp image are selected from the generated stamp list information, the control unit 11 of the server 10 may sequentially transmit first stamp content information including the first stamp image and second stamp content information including the second stamp image to the terminal 20. That is, for example, in a talk room, a stamp based on the first stamp image may be displayed, and then a stamp based on the second stamp image may be displayed.

[0404] The control unit 11 of the server 10 may transmit stamp content information including the first stamp image and the second stamp image to the terminal 20. That is, for example, a plurality of stamp images may be displayed at once in a talk room.

[0405] Fourth Example The fourth example is an example in which a stamp image is generated and sent to a talk room using an incremental learning model created using a base image provided by the user of the server 10 or the terminal 20. The content described in the fourth example is similarly applicable to any of the other examples and other modified examples.

[0406] In this embodiment, an example of the functions realized by the control unit 11 of the server 10 may be configured in the same manner as in FIG. 2-1, for example.

[0407] 4A is a diagram illustrating an example of information stored in the storage unit 15 of the server 10 in this embodiment. The storage unit 15 stores, for example, an application management processing program 151, account registration data 153, and stamp image additional learning model management data 155.

[0408] <Display Screen> Figures 4-2 and 4-3 are diagrams showing examples of screens displayed on the display unit 24 of the terminal 20A in this embodiment. In this example, it is assumed that an additional learning model has already been created in which images including a rabbit character, a bear character, etc. have been additionally learned in the image generation model unit 113. It is also assumed that the learned rabbit character is associated with the instance prompt "cpc," the bear character is associated with the instance prompt "usu," and the pet Pomeranian is associated with the instance prompt "sks."

[0409] Each additional learning model may be independent for each object (instance prompt), or a single additional learning model may be configured to additionally learn a plurality of objects collectively.

[0410] The left side of FIG. 4-2 shows, for example, an example of a chat room screen between user A.A and user B.B, who is a friend of user A.A. When the user taps the stamp image generation button SCB2, for example, the display changes to, for example, the chat room screen shown in the center of FIG. 4-2.

[0411] For example, this screen is configured to display an instance prompt selection area ISR1 above the content input area for specifying an instance prompt in the stamp image generation process. The instance prompt selection area ISR1 is configured to display images corresponding to instance prompts that have been learned in the additional learning model side by side below the character string "Please select an example stamp to generate."

[0412] The image corresponding to the instance prompt may be, for example, one of the base images associated with the instance prompt, or one of the stamp images generated based on the instance prompt.

[0413] Additionally, the instance prompt selection area ISR1 may display, for example, images corresponding to the following instance prompts so that they can be selected: - An instance prompt of an additional learning model created by the stamp sender himself / herself (in this case, user A.A). - An instance prompt of an additional learning model created by the participants in the chat room (in this case, users A.A and B.B). - An instance prompt of an additional learning model created by the operator of the messaging service (for example, a user of server 10). - Any combination of the above.

[0414] For example, when a user taps the image of a Pomeranian showing the instance prompt "sks," a stamp image generation process is executed based on the character string entered in the text message input area MIR1 and the instance prompt selected in the instance prompt selection area ISR1. Then, for example, the display changes to the chat room screen shown on the right side of FIG. 4-2.

[0415] On this screen, the indicator for the stamp image generation progress indicator progresses, and when the stamp image generation process is completed, the display changes to, for example, the chat room screen shown on the left side of Figure 4-3. On this screen, the generated stamp list information display area SSR2 is configured to display a predetermined number (for example, "5") of stamps, including stamp images generated by an incremental learning model using the instance prompt "sks" based on the character string "thank you." Note that the character string "thank you" entered in the text message input area MIR1 may not be superimposed on each stamp displayed in the generated stamp list information display area SSR2.

[0416] For example, when the user taps the rightmost stamp in the generated stamp list information display area SSR2, the display changes to the talk room screen shown on the right side of Figure 4-3. On this screen, the stamp content SCT2 selected in the generated stamp list information display area SSR2 is sent to and displayed in the talk room content display area TR1.

[0417] 4-4 is a flowchart showing an example of the flow of processing executed by each device in this embodiment. From the left, this figure shows an example of processing executed by the control unit 21 of terminal 20A, an example of processing executed by the control unit 21 of terminal 20B, and an example of processing executed by the control unit 11 of server 10.

[0418] First, the control unit 21 of the terminal 20A and the control unit 11 of the server 10 execute a stamp image additional learning process (A410 and S410) based on a user input to the terminal 20A. The stamp image additional learning process may be executed in accordance with steps A110 to A130 and S110 to S150 in FIG. 1-13.

[0419] In the stamp image additional learning process, the control unit 11 of the server 10 may, for example, read the base image set information in accordance with the image set stored in the storage unit of the server 10 based on a user input to the server 10. In other words, the base image to be learned by the additional learning model may be provided by a service operator such as a user of the server 10.

[0420] In addition, in the stamp image additional learning process, the control unit 11 of the server 10 may cause the additional learning model to learn a plurality of base images and a plurality of instance prompts or class prompts associated with the base images.

[0421] As an example, when stamp text information is received from terminal 20A via communication I / F 14, the control unit 11 of server 10 refers to the stamp image additional learning model management data 155 and sends stamp instance selection request information to terminal 20A via communication I / F 14 to select a registered instance prompt (S430).

[0422] When stamp instance selection request information is received from the server 10 via the communication I / F 22, the control unit 21 of the terminal 20A, for example, causes the stamp instance selection request information to be displayed on the display unit 24. Then, for example, based on a user input in response to the displayed stamp instance selection request information, the control unit 21 of the terminal 20A transmits stamp instance information for specifying an instance prompt to the server 10 via the communication I / F 22. Note that the stamp instance information may include the instance prompt, or may include information for identifying the instance prompt.

[0423] When the stamp instance information is received from the terminal 20A via the communication I / F 14, the control unit 11 of the server 10 executes a stamp image generation prompt generation process based on the stamp text provided in the stamp text information and the instance prompt and class prompt specified in the stamp instance information (S210). Note that the control unit 11 of the server 10 may, for example, generate a prompt for image generation without using the language model unit 115, taking into consideration the third modified example (1).

[0424] Then, the control unit 11 of the server 10 uses the generated image generation prompt to repeat the stamp image generation process (S170) and the stamp text synthesis process (S220) a predetermined number of times while changing the seed value, for example.

[0425] The control unit 11 of the server 10 may repeat the stamp image generation prompt generation process to generate different image generation prompts, and may repeat the stamp image generation process (S170) and the stamp text synthesis process (S220) a predetermined number of times. Also, the stamp text synthesis process may not be executed.

[0426] Then, the control unit 11 of the server 10 transmits generated stamp list information including the predetermined number of generated stamp images to the terminal 20A via the communication I / F 14 (S350).

[0427] Effect of Fourth Example In this example, the server 10 acquires base images (not limited to, but an example of a first image set) that are composed of images including a rabbit character and a Pomeranian (not limited to, but an example of a first element), and the image set is composed of images based on the first element. This allows the first user to select an image from the image set that is composed of images based on the first element. This improves convenience for the first user.

[0428] Furthermore, by using the first image set as an image set for additional learning for the image generation model unit 113 (not limited to, but an example of an image generation device), and having the image generation device learn the first image set based on an instance prompt or class prompt (not limited to, but an example of the fourth information) associated with the first image set, the server 10 can generate an image based on the first element using a prompt based on the fourth information.

[0429] In addition, this shows an example of a configuration in which the server 10 performs control to prevent the first image set from being learned if the first element satisfies a set condition. This makes it possible to prevent the image generating device from learning the first element included in the first image set based on the set condition. As a result, it is possible to prevent the image generating device from generating an image including the first element that satisfies the set condition.

[0430] Furthermore, this configuration can prevent the generation of images in the image set that include elements that are subject to copyright infringement or that are contrary to public order and morals. It can also prevent the first image set from being trained improperly due to an inappropriate number of images in the first image set or the first image set including multiple elements.

[0431] In this embodiment, the server 10 receives a first image set from the first terminal. This allows the server 10 to generate an image set based on the first image set, which is composed of images including a first element desired by the first user. This allows the first user to express and include the desired first element in an image. The first user can then select an image from the image set that includes the first element.

[0432] In addition, when the first element satisfies the set condition, the server 10 controls a notification to the first terminal, thereby notifying the first user of the first terminal that the first image set has been transmitted that the set condition has been satisfied.

[0433] Furthermore, the server 10 can generate an image set using the second image set by prompting the first user to send a second image set that is different from the first image set.

[0434] In addition, the server 10 can notify the first user that the first image set contains images that are subject to copyright infringement or that contain elements that are contrary to public order and morals, and that additional learning of the first image set may not be performed properly.

[0435] In the above embodiment, the user of terminal 20A selects one instance prompt related to the image generation prompt, but this is not limited to this. When multiple objects are collectively additionally learned in one additional learning model, for example, images corresponding to multiple instance prompts may be selectable in the instance prompt selection area ISR1, and a stamp image may be generated based on the multiple instance prompts.

[0436] This allows the stamp image to include multiple elements selected by the user, allowing users to send expressive stamps to their chat partners that fit the flow of the chat room.

[0437] <Fourth Modification (2)> In the above embodiment, as an example, the user of the terminal 20A selects an instance prompt or a class prompt related to the image generation prompt, but this is not limited to this. For example, in the stamp image generation prompt generation process, the control unit 11 of the server 10 may randomly select an instance prompt that is available to the user of the terminal 20A from the instance prompts stored in the stamp image additional learning model management data 155, and generate the image generation prompt.

[0438] In this case, for example, when the user taps the stamp image generation button SCB1 on the left side of Fig. 4-2, the instance prompt selection area ISR1 may not be displayed, and a stamp image including multiple objects may be displayed in the generated stamp list information display area SSR2. This allows the user to easily select a desired stamp from a wide variety of stamp images.

[0439] In addition, when multiple objects are collectively additionally learned in one additional learning model, for example, the control unit 11 of the server 10 may randomly select multiple instance prompts to generate an image generation prompt. Also, the image generation prompt may be generated using an instance prompt based on a base image registered by the business operator and an instance prompt based on a base image registered by the user of the terminal 20. This allows for the generation of stamp images with an even greater variety.

[0440] In the above embodiment, the image generation prompt is generated based on the text message (character string) entered in the text message input area MIR1. However, the present invention is not limited to this. For example, the image generation prompt may be generated based on past content information sent and received in the current chat room.

[0441] 4-5 and 4-6 are diagrams showing an example of a screen displayed on the display unit 24 of the terminal 20A in this modified example. The left side of FIG. 4-5 shows, as an example, an example of a chat room screen between user A.A and user B.B, who is a friend of user A.A. In this screen, in addition to text messages CT1 and CT2, a text message CT3 sent by user A.A to confirm the location of the campsite and a text message CT4 sent by user B.B to confirm whether or not to bring their pet Pomeranian are displayed in the chat room content display area TR1. Furthermore, the user selects the text message input area MIR1, and the character string "Of course!" is entered and displayed.

[0442] For example, when the user taps the stamp image generation button SCB3, the display changes to the chat room screen shown in the center of Figure 4-5. On this screen, for example, when the user taps the image of a Pomeranian showing the instance prompt "sks" in the instance prompt selection area ISR2, for example, a stamp image generation process is executed based on the text messages CT1 to CT4 sent and received in the chat room, the character string entered in the text message input area MIR1, and the instance prompt selected in the instance prompt selection area ISR2. Then, for example, the display changes from the chat room screen shown on the right side of Figure 4-5 to the chat room screen shown on the left side of Figure 4-6.

[0443] On this screen, the generated stamp list information display area SSR3 is configured to display a predetermined number (for example, "5") of stamps including stamp images generated by an additional learning model using the instance prompt "sks" based on the content of the chat in this chat room and the string "Of course!".

[0444] For example, when a user taps the rightmost stamp in the generated stamp list information display area SSR3, which includes an image of a Pomeranian walking through a campsite, the display changes to the chat room screen shown on the right side of Figure 4-6. On this screen, the stamp content SCT3 selected in the generated stamp list information display area SSR3 is sent to and displayed in the chat room content display area TR1.

[0445] In the process of this modified example, for example, in the stamp image generation prompt generation process, the following template may be used as a prompt template for creating a prompt for image generation: "<Rule> Create a prompt that instructs the image generation model to generate an illustration. Create an English keyword list for the prompt from the sentence you will enter. Use only adjectives, verbs, and nouns in the keyword list. The first keyword is "*instance prompt* *class prompt*". <Output> *instance prompt* *class prompt*, keyword, keyword, (loop) <Input> *dialogue 1* *dialogue 2* ... *dialogue N* *stamp text*"

[0446] For example, from *Dialogue 1* to *Dialogue N*, the content information (which may also be called text content or character strings) sent and received within the talk room may be replaced in chronological order.

[0447] Here, the N pieces of content information sent and received within the talk room to be applied to the prompt template for creating a prompt for image generation may be selected as follows, for example: Content information sent and received within a predetermined period (for example, within five minutes) before the stamp text information was sent; Content information sent and received a predetermined number of times (for example, four messages) before the stamp text information was sent; Content information selected by the user of terminal 20A. These pieces of content information may be said to be the context of the chat in the talk room.

[0448] In this example, the stamp text may be left blank (e.g., no input is made in the text message input area MIR1). In this case, the stamp image may be generated based on the content information sent and received within the chat room.

[0449] This configuration allows stamp images to be generated based on the content information sent and received in a chat room, allowing users to send stamps that match the content of the conversation, further enhancing communication.

[0450] This modification shows an example of a configuration in which the server 10 generates an image set based on the context of the chat in the chat room. As a result, each image included in the image set can be generated as an image that reflects the context of the chat room, and the first user can select an image from the image set that matches the content of the chat with the second user. The second user can then confirm the third information based on the image that matches the content of the chat.

[0451] <Fourth Modification (4)> In the above embodiment, the instance prompt selection area ISR1 can select, for example, an instance prompt of an additional learning model created by the operator of the messaging service or an instance prompt of an additional learning model created by the user of the terminal 20 sending the stamp, but this is not limited to this. For example, a user sending a stamp can purchase a license to use an additional learning model created by a general user or an official user, thereby being able to select an instance prompt in the instance prompt selection area ISR1, etc., and a stamp image can be generated based on the additional learning model for which the license has been purchased.

[0452] In this modified process, for example, the control unit 11 of the server 10 executes payment processing for purchasing a stamp when it receives stamp purchase request information from the terminal 20. The control unit 11 of the server 10 may then store the application ID of a user who can use the additional learning model (instance prompt) in association with other model information in the stamp image additional learning model management data 155. Then, when it receives stamp text information from the terminal 20A, the control unit 11 of the server 10 may refer to the other model information in the stamp image additional learning model management data 155 and search for an additional learning model (instance prompt) in which the application ID of the terminal 20A is registered. The control unit 11 of the server 10 may then send stamp instance selection request information to the terminal 20A to select the searched instance prompt.

[0453] The purchase of a usage right may include the following concepts, for example: Purchase of a permanent usage right Purchase of a usage right with a set number of uses Purchase of a usage right with a set usage period

[0454] <Others> In the above embodiments, the elements included in the image (image elements) may include, in addition to the objects described above, for example, the style or style of the image. Furthermore, the image based on the element may include an image of the style (an image reflecting the style), an image of the style (an image reflecting the style), etc. In this case, too, an image set (second image set) based on elements such as the style or style can be generated by the image generation device using a method similar to that of the above embodiments.

[0455] As another example, (1) first element: object, second element: style (2) first element: object, second element: style (3) other combinations, and the information processing device acquires, for example, a first image set including a plurality of elements (first element, second element, etc.), and then causes an image generation device to learn the first image set based on information associated with the first image set, so that a second image set composed of images based on the above-mentioned plurality of elements is generated by the image generation device.

[0456] At least a part of the processing that is to be performed by the server 10 in the above-described embodiments may be performed by the terminal 20. Conversely, at least a part of the processing that is to be performed by the terminal 20 in the above-described embodiments may be performed by the server 10.

[0457] The operator of server 10 may also be an image generation service provider in partnership with a messaging service provider. In this case, the processes described in the above embodiments may be realized by a single server, or the processes described in the above embodiments may be shared and realized by a server system consisting of multiple servers.

[0458] For example, a system consisting of one or more servers may be defined as a server system, and the server of the present invention may be a server system. Also, for example, the server system may be configured to include an information processing device, an image generation device, and a language generation device.

[0459] Furthermore, in the above-described embodiments, the server for distributing various applications (the server from which the terminal 20 downloads the applications) may be configured as a server different from the server that provides the corresponding service (application). In other words, the server for distributing applications and the server that performs the application management process described in the above-described embodiments may be configured as physically separated servers, or may be configured as a single server.

[0460] Furthermore, applications are not limited to various application programs, but may also include, for example, a program that provides a function of another service as one function of a base application (for example, a program that provides a function of an advertisement distribution service as one function of a messaging application, or vice versa), a program for updating the base application, etc. Data used in application programs (which may include data for updating applications, etc.) may also be included.

[0461] Furthermore, in the above embodiments, the present invention has been described as being implemented using a client-server system, but is not limited to this. As mentioned above, the present invention may be implemented using a system such as a distributed system in which the functions of a server or server system are provided in the terminal 20. For example, the processes described in the flowcharts of the above embodiments as being performed by a server or server system may be performed by a terminal.

[0462] Furthermore, as mentioned above, the contents described in the above-mentioned embodiments, modifications, other embodiments, etc. can be applied in combination with each other.

[0463] 1 Communication system 10 Server 20 Terminal 30 Network

Claims

1. A program executed by a server that performs processing related to a chat including a first user of a first terminal and a second user of a second terminal, the program being executed by the server to receive text information from the first terminal based on an input to the first terminal displaying a chat room including the first user and the second user, generate an image set including one or more images based on the text information, transmit first information related to the image set to the first terminal by the communication unit, receive second information from the first terminal by the communication unit related to one or more images selected from the image set based on an input to the first terminal displaying images included in the image set in the chat room based on the first information, and transmit third information based on the second information to the second terminal by the communication unit.

2. The program of claim 1, wherein the set of images is generated by an image generation device that generates images based on a prompt, the prompt being based on the text information.

3. The program according to claim 2, wherein the prompt is generated by a language processing device.

4. The program according to claim 1, wherein the server executes the steps of obtaining a first image set consisting of images including a first element, and the image set is generated based on the first image set as an image set consisting of images based on the first element.

5. The program as described in claim 4, wherein the image set is generated by an image generation device that generates images based on a prompt, the first image set is an image set for additional learning for the image generation device, and the server executes the step of having the image generation device learn the first image set based on fourth information associated with the first image set.

6. The program according to claim 5, wherein, when the first element satisfies a set condition, the server executes control to prevent the first image set from being learned.

7. The program as described in claim 6, wherein the set condition includes at least one of the following: the first element being an element that is subject to copyright infringement, the first element being an element that is contrary to public order and morals, the first element including multiple elements, and the number of images constituting the first image set being less than or equal to a set number.

8. The program according to claim 4, wherein the server executes the process of receiving the first image set from the first terminal via the communication unit.

9. The program according to claim 8, wherein, when the first element satisfies a set condition, the server controls a notification to the first terminal.

10. The program as described in claim 9, wherein the notification is a notification prompting transmission of a second image set different from the first image set, the second image set is received from the first terminal via the communication unit by the server, and the image set is generated based on the second image set.

11. The program as described in claim 9, wherein the set condition includes at least one of the following: the first element being an element that is subject to copyright infringement, the first element being an element that is contrary to public order and morals, the first element including multiple elements, and the number of images constituting the first image set being less than or equal to a set number.

12. The program of claim 1, wherein generating the set of images based on a chat context of the chat room is executed by the server.

13. A program as described in claim 1 or claim 2, wherein the image set is generated using a model trained based on input to the first terminal, and the trained model is not reflected in terminals other than the first terminal.

14. The program according to claim 1 or 2, wherein the images included in the image set contain text based on the text information.

15. The program according to claim 1 or 2, wherein the image set is a sticker image set or an emoji image set.

16. The program of claim 15, wherein an image included in the sticker image set is displayed in the chat room by a chat transmission in a manner different from text that is displayed in the chat room by the chat transmission.

17. The program according to claim 1 or 2, wherein a second button for generating the image set is displayed in the chat room, the second button being different from a first button for sending a chat to the chat room.

18. A program as described in claim 17, wherein the chat room displays an area for chat input to the chat room, and when input to the second button is made after input of text to the area, the image set is generated with the input text as a prompt, and when input to the first button is made after input of text to the area, the input text is sent as a chat to the chat room.

19. An information processing method of a server that executes processing related to a chat including a first user of a first terminal and a second user of a second terminal, comprising: receiving text information from the first terminal by a communication unit of the server based on an input to the first terminal displaying a chat room including the first user and the second user; generating an image set including one or more images based on the text information; transmitting first information related to the image set to the first terminal by the communication unit; receiving second information from the first terminal by the communication unit related to one or more images selected from the image set based on an input to the first terminal displaying images included in the image set in the chat room based on the first information; and transmitting third information based on the second information to the second terminal by the communication unit.

20. A server that executes processing related to a chat including a first user of a first terminal and a second user of a second terminal, comprising a communication unit and a control unit, wherein the communication unit receives text information from the first terminal based on an input to the first terminal displaying a chat room including the first user and the second user, the control unit generates an image set including one or more images based on the text information, the communication unit transmits first information related to the image set to the first terminal, receives second information from the first terminal related to one or more images selected from the image set based on an input to the first terminal displaying images included in the image set in the chat room based on the first information, and transmits third information based on the second information to the second terminal.

Images