Programs, methods, information processing devices, systems

Abstract

We will realize the first tag cloud, which consists of high-precision tags. [Solution] This is a program to be executed on a computer equipped with a processor and memory. The program causes the processor to perform the following steps: accept input of multiple images; input for each of the input images, an instruction sentence that instructs the output of the image and a descriptive sentence about the image to a multimodal generation AI model, and have the multimodal generation AI model output the descriptive sentences; analyze the output descriptive sentences and extract multiple characteristic words as tags; and present the extracted tags as a first list.

Description

Technical Field

[0001] The present disclosure relates to a program, a method, an information processing apparatus, and a system.

Background Art

[0002] As a user interface often used in sites that perform social tagging, there is a user interface called a tag cloud. A tag cloud is a list-form expression of tags based on the number of tag registrations, appearance frequency, popularity, importance, etc. By the way, when trying to implement a tag cloud, for example, it is necessary to attach a plurality of tags to one content. Attaching a plurality of tags to content requires effort. To address this problem, for example, in the technology disclosed in Patent Document 1, an artificial intelligence (AI) - based image classification tool is used to perform identification of specific image features and / or generation of image tags.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Specifically, image analysis using the aforementioned image classification tool can identify a plurality of image features such as smiling faces, waitresses, counters, vending machines, coffee cups, cakes, hands, food, people, cafes, etc. Therefore, an image can be tagged with each of these identified features. However, in tagging using an image classification tool, the accuracy of the set tags is not high.

[0005] An object of the present disclosure is to realize a tag cloud composed of highly accurate tags. [Means for solving the problem]

[0006] This is a program to be executed on a computer equipped with a processor and memory. The program causes the processor to perform the following steps: accept multiple images as input; input for each of the input images, an instruction sentence that instructs the output of the image and a descriptive sentence about the image to a multimodal generation AI model, and have the multimodal generation AI model output the descriptive sentences; analyze the output descriptive sentences and extract multiple characteristic words as tags; and present the extracted tags as a first list. [Effects of the Invention]

[0007] According to this disclosure, a tag cloud composed of high-precision tags can be realized as the first list. [Brief explanation of the drawing]

[0008] [Figure 1] This is a block diagram showing an example of the overall configuration of System 1. [Figure 2] This block diagram shows an example configuration of the terminal device 10 shown in Figure 1. [Figure 3] Figure 1 is a block diagram showing an example of the functional configuration of server 20. [Figure 4] This diagram shows the data structure of the first tag cloud. [Figure 5] This figure shows the data structure of the user information table 2021, as shown in Figure 3. [Figure 6] This figure shows the data structure of the image table 2022, as shown in Figure 3. [Figure 7] This figure shows the data structure of the prompt table 2023, as shown in Figure 3. [Figure 8] This figure shows the data structure of the explanatory text table 2024, as shown in Figure 3. [Figure 9] This flowchart shows an example of how server 20 operates when presenting the first tag cloud to the user. [Figure 10] This is a schematic diagram showing an example of the display screen of the display 141 when the first tag cloud 40 is presented to the user. [Figure 11] This is a schematic diagram showing an example of the display screen of the display 141 when a user searches the first tag cloud 40 shown in Figure 10. [Figure 12] This is a schematic diagram showing another example of the display screen of display 141 when the first tag cloud 401 is presented to the user. [Figure 13] A block diagram showing the basic hardware configuration of Computer 90. [Modes for carrying out the invention]

[0009] Embodiments of this disclosure will be described below with reference to the drawings. In the following description, the same parts are denoted by the same reference numerals. Their names and functions are also the same. Therefore, detailed descriptions of them will not be repeated.

[0010] <Overview> The system according to this embodiment has a function for generating tags related to images. For example, for each of a plurality of images input by the user, the system according to this embodiment inputs the image and an instruction text that instructs the output of a descriptive text about the image to a multimodal generation AI model (details described later). There are no particular limitations on the content of the images input to the multimodal generation AI model; they may be images related to restaurants, apparel stores, real estate properties, etc. Also, for example, if the image input to the multimodal generation AI model is an image related to a restaurant, then izakayas, restaurants, cafes, etc., are also included as the objects captured in the image.

[0011] The system according to this embodiment, for example, performs natural language analysis on the description text output to the multimodal generation AI model to extract a plurality of characteristic words, and uses each as a tag. Then, the system according to this embodiment accumulates the tags extracted for a plurality of description texts. The system presents the accumulated plurality of tags to the user as a list. This list becomes the first tag cloud. Thereby, a first tag cloud composed of high-precision tags can be realized.

[0012] <1 Configuration diagram of the entire system> FIG. 1 is a block diagram showing an example of the overall configuration of the system 1. The system 1 shown in FIG. 1 includes, for example, a terminal device 10, a server 20, and a multimodal generation AI system 30 (hereinafter, “MAI system 30”). The terminal device 10, the server 20, and the MAI system 30 are communicatively connected via, for example, a network 80.

[0013] In FIG. 1, an example in which the system 1 includes two terminal devices 10 is shown, but the number of terminal devices 10 included in the system 1 may be less than three or may be three or more. In FIG. 1, an example in which the system 1 includes one MAI system 30 is shown, but the number of MAI systems 30 included in the system 1 may be two or more.

[0014] In FIG. 1, an example in which the MAI system 30 is independent of the server 20 is shown, but the server 20 may include the functions of the MAI system 30. That is, the server 20 may store a multimodal generation AI model.

[0015] In this embodiment, an aggregate of a plurality of devices may be regarded as one server. The way of distributing the plurality of functions required to realize the server 20 according to this embodiment to one or more hardware can be appropriately determined according to the processing capabilities of each hardware and / or the specifications required for the server 20.

[0016] The terminal device 10 shown in Figure 1 is an information processing device operated by a user utilizing the first tag cloud (first list). The terminal device 10 can be implemented as, for example, a stationary PC (Personal Computer), a laptop PC, etc. The terminal device 10 may also be implemented as a mobile device such as a smartphone or tablet.

[0017] The terminal device 10 comprises a communication interface 12, an input device 13, an output device 14, memory 15, storage 16, and a processor 19. The input device 13 is a device for receiving input operations from the user (e.g., a touch panel, touchpad, mouse or other pointing device, keyboard, etc.). The output device 14 is a device for presenting information to the user (display, speaker, etc.).

[0018] Server 20 is an information processing device that deploys a first tag cloud service using, for example, a multimodal generation AI model. Server 20 transmits multiple images input by the user via terminal device 10 to MAI system 30. Server 20 also transmits instruction texts input by the user via terminal device 10 to MAI system 30.

[0019] An instruction statement is a statement that instructs the output of a descriptive statement (hereinafter, "descriptive statement") about an image input by the user. For example, different instruction statements may be input for each of multiple images, or the same instruction statement may be input for some or all of multiple images. Note that instruction statements do not necessarily have to be input to the server 20 via the terminal device 10. For example, one or more default instruction statements depending on the type of object being captured displayed in each of the multiple images may be stored in advance in the prompt table 2023. For example, when an image is input, the server 20 selects an instruction statement stored in the prompt table 2023 according to a predetermined rule.

[0020] Server 20 outputs a descriptive text to the MAI system 30. For example, if the object captured in the image is a restaurant, the restaurant's advertisement will serve as the descriptive text. Server 20 analyzes the descriptive text output from the MAI system 30 and extracts multiple tags. Server 20 sends the extracted tags as a first tag cloud to the terminal device 10 and presents them to the user via the output device 14.

[0021] There are no particular limitations on how the first tag cloud is presented to the user. The server 20 may design the text color, text size, font, display direction, and arrangement of the multiple tags constituting the first tag cloud in various ways, taking into consideration user convenience, etc.

[0022] The server 20 is an information processing device implemented by, for example, a computer connected to the network 80. As shown in Figure 1, the server 20 includes a communication IF 22, an input / output IF 23, memory 25, storage 26, and a processor 29. The input / output IF 23 functions as an interface for an input device that receives input operations from the user and an output device that outputs information to the user.

[0023] The MAI system 30 includes a multimodal generative AI model. A multimodal generative AI model is a large-scale language model built using deep learning that collects two or more types of information, such as text, audio, images, and videos, and processes them integrally. Examples of multimodal generative AI models include "OpenAI ChatGPT GTP-4 (registered trademark)" and "Google Gemini."

[0024] The MAI system 30 inputs the images and instruction text received from the server 20 into the multimodal generation AI model and causes the multimodal generation AI model to output an explanatory text. The MAI system 30 then sends the explanatory text output by the multimodal generation AI model to the server 20.

[0025] Each information processing device consists of a computer 90 (see Figure 11) equipped with an arithmetic unit and a memory device. The basic hardware configuration of the computer 90 and the basic functional configuration of the computer 90 realized by this basic hardware configuration will be described later. For the terminal device 10, server 20, and MAI system 30, explanations that overlap with the basic hardware configuration of the computer 90 and the basic functional configuration of the computer will be omitted.

[0026] <1.1 Terminal Device Configuration> Figure 2 is a block diagram showing an example configuration of the terminal device 10 shown in Figure 1. As shown in Figure 2, the terminal device 10 comprises a communication unit 120, an input device 13, an output device 14, a storage unit 180, and a control unit 190. Each block included in the terminal device 10 is electrically connected, for example, by a bus. The terminal device 10 may also include an audio processing unit, a microphone, a speaker, a camera, a location information sensor, or at least two combinations thereof.

[0027] The communication unit 120 performs processing such as modulation and demodulation processing for the terminal device 10 to communicate with other devices. The communication unit 120 performs transmission processing on the signal generated by the control unit 190 and sends it to an external source (for example, the server 20). The communication unit 120 performs reception processing on the signal received from an external source and outputs it to the control unit 190.

[0028] The input device 13 is a device for a user operating the terminal device 10 to input instructions or information. The input device 13 is implemented, for example, by a touch-sensitive device 131 on which instructions are input by touching the operating surface. If the terminal device 10 is a PC or the like, the input device 13 may be implemented by a reader, keyboard, mouse, etc. The input device 13 converts the instructions input by the user into electrical signals and outputs them to the control unit 190. The input device 13 may also include, for example, a receiving port that accepts electrical signals input from an external input device.

[0029] The output device 14 is a device for presenting information to the user operating the terminal device 10. The output device 14 is implemented, for example, by a display 141. The display 141 displays various information according to the control of the control unit 190. The display 141 is implemented, for example, by an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) display.

[0030] The storage unit 180 is implemented, for example, by memory 15 and storage 16, and stores data and programs used by the terminal device 10. The storage unit 180 stores, for example, user information 181. The user information 181 includes, for example, various information about the user who uses the terminal device 10. The various information about the user includes, for example, the user's name, age, address, date of birth, and contact information.

[0031] The control unit 190 is realized when the processor 19 reads a program stored in the memory unit 180 and executes the instructions contained in the program. The control unit 190 controls the operation of the terminal device 10. By operating according to the program, the control unit 190 performs the functions of an operation reception unit 191, a transmission / reception unit 192, and a presentation control unit 193.

[0032] The operation reception unit 191 performs processing to receive instructions or information input from the input device 13. Specifically, the operation reception unit 191 receives instructions or information input from the touch-sensitive device 131.

[0033] The transmitting / receiving unit 192 performs processing to enable the terminal device 10 to send and receive data with an external device such as the server 20 in accordance with a communication protocol. Specifically, the transmitting / receiving unit 192 transmits instructions or information input by the user to the server 20. The transmitting / receiving unit 192 receives information transmitted from the server 20.

[0034] The presentation control unit 193 controls the output device 14 to present the information transmitted from the server 20 to the user. Specifically, the presentation control unit 194 causes the first tag cloud transmitted from the server 20 to be displayed on the display 141.

[0035] <1.2 Functional Configuration of the Server> Figure 3 is a block diagram showing an example of the functional configuration of the server 20 shown in Figure 1. As shown in Figure 3, the server 20 performs the functions of a communication unit 201, a storage unit 202, and a control unit 203.

[0036] The communication unit 201 performs processing for the server 20 to communicate with external devices. The storage unit 202 includes, for example, a user information table 2021, an image table 2022, a prompt table 2023, and a description table 2024.

[0037] The tables held by the storage unit 202 are not limited to these. For example, the image table 2022 does not have to be stored in the storage unit 202. Depending on the content of the image, different services may use the first tag cloud. Therefore, images may be stored for each server that manages the service. Also, for example, the image table 2022 may be stored on a server related to a service for sharing information among multiple users. Also, for example, the storage unit 202 may have a first tag cloud table 2025 as shown in Figures 3 and 4. The first tag cloud table 2025 is a table that stores the first tag cloud, and as shown in Figure 4, it has columns such as tag data and target, with the tag cloud ID as the key. The tag cloud ID is an item that stores an identifier for uniquely identifying the first tag cloud. The tag data is an item that stores data about multiple tags that constitute the first tag cloud. The target is an item that stores the imaged object captured in the image.

[0038] The User Information Table 2021 is a table that stores information about the user. The Image Table 2022 is a table that stores information about multiple images entered by the user. The Prompt Table 2023 is a table that stores information about instruction statements that are input to the MAI system 30. The Description Table 2024 is a table that stores information about description statements that are output from the MAI system 30. Details of these tables will be described later.

[0039] The control unit 203 is realized when the processor 29 reads a program stored in the memory unit 202 and executes instructions contained in the program. The program includes applications such as web browser applications. The program includes programming languages ​​such as JavaScript® that are executed on the web browser application stored in the terminal device 10. By operating according to the program, the control unit 203 performs the functions of the receiving control module 2031, the transmitting control module 2032, the tag generation module 2033, and the presentation control module 2035.

[0040] The receive control module 2031 controls the process by which the server 20 receives signals from an external device according to a communication protocol. The transmit control module 2032 controls the process by which the server 20 transmits signals to an external device according to a communication protocol.

[0041] The tag generation module 2033 performs the process of generating multiple tags using the description output from the MAI system 30. Specifically, the tag generation module 2033 analyzes the description output from the MAI system 30. For example, the tag generation module 2033 performs natural language analysis on the description. Natural language analysis may be performed using existing technologies or by a predetermined trained model. Based on the analysis results, the tag generation module 2033 extracts multiple characteristic words from the description and uses each of the extracted words as a tag.

[0042] Here, "characteristic words" refer to words that represent meanings that allow users of the first tag cloud service to easily recall the imaged object. The aforementioned service users who serve as the criterion for determining whether a word is a "characteristic word" are assumed to have an average level of knowledge about the imaged object. The tag generation module 2033 determines whether a word included in the description is a characteristic word, for example, based on dictionary information pre-stored in the memory unit 202. The dictionary information may be stored for each predetermined field. The dictionary information may be updated as needed. In addition, the dictionary information may learn new information in response to user operations.

[0043] The presentation control module 2035 controls the process of presenting information to the user. For example, the presentation control module 2035 controls the process of presenting multiple tags to the user by configuring multiple tags generated by the tag generation module 2033 as a first tag cloud.

[0044] <2 Data Structure> Figures 5 to 8 show the data structure of each table stored by server 20. Note that Figures 5 to 8 are examples and do not exclude data that is not shown. Also, even if data is listed in the same table, it may be stored in separate memory areas in storage unit 202.

[0045] Figure 5 shows the data structure of the user information table 2021. The user information table 2021 shown in Figure 5 is a table that uses User ID as the key and has columns for Name, Age, Gender, Date of Birth, and Contact Information.

[0046] The User ID field stores an identifier to uniquely identify the user. The Name field stores the user's name. The Age field stores the user's age. The Gender field stores the user's gender. The Date of Birth field stores the user's date of birth. The Contact Information field stores the contact information (e.g., phone number, email address, etc.) of the terminal device 10 that the user possesses.

[0047] Figure 6 shows the data structure of image table 2022. Image table 2022 shown in Figure 6 is a table with image ID as the key and columns for user ID, date and time, subject, and image. The columns included in image table 2022 are not limited to the example in Figure 6. Image table 2022 may also include columns that store information about the image and the associated facility, such as URL, business hours, address, contact information, and recommendations.

[0048] The Image ID is an item that stores an identifier to uniquely identify an image. The User ID is an item that stores an identifier to uniquely identify the person who entered the image. The User ID stored in Image Table 2022 corresponds, for example, to the User ID stored in User Information Table 2021. Note that the name of the person who entered the image may be stored instead of the User ID. The Date and Time is an item that stores the date and time the image was taken. Note that the Date and Time may also be an item that stores the date and time the image was entered.

[0049] The "Target" field stores the image of the object captured in the image. The objects include, for example, restaurants, apparel stores, and real estate properties. For example, if the object is a restaurant, then izakayas, restaurants, cafes, etc., are also included. The "Image" field stores image data entered by the user. If the image data relates to a restaurant or apparel store, the "Image" field stores, for example, an image of the interior of a restaurant or apparel store. If the image data relates to a real estate property, the "Image" field stores, for example, an image of the exterior of the property.

[0050] Figure 7 shows the data structure of prompt table 2023. Prompt table 2023, shown in Figure 7, is a table that has prompt data and target columns, with prompt ID as the key.

[0051] The prompt ID is an item that stores an identifier for uniquely identifying an instruction. The prompt data is an item that stores data related to the instruction. The data related to the instruction is, for example, text information that describes the content of the instruction. The text information that describes the content of the instruction is, for example, an input sentence that causes the MAI system 30 to create an explanatory text, and includes, for example, the following: "The image you entered is a picture taken of... (restaurant). Please create a description for this image that includes the following points." • History of (restaurants) • Current customer demographics and visitor trends The atmosphere inside and outside (restaurant) ·menu Location

[0052] The item "Prompt Data" may store reference information (path) to a prompt data file located elsewhere.

[0053] The target is an item that stores the object being captured in the image. The object stored in prompt table 2023 corresponds, for example, to the object stored in image table 2022. In other words, prompt data is pre-stored for each object being captured.

[0054] Figure 8 shows the data structure of the description table 2024. The description table 2024 shown in Figure 8 is a table that has a description ID as the key and columns for description data and image ID.

[0055] The Description ID is an item that stores an identifier to uniquely identify a description. The Description Data is an item that stores data about the description. The data about the description is, for example, text information that describes the content of the description. The text information that describes the content of the description is the sentence that serves as the basis for extracting multiple tags that make up the first tag cloud, and includes, for example, the following: "...(restaurant) is a new cafe that opened a month ago. Located on the banks of a river that flows through the heart of the downtown area, it offers a relaxing atmosphere where you can enjoy the scenery around the river. Its convenient location has made it popular with a wide range of customers, from young people to seniors. In addition to cafe menu items such as desserts, they also have a full lunch menu, making it suitable for various occasions."

[0056] The item "Description Data" may store reference information (path) to a prompt data file located elsewhere.

[0057] The image ID is an item that stores an identifier to uniquely identify an image, similar to the image IDs stored in image table 2022 and prompt table 2023, respectively.

[0058] <3 operations> Referring to Figure 9, the operation of server 20 when presenting the first tag cloud to the user will be explained. Figure 9 is a flowchart illustrating an example of the operation of server 20 when presenting the first tag cloud to the user. In the explanation of Figure 9, it is assumed that the type of object captured in each of the multiple images input by the user is a restaurant. The object to be captured may be specified by the user when the image is input. Alternatively, the object to be captured may be recognized by image analysis of the input image. It may also be recognized based on an information site where information about the image is registered. For example, if information about the image is registered on a gourmet site, the object to be captured will be recognized as a "restaurant". Note that the user may input only one image, not multiple images.

[0059] In step S11, the server 20 accepts input of multiple images. Specifically, the user operates the input device 13 to input multiple images to the terminal device 10. The operation reception unit 191 accepts the input of multiple images from the user. The transmission / reception unit 192 transmits the multiple images received by the operation reception unit 191 to the server 20. The reception control module 2031 receives the multiple images transmitted from the terminal device 10. In other words, the server 20 accepts input of multiple images from the user.

[0060] In step S12, the server 20 accepts the input of an instruction. Specifically, the user operates the input device 13 to input an instruction to the terminal device 10. The operation reception unit 191 accepts the instruction input from the user. The transmission / reception unit 192 transmits the instruction received by the operation reception unit 191 to the server 20. The reception control module 2031 receives the instruction transmitted from the terminal device 10. In other words, the server 20 accepts the input of an instruction from the user. Note that if the instruction is not input by the user but is pre-stored in the prompt table 2023, the system 1 does not need to execute the process in step S12.

[0061] In step S13, the server 20 causes the multimodal generation AI model to output a descriptive text corresponding to each of the multiple images. Specifically, the transmission control module 2032 transmits the multiple images received by the reception control module 2031 to the MAI system 30. Basically, the transmission control module 2032 transmits each image to the MAI system 30 sequentially each time the reception control module 2031 receives an individual image. However, the transmission control module 2032 may, for example, transmit all of the multiple images to the MAI system 30 at once. The transmission control module 2032 transmits the instruction text received by the reception control module 2031 to the MAI system 30.

[0062] When receiving input of instruction texts corresponding to each of multiple images, the transmission control module 2032 transmits the instruction text to the MAI system 30 simultaneously with the corresponding image each time the reception control module 2031 receives an individual instruction text. However, the timing of the instruction text transmission and the image transmission timing may differ.

[0063] If user input of instructions is not required, the transmission control module 2032 reads prompt data from the prompt table 2023 based on the item "Target" which corresponds to the object being captured in the input image. The transmission control module 2032 then sends the read prompt data as an instruction to the MAI system 30.

[0064] The MAI system 30 inputs multiple images and instruction texts received from the server 20 into a multimodal generation AI model. Based on the multiple images and instruction texts input into the multimodal generation AI model, the MAI system 30 causes the multimodal generation AI model to output a corresponding explanatory text for each of the multiple images. The MAI system 30 sends the multiple explanatory texts output from the multimodal generation AI model to the server 20. The receiving control module 2031 receives the multiple explanatory texts sent from the MAI system 30. In other words, the server 20 causes the multimodal generation AI model to output multiple explanatory texts.

[0065] In step S14, the server 20 extracts multiple tags. Specifically, the tag generation module 2033 analyzes the received multiple descriptive texts and extracts multiple characteristic words as tags. The tag generation module 2033 associates the extracted tags with images and stores them in the first tag cloud table 2025. The server 20 generates tags for each registered image and accumulates them in the first tag cloud table 2025.

[0066] In step S15, the server 20 presents the first tag cloud to the user. Specifically, for example, when the user requests the display of the first tag cloud, that is, when the user requests a search for restaurants, the presentation control module 2035 reads the tag data by referring to the item "Target" from the first tag cloud table 2025. The presentation control module 2035 configures the read tag data as the first tag cloud and displays the configured first tag cloud on the terminal device 10. In other words, the transmission control module 2032 sends information to the terminal device 10 to display the first tag cloud configured by the presentation control module 2035. In other words, the server 20 presents the first tag cloud to the user.

[0067] Alternatively, instead of the presentation control module 2035, the presentation control unit 193 may configure multiple tags as the first tag cloud. In this case, the transmission control module 2032 transmits the tag data read from the first tag cloud table 2025 to the terminal device 10.

[0068] The transmitting / receiving unit 192 receives information for displaying the first tag cloud transmitted from the server 20. The display control unit 193 displays the first tag cloud on the display 141 based on the received information for displaying the first tag cloud. Step S15 does not necessarily have to be performed after step S14. The server 20 may perform step S15 in response to a request from the user.

[0069] <4 Screen Examples> Figure 10 is a schematic diagram showing an example of the display screen of the display 141 when the first tag cloud 40 is presented to the user. In other words, Figure 10 is an example of a search screen using the first tag cloud. In Figure 10, the example is given where the type of object captured in each of the multiple images input by the user is a cafe (restaurant). The same applies to Figures 11 and 12 described later.

[0070] The display screen shown in Figure 10 shows multiple tags 41 that evoke the image of a cafe, displayed in a list format. These multiple tags 41 are based, for example, on tags registered for the same image target. This block of multiple tags 41 displayed in the list forms the first tag cloud 40. Specifically, the first tag cloud 40 shown in Figure 10 consists of three rows of records, and the number of different types of tags 41 included in each row is all different. In addition, all tags 41 in the first tag cloud 40 have the same text color, text size, font, and display direction. For example, the display direction of all multiple tags 41 is horizontal (left to right when facing the page).

[0071] In the example shown in Figure 10, the first tag cloud 40 is displayed in area 1411 of the display screen, but the first tag cloud 40 may be displayed in any area of ​​the display screen. This is also true in the examples shown in Figures 11 and 12 described later. The placement of the first tag cloud 40 on the display screen is controlled, for example, by the presentation control module 2035 or the presentation control unit 193.

[0072] The user operates on the first tag cloud 40 displayed on the screen shown in Figure 10, for example. In other words, the user selects one of the tags 41 included in the first tag cloud 40. The user may select one tag 41 or multiple tags 41 included in the first tag cloud 40. Specifically, "user selection of tags 41" refers to the operation in which the user selects one or more desired tags 41 from among the multiple tags 41 that make up the first tag cloud 40.

[0073] Here, the control unit 203 functions as an image-related information generation module. In this embodiment, the image-related information is, for example, information about an image whose description includes one or more selected tags. Specifically, the image-related information includes, for example, an image whose description includes one or more selected tags, a description of the image, information about the store represented by the image, or at least a combination of these. The information about the store represented by the image includes the store's URL, business hours, address, contact information, recommendations, etc.

[0074] The image-related information generation module, upon receiving a tag selection from the user, generates image-related information based on the selected tag. The image-related information may be stored, for example, in an image-related information table (not shown) held by the storage unit 202. The presentation control module 2035 presents the image-related information to the user. For example, the transmission control module 2032 transmits information for displaying the image-related information to the terminal device 10. Upon receiving the information for displaying the image-related information, the presentation control unit 193 transitions the display screen shown in Figure 10 to, for example, the display screen shown in Figure 11.

[0075] Figure 11 is a schematic diagram showing an example of the display screen of the display 141 when a user operates the first tag cloud 40 shown in Figure 10. In the display screen shown in Figure 11, the tags 41 selected in the first tag cloud 40 are displayed in area 1412. Also, in the display screen shown in Figure 11, image-related information 50 generated based on the tags 41 selected in the first tag cloud is displayed in area 1413. In the example in Figure 11, as a search result using the first tag cloud, several articles introducing cafes that are expected to meet the conditions desired by the user are displayed in list format as image-related information 50.

[0076] However, the display method of the search results in the first tag cloud is not limited to the example in Figure 11, and various display methods can be adopted. For example, in the example in Figure 11, before the image-related information 50 is displayed, only the number of image-related information 50 generated by the selected tag may be displayed. The user checks the number of items displayed and decides whether to display the image-related information 50 or to select more tags 41. For example, if the number of items displayed is more than expected, the user selects more tags 41 from the first tag cloud 40. Also, if the number of items is sufficient to display the image-related information 50, the user inputs an instruction to display the image-related information 50 from the input device 13. Note that, for the sake of simplicity of explanation and illustration, two image-related information 50 items are displayed in the example in Figure 11, but the displayed image-related information is not limited to two items.

[0077] <5 Summary> In this embodiment, the operation reception unit 191 and the reception control module 2031 receive input of multiple images. The MAI system 30 inputs the image and instruction text for each of the multiple images into the multimodal generation AI model. Based on the multiple images and instruction texts input into the multimodal generation AI model, the MAI system 30 causes the multimodal generation AI model to output an explanatory text corresponding to each of the multiple images. The tag generation module 2033 analyzes the multiple explanatory texts and extracts multiple characteristic words as tags. The presentation control module 2035 configures the multiple tags as a first tag cloud and presents it to the user.

[0078] As described above, in this embodiment, tags are generated using explanatory text obtained by inputting multiple images and instructional texts into a multimodal generation AI model. Therefore, the accuracy of the tags is improved compared to cases where tags are generated using an AI-based image classification tool. Accordingly, according to this embodiment, a first tag cloud composed of highly accurate tags can be realized.

[0079] <6 First Variation> System 1 may, for example, analyze the extracted tags and present a first tag cloud in a manner corresponding to the analysis results (first modified example). The analysis of the tags may involve, for example, analyzing the number of tags extracted, their frequency of occurrence, their importance, or at least a combination of two of these.

[0080] Analysis of multiple tags is achieved, for example, by the control unit 190 or control unit 203 functioning as a tag analysis module (not shown). For example, the tag analysis module analyzes the number of occurrences and frequency of occurrences of tags generated by the tag generation module 2033. The tag analysis module also analyzes the importance of tags using information such as the frequency of occurrence. Presentation of the first tag cloud in a manner corresponding to the analysis results is achieved, for example, by the control unit 190 functioning as a presentation control unit 193. Alternatively, it can be achieved by the control unit 203 functioning as a presentation control module 2035. The analysis results for each of the multiple tags may be stored, for example, in an analysis result table (not shown) in the storage unit 202, or in the first tag cloud table 2025.

[0081] According to the first modification, the analysis results for each of the multiple tags are reflected in the presentation of the first tag cloud. Therefore, users can select tags from the first tag cloud that are more easily accessible to the content, services, etc. that they desire. In other words, it is possible to realize a first tag cloud composed of tags with higher accuracy.

[0082] In the first modified example, the presentation method of the first tag cloud according to the analysis results may be based on the number of tags extracted, their frequency of appearance, their importance, or a combination of at least two of these. This configuration improves the accuracy of tag selection by the user compared to other presentation methods based on analysis results.

[0083] In the first modified example, the presentation mode of the first tag cloud may be the size, color, font, and display direction of the words constituting the tags, or a combination of at least two of these. This configuration improves the accuracy of tag selection by the user compared to other presentation modes. Below, an example of this configuration will be explained with reference to the example in Figure 12. Figure 12 is a schematic diagram showing another example of the display screen of the display 141 when the first tag cloud 401 is presented to the user.

[0084] The display screen shown in Figure 12 shows multiple tags 411 that evoke the image of a cafe, displayed in a list format. This block of multiple tags 411 displayed in this list constitutes the first tag cloud 401. Specifically, the first tag cloud 401 is not composed of columns and records, and individual tags 411 are randomly arranged within the block of the first tag cloud 401. Furthermore, tags 411 with a vertical display direction (up and down direction when viewed from the page) and tags 411 with a horizontal display direction are mixed within the block of the first tag cloud 401.

[0085] Furthermore, in the example shown in Figure 12, the font size and color of the words constituting tag 411 are divided into three levels depending on the number of extracted tags, frequency of occurrence, importance, or at least a combination of two of these. Regarding font size, the words constituting tag 411 with high-level features have the largest font size. The words constituting tag 411 with low-level features have the smallest font size. In addition, the font size of the words constituting tag 411 with medium-level features is intermediate between that of tag 411 with high-level features and tag 411 with low-level features. Although not shown in the illustration, regarding font color, the words constituting tag 411 with high-level features are red. The words constituting tag 411 with low-level features are blue. In addition, the words constituting tag 411 with medium-level features are yellow.

[0086] Furthermore, the size and color distinctions of the words constituting tag 411 do not necessarily have to be in three stages; they may be in two or four or more stages. Also, in the example in Figure 12, the font of all the words constituting tag 411 is the same, and the thickness of the letters is proportional to the size of the letters, but this is not limited to this example. For example, the type of font may be different depending on the level of the features that tag 411 possesses. Alternatively, for example, the size of the letters may be the same regardless of the level of the features that tag 411 possesses, while the thickness of the letters may be different depending on the level.

[0087] Furthermore, the display direction of tags in the first tag cloud 401 does not have to be limited to just two types: vertical and horizontal. For example, instead of these directions, a first direction tilted by a predetermined angle from the vertical to the horizontal, and a second direction tilted by a predetermined angle from the horizontal to the vertical, may be used as the display direction of tags. Alternatively, in addition to the two types of vertical and horizontal, multiple types of display directions including at least one of the first and second directions may be used as the display direction of tags. Even if there are only two directions, the display of the first tag cloud 401 in the area 1411 of the display screen is the same as in the example in Figure 10.

[0088] <7. Second variation> System 1 may, for example, repeatedly accept the selection of tags from the first tag cloud and present the second tag cloud (second list), thereby presenting image-related information (second modification). The second tag cloud is a reconfiguration of the clusters to which the selected tags belong as tag clouds. A cluster is each group obtained by classifying and grouping the multiple tags that make up the first tag cloud according to tags whose semantic content is similar to each other. Image-related information is information about images that contain the selected tags in their descriptions. In this case, "selected tags" refers to the tags selected for cluster extraction.

[0089] Specifically, image-related information includes, for example, an image whose description contains the selected tags, a description of the image, information about the store represented by the image, or at least a combination of these. Information about the store represented by the image includes the store's URL, business hours, address, contact information, recommendations, etc.

[0090] The user's interaction with the first tag cloud, specifically the user's selection of any of the tags included in the first tag cloud, is the same as when the user interacts with the first tag cloud 40 displayed on the screen shown in Figure 10. Here, the control unit 203 functions as a cluster extraction module. Upon receiving a tag selection from the user, the cluster extraction module extracts the cluster to which the selected tag belongs. The cluster may be stored, for example, in a cluster table (not shown) of the storage unit 202.

[0091] The presentation control module 2035 presents the second tag cloud to the user. Specifically, the presentation control module 2035 reads tag data, including tags belonging to the extracted cluster, from the first tag cloud table 2025. The presentation control module 2035 configures the read tag data as the second tag cloud and presents the configured second tag cloud to the user. In other words, the transmission control module 2032 transmits information to the terminal device 10 for displaying the second tag cloud configured by the presentation control module 2035. The transmission / reception unit 192 receives information for displaying the second tag cloud transmitted from the server 20. The presentation control unit 193 displays the second tag cloud on the display 141 based on the received information for displaying the second tag cloud.

[0092] The presentation control module 2035 repeatedly performs the process of presenting the second tag cloud to the user. Specifically, for example, after a particular second tag cloud is displayed on the display 141, the operation reception unit 191 accepts the selection of a tag other than the tags that constitute the particular second tag cloud. When the cluster extraction module accepts the selection of another tag from the user, it extracts a new cluster to which the other tag belongs from the particular second tag cloud and stores it in the cluster table. When the user requests the display of a new second tag cloud, the presentation control module 2035 reads other tag data from the cluster table to construct a new second tag cloud and presents the new second tag cloud to the user. Repeating this series of processes constitutes "repeating the process of presenting the second tag cloud." The second tag cloud may be stored, for example, in a second tag cloud table (not shown) in the storage unit 202.

[0093] The presentation control module 2035 presents image-related information to the user. Specifically, the transmission control module 2032 transmits, for example, information for displaying the image-related information generated by the presentation control module 2035 to the terminal device 10. The presentation control unit 193 displays a second tag cloud on the display 141 based on the received information for displaying the image-related information. Note that, as image-related information, for example, only information about images whose description contains the initially selected tag may be displayed on the display 141. Alternatively, for example, each time another tag is selected, new image-related information about images whose description contains that other tag may be displayed on the display 141. Also, for example, the second tag cloud and the image-related information may be displayed on the display 141 simultaneously, or at different timings. Furthermore, the image-related information may be stored, for example, in an unillustrated image-related information table held by the storage unit 202.

[0094] According to the second modification, with each repeated presentation of the second tag cloud to the user, the user can receive a tag cloud service with a higher degree of similarity between tags. This improves the accuracy of tag selection by the user. In addition, the user can refer to image-related information when selecting tags from the second tag cloud. This improves the user's convenience when using the second tag cloud.

[0095] In the second modified example, System 1 may display the number of image-related information items that include the selected tag in their description. The count of image-related information items is achieved, for example, by the control unit 190 or control unit 203 functioning as an item counting module (not shown). The display of the number of image-related information items is achieved, for example, by the control unit 190 functioning as a display control unit 193. Alternatively, it can also be achieved by the control unit 203 functioning as a display control module 2035.

[0096] With this configuration, when a user selects a tag from the second tag cloud, they can refer not only to the image-related information but also to the number of image-related items associated with the second tag cloud. This allows System 1 to narrow down the number of image-related items presented to the user each time it processes a tag selection from the second tag cloud. As a result, users can more easily find the desired image-related information. Therefore, user convenience when using the second tag cloud is further improved.

[0097] In the second modified example, System 1 may receive a request from the user for the presentation of image-related information and present the image-related information to the user based on the received request. The reception of the request is achieved, for example, by the control unit 190 functioning as an operation reception unit 191. Alternatively, it can be achieved by the control unit 203 functioning as a reception control module 2031. Specifically, for example, suppose that the number of image-related information items is displayed on the display 141 when a tag is selected for the second tag cloud. When the user determines that they can check the displayed number of image-related information items, they request the operation reception unit 191 to present the image-related information. When the image-related information generation module receives the selection of tags from the user, it generates image-related information based on the selected tags. The presentation control module 2035 presents the user with image-related information that includes the multiple tags selected by the user in the description. With this configuration, when the user selects a tag from the second tag cloud, the image-related information can be presented at the time the user wants to refer to it. This further improves the user's convenience when using the second tag cloud.

[0098] <8. Third variation> System 1 may, for example, input text information, an image, and an instruction that instructs the output of an explanatory text that takes into account the content of the text information for each of the multiple input images (third modified example). The text information is character information associated with the image. Specifically, the "explanatory text that takes into account the content of the text information" is a sentence that explains the content that comprehensively captures the text information and the image. The acquisition of text information is achieved, for example, by the control unit 190 functioning as an operation reception unit 191. Alternatively, it can be achieved by the control unit 203 functioning as a reception control module 2031.

[0099] In Modification 3, multiple tags are extracted based on a descriptive text that takes into account the content of the text information, thus improving the accuracy of the tags. As a result, a first tag cloud composed of more accurate tags can be realized.

[0100] In Modification 3, the text information may include, for example, information about the facility associated with the image. Alternatively, the text information may include, for example, user reviews of the facility associated with the image. An example of a "facility associated with the image" would be a commercial facility such as a department store that has a restaurant as a tenant, if the object captured in the image is a restaurant.

[0101] <9 Basic Computer Hardware Configuration> Figure 13 is a block diagram showing the basic hardware configuration of computer 90. Computer 90 comprises at least a processor 91, main memory 92, auxiliary memory 93, and a communication interface IF99. These components are electrically connected to each other by a bus.

[0102] The processor 91 is hardware for executing the instruction set described in the program. The processor 91 consists of an arithmetic unit, registers, peripheral circuits, etc. The main memory 92 is for temporarily storing the program and data processed by the program, etc. For example, it is a volatile memory such as DRAM (Dynamic Random Access Memory). The auxiliary memory 93 is a memory device for storing data and programs. For example, it is a flash memory, HDD (Hard Disc Drive), magneto-optical disk, CD-ROM, DVD-ROM, semiconductor memory, etc. The communication IF 99 is an interface for inputting and outputting signals for communicating with other computers via a network using wired or wireless communication standards.

[0103] A network consists of various mobile communication systems built on the internet, LANs, wireless base stations, etc. For example, networks include 3G, 4G, and 5G mobile communication systems, LTE (Long Term Evolution), and wireless networks that can connect to the internet via designated access points (e.g., Wi-Fi®). When connecting wirelessly, communication protocols include, for example, Z-Wave®, ZigBee®, and Bluetooth®. When connecting via wired connections, networks also include those directly connected via USB (Universal Serial Bus) cables, etc.

[0104] Furthermore, by distributing all or part of each hardware configuration across multiple computers 90 and connecting them to each other via a network, the computers 90 can be virtually realized. Thus, the concept of computer 90 includes not only computers in which each hardware component is housed in a single enclosure, but also virtualized computer systems.

[0105] <Basic Functional Configuration of Computer 90> This section describes the functional configuration of a computer realized by the basic hardware configuration of computer 90. The computer performs the functions of a control unit, a memory unit, and a communication unit. Note that the functions of computer 90 can also be performed by distributing all or part of those functions across multiple computers 90 interconnected via a network. Thus, computer 90 is a concept that includes not only a single computer but also a virtualized computer system.

[0106] The memory unit is realized by the main memory 92 and the auxiliary memory 93. The memory unit stores data, various programs, and various databases. The control unit is realized when the processor 91 reads various programs stored in the auxiliary memory 93, loads them into the main memory 92, and executes processing according to those programs. The control unit can perform various information processing functions depending on the type of program. In this way, the computer is realized as an information processing device that performs information processing.

[0107] Furthermore, the control unit can instruct the processor 91 to allocate a memory area corresponding to the memory unit in the main memory 92 or auxiliary memory 93 according to a program. In addition, the control unit can instruct the processor 91 to add, update, or delete data stored in the memory unit according to various programs.

[0108] A database, specifically a relational database, is used to manage and associate data sets called tables, which are structured in a tabular format defined by rows and columns. In a database, tables are called tables, the columns of a table are called columns, and the rows of a table are called records. In a relational database, relationships can be established and linked between tables.

[0109] Typically, each table in a database has a key column to uniquely identify a record, but setting a key on a column is not mandatory. The control unit can instruct the processor 91 to add, update, or delete records in specific tables stored in the memory unit according to various programs.

[0110] The communication unit is implemented by the communication IF99. The communication unit performs the function of communicating with other computers 90 via the network. The communication unit can receive information transmitted from other computers 90 and output it to the control unit. The control unit can cause the processor 91 to perform information processing on the received information according to various programs. The communication unit can also transmit information output from the control unit to other computers 90.

[0111] While several embodiments of this disclosure have been described above, these embodiments can be implemented in a variety of other forms, and various omissions, substitutions, and modifications are permitted without departing from the spirit of the invention. These embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims and their equivalents.

[0112] <Note> The details described in each of the above embodiments are noted below. (Note 1) A program for execution on a computer having a processor and memory, the program causing the processor to perform the following steps: receiving input of a plurality of images; inputting, for each of the input plurality of images, the image and an instruction sentence that instructs the output of an explanatory sentence about the image to a multimodal generation AI model, and causing the multimodal generation AI model to output the explanatory sentences; analyzing the plurality of explanatory sentences that have been output and extracting a plurality of characteristic words as tags; and presenting the extracted plurality of tags as a first list. (Note 2) The program described in (Appendix 1) causes the processor to perform the step of analyzing a plurality of extracted tags, and in the step of presenting the first list, presents the first list in a manner corresponding to the analysis results in the analysis step. (Note 3) In the step of presenting the first list, the configuration according to the analysis results is a configuration according to the number of tags extracted, the frequency of occurrence, the importance, or at least a combination of these (as described in Appendix 2). (Note 4) In the step of presenting the first list, the embodiment is the size, color, font, display direction, or at least two combinations thereof of the characters of the words constituting the tag, as described in (Appendix 2) or (Appendix 3). (Note 5) A program according to any one of (Appendix 1) to (Appendix 4) that causes the processor to perform the steps of: accepting the selection of one or more tags from the first list; and presenting image-related information relating to the image that includes the selected one or more tags in the description. (Note 6) In the step of presenting the image-related information, the program described in (Appendix 5) includes the descriptive text as the image-related information. (Note 7) In the step of presenting the image-related information, the image-related information includes the image (as described in Appendix 6). (Note 8) A program according to any one of (Appendix 1) to (Appendix 4) that causes the processor to perform the steps of accepting the selection of the tag from the first list, repeatedly presenting the cluster to which the selected tag belongs as a second list, and presenting image-related information relating to the image that includes the selected tag in the description. (Note 9) The program described in (Appendix 8) which, in the step of presenting the image-related information, presents the number of image-related information items related to the second list. (Note 10) The program described in Appendix 9, wherein each time the process of accepting the selection is repeated, the number of items to be presented in the step of presenting the image-related information is reduced. (Note 11) A program according to (Appendix 9) or (Appendix 10) that causes the processor to perform the step of receiving a request for the presentation of the aforementioned image-related information, and in the step of presenting the aforementioned image-related information, presents the aforementioned image-related information based on the received request. (Note 12) A program according to any of (Appendix 1) to (Appendix 11), wherein, in the step of outputting, text information, an instruction statement that instructs the output of an explanatory text that takes into account the content of the text information for each of the multiple images that have been input, is input to a multimodal generation AI model, and the explanatory text is output from the multimodal generation AI model. (Note 13) In the step of outputting the information, the text information includes information about the image and the facility associated with it (the program as described in Appendix 12). (Note 14) In the step of outputting the above, the text information includes the image and user reviews of the facility as described in (Appendix 12) or (Appendix 13). (Note 15) The program described in any of (Appendix 1) to (Appendix 14), wherein in the step of receiving the input, each of the plurality of images is an image relating to a restaurant. (Note 16) In the step of receiving the aforementioned input, each of the plurality of images is an image relating to a real estate property. A program as described in any of (Appendix 1) to (Appendix 14). (Note 17) A method to be performed on a computer comprising a processor and memory, wherein the processor performs all steps performed in any of the inventions described in (Appendix 1) to (Appendix 16). (Note 18) An information processing device comprising a control unit and a storage unit, wherein the control unit performs all steps performed in any of the inventions described in (Appendix 1) to (Appendix 16). (Note 19) A system comprising means for performing all steps performed in any of the inventions described in (Appendix 1) to (Appendix 16). [Explanation of Symbols]

[0113] 1... System 10…Terminal device 120... Communications Department 13…Input device 131…Touch-sensitive devices 14…Output device 15…Memory 16…Storage 19… Processor 20... Server 22...Communication IF 23…Input / Output Interface 25…Memory 2 hours… storage 29… Processor 40, 401... First Tag Cloud (First List) 41, 411… tags 50…Image-related information

Claims

1. A program to be executed by a computer having a processor and memory, wherein the program is to be executed by the processor, A step that accepts multiple image inputs, For each of the multiple images that have been input, the image and an instruction that instructs the output of an explanatory text about the image are input to a multimodal generation AI model, and the explanatory text is output from the multimodal generation AI model. The steps include analyzing the multiple explanatory texts output and extracting several characteristic words as tags, The steps include presenting the extracted multiple tags as a first list, and A program that executes the command.

2. The processor is made to perform the step of analyzing the extracted multiple tags. The program according to claim 1, wherein in the step of presenting the first list, the first list is presented in a manner corresponding to the analysis results in the analysis step.

3. The program according to claim 2, wherein in the step of presenting the first list, the embodiment corresponding to the analysis results is an embodiment corresponding to the number of tags extracted, the frequency of occurrence, the importance, or at least a combination of these.

4. The program according to claim 2, wherein in the step of presenting the first list, the embodiment is the size, color, font, display direction of the characters of the words constituting the tag, or at least a combination of these.

5. A step of accepting the selection of one or more tags from the first list, A step of presenting image-related information relating to the image, which includes one or more selected tags in the description; The program according to claim 1, which causes the processor to execute the following.

6. The program according to claim 5, wherein the step of presenting the image-related information includes the explanatory text.

7. The program according to claim 6, wherein the step of presenting the image-related information includes the image.

8. A step of accepting the selection of the tag from the first list, The process of repeatedly presenting the cluster to which the selected tag belongs as a second list, and the step of presenting image-related information concerning the image that includes the selected tag in the description, The program according to claim 1, which causes the processor to execute the following.

9. The program according to claim 8, wherein in the step of presenting the image-related information, the number of image-related information items related to the second list is presented.

10. In the step of accepting the selection, each time the process of accepting the selection is repeated, The program according to claim 9, wherein the number of items to be presented in the step of presenting the image-related information is reduced.

11. The processor is instructed to perform the step of receiving a request to present the aforementioned image-related information. The program according to claim 9, which, in the step of presenting the image-related information, presents the image-related information based on the received request.

12. The program according to claim 1, wherein in the step of outputting, for each of the plurality of input images, text information, the image, and the instruction statement that instructs the output of the explanatory text taking into account the content of the text information are input to the multimodal generation AI model.

13. The program according to claim 12, wherein in the step of outputting, the text information includes information relating to the image and the facility associated therewith.

14. The program according to claim 12, wherein in the step of outputting, the text information includes reviews from users of the facility associated with the image.

15. The program according to claim 1, wherein in the step of receiving the input, each of the plurality of images is an image relating to a restaurant.

16. The program according to claim 1, wherein in the step of receiving the input, each of the plurality of images is an image relating to a real estate property.

17. A method to be performed on a computer comprising a processor and memory, wherein the processor performs all steps performed in any of the inventions according to claims 1 to 16.

18. An information processing apparatus comprising a control unit and a storage unit, wherein the control unit performs all steps performed in any of the inventions according to claims 1 to 16.

19. A system comprising means for performing all steps performed in the invention according to any one of claims 1 to 16.

Images