Information processing system, information processing method, and program

Abstract

To provide a summary that makes it easy for viewers to identify the content of an image. [Solution] An information processing system that provides summary texts for images identifies specific information, including data for a predetermined image from which a summary text should be generated, and information about a viewer who views the predetermined image. It creates an instruction statement containing the specific information for a generative artificial intelligence (generative AI) to generate a summary text for the predetermined image, supplies the instruction statement to the generative AI, and provides the summary text generated by the generative AI based on the instruction statement to the viewer in association with the predetermined image.

Description

Technical Field

[0004] , , , , , , ,

[0003] , , ,

[0001] The present invention relates to an information processing system, an information processing method, and a program for managing and utilizing image data.

Background Art

[0002] With the digitization and performance improvement of devices for shooting and recording images such as cameras and video camcorders, the opportunities for utilizing image data such as still images and moving images taken are expanding. In recent years, as a form of image management service related to the management of image data, a service has been provided that enables users to store, edit, share, etc. the data of images taken on the cloud. In such a service, a user can store a large amount of image data on the cloud. By using the image display function and playback function provided by the image management service, the user can confirm the content of the image data stored on the cloud. Patent Document 1 describes a technique for generating a summarized video by converting the audio included in a video into text using speech recognition technology and extracting and combining the videos of the sections in which the audio is included in order for the user to grasp the content of a long video in a short time. By using such a technique, it becomes possible for the user to efficiently identify the content of a large amount of image data stored on the cloud.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Non-Patent Documents

[0004]

Non-Patent Document 1

[0005] In the technology described in Patent Document 1, viewers of a video cannot identify the content of the video unless they play the video. Furthermore, even when the video is played, it is not always summarized in a way that is easy for the viewer to understand. Moreover, if still images cannot be displayed or videos cannot be played, the content cannot be identified. The present invention provides a technology for providing a summary text that makes it easy for viewers of an image to identify the content of that image. [Means for solving the problem]

[0006] An information processing system according to one aspect of the present invention is an information processing system that provides an image summary, comprising: identification means for identifying specific information including data of a predetermined image from which an image summary should be generated and information relating to a viewer who views the predetermined image; creation means for creating an instruction statement for a generative artificial intelligence (generative AI) to generate an image summary, the instruction statement including the specific information; supply means for supplying the instruction statement to the generative AI; and providing means for providing the summary statement generated by the generative AI based on the instruction statement to the viewer in association with the predetermined image. [Effects of the Invention]

[0007] According to the present invention, it is possible to provide a summary text that makes it easy for viewers of an image to identify the content of that image. [Brief explanation of the drawing]

[0008] [Figure 1] This is a diagram showing an example of the configuration of an information processing system. [Figure 2]It is a diagram showing an example of the hardware configuration of an information processing apparatus. [Figure 3] It is a diagram showing an example of the configuration of a content management system. [Figure 4] It is a diagram showing an example of the configuration of a user management system. [Figure 5] It is a diagram showing an example of the configuration of a prompt creation system. [Figure 6] It is a diagram showing an example of a processing flow executed by an information processing system. [Figure 7] It is a diagram showing an example of a screen for managing user information. [Figure 8] It is a diagram showing an example of the configuration of a prompt. [Figure 9] It is a diagram showing an example of the configuration of a prompt. [Figure 10] It is a diagram showing an example of a screen of an image list. [Figure 11] It is a diagram showing an example of a screen of image details. [Figure 12] It is a diagram showing an example of a screen of image details. [Figure 13] It is a diagram showing an example of the configuration of a prompt. [Figure 14] It is a diagram showing an example of a processing flow executed by an information processing system. [[ID=)37]] [Figure 15] It is a diagram showing an example of the configuration of an information processing system. [Figure 16] It is a diagram showing an example of a screen for image sharing [Figure 17] It is a diagram showing an example of the configuration of a prompt. [Figure 18] It is a diagram showing an example of the configuration of a message to notify the sharing destination of image data. [Figure 19] It is a diagram showing an example of a processing flow executed by an information processing system.

Embodiments for Carrying Out the Invention

[0009] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims. Although a plurality of features are described in the embodiments, not all of these plurality of features are essential for the invention, and the plurality of features may be arbitrarily combined. Further, in the accompanying drawings, the same or similar configurations are denoted by the same reference numerals, and redundant descriptions are omitted.

[0010] (System Configuration) FIG. 1 shows a configuration example of an information processing system according to this embodiment. The information processing system in FIG. 1 can be configured to include, for example, a content management system 101, a user management system 102, a prompt creation system 103, a generative AI system 104, a user terminal 111, and a user terminal 112. AI is an abbreviation for Artificial Intelligence. Generative AI may also be referred to as Generative Artificial Intelligence. Each element (each system and each terminal) constituting the information processing system according to this embodiment is interconnected via a network 121.

[0011] The user terminal 111 is a recording device that captures still images and videos. The user terminal 111 includes, for example, cameras, video cameras, and video camcorders. Still images and videos are sometimes collectively referred to as images. The user terminal 111 can record the captured image data onto a recording medium it possesses. The user terminal 111 may also have an application for transmitting the captured image data to the content management system 101. In this case, the user terminal 111 can transmit the image data to the content management system 101 based on the generation of the image data within its own device. Furthermore, the user terminal 111 may transmit image data selected by the user to the content management system 101 based on user input. When transmitting image data to the content management system 101, the user terminal 111 may also transmit information related to the image data, information about the user, information about the user terminal 111, etc., associated with the image data to the content management system 101. For example, information related to the image data may include the location and date and time the image data was taken. Furthermore, user information includes information that identifies the user's account in the content management system 101. In addition, user terminal information includes information that identifies the model of the user terminal 111.

[0012] User terminal 112 is a device used to manage image data owned by the user that is stored on the content management system 101. For example, user terminal 112 includes personal computers (PCs), smartphones, tablets, etc. User terminal 111 may also function as user terminal 112. For example, a device that integrates user terminal 111 and user terminal 112 may be a camera or smartphone with an image data management application (content management application) installed. For example, a user can log in to the content management system 101 using the content management application installed on user terminal 112 and register image data, view registered image data, and check and edit information associated with the image data.

[0013] The content management system 101 manages image data and information associated with that image data. For example, the content management system 101 stores image data registered by users, stores and updates information associated with that image data, and provides image data and information associated with that image data to image viewers. A detailed explanation of the configuration of the content management system 101 will be given later.

[0014] The user management system 102 manages information about users who use the content management system 101. For example, the user management system 102 issues accounts to users who use the content management system 101. The user management system 102 also receives and stores user information entered by users of the content management system 101. Furthermore, the user management system 102 modifies and deletes stored user information. A detailed explanation of the configuration of the user management system 102 will be provided later.

[0015] The prompt creation system 103 creates instruction texts to cause the generation AI system 104 to generate summary texts of images registered in the content management system 101. The instruction texts supplied to the generation AI system 104 may be called prompts. A detailed explanation of the configuration of the prompt creation system 103 will be given later.

[0016] The generative AI system 104 generates a summary of an image based on a prompt supplied by the prompt generation system 103. For example, the generative AI system 104 could be a device that generates text using a large-scale language model. As an example, the generative AI system 104 could be a multimodal generative AI device that enables input and output of data such as images, audio, and text through machine learning that reads a large amount of input data. Gemini, described in Non-Patent Literature 1, is an example of a multimodal generative AI. In Gemini, a trained model generates a response to an input called a prompt, which instructs the generation process.

[0017] In the information processing system shown in Figure 1, a prompt creation system 103 creates a prompt for generating a summary text about a predetermined image data stored in the content management system 101. For example, the prompt creation system 103 can create a prompt containing information that identifies the data of a predetermined image and instruct the generation AI system 104 to generate a summary text about this predetermined image. The generation AI system 104 generates a summary text based on the prompt provided by the prompt creation system 103. The generated summary text is stored in the content management system 101, for example, associated with the data of the predetermined image. When a user requests the display or playback of a predetermined image via the user terminal 112, the content management system 101 responds to this request by providing the user terminal 112 with the data of the predetermined image and information related to this image data. For example, the content management system 101 provides the summary text generated by the generation AI system 104 as information related to the data of the predetermined image. By displaying the summary text of the predetermined image along with the image data on the user terminal 112, the user can identify the content of the image by the summary text without having to play the predetermined image.

[0018] However, the generation AI system 104 does not always generate a summary that is easy for the user to identify the content of the image. For example, if a video contains multiple scenes, the scene selected by the generation AI system 104 for creating the summary may not be easily identifiable to the user.

[0019] In light of these circumstances, the information processing system in this embodiment creates a prompt that includes data for a predetermined image from which a summary should be generated, and information about the viewer who views the predetermined image and summary, and supplies it to the generation AI system 104. For example, the information processing system first identifies specific information that includes data for a predetermined image from which a summary should be generated, and information about the viewer. For example, the information about the viewer may include the viewer's age, gender, or family members. The information about the viewer may also include the age and gender of each member of the viewer's family. Furthermore, the information about the viewer may include the language used by the viewer, the viewer's preferences regarding summary texts, or the viewer's preferences for subjects to photograph. The information processing system creates a prompt that includes such specific information and supplies it to the generation AI system 104. The information processing system then provides the summary text generated by the generation AI system 104 based on the prompt, associated with the predetermined image, to the viewer. With this configuration, information about the viewer and their family, as well as information about the viewer's preferences, is supplied to the generation AI system 104, enabling the generation AI system 104 to generate a summary that is easy for the viewer to understand. The configuration of the information processing system that operates in this way is described below.

[0020] Figure 2 is a block diagram showing an example of the hardware configuration of each element (each system and each terminal) constituting the information processing system of this embodiment. In Figure 2, the information processing device 201 is described as operating as one of the elements constituting the information processing system. In this embodiment, the processing performed by the content management system 101, the user management system 102, the prompt creation system 103, and the generation AI system 104 may each be implemented by a single information processing device 201. Alternatively, a single information processing device may perform processing performed by multiple of these systems. Furthermore, processing performed by one component may be distributed and executed across multiple information processing devices 201. As described above, the multiple information processing devices 201 are connected to each other so as to be able to communicate via the network 121. The information processing device 201 includes, for example, a CPU 202, ROM 203, RAM 204, HDD 205, display unit 206, operation unit 207, communication unit 208, and system bus 209.

[0021] The CPU 202 is a control unit that comprehensively controls the entire information processing unit 201. The CPU 202 includes arithmetic processing processors such as CPUs and MPUs. CPU is an abbreviation for Central Processing Unit. MPU is an abbreviation for Micro-Processing Unit. The ROM 203 is a non-volatile memory that stores programs and parameters executed by the processor of the CPU 202. A program is an instruction that executes each process shown in the information processing system functions and flowchart described later. The RAM 204 is a work memory that temporarily stores programs and data supplied from external devices. The HDD 205 is a storage device that includes internal devices such as hard disks and memory cards built into the information processing unit 201, or external devices such as hard disks and memory cards that are detachably connected to the information processing unit 201. The HDD 205 may be a memory card or hard disk made of semiconductor memory or magnetic disks. The HDD 205 may also be a storage medium consisting of a disk drive that reads / writes data to / from optical disks such as DVDs and Blu-ray® Disks.

[0022] The display unit 206 is a display device such as an LCD or organic EL display or monitor, and is an output device that displays data held by the information processing device 201 or data supplied from external devices. LCD is an abbreviation for Liquid-Crystal Display. EL is an abbreviation for Electro-Luminescence. The operation unit 207 is an operating component such as a mouse, keyboard, or touch panel that accepts user input and outputs instructions from the user to the CPU 202. The communication unit 208 is a network interface that enables communication with networks such as the Internet or a LAN (Local Area Network). The system bus 209 includes an address bus, a data bus, and a control bus that connect the various components 202 to 208 of the information processing device 201 so that data can be exchanged.

[0023] Furthermore, ROM203 stores the operating system (OS), which is the basic software executed by the CPU202, and applications that work in cooperation with the OS to realize advanced functions. ROM203 also stores applications that enable the information processing device 201 to manage image data, including the creation of prompts and the generation of summary texts, as described later. Additionally, ROM203 stores programs that enable the operation of the software constituting the content management system 101, user management system 102, prompt creation system 103, and generation AI system 104.

[0024] The processing of the information processing device 201 in this embodiment is realized by loading software provided by an application. The application is assumed to have software for utilizing the basic functions of the OS installed on the information processing device 201. The OS of the information processing device 201 may also have software for realizing the processing in this embodiment.

[0025] Figure 3 shows an example configuration of a content management system 101 configured by the information processing device 201 in this embodiment. The content management system 101 may include an image registration unit 301, an image analysis unit 302, an image management unit 303, a content management database (DB) 304, a summary editing unit 305, an image sharing processing unit 306, a shared message creation unit 307, and a shared information receiving unit 308. The software constituting the information processing device 201 may be installed on a single information processing device 201 or may be divided and installed on multiple information processing devices 201. The image registration unit 301 receives registration of image data to be stored in the content management system 101 from a user. For example, the image registration unit 301 can receive image data received from a user who has logged into the content management system 101 as image data to be registered. The image registration unit 301 can store the received image data in the content management DB 304.

[0026] The image analysis unit 302 analyzes image data registered by the user. For example, the image analysis unit 302 can acquire metadata attached to the image data and extract subjects contained in the image. For example, metadata that may be attached to image data may include information that identifies the date and time the image data was taken (shooting date and time information) and information that identifies the equipment used to take the image (shooting equipment information). Metadata that may be attached to image data may also include the image resolution, the size of the image data, the recording format of the image data, color information, and brightness information. Furthermore, the extraction of subjects contained in the image can be performed by performing subject detection processing, recognition processing, identification processing, etc. If the image is a video, data may be divided into scenes, speakers may be identified by analyzing audio information, audio types may be identified, and spoken text may be created by speech recognition. The results of the analysis performed by the image analysis unit 302 may be stored in the content management DB 304 as analysis result data associated with each image data.

[0027] The image management unit 303 provides functions for users to manage image data. For example, the image management unit 303 may provide the user with image data stored in the content management DB 304 and information associated with that image data, based on a request from the user via a content management application installed on the user terminal 112. As an example, the image data and information associated with that image data provided by the image management unit 303 may be provided to the user in the form of an image list or image details on the content management application. In this case, the image management unit 303 may generate data for displaying the image list or image details, or the content management application may generate data for displaying the image list or image details based on the information provided by the image management unit 303.

[0028] The content management DB304 stores image data registered by each user, along with related metadata and other information about that image. For example, the information about the image may include a reduced image (thumbnail, etc.) or a summary text for reviewing the content of the image data. The content management DB304 may store the image data and the information about this image data together or separately. For example, if stored separately, the content management DB304 may store the image data in data storage suitable for BLOB storage management, and the information about this image data in key-value type storage linked to the BLOB data. BLOB is an abbreviation for Binary Large Object.

[0029] The summary editing unit 305 edits the summary text associated with the image data. For example, the summary editing unit 305 may output the summary text to the user who owns the image data and obtain information from the user for editing that summary text. The summary editing unit 305 may also execute a process to regenerate the summary text based on the information obtained for editing the summary text.

[0030] The image sharing unit 306 to the information receiving unit 308 perform processing related to the sharing of image data among multiple users, as described later. For example, the image sharing unit 306 performs processing to share the image data with other users specified by the user who owns the image data. As an example, when the image sharing processing unit 306 receives an instruction from the user who owns the image data via the content management application to share the image data with other users, it notifies the specified other users of a message to share the image. The sharing message creation unit 307 creates a message for sharing image data among multiple users. For example, the sharing message creation unit 307 creates a message that includes a summary text generated for the other users to whom the image data is to be shared. The message may be the body of an email. The sharing message creation unit 307 may also create a prompt to generate a summary text for the other users to whom the image data is to be shared and supply it to the generation AI system 104. In this case, the sharing message creation unit 307 may obtain the summary text generated by the generation AI system 104 and create a message that includes this summary text. The sharing information receiving unit 308 receives information about the sharing of image data entered by the user who owns the image data. For example, the sharing information receiving unit 308 can receive information from the user via a content management application installed on the user terminal 112 that identifies other users who should share the image data, as well as information that identifies the purpose and intent of the sharing.

[0031] Figure 4 shows an example configuration of a user management system 102 configured by the information processing device 201 in this embodiment. The user management system 102 may include a user authentication processing unit 401, a user information management unit 402, a user operation history recording unit 403, and a user profile analysis unit 404. The user management system 102 may also include an operation history database (DB) 405, a user profile database (DB) 406, and a user address book database (DB) 407. The software constituting the information processing device 201 may be located on a single information processing device 201 or may be divided and located on multiple information processing devices 201. The user authentication processing unit 401 performs authentication processing for users who use the content management system 101. For example, the user authentication processing unit 401 performs account registration processing to allow new users to use the content management system. The user authentication processing unit 401 also performs user authentication when a user with a registered account uses the content management system 101. For example, the user authentication processing unit 401 performs user authentication based on whether the combination of user identification information (account ID) and password entered by the user matches the combination of account ID and password stored in the user information management unit 402. The user authentication processing unit 401 may also perform user authentication based on biometric information such as the user's fingerprints or facial features. Multiple user authentication methods may be used in combination. Information about the user necessary for user authentication may be entered by the user when registering their account or after registration and stored in the user profile DB 406.

[0032] The User Information Management Unit 402 manages information about users who use the Content Management System 101. For example, user information includes username, email address, camera equipment information, lens information owned by the user, family members, languages ​​used, photography preferences, and summary writing preferences. User information is not limited to these and may include other information. The User Information Management Unit 402 can accept input of user information when a user registers an account or after registration. The User Information Management Unit 402 can also accept changes or deletions of registered user information. The User Information Management Unit 402 can add and manage user preference data obtained using the User Operation History Recording Unit 403 and the User Profile Analysis Unit 404 to the user information. User information may be associated with the user's account ID and stored in the User Profile DB 406.

[0033] The user operation history recording unit 403 records logs of the processes performed for each user in the information processing system. For example, a user may make requests to the content management system 101, etc., via the content management application on the user terminal 112 for the registration and management of image data and the management of information associated with those images. The content management system 101 executes processing in response to the user's request. The content management system 101 executes processing in cooperation with the user management system 102, prompt creation system 103, generation AI system 104, etc., as needed. In addition, components other than the content management system 101 may execute processing in response to requests from the user. The user operation history recording unit 403 records logs of the processing of the information processing system performed in response to such user requests for each user. The user's request and the information processing system's response may be executed and recorded as interactive processing. For example, the user operation history recording unit 403 may record information identifying image data registered by the user, information about that image provided to the user, and information identifying images shared with other users in the operation history DB 405, associating these with the user's account ID. The records of the processes created in this way may be called operation history data.

[0034] The user profile analysis unit 404 analyzes user preferences. For example, the user profile analysis unit 404 can identify a user's implicit preferences by performing mining analysis on user-specific operation history data recorded in the operation history DB 405. As an example, the user profile analysis unit 404 periodically performs user preference analysis. In this case, when the timing for performing the analysis process arrives at a predetermined interval, the user information management unit 402 activates the user profile analysis unit 404. The user profile analysis unit 404 obtains user-specific operation history data from the operation history DB 405. If the operation history data includes operations such as registering or viewing a specific image, the user profile analysis unit 404 obtains the image data that was the target of the operation and information related to that image from the content management DB 304. The user profile analysis unit 404 can perform data mining processes such as regression analysis, classification, and statistical methods using the obtained operation history data and the image data that was the target of the operation and information related to that image. The user profile analysis unit 404 identifies characteristics such as the user's preferences for registered images and their preferences for how to use image data through data mining processing, and stores them in the user profile DB 406 as information about the user. The user profile analysis unit 404 may perform data mining processing using, for example, the method described in Patent Document 2.

[0035] The operation history DB 405 stores operation history data generated by the user operation history recording unit 403. For example, the operation history DB 405 can store logs of processes executed by the information processing system in response to requests made by the user via the content management application on the user terminal 112. The operation history DB 405 can also store requests received from the content management application as logs. For example, the operation history DB 405 can store requests received from the content management application, the processes executed by the information processing system in response, and the responses sent to the content management application as interactive logs. As an example, the operation history DB 405 can store interactive logs for each user, including the user's account ID, the time when each request, process, and response was made, and a set of processing parameters such as the image specified as the target of the processing and the type of processing.

[0036] The user profile database 407 stores information about users. For example, the user profile database 407 may store information about users received from users by the user information management unit 404, as well as information about users' preferences identified using the user profile analysis unit 404.

[0037] User Address Book DB407 manages information about other users used when users share image data and other data with other users. For example, User Address Book DB407 holds email addresses and SNS account information used to send messages to recipients when users share image data they own with other users. SNS is an abbreviation for Social Networking Service.

[0038] Figure 5 shows an example configuration of the prompt creation system 103, which is configured by the information processing device 201 in this embodiment. The prompt creation system 103 may include a user information acquisition unit 501, a prompt creation unit 502, a prompt transmission unit 503, a response acquisition unit 504, and a similar summary text acquisition unit 505. The software constituting the information processing device 201 may be located on a single information processing device 201 or may be divided and located on multiple information processing devices 201. The prompt creation system 103 creates prompts for generating summary texts according to the profile of the user viewing the image, using the generation AI system 104. The user information acquisition unit 501 acquires information about the user viewing the image. For example, if the viewer of the image is the owner of the image data, the user information acquisition unit 501 acquires information about the owner of the image data from the user profile DB 406 of the user management system 102. In this case, the user information acquisition unit 501 can identify the image for which a summary text should be generated based on the fact that the image data was registered by the owner of the image data. In other words, the user information acquisition unit 501 can generate a summary text for the owner of the image data to identify the content of the image data, based on the fact that the content management system 101 has received registration of image data from a user. Furthermore, if the viewer of the image is another user designated by the owner of the image data, the user information acquisition unit 501 can obtain information about that other user from the user profile DB 406 of the user management system 102. For example, the owner of the image data can specify the recipient of the image data by selecting a user registered in the user address book DB 407. In this case, the user information acquisition unit 501 can identify the target user based on the email address of the selected user and obtain information about the identified user from the user management system 102. In this case, the user information acquisition unit 501 can identify the image and the recipient for which a summary text should be generated, based on the fact that the recipient of the image data has been selected by the owner of the image data. In other words, the user information acquisition unit 501 can generate a summary text for the recipient of the image data to identify the content of the image data, based on the fact that it has received the designation of the image data and the recipient from the user.

[0039] The prompt creation unit 502 creates a prompt (instruction text) to instruct the generation AI system 104 to generate a summary text. For example, the prompt creation unit 502 may create a prompt using user information obtained by the user information acquisition unit 501 from the user profile DB 406 of the user management system 102. The prompt creation unit 502 may also accept input from the user to create a prompt. In this case, the prompt creation unit 502 may further create a prompt using information contained in the input received from the user. User input to create a prompt may include, for example, information indicating the purpose of generating a summary text, the purpose of sharing image data, or the intention to share image data.

[0040] The prompt sending unit 503 instructs the generation AI system 104 to generate a summary text. For example, the prompt sending unit 503 may instruct the generation AI system 104 by supplying a prompt created by the prompt creation unit 502 to an API provided by the generation AI system 104. API is an abbreviation for Application Programming Interface. The response acquisition unit 504 acquires the summary text generated by the generation AI system 104. For example, the response acquisition unit 504 may acquire the response output by the generation AI system 104 to the prompt supplied by the prompt sending unit 503 as a summary text.

[0041] The Similar Summary Acquisition Unit 505 performs processing to make the summary text acquired from the Generation AI System 104 distinguishable from other summary texts. For example, the Similar Summary Acquisition Unit 505 acquires summary texts of other images held in the Content Management DB 304 of the Content Management System 101 and analyzes their similarity to the summary text acquired from the Generation AI System 104. As an example, the Similar Summary Acquisition Unit 505 divides each summary text into token units. A token is a specific sentence component such as a word. The Similar Summary Acquisition Unit 505 calculates the value of the dot product of vectors between one token of one summary text and one token of the other summary text. If the value is below a predetermined threshold, it is determined that there is no similarity; if the value is above the predetermined threshold, it may be determined that there is similarity. If the number or ratio of combinations of tokens determined to be similar is above a predetermined threshold, the Similar Summary Acquisition Unit 505 may determine that those summary texts are similar. The method of similarity determination by the Similar Summary Acquisition Unit 505 is not limited to this, and other methods may be applied. If the Similar Summary Acquisition Unit 505 determines that the summary acquired from the Generation AI System 104 is similar to one or more other summary sentences, it creates a prompt containing information about that summary sentence and instructs the Generation AI System 104 to regenerate the summary sentence. If the Similar Summary Acquisition Unit 505 determines that the summary acquired from the Generation AI System 104 is not similar to any of the other summary sentences, it provides that summary sentence to the Content Management System 101. This summary sentence may be associated with the target image data and stored in the Content Management DB 304 of the Content Management System 101.

[0042] (Process flow) (Example of processing in the first case) Figure 6 shows an example of the processing flow executed in the information processing system in this embodiment. This processing example will be explained using an example in which a summary text for the owner of an image is generated based on the fact that a user who has logged into the content management system 101 has registered image data in the content management system 101. First, the content management system 101 receives image data sent by the logged-in user (S601). For example, a user can register image data by logging into the content management system 101 via the content management application on the user terminal 112 and uploading the image data to the content management system 101. The content management system 101 can store the image data input by the user in the content management DB 304 in association with the user's account ID. For example, the content management system 101 can assign a unique identification address to the received image data to distinguish it from other data. As an example, the content management system 101 can assign a unique URL (Uniform Resource Locator) to each image data and manage it in association with the user's account ID.

[0043] The content management system 101 obtains information about the user who registered the image data (S602). For example, the content management system 101 may obtain information about the user associated with the account ID of the user who registered the image data from the user profile DB 406 of the user management system 102.

[0044] The content management system 101 performs analysis on the registered image data (S603). For example, by analyzing the image data, the content management system 101 can generate analysis result data of metadata attached to the image, such as the date and time of shooting, shooting equipment information, resolution, size, recording format, color information, and brightness information. The content management system 101 can also use user information obtained from the registered image data to detect, recognize, and identify subjects contained in the image. For example, the content management system 101 can generate analysis result data of information that identifies subjects contained in the image, based on the user's family members and subjects related to the user's preferences contained in the user information. If the image data is video, the content management system 101 can generate analysis result data by dividing the data by scene, identifying speakers through analysis of audio information, identifying audio types, and creating spoken text through speech recognition. The analysis result data generated by the analysis of the image data can be stored in the content management DB 304 in association with the image data.

[0045] The prompt generation system 103 creates a prompt to cause the generation AI system 104 to generate a summary sentence for the user who owns the image data (S604). For example, as will be described later, the prompt generation system 103 may create a prompt based on information that identifies the image data for which a summary sentence should be generated, information about the user who owns the image data, and information contained in the image data analysis result data. The prompt generation system 103 supplies the created prompt to the generation AI system 104 (S605). The prompt generation system 103 also obtains the response generated by the generation AI system 104 based on the supplied prompt as a summary sentence (S606).

[0046] The prompt creation system 103 may analyze whether the generated summary is similar to other summary sentences (S607). For example, based on the fact that the purpose of generating the summary sentence is for the owner of the image data to identify the content of the image, the prompt creation system 103 may analyze whether the generated summary sentence is similar to other summary sentences. This may make it even easier for the owner of the image data to identify a specific image from among many images. For example, the prompt creation system 103 retrieves summary sentences of other images stored in the content management DB 304 of the content management system 101 and analyzes their similarity to the summary sentence obtained from the generation AI system 104. If the prompt creation system 103 determines that the summary sentence obtained from the generation AI system 104 is similar to one or more other summary sentences (YES in S608), it regenerates the summary sentence. For example, the prompt creation system 103 creates a prompt that includes information about summary sentences of other images that are similar to the summary sentence obtained from the generation AI system 104, with an added instruction that this summary sentence is not similar. The prompt creation system 103 then supplies the created prompt to the generation AI system 104 and instructs it to regenerate the summary sentence (S609). On the other hand, if the prompt creation system 103 determines that the summary sentence obtained from the generation AI system 104 is not similar to any of the other summary sentences (NO in S608), it provides that summary sentence to the content management system 101. This summary sentence may be associated with the target image data and stored in the content management DB 304 of the content management system 101 (S610).

[0047] In this example, based on the registration of image data by the user, a prompt is created to instruct the generation of a summary text for the image using information about the user who owns the image data, and this prompt is supplied to the generation AI system 104. Furthermore, if the summary text generated by the generation AI system 104 is similar to the summary text of other image data, a new prompt is created to generate a different summary text that is not similar to the previous one, and this prompt is supplied to the generation AI system 104. The generated summary text is then stored in association with the image data. This configuration makes it easy for the user to identify a specific image from among many images. The following describes the process of receiving user information entered by the user and the process of creating a prompt for generating a summary text for the owner, both of which are performed by the information processing system. Following this, an example of the configuration of the operation screen of the content management application used by the user is described.

[0048] (Processing of information received from users) Figure 7 shows an example of a user information operation screen when a user registers information about themselves in the user management system 102 via the content management application on the user terminal 112, or when editing information about a registered user. The user information operation screen 701 may include a user icon image 702, operation parts 703 and 704, username 705, email address 706, photographic equipment 707, owned lenses 708, and photographic genre 709. The user information operation screen 701 may also include family members 710, images of family members 711, operation parts 712, language 713, summary text preferences 714-716, and operation parts 717. The user icon image 702 may display a photograph of the user's face or any image registered by the user. For example, the user can specify the image to be displayed as the user icon image 702 by selecting one image from a list of images displayed by clicking on operation part 703. If the user does not specify a particular image, a default image indicating that the user has not specified a user icon image may be displayed. For example, a user can change the state from specifying a particular image to not specifying a particular image by clicking on the control component 704.

[0049] Username 705 indicates the user's name. The user's name may be the user's actual name, or it may be any string entered by the user, such as a nickname. The user management system 102 can accept any string entered in Username 705 as the username. Email address 706 indicates the user's email address. The user management system 102 can accept any string entered in Email address 706 as the user's email address.

[0050] The "Shooting Equipment 707" field indicates information that identifies the shooting equipment, such as a camera or video camcorder, used by the user for shooting. The "Owned Lenses 708" field indicates information that identifies the type of lens the user attaches to the shooting equipment. For example, if the metadata of the image data registered by the user does not include information identifying the equipment and lenses used for shooting, the information shown in "Shooting Equipment 707" and "Owned Lenses 708" can be associated with the equipment used to shoot that image data. "Shooting Equipment 707" and "Owned Lenses 708" may be entered using a dropdown list or by direct input. The user management system 102 can accept the information entered in "Shooting Equipment 707" and "Owned Lenses 708" as information about the shooting equipment used by the user.

[0051] The shooting genre 709 indicates the user's preference for the genre of images they take. For example, the shooting genre may include family snapshots, family movies, landscape snapshots, etc. The shooting genre 709 may be entered using a dropdown list or by direct input. The user management system 102 can accept the information entered in the shooting genre 709 as information about the user's preferred shooting genre.

[0052] The family member 710 indicates information about the members of the user's family. For example, the information about family members may include the names, ages, genders, etc. of the user's family. In FIG. 7, an example is shown where the information about family members is concatenated with underscore characters for display or input. For example, "B_7 years old_male" indicates that there is a family with the name "B", an age of "7 years old", and a gender of "male". When a part of the information about family members is not input, specific characters or symbols such as "?" may be used. For example, in the case of "A_?_male", it is indicated that the age is not input. When there are two or more family members, the family member 710 may be added. For example, the user can add a new family member 710 on the operation screen 701 by clicking the operation component 712. Also, the user can register images of each family member as information about family members. For example, the user can register an image of a family member by selecting one image from the list of images displayed by clicking on the image 711 of the family member. By registering an image of a family member, the user's family can be detected in the registered image data, and the description of the scene can be included in the summary text. The user management system 102 can accept the information and image data input to the family member 710 and the image 711 of the family member as information about the user's family members.

[0053] The language 713 indicates the language used by the user. For example, the language indicated by the language 713 can be used as the language for displaying information and summarizing images in the content management application. The user management system 102 can accept the language input to the language 713 as the language used by the user.

[0054] Summary preferences 714-716 indicate the user's preferences regarding summaries. For example, summary preference 714 indicates that the user prefers shorter summaries to longer ones. For instance, summary preference 714 could be configured as a dropdown list allowing the user to select the length (number of characters) of the summary from options such as "longer," "shorter," or "not specified." Also, summary preference 715 indicates that the user prefers summaries written in a more informal style to those written in a more formal style. Furthermore, summary preference 716 indicates that the user prefers objective summaries to subjective ones. The user management system 102 can accept the information entered into summary preferences 714-716 as information regarding the user's preferences regarding summaries.

[0055] (Process to create a prompt to generate a summary for the owner) Figure 8 shows an example of a prompt created by the prompt creation system 103 to generate a summary text for the owner. The prompt creation system 103 can create a prompt by obtaining the necessary information from the content management system 101 and the user management system 102. When supplying the created prompt to the generation AI system 104, the prompt creation system 103 may also supply other information related to the image. For example, the prompt creation system 103 may provide an image that can identify the owner of the image data, or an image that can identify the owner's family or a specific subject included in the image. The prompt 801 may include description sections for the summary text generation instruction 802, the image to be summarized 803, information about the image to be summarized 804, conditions 805, the purpose of the summary text 806, and information about the person who requested the summary 807. Each description section may include one or more items following the bullet point "·". For example, the description section for the summary text generation instruction 802 includes three items. Also, parts enclosed in curly braces "{}" indicate variables to which strings are assigned. For example, {URL} indicates that the URL of the image data to be summarized will be substituted.

[0056] Summary generation instruction 802 indicates the main instruction of the prompt. For example, summary generation instruction 802 may be an instruction included in the prompt by default. Note that {URL} may be replaced with a URL attached to the image data registered by the user. For example, summary generation instruction 802 includes an instruction that the summary of the image and information that identifies the scene in which the summary is described (the time code in the image) should be output.

[0057] The image to be summarized 803 indicates information that identifies the image data from which a summary should be generated. {URL} may be replaced with the URL attached to the image data registered by the user. Note that the information identifying the image data does not have to be the URL attached to the image. For example, the image data itself from which a summary should be generated may be supplied to the generation AI system 104.

[0058] Information 804 regarding the image to be summarized indicates information about the image data. For example, information about the image data may include information contained in the analysis result data generated by the analysis of the image data. For example, {SHOOTING DATE} may contain the date and time information of the image data being taken. {TIME LENGTH} may contain information indicating the length of the video. {MODEL} and {LENS} may contain information identifying the camera equipment and lens used for shooting, respectively.

[0059] Condition 805 specifies the conditions for creating the summary. For example, the conditions may include information about the user who owns the image data. Specifically, the conditions may include information indicating the user's preferences regarding the summary or information indicating the language used by the user.

[0060] The purpose 806 of the summary indicates information that identifies the purpose for which the summary is generated. For example, in the case of a summary intended for the owner of image data, the purpose for generating the summary may be to identify the image being summarized from other images. However, the purpose for generating a summary intended for the owner is not limited to this, and any purpose may apply. For example, a user may input the purpose for generating the summary as information about the user into the user management system 102. Also, when a user specifies specific image data to generate a summary, they may specify the image for which the summary is to be generated and the purpose for which the summary is to be generated, and have the prompt generation unit 103 create a prompt.

[0061] Information 807 regarding the requester of the summary indicates information about the owner of the image data. For example, the information about the owner may include information about the user who owns the image data. As an example, the information about the owner may include information about the owner related to the analysis result data obtained by analyzing the image data that should be used to generate the summary. For example, based on the fact that the analysis result data includes a family member named B, information about B may be included in the information 807 regarding the requester of the summary. Also, based on the fact that the analysis result data includes subjects related to the photography genre preferred by the owner, information about the photography genre preferred by the owner may be included in the information 807 regarding the requester of the summary.

[0062] In this way, the prompt supplied to the generation AI system 104 includes user information that the generation AI system 104 can use when creating the summary, so that the generation AI system 104 creates the summary based on this information. This makes it easier to generate a summary that is easier for the user to identify the images. Furthermore, by selecting user information included in the prompt based on the information contained in the analysis result data obtained by analyzing the image data in advance, more useful information is supplied to the generation AI system 104 in generating the summary. This makes it easier to generate a summary that is easier for the user to identify the images.

[0063] Figure 9 shows an example of a prompt newly created by the prompt creation unit 103 when the summary text generated by the generation AI system 104 is determined to be similar to the summary text of another image. In Figure 9, the same reference numbers are used for components similar to those in Figure 8, and their explanations are omitted. Prompt 901 includes the instruction that condition 902 is "a summary text that can be distinguished from similar summary texts," and includes a similar summary text 903. The similar summary text 903 is a summary text of another image that is similar to the summary text generated by the generation AI system 104. In this way, by adding a summary text to be compared for similarity determination and an instruction to generate a summary text that is not similar to this summary text to the prompt, it becomes easier for the generation AI system 104 to generate a summary text that can be distinguished from the summary texts of other images. Note that the instruction to generate a summary text that is not similar to the summary text of other images may be included in a description section other than condition 902. For example, the instruction to generate a summary text that is not similar to the summary text of other images may be included in the summary text generation instruction 802. Including instructions in the main instruction section that the summary should not be similar to the summaries of other images increases the likelihood that the generating AI system 104 will generate a summary that is distinguishable from other summaries. Furthermore, the similar summary 903 does not necessarily have to include the summary that is used as the comparison target for similarity determination. For example, by listing keywords that should and should not be used in the summary in the similar summary 903, the summary generated by the generating AI system 104 may become easier to distinguish from other summaries.

[0064] (Content management application configuration) This section describes an example configuration of the operation screen of a content management application installed on user terminal 112. Figure 10 shows an example configuration of the image list displayed on the content management application. The image list may be displayed based on the user's actions to display the image list after logging into the content management system 101. The image list is a screen that displays a list of images registered by the user. For example, within the content management application window 1001, icons representing each image registered by the user and operation components used by the user are displayed. The user icon image 1002 may display the user's face photograph or any image registered by the user. For example, the user icon image 1002 may be the same as the user icon image 702. Also, information indicating the username of the logged-in user ("User A") may be displayed near the user icon image 702.

[0065] Icon 1003 represents each of the still images registered by the user in the content management system 101. Icon 1003 may be a common icon for all still images, or it may be a thumbnail of each still image. Icon 1004 represents each of the videos registered by the user in the content management system 101. Icon 1004 may be a common icon for all videos, or it may be a thumbnail of a still image from any scene in each video. By selecting any of the icons, the image details shown in Figure 11 may be displayed. If it is not possible to display all the images registered by the user in the icon display area of ​​window 1001, the system may be configured to display images by moving between multiple pages using the operation elements. For example, the screen may be configured so that clicking the "Prev" operation element displays the image of the previous page, and clicking the "Next" operation element displays the image of the next page.

[0066] Figure 11 shows an example of the configuration of image details displayed on a content management application. Image details may be displayed based on the user selecting a specific image in the image list. In Figure 11, the same reference numbers are used for configurations similar to those in Figure 10, and explanations are omitted. Area 1101 is the area where the content of the image selected by the user is displayed. For example, if the image is a video, the video currently playing is displayed in area 1101. Operation component 1102 is an operation component for controlling the playback and pause of the video displayed in area 1101. For example, in Figure 11, the video displayed in area 1101 is paused, and the video playback starts when the user clicks operation component 1102. Area 1103 is the area where each of the multiple scenes contained in the video currently playing is displayed. For example, the video may start playing from a scene when the user selects one of the scenes. Each scene may be arranged along the time axis. Rectangle 1104 indicates the selected scene. For example, the user may select a scene by moving rectangle 1104 over another scene. Furthermore, if the image selected by the user is a still image, area 1103 may be configured not to exist, or nothing may be displayed.

[0067] Area 1105 is an area where information about the image selected by the user is displayed. For example, area 1105 may display information stored in the content management system 101 that is associated with the image selected by the user. For example, the image file name, file size, playback time, equipment used for shooting, lens, shooting date and time, etc., may be displayed. Area 1106 is an area where a summary text stored in association with the image selected by the user is displayed. For example, a summary text generated by the generation AI system 104 may be displayed. In the summary text of Figure 11, it is shown that "B's tug-of-war competition" is included in the video. In this way, by including information about the family members of the video data owner in the prompt supplied to the generation AI system 104, the summary text created by the generation AI system 104 will include information that makes it easier for the owner to identify the content of the image.

[0068] Icon 1107 indicates a characteristic scene contained in the image. For example, icon 1107 may be generated based on information that identifies a scene whose description is included in the summary text output by the generation AI system 104. For example, by instructing the generation AI system 104 to output the timecode of the scene whose description is included in the summary text in the prompt supplied to the generation AI system 104, a timecode corresponding to a characteristic scene contained in the image may be provided. The content management application may create and display a thumbnail from the still image of the scene corresponding to the identified timecode. When the user selects icon 1107, playback of the image from that scene may start in area 1101, similar to when the user selects a scene by manipulating rectangle 1104. Area 1108 indicates the timecode corresponding to the scene displayed in icon 1107. Operation components 1109 to 1113 will be described later. In this way, by displaying the summary text generated by the generation AI system 104 on the image details screen, it becomes easy for the user, who owns the image data, to identify the content of the image based on the summary text. Furthermore, on the image details screen, representative scenes included in the image output by the generation AI system 104 are displayed as thumbnails, making it easy for the user who owns the image data to play those scenes and identify the content of the image.

[0069] (Processing to modify the summary) A user can request a change to the summary text of an image stored in the content management system 101 using a content management application installed on the user terminal 112. For example, a user can request a change to the summary text of an image by specifying which scenes from among several scenes included in the image should be used to generate the summary text. Alternatively, a user can request a change to the summary text of an image by specifying which scenes from among several scenes included in the image should not be used to generate the summary text. The operation when a user requests a change to the summary text of an image stored in the content management system 101 will be explained using Figure 12. In Figure 12, the same reference number is used for configurations similar to those in Figure 11, and the explanation is omitted. In this example, the user wants to change the summary text shown in area 1106 so that it includes a description of foot races as well as tug-of-war.

[0070] In area 1201, the user can identify a scene related to an explanation they want to add to the summary and request a change to the summary. For example, the user selects a scene in area 1104 to which they want to add an explanation to the summary. Then, the user clicks the "Select" operation component 1202. This adds the scene selected by the user in area 1104 to area 1201. In fact, compared to Figure 11, the user selected a scene related to a foot race using rectangle 1104 and clicked operation component 1202, resulting in the addition of icon 1204, timecode 1205, and operation components 1206-1207 to area 1201. Operation components 1206-1207 indicate whether the explanation for the scene corresponding to icon 1204 is included in the summary. For example, as shown in Figure 11, if the explanation for the scene corresponding to icon 1107 is included in the summary, operation component 1109 is highlighted. For example, operation component 1109 is highlighted by being displayed in a different color than operation component 1110. On the other hand, if the corresponding scene is not included in the summary text, the operation component 1110 is highlighted. When the user adds an icon 1204 corresponding to a new scene to area 1201 through user operation, the operation component 1207 is highlighted to indicate that the description of this scene is not included in the summary text. The user selects operation component 1206 to include the added scene in the summary text. This highlights operation component 1206. The user then clicks the "Generate Summary" operation component 1112. This sends a request to the prompt creation system 103 to generate a summary text that includes the description of the scene corresponding to the "T" operation component 1206 (and operation component 1109 in Figure 11) in area 1201. If the user wishes to remove the icon 1204 shown in area 1201 from area 1201, they can select the icon 1204 and click the "Cancel" operation component 1203.

[0071] The prompt creation system 103 creates a new prompt when it receives a request from a user to change the summary text via the content management application. For example, if the prompt creation system 103 receives a request from a user that includes information identifying scenes that should or should not be included in the summary text, it creates a prompt that includes information identifying these scenes. Figure 13 shows an example of a prompt that includes information identifying scenes that should be included in the summary text. In Figure 13, the same reference numbers are used for components similar to those in Figure 9, and their explanations are omitted. Prompt 1301 includes a scene specification 1303 in condition 1302. The scene specification 1303 identifies scenes that should be included in the summary text, for example, by time codes in images. Figure 13 indicates that descriptions of two scenes, indicated by the time codes "00:19:23" and "00:23:00" as specified by the user, should be included in the summary text. In this way, based on user input, prompts containing information identifying scenes to be included in the summary are supplied to the generation AI system 104, so that the generation AI system 104 provides a summary including descriptions of these scenes. The prompt creation system 103 provides the content management system 101 with the summary generated based on the new prompt and time codes identifying the scenes included in the summary, associating them with the target image data. The prompt creation system 103 also provides the summary generated based on the new prompt and time codes identifying the scenes included in the summary as a response to the user's request to the content management application on the user terminal 112. The content management application may display the provided summary and time codes identifying the scenes included in the summary to the user. For example, the content management application may update the display of area 1106 and area 1201 based on the provided summary and time codes. The content management application may also confirm with the user whether or not to update the summary stored in the content management system associated with the target image based on the provided summary.In this case, based on the user's input of an instruction to update, the content management application may notify the content management system 101 that the summary should be updated. If the user does not input an instruction to update, the content management application will not notify the content management system 101 that the summary should be updated. Based on the notification from the content management application, the content management system 101 may update the stored summary.

[0072] Figure 14 shows an example of the processing flow executed in the information processing system when modifying a summary text based on a user request. First, the content management application on the user terminal 112 displays the image details of the image selected by the user (S1401). For example, the content management application may display the image details screen shown in Figure 11 on the user terminal 112. The content management application receives instructions from the user to modify the summary text (S1402). For example, the content management application may receive a request from the user that includes information specifying which scenes should be included in the summary text description and which should not be included. The user's request to modify the summary text is notified to the content management system 101 via the content management application. For example, the content management application may send a notification that includes information specifying the target image data, information specifying the user who made the request, and information specifying conditions related to the generation of the summary text, such as which scenes should be included in the summary text description and which should not be included.

[0073] The content management system 101 initiates processing for modifying the summary text based on the request. For example, the content management system 101 instructs the prompt creation system 103 to create a prompt for generating a new summary text. For example, the content management system 101 may provide instructions that include information identifying the target image data, information identifying the user who made the request, and information identifying the conditions for generating the summary text, such as scenes to be included in the summary text's description and scenes that should not be included. The prompt creation system 103 creates a new prompt based on the instructions from the content management system 101. For example, the prompt creation system 103 acquires information necessary for creating the prompt (S1403). As an example, the prompt creation system 103 may acquire analysis result data of the image for which a summary text should be generated, information about the user who owns the image, etc., based on the information included in the instructions from the content management system 101. The prompt creation system 103 instructs the generation AI system 104 to generate the summary text (S1404). For example, the prompt creation system 103 instructs the generation AI system 104 to generate the summary text by supplying the newly created prompt to the generation AI system 104. The prompt creation system 103 acquires the summary text generated by the generation AI system 104 and information identifying the scene in which the summary text contains the explanation (S1405), and provides it to the content management system 101. The content management system 101 provides the acquired summary text and information identifying the scene in which the summary text contains the explanation to the content management application on the user terminal 112. For example, as a response to a request from the user, the content management system 101 may provide information identifying the target image data, a newly generated summary text, information identifying the scene in which the summary text contains the explanation, etc.

[0074] The content management application on the user terminal 112 displays the summary text provided by the prompt creation system 103 and the scene in which the summary text contains an explanation in image detail (S1405, S1406). The content management application may then accept input from the user. For example, the content management application may ask the user whether or not to update the information stored in the content management system 101 using the newly generated summary text and information identifying the scene in which the summary text contains an explanation. Based on the user input, the content management application may update the information stored in the content management system 101. In this case, the content management application may send a notification to the content management system 101 that includes information identifying the target image, the summary text to be used for the update, and information identifying the scene in which the summary text contains an explanation. Based on the notified information, the content management system 101 may update the information stored in its device (S1408). The content management application may also make further changes to the summary text based on the user input. Based on the user input, the content management application terminates the display of image details (S1407). For example, when the "Back" button 1113 in Figure 12 is clicked, the content management application may end the display of the image details screen and display the image list screen in Figure 10.

[0075] In the above explanation, we used an example where the content management application instructs the content management system 101 to regenerate the summary text based on a user's instruction to change the summary text. However, changes to the summary text can be performed by other means. For example, the content management application may instruct the prompt creation system 103 to regenerate the summary text. In this case, the prompt creation system 103 obtains the information necessary to create a prompt based on the instruction from the content management application and creates a new prompt for regenerating the summary text. The prompt creation system 103 also provides the content management application with the summary text generated based on the new prompt.

[0076] In this example, the user can request a change to the summary generated by the generation AI system 104. The user's request to change the summary may include information identifying scenes in which explanations should be included in the summary or scenes in which explanations should not be included. The explanations to be included in the summary may be indicated by means other than time codes. For example, keywords may be used to indicate scenes in which explanations should be included or not included in the summary. The prompt creation system 103 creates a new prompt based on the user's request and causes the generation AI system 104 to regenerate the summary. This configuration allows the summary to be changed according to the user's request.

[0077] (Second example of processing) The first processing example described an example in which a summary text for the owner of image data is generated based on a user who has logged into the content management system 101 and registered image data in the content management system 101. This processing example describes an example in which a summary text for a second user is generated based on a first user who has registered image data in the content management system 101 and has performed an operation to share that image data with a second user. Figure 15 shows an example of the configuration of the information processing system in this embodiment. For configurations that are the same as in Figure 1, the same reference number is assigned and the explanation is omitted. That is, in Figure 15, there is a user terminal 113. The user terminal 113 is a user terminal used by a second user who will be shared the image data by the owner of the image data. The user terminal 113 includes personal computers (PCs), smartphones, tablets, etc. It is assumed that the user terminal 113 has an application installed for displaying images from image data and an application for displaying messages received from the content management system 101. Assume that the second user is registered as a user of the content management system 101, just like the first user who uses the user terminal 112, and that information about the second user is registered in the user management system 102.

[0078] (Image data sharing process) This section describes the process by which a first user, who owns the image data, shares the image data with a second user. Figure 16 shows an example of the configuration of a screen for image sharing displayed on the content management application of the user terminal 112. For example, the screen for image sharing may be displayed when the user clicks the "Share" operation component 1111 shown in Figure 11, etc. In Figure 16, the same reference number is used for configurations similar to those in Figure 12, and the explanation is omitted. Note that "sharing" image data includes making the image data registered by the first user in the content management system 101 available for display and use by a second user other than the first user. As a method for making the image data available for display and use by the second user, the content management system 101 may grant the second user access rights to the image data it manages, or it may send a copy of the image data to the second user. In other words, until the image data is shared, only the first user has access rights to that image data.

[0079] Area 1601 is an area for a first user, who is the owner of the image data, to specify a second user with whom the image data will be shared. For example, the first user can specify a second user using information stored in the user address book DB 407 that is associated with the first user. As an example, the user address book DB 407 may store information such as email addresses and SNS accounts (account IDs and usernames) of other users registered by the first user, associated with the first user. In this case, the content management application can retrieve the information stored in the user address book DB 407 of the user management system 102 based on the selection made by the first user in the pull-down list, and display it to the first user as a pull-down list. The user can then select the information corresponding to the second user from the displayed pull-down list.

[0080] Area 1602 is an area for the first user to select the purpose for which they want to share image data with the second user. Area 1602 displays a dropdown list of options for the purpose of sharing image data, from which the first user can select one purpose. The default purpose for sharing image data may be "I want to share the image content." Possible purposes for sharing image data include "I want to give the image as a gift," "I want to hear feedback on the image content," "I want to share the shooting techniques related to the image," "I want to discuss the shooting techniques related to the image," and "I want to share information related to the equipment used to take the image."

[0081] Area 1603 is an area for the first user to indicate their intention to share image data with the second user. While Area 1602 is configured for the first user to select from multiple common options, Area 1603 allows the first user to input arbitrary information. For example, in Area 1603, the first user may indicate specific circumstances for the second user or a sharing intention unique to the first user. The purpose and intention of sharing indicated by the first user in Areas 1602 and 1603 may be notified to the prompt creation system 103. The prompt creation system 103 may create a prompt using the purpose and intention of sharing indicated by the first user.

[0082] The first user can identify in area 1201 which scenes should include descriptions in the summary and which should not. For example, if the intention to share the image data is to show the second user's family, which includes a tug-of-war, the first user can select which scenes to include a description of, such as the tug-of-war scene, but not the foot-racing scene.

[0083] The first user can request the system to begin processing to share specified image data with a second user by clicking the "Share" operation component 1111. For example, when operation component 1111 is clicked, a request is made to the content management system 101 to begin processing to share the image data with the second user. This request may include information entered by the first user, such as the recipient, purpose of sharing, intention to share, and information specifying which scenes should and should not be included in the summary. Based on the information included in the request, the content management system 101 executes the processing to share the image data with the other specified user. As an example, the content management system 101 instructs the prompt generation unit 103 to create a summary about the image data to be shared. Based on the instruction, the prompt generation unit 103 obtains information about the second user who is the recipient and analysis result data associated with the image data. The prompt generation unit 103 also creates a prompt to generate the summary based on the information included in the request and the obtained information about the second user and analysis result data. The prompt creation unit 103 supplies the created prompt to the generation AI system 104 to generate a summary sentence. The prompt creation unit 103 then acquires information identifying the summary sentence generated by the generation AI system 104 and the scenes in which the summary sentence contains explanations, and provides this information to the content management system 101. The content management system 101 creates a message for a second user that includes the acquired summary sentence and the scenes in which the summary sentence contains explanations, and sends it to the second user.

[0084] Furthermore, the first user can request the generation of a summary text for a specified image by clicking the "Generate Summary" operation component 1112. For example, when operation component 1112 is clicked, a request is made to the content management system 101 to generate a summary text for sharing the image data with the second user. This request may include information entered by the first user, such as the recipient, purpose of sharing, intention to share, and information identifying scenes that should and should not be described in the summary text. In this case, the content management system 101 performs the process to generate the summary text in the same way as when the "Share" operation component 1111 is clicked. The content management system 101 then provides the generated summary text to the content management application on the user terminal 112. The content management application displays the retrieved summary text to the user. After confirming the displayed summary text, the user can click the "Share" operation component 1111 to start sharing the image data. The user can also request to change the displayed summary text. If a user wishes to modify the displayed summary, they can add or delete new scenes in area 1201 and then click the "Generate Summary" operation component 1112 again. This allows the user to review the summary notified to the second user and then share the image data. In this case, the summary and the prompt for generating the summary may be configured to exclude certain types of information about the second user. For example, the prompt creation system 103 may instruct the system not to include certain information about the second user in the summary when creating a prompt. This prevents the first user from being presented with information about the second user that they would not ordinarily know. Certain types of information about the second user may include, for example, information about the second user's preferences. Furthermore, when the second user registers information about themselves in the user management system 102, they may be able to choose whether or not to make each piece of information they register public. In this case, certain types of information about the second user may include information that the second user has chosen not to make public.

[0085] Figure 17 shows an example of prompts created by the prompt creation system 103 to cause the generation AI system 104 to generate a summary text for a second user. In Figure 17, the same reference numbers are used for components similar to those in Figure 8, and their explanations are omitted. Prompt 1701 may include instructions that the summary text generation instruction 1702 "include information about the reader of the summary text" and "do not include information about the person who requested the summary text." As a result, the summary text generated by the generation AI system 104 will include information about the second user and not information about the first user. For example, the summary text will not include information about B, a family member of the first user, but will include information about D, a family member of the second user. Note that the summary text generation instruction 1702 may also include an instruction indicating that information about the first user may be included in the summary text. For example, if the first user's sharing intention indicates that they want the summary to include a scene in which both B and D are visible, they may be instructed to create the summary based on the information about the first user and the information about the second user.

[0086] Condition 1702 specifies the conditions for a summary text that makes the image content easily identifiable to a second user. For example, Condition 1702 may be created based on information about the second user. This could include information indicating the second user's preferences for summary texts or information indicating the language used by the second user. Condition 1702 may also include a scene specification 1703, which is information that identifies the scenes in which the summary text should include a description. The scene specification 1703 may be created based on the first user's selection in area 1201 of Figure 16.

[0087] The purpose of the summary 1704 and the shared intent of the requester of the summary 1706 indicate the purpose for which the image data is shared. For example, the purpose of the summary 1704 and the shared intent of the requester of the summary 1706 may be created based on the input of the first user in areas 1602 and 1603 of Figure 16.

[0088] Information 1705 concerning the reader of the summary indicates information about a second user. For example, the information concerning the reader of the summary may include information about a second user related to the analysis result data obtained by analyzing the image data to be used to generate the summary. For example, based on the fact that the analysis result data includes a second family member, Mr. D, information about Mr. D may be included in the information 1705 concerning the reader of the summary. Also, based on the fact that the analysis result data includes subjects related to a photography genre that suits the second user's preferences, information about a photography genre that suits the second user's preferences may be included in the information 1705 concerning the reader of the summary.

[0089] Figure 18 shows an example of the structure of a message to a second user generated by the content management system 101. The message to the second user 1801 can be the body of an email or SNS message. Message 1801 may include main text 1802, an image summary 1803, and an embedded image 1804. Text 1802 includes information indicating the second user to whom the message is addressed and the first user to whom the message is sent. Image summary 1803 includes a summary generated by the generation AI system 104. In summary 1803, there is no description of B, a family member of the first user, but there is a description of D, a family member of the second user. In this way, by instructing the generation AI system 104 to generate a summary that does not include information about the first user but includes information about the second user in the prompt supplied, the summary is generated based on information about the second user. Embedded image 1804 is a thumbnail of a representative scene contained in the image. For example, embedded image 1804 may be a thumbnail corresponding to a scene output by the generation AI system 104 as a scene included in the description of summary text 1803. As an example, if embedded image 1804 has a hyperlink set to image data stored in content management DB 304, a second user can click embedded image 1804 to start downloading that image data.

[0090] Figure 19 shows an example of the processing flow executed in the information processing system when image data is shared with a second user based on a request from a first user. First, the content management application on the user terminal 112 displays a screen for sharing the image based on the first user's operation to share the image data (S1901). For example, the content management application may display the image sharing screen shown in Figure 16 on the user terminal 112. The content management application may accept input of information regarding sharing from the first user (S1902). For example, the content management application may accept input from the first user such as information identifying the second user to whom the image will be shared, information indicating the purpose and intent of sharing, and information identifying scenes that should and should not be included in the summary description. The first user's request to share the image data is notified to the content management system 101 via the content management application. For example, the content management application may send a notification that includes information identifying the target image data, information identifying the user to whom the image will be shared, information indicating the purpose and intent of sharing, and information identifying scenes that should and should not be included in the summary description.

[0091] The content management system 101 creates a message to notify a second user who will be the recipient of the shared content, based on a notification from the content management application. First, the content management system 101 instructs the prompt creation system 103 to create a summary of the image to be shared. The prompt creation system 103 obtains information necessary to create a prompt for the generation AI system 104 to generate the summary (S1903). For example, the prompt creation system 103 may obtain information about the image for which a summary should be generated, information about the second user to whom the image will be shared, etc., based on the information contained in the notification from the content management application. The prompt creation system 103 instructs the generation AI system 104 to generate the summary (S1904). For example, the prompt creation system 103 instructs the generation AI system 104 to generate the summary by supplying the created prompt to the generation AI system 104. The prompt creation system 103 obtains information that identifies the summary generated by the generation AI system 104 and the scene in which the summary contains an explanation (S1905), and provides it to the content management system 101.

[0092] The content management system 101 creates a message that includes a summary text generated by the generation AI system 104 and a scene that contains an explanation of this summary text (S1906). For example, the content management system 101 may create the message shown in Figure 18. The content management system 101 then notifies the second user who is the recipient of the created message (S1907). For example, if the information indicating the second user who is the recipient of the shared content, selected by the first user, is an email address, the content management system sends an email containing the created message to that email address. Furthermore, if the content management system 101 provides a function that enables the sharing of image data between users, the content management system 101 may use that function to notify the second user who is the recipient of the shared content of the message. For example, the content management system 101 may notify the account of the second user who is the recipient of the shared content, selected by the first user, of the created message. The second user who receives the message can identify the content of the image from the summary text included in the message and decide whether or not to view the image.

[0093] In this example, the summary generation process is executed based on the fact that the first user, who owns the image data, has performed an operation to share that image data with a second user. For example, a prompt is created that instructs the generation of a summary of the image using information about the second user who will view the image, and this prompt is supplied to the generation AI system 104. The generated summary is included in a message containing information that identifies the image data and is notified to the second user. This configuration makes it easy for the second user to determine whether or not they should access the image included in the message.

[0094] As described above, according to this embodiment, the information processing system creates a prompt that includes data for a predetermined image for which a summary should be generated, and information about the viewer who will view the predetermined image and summary, and causes the generating AI to create the summary. With this configuration, the generating AI generates a summary that includes information about the viewer of the image and summary. As a result, a summary is provided that makes it easy for the viewer to identify the content of the image.

[0095] The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0096] (Summary of the embodiments) At least some of the embodiments described above can be summarized as follows: (Item 1) An information processing system that provides a summary text of an image, A means for identifying specific information, which includes data of a predetermined image from which a summary should be generated, and information about a viewer who views the predetermined image. An instruction statement for causing a generative artificial intelligence (generative AI) to generate a summary text of a predetermined image, comprising a creation means for creating the instruction statement which includes the specific information, A supply means for supplying the instruction text to the generating AI, The system includes a providing means that provides the viewer with a summary text generated by the generating AI based on the instruction text, associated with a predetermined image. An information processing system characterized by the following: (Item 2) The information relating to the viewer includes at least one of the following: the viewer's age, the viewer's gender, the members of the viewer's family, the ages of each member, the genders of each member, the language the viewer uses, the viewer's preferences for summaries, and the viewer's preferences for subjects to photograph. The information processing system described in item 1, characterized by the features described herein. (Item 3) The aforementioned viewer is the owner of the data of the predetermined image, The identifying means identifies the information relating to the owner as the information relating to the viewer. An information processing system according to item 1 or 2, characterized by the features described above. (Item 4) The identification means identifies information about the owner by receiving information about the owner entered by the owner. The information processing system described in item 3, characterized by the features described herein. (Item 5) A storage means for storing a summary text generated by the generation AI for each of one or more images, The system further includes an analysis means for analyzing the similarity between the summary text generated by the generation AI for the predetermined image and the summary texts of each of the one or more images that are held, The creation means further creates a new prompt which includes information about the summary sentences of one or more images that are determined to be similar by the similarity value, and an instruction to generate a summary sentence that is not similar to the summary sentences of the one or more images. The providing means provides the owner with a summary text regenerated by the generating AI using the new prompt, associated with the predetermined image. An information processing system according to item 3 or 4, characterized by the features described above. (Item 6) The providing means, as part of the provision, displays to the owner the predetermined image and the summary text generated by the generating AI. An information processing system characterized by any one of items 3 to 5. (Item 7) The aforementioned viewer is a designated user who is shared with the designated image by the owner of the data of the designated image, The aforementioned identification means identifies the information relating to the predetermined user as the information relating to the viewer. An information processing system according to item 1 or 2, characterized by the features described above. (Item 8) The identification means identifies information about the predetermined user by obtaining information about the predetermined user registered in the information processing system based on information identifying the predetermined user entered by the owner. The information processing system described in item 7, characterized by the features described herein. (Item 9) The identifying means further identifies information relating to the owner as the identifying information. An information processing system according to item 7 or 8, characterized by the features described above. (Item 10) The creation means creates the instruction statement which includes an instruction to generate a summary statement without including specific types of information relating to the predetermined user. An information processing system characterized by any one of items 7 to 9. (Item 11) The providing means provides, as part of the provision, a notification to the designated user that includes information that can identify the data of the designated image and a summary text generated by the generating AI. An information processing system characterized by any one of items 7 to 10. (Item 12) The identifying means further specifies the purpose of generating the summary text as the identifying information, based on input from the owner of the data of the predetermined image. An information processing system according to item 1 or 2, characterized by the features described above. (Item 13) The aforementioned image is a video, If the predetermined image includes multiple scenes, the identifying means further identifies a first scene to be included in the summary as the identifying information. An information processing system according to item 1 or 2, characterized by the features described above. (Item 14) The aforementioned image is a video, If the predetermined image includes multiple scenes, the identifying means further identifies a second scene that should not be included in the summary as identifying information. An information processing system according to item 1 or 2, characterized by the features described above. (Item 15) Output means for outputting the summary text generated by the generation AI to the owner of the data of the predetermined image, The system includes a receiving means for receiving input from the owner, which includes information for modifying the summary text generated by the generation AI, The creation means creates a new instruction sentence that has been modified based on the information for modifying the summary sentence, The supply means supplies the new instruction text to the generating AI, The providing means provides the viewer with a summary text generated by the generating AI based on the new instruction text, associated with the predetermined image. An information processing system characterized by any one of items 1 to 14. (Item 16) An information processing method performed by an information processing system that provides summary texts of images, Identifying specific information that includes data for a predetermined image from which a summary should be generated, and information about the viewer who views the predetermined image; An instruction statement for causing a generative artificial intelligence (generative AI) to generate a summary text of the predetermined image, the instruction statement containing the specific information, The instruction text is supplied to the generating AI, The system provides the viewer with a summary text generated by the AI ​​based on the instructions, associated with a predetermined image. An information processing method characterized by the following: (Item 17) A program that causes a computer to function as one of the means of an information processing system described in any one of items 1 through 15.

[0097] The invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, claims are attached to disclose the scope of the invention. [Explanation of symbols]

[0098] 101: Content management system, 102: User management system, 103: Prompt creation system, 104: Generative AI system, 111: User terminal, 112: User terminal, 121: Network

Claims

1. An information processing system that provides a summary text of an image, A means for identifying specific information, which includes data of a predetermined image from which a summary should be generated, and information about a viewer who views the predetermined image. An instruction statement for causing generative artificial intelligence (generative AI) to generate a summary text of the predetermined image, comprising a means for creating the instruction statement which includes the specific information, A supply means for supplying the instruction text to the generating AI, The system includes a means for providing to the viewer a summary text generated by the generating AI based on the instruction text, associated with a predetermined image. An information processing system characterized by the following:

2. The information relating to the viewer includes at least one of the following: the viewer's age, the viewer's gender, the members of the viewer's family, the ages of each member, the genders of each member, the language the viewer uses, the viewer's preferences for summaries, and the viewer's preferences for subjects to photograph. The information processing system according to feature 1.

3. The aforementioned viewer is the owner of the data of the predetermined image, The identifying means identifies the information relating to the owner as the information relating to the viewer. The information processing system according to feature 1.

4. The identification means identifies information about the owner by receiving information about the owner entered by the owner. The information processing system according to feature 3.

5. A storage means for storing the summary text generated by the generation AI for each of one or more images, The system further includes an analysis means for analyzing the similarity between the summary text generated by the AI ​​for the predetermined image and the summary texts of each of the one or more images that are held, The creation means further creates a new prompt which includes information about the summary sentences of one or more images that are determined to be similar by the similarity value, and an instruction to generate a summary sentence that is not similar to the summary sentences of the one or more images. The providing means provides the owner with a summary text regenerated by the generating AI using the new prompt, associated with the predetermined image. The information processing system according to feature 3.

6. The providing means, as part of the provision, displays to the owner the predetermined image and the summary text generated by the generating AI. The information processing system according to feature 3.

7. The aforementioned viewer is a designated user who is shared with the designated image by the owner of the data of the designated image, The aforementioned identification means identifies the information relating to the predetermined user as the information relating to the viewer. The information processing system according to feature 1.

8. The identification means identifies information about the predetermined user by obtaining information about the predetermined user registered in the information processing system based on information identifying the predetermined user entered by the owner. The information processing system according to feature 7.

9. The identifying means further identifies information relating to the owner as the identifying information. The information processing system according to feature 7.

10. The creation means creates the instruction statement which includes an instruction to generate a summary statement without including specific types of information relating to the predetermined user. The information processing system according to feature 7.

11. The aforementioned provision means provides a notification to the predetermined user that includes information that can identify the data of the predetermined image and a summary text generated by the generation AI. The information processing system according to feature 7.

12. The identifying means further specifies the purpose of generating the summary text as the identifying information, based on input from the owner of the data of the predetermined image. The information processing system according to feature 1.

13. The aforementioned image is a video, If the predetermined image includes multiple scenes, the identifying means further identifies a first scene to be included in the summary as the identifying information. The information processing system according to feature 1.

14. The aforementioned image is a video, If the predetermined image includes multiple scenes, the identifying means further identifies a second scene that should not be included in the summary as identifying information. The information processing system according to feature 1.

15. Output means for outputting the summary text generated by the generation AI to the owner of the data of the predetermined image, The system includes a receiving means for receiving input from the owner, which includes information for modifying the summary text generated by the AI, The creation means creates a new instruction sentence that has been modified based on the information for modifying the summary sentence, The supply means supplies the new instruction statement to the generating AI, The providing means provides the viewer with a summary text generated by the generating AI based on the new instruction text, associated with the predetermined image. The information processing system according to feature 1.

16. An information processing method performed by an information processing system that provides summary texts of images, Identifying specific information that includes data for a predetermined image from which a summary should be generated, and information about the viewer who views the predetermined image; An instruction statement for causing generative artificial intelligence (generative AI) to generate a summary text of the predetermined image, wherein the instruction statement includes the specific information, The instruction statement is supplied to the generating AI, The system provides the viewer with a summary text generated by the AI ​​based on the instructions, associated with a predetermined image. An information processing method characterized by the following:

17. A program for causing a computer to function as one of the means of the information processing system described in claim 1.

Images