Program, information processing method, and information processing apparatus

Abstract

This program provides information and categories related to the year a film photograph was taken. [Solution] A program according to one embodiment of the present disclosure acquires capture data generated by capturing a film photograph, and provides the acquired capture data to a language model, causing the computer to perform a process that outputs information about the year the film photograph was taken and its category. A program according to one embodiment of the present disclosure outputs the category and the year the film photograph was taken by providing the language model with prompts that include instructions for specifying the year the film photograph was taken and its category.

Description

[[Technical Field]]

[0001] The present technology relates to a program, an information processing method, and an information processing apparatus. [[Background Art]]

[0002] Conventionally, techniques for scanning film photographs to obtain image data have been proposed. For example, the image processing apparatus described in Patent Document 1 scans image data of a photographic image and extracts date information described in the photographic image. [[Prior Art Documents]] [[Patent Documents]]

[0003] [[Patent Document 1]] Japanese Unexamined Patent Publication No. 2003-283789 [[Summary of the Invention]] [[Problem to be Solved by the Invention]]

[0004] However, the technique described in Patent Document 1 does not take into consideration obtaining information and categories related to the shooting year of a film photograph.

[0005] The present disclosure has been made in view of such circumstances, and an object thereof is to provide a program or the like capable of obtaining information and categories related to the shooting year of a film photograph. [[Means for Solving the Problem]]

[0006] A program according to an embodiment of the present disclosure, when executed by a computer, causes the computer to perform processing of obtaining capture data generated by capturing a film photograph, inputting the obtained capture data to a language model, and outputting information related to the shooting year of the film photograph and a category of the film photograph. [[Effects of the Invention]]

[0007] In a program according to one embodiment of this disclosure, it is possible to obtain information regarding the year and category of a film photograph. [Brief explanation of the drawing]

[0008] [Figure 1] This is an explanatory diagram showing an overview of the information processing system. [Figure 2] This is a block diagram showing an example of the configuration of an information processing device. [Figure 3] This is a block diagram showing an example configuration of a terminal device. [Figure 4] This is an explanatory diagram showing an example of a user table. [Figure 5] This is an explanatory diagram showing an example of a captured data table. [Figure 6] This is an explanatory diagram showing an example of a comment table. [Figure 7] This is an explanatory diagram showing an example of a language model. [Figure 8] This is an explanatory diagram showing an example of a language model. [Figure 9] This is an explanatory diagram showing an example of a capture data registration screen. [Figure 10] This is an explanatory diagram showing an example of the capture data list screen. [Figure 11] This is an explanatory diagram showing an example of a comment screen. [Figure 12] This is an explanatory diagram showing an example of a user comment screen. [Figure 13] This flowchart shows an example of the capture data registration process. [Figure 14] This flowchart shows an example of the process for displaying captured data. [Figure 15] This flowchart shows an example of the process for displaying captured data. [Figure 16] This is a flowchart illustrating an example of the language model retraining process. [Modes for carrying out the invention]

[0009] (Embodiment 1) When capture data obtained by capturing a film photograph is input, the language model M according to Embodiment 1 outputs information relating to the shooting year of the film photograph and a category. Embodiment 1 will be described below with reference to the drawings.

[0010] FIG. 1 is an explanatory diagram showing an overview of an information processing system S. The information processing system S includes an information processing device 1, a terminal device 2, a capture device 3, and a language model server 4. The capture device is a device (film scanner) that converts a film photograph (a photograph recorded on a film) into capture data by capturing the film photograph. Note that the capture device may be a scanner or an imaging device capable of capturing a developed photograph, or a camera incorporated in a smart glass, a smartphone, a tablet terminal, a personal computer, or the like.

[0011] The information processing device 1 transmits capture data of the film photograph captured by the capture device 3 to the language model server 4, and inputs the capture data to the language model M of the language model server 4. Hereinafter, transmission of information to the language model server 4 and input of information to the language model M shall be synonymous. The information processing device 1 stores information relating to the shooting year (shooting year information) of the film photograph pertaining to the capture data output by the language model M and the category of the film photograph, and transmits the same to the terminal device 2 in response to a request from the terminal device 2.

[0012] Terminal device 2 is a terminal device used by the user. Terminal device 2 accepts input of comments, for example, voice comments, on the film photographs related to the captured data, and transmits the voice-input comments (voice comments) to the information processing device 1. The information processing device 1 stores the voice comments and the identification information of the user who entered the voice comments, associating them with the film photograph's captured data. Terminal device 2 also accepts input of comments, for example, text comments, on the film photographs related to the captured data, and transmits the text-input comments (text comments) to the information processing device 1. The information processing device 1 stores the content of the text comments and the identification information of the user who entered the text comments, associating them with the film photograph's captured data.

[0013] Figure 2 is a block diagram showing an example configuration of the information processing device 1. The information processing device 1 is, for example, a server computer and comprises a processing unit 11, a storage unit 12, and a communication unit 13. The processing unit 11 is composed of a CPU (Central Processing Unit), MPU (Micro Processing Unit), GPU (Graphical Processing Unit), or quantum processor, and performs various control processing, arithmetic processing, etc., by reading and executing a program P (program product) and a database pre-stored in the storage unit 12. A database server or the like may be provided outside the information processing device 1, and the database may be read from the database server or the like. Furthermore, the information processing device 1 may have its functions realized by multiple server devices or computers. Also, the information processing device 1 may correspond to a node on a blockchain. Furthermore, the processing unit 11 of the information processing device 1 may execute some or all of the processing performed by the terminal device 2.

[0014] The storage unit 12 of the information processing apparatus 1 is, for example, a volatile memory or a non-volatile memory. The storage unit 12 stores a program P, a user table T1, a capture data table T2, and a comment table T3. The program P may be provided to the information processing apparatus 1 using a storage medium 12a stored in a computer-readable manner. The storage medium 12a is, for example, a portable memory. Examples of portable memories include CD-ROM, USB (Universal Serial Bus) memory, SD card, micro SD card, CompactFlash (registered trademark) memory, and the like. When the storage medium 12a is a portable memory, the processing element of the processing unit 11 may read the program P from the storage medium 12a using a reader not shown in the figures. The read program P is written into the storage unit 12. Furthermore, the program P may be provided to the information processing apparatus 1 through the communication between the communication unit 13 and an external device. Details of the user table T1, the capture data table T2, and the comment table T3 will be described later. In the present embodiment, the storage unit 12 of the information processing apparatus 1 stores various tables in a relational database format, but the configuration is not limited thereto. The storage unit 12 may store information stored in the various tables described above by a NoSQL database including a document database such as one in JSON (JavaScript Object Notation) format, for example.

[0015] The communication unit 13 is a communication module or communication interface for communicating with the terminal device 2, the capture device 3, and the language model server 4 by wire or wirelessly, and is, for example, a wide-area wireless communication module such as LTE (registered trademark), 4G, or 5G. The processing unit 11 communicates with the terminal device 2, the capture device 3, and the language model server 4 via the communication unit 13 through an external network N such as the Internet.

[0016] The language model server 4 stores a language model M. Details of the language model M will be described later. Note that the language model M may be stored in the information processing apparatus 1 or the terminal device 2.

[0017] Figure 3 is a block diagram showing an example configuration of terminal device 2. Terminal device 2 is, for example, a smartphone, a tablet terminal, or a personal computer. In this embodiment, the example of terminal device 2 being a smartphone will be described. Terminal device 2 comprises a terminal processing unit 21, a storage unit 22, a communication unit 23, an input unit 24, a display unit 25, and an audio input unit 26. The terminal processing unit 21 is composed of a CPU or MPU, etc., and performs various control processing, calculation processing, etc. Terminal device 2 may perform all or part of the processing performed by the information processing device 1 or the language model server 4. Furthermore, terminal device 2 may be configured as an integral part with the capture device 3.

[0018] The storage unit 22 stores an application program Pa that displays the captured data acquired by the information processing device 1, information regarding the year the film photographs of the captured data were taken, and the category, and accepts voice or text comments. The application program Pa is provided to the terminal device 2, for example, using the storage medium 22a. The terminal processing unit 21 of the terminal device 2 may acquire the application program Pa using the internet and store it in the storage unit 22.

[0019] The input unit 24 receives various information, selections, or instructions from the user. The display unit 25 displays the captured data acquired from the information processing device 1, as well as information about the year the film photographs in the captured data were taken and their categories. If the terminal device 2 is a smartphone, the input unit 24 and the display unit 25 are integrated by a touch panel.

[0020] The voice input unit 26 accepts voice input of comments from the user. If the terminal device 2 is a smartphone, the voice input unit 26 is composed of the microphone built into the smartphone.

[0021] Figure 4 is an explanatory diagram showing an example of user table T1. User table T1 stores information about users of the information processing system S, and information about captured data that users can view. The management items (fields) of user table T1 include the user ID field, the username field, and the group ID field.

[0022] The User ID field in User Table T1 stores the User ID (identification information) assigned to the user. The User Name field stores the user's name. The User Name may be the user's real name or a username. The Group ID field stores the Group ID of the group to which the user belongs. Capture data registered by a user can be viewed by the user and other users in the group to which that user belongs, and a user can view capture data registered by the user and other users in the group to which they belong. The User Table T1 may also store the user's email address or password. In this case, the email address may be included in the identification information.

[0023] Figure 5 is an explanatory diagram showing an example of a capture data table T2. The capture data table stores information about the year (shooting date or shooting year) and category of the film photograph related to the capture data output by the language model M, as well as information about comments entered for the film photograph. The management items (fields) of the capture data table T2 include the capture data ID field, registrant field, capture data field, shooting year information field, category field, comment ID field, and group ID field.

[0024] The Capture Data ID field in Capture Data Table T2 stores the ID (Capture Data ID) assigned to the film photograph's captured data. The Registrant field stores the User ID of the user who processed the registration of the captured data. The Capture Data field stores the film photograph's captured data, for example, in file format. The Year of Shooting Information field stores information (shooting date or year of shooting) about the year the film photograph related to the captured data, output by Language Model M (see Figure 7), was taken. The Category field stores the category of the film photograph related to the captured data, output by Language Model M (see Figure 7). The Comment ID field stores the ID (Comment ID) assigned to the comment entered for the film photograph related to the captured data. Note that the Comment ID field may store multiple Comment IDs. The Group ID field stores the Group ID of the group to which the registrant user belongs, i.e., the Group ID of the group to which users who can view the captured data belong. Note that Capture Data Table T2 may also store the year, month, and day the film photograph was taken.

[0025] Figure 6 is an explanatory diagram showing an example of comment table T3. Comment table T3 stores information about comments entered for film photographs related to captured data. The management items (fields) of comment table T3 include a comment ID field, a capture data ID field, an inputter field, an input date and time field, a type field, a comment content field, and an audio comment field.

[0026] The Comment ID field in Comment Table T3 stores the ID (Comment ID) assigned to comments entered via text or voice on the comment screen (see Figure 11). The Capture Data ID field stores the Capture Data ID of the film photograph to which the comment was entered. The Inputter field stores the User ID of the user who entered the comment (the user using terminal device 2 to which the comment was entered). The Input Date and Time field stores the date and time the comment was entered.

[0027] The Type field in comment table T3 stores the type of method used to input the comment (in this embodiment, text input or voice input). The Comment Content field stores the string of the comment if it was entered using text. The Voice Comment field stores the voice data of the comment entered by the user in file format. In this embodiment, if the comment was entered using voice, an empty value is stored in the Comment Content field, and if it was entered using text, an empty value is stored in the Voice Comment field. If the comment was entered using voice, the string obtained by processing the entered voice comment using speech recognition may be stored in the Comment Content field. Alternatively, if the comment was entered using text, the file containing the voice data obtained by processing the entered string using text-to-speech may be stored in the Voice Comment field.

[0028] Figures 7 and 8 are explanatory diagrams illustrating an example of a language model M. The language model M is composed of, for example, GPT (Generative Pretrained Transformer), PaLM (Scaling Language Modeling with Pathways), or Deep Seek. The language model M includes both an aspect as an LLM (Large Language Model) that produces output according to the instructions contained in the prompt based on the input prompt and data, and an aspect as a VLM (Vision Language Model) that produces output according to the instructions contained in the prompt based on the input prompt and image. Furthermore, the language model M may also be an MLLM (Multimodal Large Language Model).

[0029] The language model M receives capture data and prompts. The capture data input to the language model M includes the capture data selected by the user on the input screen (see Figure 9).

[0030] The prompts input to the language model M include commands, examples of dates, and examples of categories. The prompt commands (specific commands) shown in Figure 7 include commands that output information about the year the film photograph of the input capture data was taken (Please answer the year in the Western calendar when the input film photograph was taken. If the year is written on the photograph, please answer that year. If the year is not written on the photograph, please answer the estimated year in 10-year increments according to the <Examples of Dates>) and commands that output the category of the event or scene depicted in the film photograph (Please select the category of the input film photograph from the <Examples of Categories> and answer with exactly one word). The prompt commands also include commands that always output the year and category of the photograph, even if estimation is difficult. As a result of these commands, the language model M always outputs the year and category of the photograph, allowing the user to obtain information related to the film photograph, thus improving convenience.

[0031] The date examples input to the language model M include example outputs of information about the year the film photograph was taken (date of shooting) (e.g., "If it is estimated to have been taken in the 1980s: 1980s", "If it is estimated to have been taken in the 2010s: 2010s"). The language model M outputs the estimated year in which the film photograph was taken, according to the date examples.

[0032] The category examples input to the language model M include, for example, daily life, anniversaries, events, growth records, and travel. In other words, the categories output by the language model M indicate the type of event or scene depicted in the film photographs of the captured data. The category examples only need to include at least two of the following categories: daily life, anniversaries, events, growth records, or travel. Furthermore, the category examples are not limited to those listed above and may include examples of other categories such as school events, presentations, meals, or scenery. In addition, the category examples may include, for example, examples of categories entered by the user on terminal device 2.

[0033] As shown in Figure 7, when the captured data and prompts described above are input to the language model M, if the film photograph in the captured data does not have the year of shooting indicated, the language model M outputs the year of shooting (information about the year of shooting) and category of the film photograph in the captured data. In the example shown in Figure 7, the output is separated into the items of the shooting year or shooting year and the category, but the language model M may also output a string that concatenates the shooting year or shooting year and the category without including the names of the items.

[0034] As shown in Figure 8, when the language model M receives the captured data and prompts described above, if the film photograph in the captured data has the year of shooting printed on it, it outputs the year of shooting. In the example shown in Figure 8, the film photograph related to the captured data has a sequence of numbers (2003-08-15) indicating the date it was taken printed on it. The language model M extracts the year of shooting (2003) from this sequence of numbers and outputs the extracted year of shooting. The language model M may also output the date the film photograph was taken.

[0035] In this embodiment, the prompts input to the language model M include, but are not limited to, commands to output the year the film photograph was taken or the year the film photograph was taken as indicated on it. The prompts may also include commands to estimate and output the year the film photograph was taken even if the year is not indicated on the film photograph, and may further include commands to output the season in which the film photograph was taken. In this case, the language model M outputs the year or season in which the film photograph was taken. That is, the information regarding the year the film photograph was taken may include the year the film photograph was taken in one-year increments, or it may include the season in which the film photograph was taken. Furthermore, the prompts input to the language model M may also include commands to estimate and output the year and month, or the year, month, and day, in which the film photograph was taken. If the prompts include commands to estimate and output the year, month, and day, the language model M may estimate the year, month, and day based, for example, on objects depicted in the film photograph related to the captured data. For example, if a film photograph shows New Year's dishes, a New Year's pine decoration, or a kagami mochi (rice cake offering), the estimated date may be output as January 1st. If a film photograph shows Hina dolls or chirashi sushi, the estimated date may be output as March 3rd. If a film photograph shows a helmet or carp streamers, the estimated date may be output as May 5th. Furthermore, if the estimated category is an event, the language model M may identify the type of event and output the date associated with that type of event. For example, if the identified type of event is Halloween, the estimated date may be output as October 31st. If the identified type of event is Christmas, the estimated date may be output as December 25th. In addition, the prompts input to the language model M may include instructions to output only the identifiable numerical values ​​of the year, month, or day.

[0036] Figure 9 is an explanatory diagram showing an example of a capture data registration screen. The capture data registration screen displays the capture data of multiple film photographs captured by the capture device 3. When the terminal device 2 receives multiple capture data from, for example, the information processing device 1, it displays the capture data registration screen on its display unit. The terminal processing unit 21 of the terminal device 2 may read photographs that have been pre-stored in the storage unit 22 and display the read photographs on the capture data registration screen.

[0037] On the capture data registration screen, the user selects one capture data from several options and presses the output button. Terminal device 2 then sends the capture data ID of the selected capture data to information processing device 1. In Figure 9, the selected capture data is displayed with a check mark. Based on the received capture data ID, information processing device 1 provides the capture data to language model M, outputting information about the year the film photograph was taken (shooting year information) and its category. Information processing device 1 then sends the outputted shooting year information (shooting date or shooting year) and category to terminal device 2. Terminal device 2 displays the received shooting year information and category on the capture data registration screen. Note that terminal device 2 may accept input for the selection of multiple capture data on the capture data registration screen, and information processing device 1 may input the selected multiple capture data to language model M.

[0038] Furthermore, the capture data registration screen displays a field for correcting the shooting year information. The shooting year information correction field accepts user input for the shooting year or year of shooting, which will be registered in the capture data table T2 as the shooting year information for the film photograph if the user determines that the shooting year information of the film photograph output by the language model M is incorrect. If the user determines that the shooting year information of the film photograph output by the language model M is correct, no input is made in the shooting year information correction field. The terminal processing unit 21 may also accept input of the shooting year information of the film photograph from the user on the capture data registration screen.

[0039] A registration button is displayed on the capture data registration screen. When the registration button is pressed, the terminal processing unit 21 of terminal device 2 transmits a message to the information processing device 1 indicating that the registration button has been pressed. At this time, the processing unit 11 of information processing device 1 associates the capture data ID of the captured data (selected capture data) input to language model M, the user ID of the user (registrant), the year of shooting information, the category, and the user's (registrant's) group ID, and stores them in the capture data table T2. Note that multiple capture data tables T2 may be stored in the storage unit 12 of information processing device 1 for each group ID. That is, there may be a capture data table T2 (folder) for each group that stores capture data viewable by multiple users associated with the same group ID. In this case, a single user ID may be associated with multiple group IDs. Therefore, on the capture data registration screen, terminal device 2 may accept input of an identifier (identification number or folder name, etc.) of the capture data table T2 (folder) to which the capture data is to be registered. Furthermore, terminal device 2 may accept comments on the film photograph related to the captured data on the capture data registration screen.

[0040] Figure 10 is an explanatory diagram showing an example of a capture data list screen. Terminal device 2 transmits the user ID of the user using terminal device 2 to information processing device 1. Information processing device 1 reads the capture data from capture data table T2 using the group ID (group ID of the group to which the user belongs) corresponding to the received user ID as the key, and transmits it to terminal device 2. Terminal device 2 displays the received capture data on the capture data list screen.

[0041] The capture data list screen displays a chronological sort button and a category sort button. When the chronological sort button is pressed, terminal device 2 sends information that the chronological sort button has been pressed to information processing device 1. Information processing device 1 sorts the multiple capture data stored in the selected capture data table T2 in chronological order from oldest to newest, and sends (outputs) the sorted multiple capture data to terminal device 2.

[0042] When the category sort button is pressed, terminal device 2 transmits information that the category sort button has been pressed to information processing device 1. Information processing device 1 sorts the read capture data in category order and transmits (outputs) the sorted capture data to terminal device 2. Here, category order refers to the order of categories included in the example categories of the prompt (see Figure 7) (in this embodiment, the order is Daily Life, Anniversaries, Events, Growth Records, and Travel). Note that the category order is not limited to this, and may also be alphabetical by category, for example.

[0043] Furthermore, the capture data list screen displays a year selection field and a category selection field. Terminal device 2 accepts input in the year selection field for selecting the shooting year of the film photograph related to the capture data to be displayed on the capture data list screen. When terminal device 2 accepts input for the shooting year selection, it sends the selected shooting year to the information processing device 1. The information processing device 1 sends (outputs) to terminal device 2 the capture data from among the multiple capture data that is associated with the selected shooting year (the shooting year information field stores the shooting year of the selected shooting year or a shooting year included in the shooting year). Terminal device 2 displays the received capture data on the capture data list screen. Note that the year selection field may also accept input for selecting the shooting year, shooting year, or range of shooting years of the film photograph related to the capture data to be displayed on the capture data list screen.

[0044] Terminal device 2 accepts input in the category selection field for selecting the category of the film photograph related to the captured data to be displayed on the captured data list screen. When terminal device 2 accepts input for a category selection, it sends the selected shooting category to information processing device 1. Information processing device 1 sends (outputs) to terminal device 2 the captured data from among the multiple captured data that is associated with the selected category (the selected category is stored in the category field). Terminal device 2 displays the received captured data on the captured data list screen. Note that the category selection field may accept selections for a range of multiple categories.

[0045] In this embodiment, terminal device 2 transmits information that the year sort button or category sort button has been pressed, or information entered in the year selection field or category selection field, to information processing device 1, and information processing device 1 transmits (outputs) captured data to terminal device 2 based on the received information, but is not limited to this. For example, information processing device 1 may output all read captured data to terminal device 2 along with the year of shooting information and category associated with the captured data, and terminal processing unit 21 of terminal device 2 may sort the captured data or select the captured data to display based on information that the year sort button or category sort button has been pressed, or information entered in the year selection field or category selection field.

[0046] A comment button is displayed on the capture data list screen. When the user selects one of several capture data on the capture data list screen and presses the comment button, the terminal device 2 displays a comment screen (see Figure 11) on the display unit 25. In Figure 10, the selected capture data is displayed with a check mark.

[0047] Figure 11 is an explanatory diagram showing an example of a comment screen. When the comment button is pressed on the capture data list screen, terminal device 2 sends information to information processing device 1 requesting information about the selected capture data. Information processing device 1 reads the year of capture information, category, registrant's username, comment (comment content or audio comment), comment inputter's username, and comment input date and time from the capture data table T2 or comment table T3 and sends it to terminal device 2.

[0048] Terminal device 2 displays the capture data, year of shooting information, category, registrant's username, comment, the username of the person who entered the comment, and the date and time the comment was entered on the comment screen. If the comment was entered in text, the username of the person who entered it, the date and time the comment was entered, and a string indicating the content of the comment will be displayed. If the comment was entered by voice, the username of the person who entered it, the date and time the comment was entered, and a statement that the comment is a voice comment and a play button will be displayed. If the play button is pressed, the voice comment (file) will be played. In addition, on the comment screen, the elapsed time from when the comment was registered to the present may be displayed instead of, or in addition to, the date and time the comment was entered.

[0049] The comment screen displays a comment text input field and a comment voice input (recording) button. Terminal device 2 accepts comments entered in text in the comment input field. Comments entered in the comment text input field are sent to the information processing device 1 and stored in comment table T3 along with the capture data ID of the capture data to which the comment was entered and the user ID of the user who entered the comment. Comments entered as strings are stored in the comment content field of comment table T3.

[0050] When the recording button displayed on the comment screen is pressed, the voice input unit 26 of terminal device 2 is activated, and terminal device 2 accepts voice input of comments. For example, the user can input comments by voice while holding down the recording button. The file of the voice-input comment (voice comment) is sent to the information processing device 1 and stored in comment table T3 along with the capture data ID of the capture data to which the comment was entered and the user ID of the user who entered the comment. The voice-input voice comment file is stored in the voice comment field of comment table T3.

[0051] Figure 12 is an explanatory diagram showing an example of a user comment screen. The user comment screen displays a user ID input field. Terminal device 2 accepts the user ID of the user who entered the voice comment for the film photograph related to the captured data in the user ID input field. The user ID input field on the user comment screen may contain the user's own user ID, or it may contain the user ID of another user. The user ID entered in the user ID input field is transmitted to the information processing device 1. In addition, the username of the user who entered the voice comment for the film photograph may be entered on the user comment screen and transmitted to the information processing device 1.

[0052] Upon receiving a user ID from terminal device 2, information processing device 1 uses the received user ID as a key to read the capture data ID and voice comment from comment table T3 for the capture data in which the user with that user ID entered a voice comment. Information processing device 1 reads the capture data related to the capture data ID from capture data table T2 and sends it to terminal device 2 along with the voice comment. Terminal device 2 displays a play button for the received capture data and voice comment on the user comment screen. As shown in Figure 12, if the user with the entered user ID has entered voice comments for multiple film photographs, information processing device 1 reads the capture data for each of the multiple film photographs and sends it to terminal device 2, and terminal device 2 displays the capture data for each of the multiple film photographs.

[0053] Figure 13 is a flowchart illustrating an example of the capture data registration process. The processing unit 11 of the information processing device 1 acquires capture data of film photographs from the capture device 3 (S1). The processing unit 11 transmits the capture data of film photographs acquired from the capture device 3 to the terminal device 2 (S2).

[0054] The terminal processing unit 21 of terminal device 2 displays a capture data registration screen containing the capture data received from the information processing device 1 on the display unit 25 (S3). The terminal processing unit 21 accepts input for the selection of capture data (S4). The terminal processing unit 21 transmits the capture data ID of the selected capture data to the information processing device 1 (S5).

[0055] The processing unit 11 of the information processing device 1 inputs the captured data and prompt related to the received captured data ID to the language model M (S6), and outputs information regarding the year the film photograph was taken (year of shooting information) and the category related to the captured data (S7). The processing unit 11 of the information processing device 1 transmits the year of shooting information and category related to the captured data to the terminal device 2 (S8).

[0056] The terminal processing unit 21 of terminal device 2 displays the received year of shooting information and category on the display unit 25 (S9). The terminal processing unit 21 accepts input for correction of the year of shooting information (S10), transmits the corrected year of shooting information to the information processing device 1 (S11), and terminates the process. If the user determines that the year of shooting information of the film photograph output by language model M is correct, steps S10 and S11 are not executed.

[0057] The processing unit 11 of the information processing device 1 stores (registers) the captured data, as well as the year of shooting information and category of the film photograph related to the captured data, in the captured data table T2 (S12), and then terminates the process. In S14, if the processing unit 11 has received corrected year of shooting information from the terminal device 2, it stores the corrected year of shooting information in the year of shooting information field of the captured data table T2. If it has not received corrected year of shooting information from the terminal device 2, it stores the year of shooting information output by the language model M.

[0058] Figures 14 and 15 are flowcharts illustrating an example of the capture data display process. The terminal processing unit 21 of the terminal device 2 displays a capture data list screen, including a registration table selection field, on the display unit 25 (S21). The terminal processing unit 21 sends information requesting capture data to the information processing device 1 (S22).

[0059] The processing unit 11 of the information processing device 1 identifies the group ID of the group to which the user belongs by referring to the user table T1, based on the user ID of the user related to the terminal device 2 that sent the information requesting the capture data (S23). The processing unit 11 of the information processing device 1 reads multiple capture data associated with the identified group ID from the capture data table T2 (S24), and transmits the read multiple capture data to the terminal device 2 (S25).

[0060] The terminal processing unit 21 of the terminal device 2 displays the received capture data on the display unit 25 (S26). The terminal processing unit 21 accepts the press of the chronological sort button or the category sort button (S27). The terminal processing unit 21 transmits the information that the chronological sort button or the category sort button has been pressed to the information processing device 1 (S28).

[0061] When the processing unit 11 of the information processing device 1 receives information from the terminal device 2 that a year sort button or a category sort button has been pressed, it sorts the multiple capture data corresponding to the group ID according to the button pressed on the terminal device 2 (S29), and transmits (outputs) the sorted multiple capture data to the terminal device 2 (S30).

[0062] The terminal processing unit 21 of terminal device 2 displays sorted capture data received from information processing device 1 (S31). The terminal processing unit 21 also accepts input for the year of capture or category of the capture data to be displayed in the year selection field or category selection field (S32). The terminal processing unit 21 transmits the selected year of capture or category to information processing device 1 (S33).

[0063] The processing unit 11 of the information processing device 1 reads the capture data corresponding to the selected shooting year or category (capture data of the record containing the shooting year or category included in the selected shooting year) from the capture data table T2 (S34). The processing unit 11 transmits the read capture data to the terminal device 2 (S35).

[0064] The terminal processing unit 21 of terminal device 2 displays the captured data received from the information processing device 1 on the display unit 25 (S36). Note that the processing related to S27-S31, or one or both of the processing related to S32-S36, may not be executed. The terminal processing unit 21 accepts input for the selection of captured data (S37) and accepts the pressing of the comment button (S38). The terminal processing unit 21 sends information to the information processing device 1 requesting information about the selected captured data (S39).

[0065] The processing unit 11 of the information processing device 1 reads the year the film photograph was taken, category, registrant's username, comment (content of the comment or audio comment), comment inputter's username, and comment input date and time from the capture data table T2 or comment table T3 (S40). The processing unit 11 transmits the read year the film photograph was taken, category, registrant's username, comment (content of the comment or audio comment), comment inputter's username, and comment input date and time from the capture data to the terminal device 2 (S41).

[0066] The terminal processing unit 21 of terminal device 2 displays a comment screen on the display unit 25, which includes the capture data acquired from the information processing device 1, the year of shooting information, the category, the registrant's username, the comment, the inputter's username, and the input date and time (S42). The terminal processing unit 21 accepts comment input in text or voice (S43). The terminal processing unit 21 transmits the input comment (comment content or voice comment) to the information processing device 1 (S44).

[0067] The processing unit 11 of the information processing device 1 stores the comments received from the terminal device 2 in the comment table T3, associating them with the user ID of the user who entered the comment and the capture data ID of the capture data (S45).

[0068] The terminal processing unit 21 of terminal device 2 displays a user comment screen on the display unit 25 (S46). The terminal processing unit 21 accepts the input of a user ID (S47). The terminal processing unit 21 transmits the input user ID to the information processing device 1 (S48).

[0069] The processing unit 11 of the information processing device 1 uses the user ID received from the terminal processing unit 21 as a key to read the capture data ID and the voice comment of the capture data for which the user with that user ID entered a voice comment from the comment table T3 (S49). The processing unit 11 reads the capture data related to the capture data ID read in S49 from the capture data table T2 (S50), and transmits (outputs) the read capture data and voice comment to the terminal device 2 (S51) to terminate the process.

[0070] The terminal processing unit 21 of terminal device 2 displays a playback button for the captured data and audio comment received from the information processing device 1 (S52), and then terminates processing.

[0071] Figure 16 is a flowchart showing an example of the retraining process for the language model M. The processing unit 11 of the information processing device 1 starts the retraining process for the language model M when, for example, the terminal device 2 transmits corrected year-of-shoot information to the information processing device 1 in S11 shown in Figure 13. The processing unit 11 of the information processing device 1 generates training data that associates the capture data transmitted to the terminal device 2 in S8 with the received corrected year-of-shoot information (S61). The processing unit 11 inputs the generated training data into the language model M (S62), retrains the language model M (S63), and terminates the process. In S63, the processing unit 11 retrains the language model M by adjusting the parameters (bias and weights between layers) in each layer of the language model M so that it outputs corrected year-of-shoot information when capture data is input. This makes it possible for the language model M to output the year-of-shoot information with higher accuracy. Alternatively, the language model M may be retrained to output the corrected category when capture data is input, based on the training data that associates the capture data with the corrected category.

[0072] With the above configuration and processing, the processing unit 11 of the information processing device 1 can acquire information regarding the year the film photograph was taken and its category by inputting the captured data into the language model M. Furthermore, by storing comments input via voice for the film photograph related to the captured data, it is possible to share information about the film photograph among multiple users.

[0073] The embodiments disclosed herein should be considered in all respects as illustrative and not restrictive. The technical features described in each embodiment can be combined with each other, and the scope of the present invention is intended to include all modifications within the claims and scope equivalent to the claims. Furthermore, the independent and dependent claims described in the claims can be combined with each other in any combination, regardless of the form of reference. In addition, the claims use a multi-claim format in which claims refer to two or more other claims (multi-claim format), but are not limited to this. They may also be described using a multi-claim format in which at least one multi-claim refers to another multi-claim (multi-multi-claim format). [Explanation of Symbols]

[0074] 1: Information Processing Device 2: Terminal device 3: Capture device 4: Language Model Server 11: Processing Section 12: Storage section 12a:Storage medium 13: Communications Department 21: Terminal Processing Unit 22: Storage section 22a:Storage medium 23: Communications Department 24: Input section 25:Display section 26: Voice input section M: Language model N: Network P: Program Pa: App Program S: Information Processing System T1: User Table T2: Capture data table T3: Comment Table

Claims

1. By capturing film photographs, we obtain the captured data generated from the images. By providing the acquired capture data to the language model, information regarding the year the film photograph was taken and its category are output. A program that instructs a computer to perform a process.

2. By providing the language model with prompts that include information about the year the film photograph was taken and a command to specify its category, the category and year the film photograph was taken are output. The program according to claim 1.

3. The year the film photograph was taken is output by providing the language model with a prompt that includes a command to output information about the year of shooting based on the year of shooting printed on the film photograph. The program according to claim 2.

4. The prompt includes a command to output the year the film photograph was taken. The program according to claim 2 or 3.

5. The prompt includes examples of multiple categories, including at least two of the following: daily life, anniversaries, events, growth records, or travel. The program according to claim 2 or 3.

6. The language model receives input for corrections to the information regarding the year the film photograph was taken, and the language model is retrained using training data that associates the captured data with the corrected information regarding the year the photograph was taken. The program according to claim 1 or 2.

7. The multiple capture data are sorted based on the information regarding the year of capture and the category output by the language model. Output the sorted multiple capture data. The program according to claim 1 or 2.

8. The system stores the voice comments input via voice for the aforementioned film photograph, the identification information of the user who input the voice comments, and the captured data of the film photograph in association with each other. The program according to claim 1 or 2.

9. The system accepts user identification information input. The system outputs captured film photograph data that has been stored in association with the received identification information and the user's voice comment associated with that identification information. The program according to claim 8.

10. By capturing film photographs, we obtain the captured data generated from the images. By providing the acquired capture data to the language model, information regarding the year the film photograph was taken and its category are output. Information processing methods.

11. By capturing film photographs, we obtain the captured data generated from the images. By providing the acquired capture data to the language model, information regarding the year the film photograph was taken and its category are output. Processing section An information processing device equipped with the following features.

Images