Information processing system, information processing device, information processing method, and information processing program

Abstract

The information processing system comprises a starting node determination unit that determines a certain media information selected by the user as the starting node, an ending node determination unit that determines another media information selected by the user as the ending node, and a route GUI control unit that displays a route GUI on the display device of an electronic device, indicating that the starting node and the ending node are connected along a route.

Description

Technical Field

[0001] The present disclosure relates to an information processing system, an information processing apparatus, an information processing method, and an information processing program capable of searching for media information and generating new media information based on the search results.

Background Art

[0002] A method for searching for media information has been proposed that includes in search results multimedia with guaranteed relevance to the search target, even though the user did not expect it, thereby providing the user with the discovery of new connections and the expansion of exploration (paragraph 0009 of Patent Document 1).

[0003] According to Patent Document 1, as media information recommended to the user, in addition to media information directly related to the media information of the search target, media information with an indirect relationship is selected. For this reason, rather than presenting common related information, media information that is rather unexpected for the user is presented. For the user, there may be presented another topic that the user was not aware had a connection with what the user knew about a topic of interest. Thus, according to Patent Document 1, it is possible to stimulate the user's intellectual curiosity and provide the user with the discovery of new connections and the expansion of exploration. This creates an incentive for the user to use the service more (paragraph 0112 of Patent Document 1).

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] According to Patent Document 1, unexpected media information is presented to the user in relation to the media information being searched. While Patent Document 1 can provide users with the discovery of new connections, it is likely that users may often not understand the relationship between the media information being searched and the media information presented. Therefore, even if a user wants to search for the relationship between the media information being searched and the media information presented, it may be difficult for the user to determine what search query to enter appropriately.

[0006] In light of the circumstances described above, the purpose of this disclosure is to enable users to easily search for the relationships between multiple media sources and to present those relationships to the user. [Means for solving the problem]

[0007] An information processing system relating to one form of this disclosure is: A starting node determination unit that determines a certain media information selected by the user as the starting node, An endpoint node determination unit that determines another media information selected by the user as the endpoint node, A route GUI control unit that displays a route GUI on the display device of an electronic device, indicating that the starting node and the ending node are connected along a route. It is equipped with. [Effects of the Invention]

[0008] According to this disclosure, users can easily search for the relationships between multiple media pieces of information, and the relationships between multiple media pieces of information can be presented to the user.

[0009] The effects described herein are not necessarily limited to those described herein and may include any of the effects described herein. [Brief explanation of the drawing]

[0010] [Figure 1]An information processing system according to an embodiment of the present disclosure is shown. [Figure 2] This is an example of the top screen of the GUI of an electronic device. [Figure 3] This is an example of the screen after the transition of the GUI of an electronic device (operation toward a functional object). [Figure 4] The functional configuration on the foreground side of the information system is shown. [Figure 5] The operation flow on the foreground side of the information processing system is shown. [Figure 6] The root GUI is schematically shown. [Figure 7] The root GUI is schematically shown. [Figure 8] The functional configuration on the background side of the information system is shown. [Figure 9] The operation flow on the background side of the information processing system is shown. [Figure 10] The media information database is shown. [Figure 11] The configuration of the media information analysis unit is shown. [Figure 12] The concept database is shown. [Figure 13] The concept relationship database is shown. [Figure 14] The analysis information database is shown. [Figure 15] The configuration of the concept relationship calculation unit is shown. [Figure 16] The configuration of the account creation unit is shown. [Figure 17] The account database is shown. [Figure 18] The configuration of the account usage history analysis unit is shown. [Figure 19] The configuration of the startup recommendation unit is shown. [Figure 20] A modification example of the configuration of the startup recommendation unit is shown. [Figure 21] The configuration of the search-time recommendation unit is shown. [Figure 22] A modification example of the configuration of the search-time recommendation unit is shown. [Figure 23] An example of new media information generated by prompts is shown. [Modes for carrying out the invention]

[0011] Embodiments of this disclosure will be described below with reference to the drawings.

[0012] 1. Information Processing System

[0013] Figure 1 shows an information processing system according to one embodiment of the present disclosure.

[0014] Information processing system 1 comprises an information processing device 10, a storage device 20, and a web server 30, all of which are connected via a network or the like for communication. The information processing device 10 functions as a backend computing server and performs arithmetic processing. The storage device 20 is a large-capacity non-volatile storage device that functions as a server-side database server. The information processing device 10 and the storage device 20 may be connected independently via a network or the like (as shown in the diagram), or they may be constructed as an integrated information processing device (not shown). The web server 30 is a frontend web server and provides data generated by the information processing device 10 through arithmetic processing and stored in the storage device 20 in response to requests from client electronic devices 40. The information processing device 10 is configured to select multiple media information for the electronic devices 40 and supply display data for displaying the selected multiple media information to the electronic devices 40 via the web server 30.

[0015] The electronic device 40 is a device having a display device 41, such as a smartphone, tablet computer, personal computer, head-mounted display, or VR glasses, and is a terminal device used by an end user. The control circuit 42 of the electronic device 40 works in cooperation with the web server 30 to execute an application program, thereby changing the GUI displayed on the display device 41. The application program may be executed via a web browser or installed and executed on the electronic device 40. By executing the application program, the electronic device 40 recommends and displays multiple media information on the display device 41 to the user at startup, and further recommends and displays multiple different media information related to the media information selected by the user on the display device 41 from among the multiple media information during searches. This aims to provide the user with a user experience such as discovering new connections and expanding exploration. The display device 41 may be a touch panel, a display without a touch panel, a head-mounted display, a display unit built into VR glasses, etc.

[0016] In this embodiment, the media information is single-modality or multi-modality media information consisting of at least one of several different modalities (formats), such as still images, videos, music, text (summaries, articles, narratives), map information, list information, location information, and tag information. The type of modality is not limited to this example and may be any format. The media information is linked to web pages such as videos, magazines, auctions, e-commerce sites, art exhibitions, encyclopedias, social networking services, and advertisements on the network. The media information is not limited to data on the network and may also be data stored in the storage device 20.

[0017] 2. GUI of electronic devices

[0018] The following describes the GUI and the changes in the GUI that are displayed on the display device 41 by the control circuit 42 of the electronic device 40 when it executes an application program.

[0019] Figure 2 shows an example of the top screen of a GUI for an electronic device.

[0020] In this embodiment, the electronic device 40 is a smartphone, and the display device 41 is a vertically elongated rectangle. In this embodiment, the terms rectangle, square, quadrilateral, polygon, etc., also include approximate rectangles, approximate squares, approximate quadrilaterals, approximate polygons, etc., which have rounded corners (R) but visually give the impression of the shape.

[0021] The control circuit 42 of the electronic device 40 displays a top screen 400 on the display device 41, which includes multiple cards 410 and functional objects 430.

[0022] Multiple (four in this example) roughly rectangular cards 410 are displayed around the functional object 430. Each card 410 displays different media information. Each card 410 includes, for example, an image 411 and non-image information 412. For example, card 410 includes an image 411 and text information 412. The non-image information 412 (for example, text information) is information associated with the image 411, such as the name, description, and link information of the image 411.

[0023] Each of the roughly square cards 410 displays media information different from that of the roughly rectangular cards 410. For example, each card 410 contains an image 421.

[0024] Functional object 430 functions as a search window. Furthermore, users can input arbitrary media information into the plus object 431 of functional object 430.

[0025] Next, we will explain the evolution of GUIs based on user interaction.

[0026] Figure 3 shows an example of a screen after the evolution of GUIs in electronic devices (operation on a functional object).

[0027] As shown in (A), the control circuit 42 of the electronic device 40 detects that the user has performed an operation on the display device 41 that directs one card 410 toward a functional object 430 (an operation by the user to display new information). Then, as shown in (B), the control circuit 42 of the electronic device 40 switches the display to show multiple new cards 410, each showing different media information related to the media information shown by the one card 410 that has been directed toward. Note that the "operation of directing a card 410 toward a functional object 430" can be a drag operation with an operator (fingers, stylus pen, mouse, etc.), an operation by a handheld controller if the electronic device 40 is a head-mounted display or VR glasses, an operation by a part of the human body, or an operation by moving the gaze.

[0028] The arrangement of the multiple new cards 410 remains the same as on the top screen 400. The method for selecting the new cards 410 will be explained in detail later.

[0029] As a result, according to this embodiment, a search can be performed by moving the surrounding cards 410 toward the central functional object 430. Since the search is performed simply by moving the cards 410 toward the functional object 430 (e.g., drag and drop), no further operation steps are required after the movement of the cards 410 (e.g., drag and drop), resulting in highly intuitive operation for the user. Furthermore, since the target functional object 430 is located in the center and equidistant from all the cards 410, there is no variation in the distance traveled depending on which card 410 is to be moved, eliminating the need for long-distance movement and resulting in highly intuitive operation.

[0030] As an example of a variation, the card 410 and functional object 430 may be ellipses, circles, polygons (including those with rounded corners), etc. The card 410 and functional object 430 may consist of one or more colors. The color of the card 410 may be changed according to the information displayed by the shape (displayed information). For example, the color may be changed according to the category of media information. The boundaries of the card 410 and functional object 430 do not need to be explicit. When the card 410 is held for movement, the user may be notified that the card 410 is being held by playing a sound effect, changing the shape of the shape, changing the color or an animation such as the shape shaking, or by vibration of the electronic device 20.

[0031] 3. Functional configuration of the background side of the information processing system

[0032] Figure 8 shows the functional configuration of the background side of the information system.

[0033] The information processing device 10 operates as a background data collection unit 110, a media information analysis unit 120, a conceptual relationship calculation unit 130, an account creation unit 140, an account usage history analysis unit 150, a startup recommendation unit 160, and a data generation unit 180, by having the processor of the control circuit 100 load various information processing programs recorded in the auxiliary storage device into the main storage device and execute them.

[0034] The storage device 20 stores a media information database 210, a concept database 220, a concept relationship database 230, an analysis information database 240, and an account database 250.

[0035] 4. Background operation of the information processing system

[0036] Figure 9 shows the background operation flow of the information processing system.

[0037] The collection unit 110 collects media information (step S1). The media information is single-modality or multi-modality media information consisting of at least one of several different modalities (styles), such as still images, videos, music, text (summaries, articles, narratives), map information, list information, location information, and tag information. The media information is linked to any data that exists in the real world and virtual world. For example, this includes data stored on the web (also called the internet or network), data stored on personal devices, and data measured in the real world or virtual world using predetermined sensors, etc. The collection unit 110 stores the collected media information in the media information database 210.

[0038] Figure 10 shows the media information database.

[0039] Multiple media data entries 210A are registered in the media information database 210. Each media data entry 210A is registered with its media ID 211, title 212, text 213, image identifier 214, sender account ID 215, and update date 216 associated with each other.

[0040] Media ID 211 may be a serial number that uniquely identifies a single piece of media information within the media information database 210. Title 212 may be defined by metadata included in the collected media information. Text 213 is included in the collected media information. Image identifier 214 may be a URL (link destination address) uniquely assigned to the media information, or it may be actual data. Each piece of media data 210A may include either text 213 or image identifier 214, or both. If the media information is music, it may not include both text 213 and image identifier 214, but may include a music identifier (not shown) indicating a URL or actual data.

[0041] Sender account ID 215 is the account ID of the end user who sent the media information. In addition to sender account ID 215, sender classifications may be further registered, such as being a member of this service or having undergone KYC (Know Your Customer) procedures. Update date 216 is the update date of the media information. Update date 216 may be classified based on whether the update date occurred within a specified time period from the collection date.

[0042] The types and number of media data 210A may be classified according to the type and number of media by which the media data 210A is composed and provided. If each media data 210A has a separate address assigned to it (for example, SNS), this address information may also be collected.

[0043] Figure 11 shows the configuration of the media information analysis unit.

[0044] The media information analysis unit 120 calculates media features, which are the characteristic quantities of each piece of media information, for multiple pieces of media information present on the network, using a model learned by machine learning, for example (step S2). Specifically, the media information includes multiple sub-media information M1 and M2 (for example, text M1 and image M2). The media information analysis unit 120 calculates media features 245 and 246 for each of the multiple sub-media information M1 and M2 (step S21). That is, the media information has multiple media features 245 and 246 for each of the multiple sub-media information M1 and M2.

[0045] The media information analysis unit 120 extracts the concepts of each media information (step S3). Specifically, the media information analysis unit 120 refers to the concept database 220, extracts the concepts that each media information possesses, and decomposes the extracted concepts into dominant concepts 243 and sub-concepts 244 (step S22). For example, the media information analysis unit 120 may classify the concepts possessed by objects with a high occurrence rate within a single media information as dominant concepts 243, and the concepts possessed by objects with a relatively low occurrence rate as sub-concepts 244. That is, the concepts of media information include dominant concepts 243 and sub-concepts 244. The set of dominant concepts 243 and sub-concepts 244 possessed by a certain media information constitutes the concept of that media information. For example, if the dominant concepts 243 of multiple media information are different, but the sub-concepts 244 of these media information are common, then the different dominant concepts 243 of these media information are connected.

[0046] Figure 12 shows the conceptual database.

[0047] Multiple conceptual data 220A are registered in the conceptual database 220. Each conceptual data 220A is registered with a serial ID 221, a conceptual ID 222, and a conceptual name 223 associated with each other. There is no distinction between dominant concepts 243 and sub-concepts 244 in the conceptual data 220A registered in the conceptual database 220. For example, conceptual data 220A does not have a flag or anything to distinguish between dominant concepts 243 and sub-concepts 244.

[0048] Figure 13 shows a conceptual relationship database.

[0049] The conceptual relationship database 230 registers multiple conceptual relationship data 230A. A "concept" is a concept expressed within media information in text or images. Each conceptual relationship data 230A is registered with a serial ID 231, conceptual relationship ID 232, concept from 233, concept to 234, relationship strength 235, fixed tag 236, and free tag 237 associated with each other. Relationship strength 235 represents the strength of a directional connection (strength of the bond).

[0050] For example, the strength of the relationship 235 from concept ID "0" (today) to concept ID "1" (event) in Figure 12 is "0.61". Also, just because a strength of relationship 235 from concept "today" to concept "event" is registered does not necessarily mean that a strength of relationship 235 from concept "event" to concept "today" is not registered. Furthermore, the strength of the relationship 235 from concept "today" to concept "event" and the strength of the relationship 235 from concept "event" to concept "today" in the reverse direction do not necessarily coincide.

[0051] Fixed tag 236 and free tag 237 provide information that supports the numerical value of 235, which indicates the strength of the conceptual relationship. Fixed tag 236 is mandatory and is the definition (higher-level concept) of concepts from 233 or concepts to 234, such as "place / object," "facility / canvas," or "event." Free tag 237 is optional and is the specific content (lower-level concept) of fixed tag 236, such as "place / object," "facility / canvas," or "event." For example, for the combination of concept from 233 "Tokyo Dome" and concept to 234 "Yomiuri Giants," fixed tags 236 "place / object," "facility / canvas," and free tag 237 "home ground" are registered. The numerical value of 235, which indicates the strength of the conceptual relationship between concept from 233 "Tokyo Dome" and concept to 234 "Yomiuri Giants," is supported by fixed tags 236 "place / object," "facility / canvas," and free tag 237 "home ground."

[0052] The media information analysis unit 120 refers to the concept database 220 and the concept relationship database 230 and determines that concepts having a predetermined strength of conceptual relationship 235 with respect to the extracted dominant concept 243 are additional sub-concepts 244 that the media information possesses (step S23). For example, when the extracted dominant concept 243 is defined as concept from 233, the media information analysis unit 120 determines that concepts to 234 whose strength of relationship 235 from concept from 233 is greater than or equal to a predetermined value are additional sub-concepts 244. The media information analysis unit 120 registers the media features 245, 246, dominant concept 243, and sub-concepts 244 in the analysis information database 240.

[0053] Figure 14 shows the analysis information database.

[0054] Multiple analysis information data sets 240A are registered in the analysis information database 240. Each analysis information data set 240A is registered with the serial ID 241, media ID 242, dominant concept set 243, sub-concept set 244, text features 245, image features 246, and features 247 of other media (e.g., music) associated with each other. Media ID 242 is the same as media ID 211 in media information database 210. Dominant concept set 243 includes concept IDs 222 (Figure 12) of all dominant concepts extracted for the media information of media ID 242. Sub-concept set 244 includes concept IDs 222 (Figure 12) of all sub-concepts extracted for the media information of media ID 242. Text features 245, image features 246, and other media features 247 include features of all sub-media included in the media information. The text features (245), image features (246), and other media features (e.g., music) (247) are collectively referred to as "media features." While vectors are one form of feature format, any format is acceptable.

[0055] Figure 15 shows the configuration of the conceptual relationship calculation unit.

[0056] The conceptual database 220 and the conceptual relationship database 230 are created in advance by the conceptual relationship calculation unit 130 and updated as needed. That is, the conceptual relationship calculation unit 130 calculates and updates the conceptual relationships of concepts of multiple media information present on the network.

[0057] Specifically, the conceptual relationship calculation unit 130 reads the media information database 210 and the analysis information database 240. From the media information database 210, the conceptual relationship calculation unit 130 extracts the sender account ID 215 and update date 216 of media data 210A identified by a certain media ID 212. From the analysis information database 240, the conceptual relationship calculation unit 130 extracts the dominant concept set 243, sub-concept set 244, text features 245, image features 246, and other media features 247 of analysis information data 240A identified by a certain media ID 242. The conceptual relationship calculation unit 130 calculates the strength of the conceptual relationship 235 for each dominant concept set 243 and sub-concept set 244, and generates fixed tags 236 and free tags 237 that support the strength of the conceptual relationship 235. The conceptual relationship calculation unit 130 may influence the calculation of the strength of the conceptual relationship 235 and the generation of fixed tags 236 and free tags 237 by using the sender account ID 215 and update date 216, as well as the text features 245, image features 246, and other media features 247. Based on the calculation results, the conceptual relationship calculation unit 130 updates the conceptual database 220 and the conceptual relationship database 230 as needed.

[0058] Figure 16 shows the configuration of the account creation section.

[0059] Meanwhile, the account creation unit 140 registers user accounts for users who log in to services provided by the information processing system 1 using the electronic device 40 (step S4). The user enters user information using the electronic device 40 (step S41).

[0060] By logging in, users are assigned a unique account and can use unique services provided by the information processing system 1. Unique services include history management, media information tagging, viewing tagged media information, writing comments, provision of new media information tailored to preferences, and advertisement delivery. When registering for the first time, users may be asked to provide their name (username), email address, mobile phone number, SNS account, personal identification, payment information, etc., as needed. When registering for the first time, users may specify one or more keywords to be used as criteria for determining their preferences. When defining keywords, preferences may be categorized by simple questions. Preference information may be calculated using a model learned by machine learning from the network usage history, such as the media information the user has viewed, other users the user follows, and other users the user follows. When a user logs in, two-factor authentication may be required based on the registered information. The device information used by the user to log in is registered with the server operating the service, and if a user logs in with a device that has no prior history, two-factor authentication may be required. Account information is stored in the account database 250.

[0061] Figure 17 shows the account database.

[0062] The account database 250 contains multiple account data entries 250A. Each account data entry 250A is registered with its account ID 251, username 252, preference features 253, browsing history media ID list 254, following account list 255, and follower account list 256 all associated with each other.

[0063] The account creation unit 140 calculates preference features 253 for users who log in to the service using an electronic device 40, based on the browsing history media ID list 254, following account list 255, follower account list 256, preference information at the time of initial registration, etc., registered in the account database 250, using a model trained by machine learning, and registers them in the account database 250 (step S42).

[0064] Figure 18 shows the configuration of the account usage history analysis unit.

[0065] The account usage history analysis unit 150 recalculates preference features based on the user's usage history when a user logs into their account and uses the services provided by the information processing system 1 (step S5). Specifically, the account usage history analysis unit 150 reads the account ID 251, preference features 253, and browsing history media ID list 254 from the account database 250, and recalculates the preference features 253 based on the browsing history media ID list 254, which is the usage history (step S51). The account usage history analysis unit 150 may further recalculate the preference features 253 based on the following account list 255 and the follower account list 256. The account usage history analysis unit 150 registers (updates) the recalculated preference features 253 in the account database 250. In this way, the user's preferences are analyzed based on the keywords selected at the time of initial registration and the past usage history of the services.

[0066] Figure 19 shows the configuration of the startup recommendation section.

[0067] The startup recommendation unit 160 selects multiple media information items, each having media features that satisfy predetermined conditions for the preference features 253, as multiple media information items to be initially displayed as multiple cards 410 (step S6). Specifically, when a user starts the application program using the electronic device 40 (step S61), the startup recommendation unit 160 reads the account database 250. The startup recommendation unit 160 reads the preference features 253 associated with the logged-in user's account ID 251 from the account database 250. The startup recommendation unit 160 extracts multiple analysis information data 240A from the analysis information database 240, each having media features 245, 246, and 247 that satisfy predetermined conditions for the read preference features 253. This enables searching for unconscious search needs and provides a method for recommending candidate search needs to the user (displaying an initial screen based on preferences) and a display method (UI).

[0068] At startup, the recommendation unit 160 extracts multiple media data 210A with the same media ID 211 as the extracted analysis information data 240A from the media information database 210. The data generation unit 180 generates multiple cards 410 based on the title 212, text 213, and image identifier 214 of the extracted media data 210A (step S10). The multiple cards 410 are display data for displaying the selected multiple media information, supplied to the electronic device 40 via the web server 30. The data generation unit 180 causes the display device 41 of the electronic device 40 to display a top screen 400 containing the multiple cards 410 based on the multiple media data 210A via the web server 30 (step S62). That is, the web server 30 supplies the electronic device 40 with display data for displaying the multiple media information selected by the information processing device 10.

[0069] Figure 20 shows a modified configuration of the startup recommendation section.

[0070] Furthermore, the startup recommendation unit 160 may not only extract analysis information data 240A from the analysis information database 240, but may also generate media information data that matches the user's preference features 253 (step S63). In this case, the data generation unit 180 only needs to generate a card 410 for displaying the generated media information data.

[0071] The user views the top screen 400 (step S7). The user selects the functional object 430 by manipulating card 410 to move it over the functional object 430 (for example, by dragging and dropping) and performs a search (step S8). When transitioning from the top screen 400, the functional object 430 does not change from its specified position, but the information in card 410 is changed as follows (step S9). Steps S8 and S9 will be explained in detail below in the functional configuration and operation of the foreground side.

[0072] 5. Functional configuration of the foreground side of the information processing system

[0073] Figure 4 shows the functional configuration of the foreground side of the information system.

[0074] The information processing device 10 operates as a foreground card GUI control unit 101, a starting node determination unit 102, a search recommendation unit 103, a destination node determination unit 109, a route search unit 104, a route GUI control unit 105, a prompt generation unit 106, a new media information generation unit 107, and a new media information output unit 108, by having the processor of the control circuit 100 load various information processing programs recorded in the auxiliary storage device into the main memory and execute them.

[0075] 6. Foreground operation of the information processing system

[0076] Figure 5 shows the foreground operation flow of the information processing system.

[0077] The card GUI control unit 101 displays multiple cards 410, each representing a different media information GUI, on the display device 41 of the electronic device 40 so that the user can select one (step S101). The multiple different media information may be media information selected by the startup recommendation unit 160 (step S6). The multiple cards 410 displayed may be cards generated by the data generation unit 180 (step S10).

[0078] The card GUI control unit 101 displays a functional object 430 and a plurality of cards 410 around the functional object 430 on the display device 41, and determines which card 410 is selected by the user from the plurality of cards 410 (the card 410 selected by the user is referred to as the first card 410) (step S102). Specifically, the card GUI control unit 101 determines that a predetermined operation has been performed on the first card 410, which is one of the plurality of cards 410 displayed on the display device 41. More specifically, the card GUI control unit 101 determines that an operation (for example, a drag-and-drop operation) has been performed on the first card 410 so that it moves onto the functional object 430 (step S8).

[0079] The starting node determination unit 102 determines the media information represented by the first card 410 on which this operation was performed as the starting node 501 (step S103). The starting node 501 is the node that serves as the starting point for the search prompt.

[0080] Furthermore, the starting node determination unit 102 can determine the media information input by the user as the starting node, instead of the media information represented by the first card 410 selected by the user. Specifically, if the user inputs arbitrary media information into the plus object 431 of the functional object 430, the starting node determination unit 102 can determine that arbitrary media information as the starting node.

[0081] During a search, the recommendation unit 103 determines multiple media information items that have a predetermined relationship to the starting node 501, which is media information represented by the first card 410, as multiple next-hop nodes 502. The recommendation unit 103 determines multiple media information items of the final generation node 502C reached by repeating the process of determining multiple media information items that have a predetermined relationship to the determined multiple next-hop nodes 502 as the next generation of multiple next-hop nodes 502 a predetermined number of times (step S104).

[0082] During the search, the recommendation unit 103 determines multiple media information items that have a predetermined relationship to the starting node 501 as multiple next-hop nodes 502A (first generation). During the search, the recommendation unit 103 determines multiple media information items that have a predetermined relationship to each of the multiple next-hop nodes 502A (first generation) as multiple next-generation next-hop nodes 502B (second generation). After repeating this a predetermined number of times (twice in this example), the recommendation unit 103 determines multiple media information items that have a predetermined relationship to each of the multiple next-hop nodes 502B (second generation) as multiple final-generation nodes 502C.

[0083] "Predetermined relationship" refers to the conceptual relationship analyzed by the media information analysis unit 120 and the conceptual relationship calculation unit 130. As described above, the media information analysis unit 120 extracts multiple concepts from multiple media information, classifies the extracted concepts into dominant concepts 243 and sub-concepts 244, and additionally determines that concepts having a predetermined conceptual relationship with the dominant concepts 243 of each media information are sub-concepts 244 of the media information. The set containing all dominant concepts 243 and all sub-concepts 244 of the media information is defined as the concept set of the media information. The conceptual relationship calculation unit 130 calculates the conceptual relationship between individual concepts included in a concept set of one media information and individual concepts included in a concept set of another media information.

[0084] Specifically, the search recommendation unit 103 determines multiple media information items, each containing a predetermined conceptual relationship with an individual concept included in the conceptual set of media information representing the selected first card 410, as multiple next-hop nodes 502. The search recommendation unit 103 determines multiple media information items, each containing a predetermined conceptual relationship with an individual concept included in the conceptual set of media information of the multiple next-hop nodes 502 determined by the multiple next-hop nodes 502, by repeating this process a predetermined number of times until it reaches the final generation node 502C (step S104). The operation of the search recommendation unit 103 will be explained in more detail later.

[0085] The card GUI control unit 101 switches the display on the display device 41 to show multiple new cards 410, each representing multiple media information for the final generation node 502C, instead of the multiple cards 410 currently displayed on the display device 410, with the exception of the first card 410 (step S105). The card GUI control unit 101 displays the multiple new cards 410 around the functional object 430. Among the multiple cards 410 displayed on the display device 41, there are no cards 410 that represent media information for one or more next-hop nodes 502. In other words, there are only multiple new cards 410, each representing multiple media information for the final generation node 502C.

[0086] The card GUI control unit 101 determines that a predetermined operation has been performed on one of the multiple cards 410 (cards 410 showing media information of the final generation node 502C) displayed on the display device 41 (the card 410 selected by the user is referred to as the second card 410) (step S106). The predetermined operation may be any method other than a move operation, which is an operation to select the first card 410 (for example, a long press).

[0087] The endpoint node determination unit 109 determines the media information represented by the second card 410 as the endpoint node 503 (step S107). The endpoint node 503 is one of the final generation nodes 502C.

[0088] The route search unit 104 determines that one or more media information points that exist sequentially along one or more routes from the media information of the starting node 501 to the media information of the ending node 503 are one or more next-hop nodes 502 (step S108). In the example shown in the figure above, the next-hop nodes 502A and 502B that exist sequentially along one or more routes from the starting node 501 to the ending node 503 (one of the final generation nodes 502C) are determined.

[0089] Figures 6 and 7 schematically show the root GUI.

[0090] The route GUI control unit 105 displays a route GUI 504 on the display device 41 indicating that the route from the starting node 501 through one or more next-hop nodes 502A, 502B to the ending node 503 (one of the final generation nodes 502C) is connected by one or more routes (step S109) (left diagram in Figure 7).

[0091] The user wants to know the relationship between the starting node 501 and the ending node 503, and the relationship between the starting node 501, the next hop nodes 502A and 502B, and the ending node 503. In that case, the user performs a predetermined action on the root GUI 504. The root GUI control unit 105 detects the predetermined action (step S110). The predetermined action may be, for example, operating (tapping, etc.) any location on the root GUI 504, or displaying a search button 505 (Figure 6(A)) or a menu button 506 (Figure 6(B)) on the root GUI 504 and operating (tapping, etc.) these buttons 505 and 506.

[0092] When the menu button 506 is operated (tapped, etc.), the modality list 520 is displayed (Figure 6(C)). The modality list 520 displays multiple modalities 521-529, specifically, outline 521, article 522, narrative (a story starting from the first-person perspective node 501) 523, still image 524, music 525, video 526, map (relevant locations traced on a map) 527, list 528, and random (any one or more of modalities 521-528) 529. The user may also select one or more modalities 521-529 from the modality list 520.

[0093] List 528 can be a list displaying suggested image and title combinations. Each suggested combination can be manipulated (e.g., by clicking) to redirect to an external website, open a corresponding card, or create new media information. Examples of list display formats include image-title-subtitle, title-music, summary (text), video, and a summary of content links.

[0094] The prompt generation unit 106 generates a prompt for generating new media information based on the relationship between the starting node 501 and the ending node 503 when the user performs the predetermined operation described above for the root GUI 504 (step S111). The prompt may be a prompt for generating new media information based on the relationship between each of the one or more next-hop nodes 502 and the starting node 501 and / or the ending node 503. And / or, the prompt may be a prompt for generating new media information based on the relationship between each of the two adjacent nodes from the starting node 501 through one or more next-hop nodes 502 to the ending node 503. The prompt may be a prompt for generating new media information based on the media information of the starting node 501 and / or the ending node 503. For example, if the media information of the starting node 501 or ending node 503 relates to manga, the prompt may be for generating new media information (e.g., a narrative, i.e., a story with the starting node 501 in the first person) in a style similar to the speech pattern of the main character of that manga (e.g., a casual speech pattern). The prompt may specify the modality (single modality or multimodality) of the new media information. The modality (single modality or multimodality) of the media information represented by multiple cards 410 and the modality (single modality or multimodality) of the new media information may all be the same, all different, or partially the same and partially different. The modality of the new media information may be one or more modalities 521-529 selected by the user from the modality list 520, or one or more random modalities if no selection has been made. The prompt may include fixed tags 236 and free tags 237 that support the strength 235 of the conceptual relationship between each of the two adjacent nodes from the starting node 501 through one or more next-hop nodes 502 to the ending node 503.

[0095] Furthermore, the user wants to know the relationship between any two of the following nodes: the starting node 501, the next-hop nodes 502A and 502B, and the ending node 503. In this case, the user performs an operation (such as tapping) to select any two nodes (for example, the starting node 501 and the next-hop node 502B, or the next-hop node 502A and the ending node 503). The root GUI control unit 105 detects this operation (step S110). In this case, the prompt generation unit 106 generates a prompt to generate new media information based on the relationship between the new starting node and the new ending node, designating the node on the starting node 501 side of the two operated nodes as the new starting node and the node on the ending node 503 side as the new ending node (step S111).

[0096] The new media information generation unit 107 generates new media information 510 by executing prompts, for example, using a large-scale language model (LLM) (step S112). The new media information output unit 108 outputs the new media information 510 to the electronic device 40 to present it to the user (step S113) (right diagram in Figure 7). The card GUI control unit 101 can display the card 410 representing the new media information generated based on the prompts on the display device 41 of the electronic device 40 so that the user can select it as the first card 410 or the second card 410 on subsequent occasions.

[0097] 7. Recommendation section during search

[0098] Figure 21 shows the configuration of the search recommendation section.

[0099] When an operation is performed to move one card 410 (starting node) toward a functional object 430, the search recommendation unit 103 selects multiple media information (ending nodes) each having a concept that has a predetermined conceptual relationship with the concept of media information represented by one card 410. Specifically, the user operates the card 410 so that it moves toward the functional object 430 (step S81).

[0100] During a search, the recommendation unit 103 extracts the media ID 211 of the media data 210A indicated by the card 410 selected by the user from the media information database 210. During a search, the recommendation unit 103 extracts the dominant concept 243 and sub-concepts 244, and media features 245, 246, and 247 of the same media ID 242 as the extracted media ID 211 from the analysis information database 240. During a search, the recommendation unit 103 extracts the concept ID 222 of the extracted dominant concept 243 and sub-concepts 244 from the concept database 220.

[0101] When the search recommendation unit 103 identifies the dominant concept 243 and sub-concept 244 specified by the extracted concept ID 222 as concept from 233, it extracts concepts to 234 from the concept relationship database 230 whose relationship strength 235 from concept from 233 satisfies a predetermined condition (e.g., above a threshold). When the search recommendation unit 103 identifies the media ID 242 associated with the extracted concept ID to 234 (i.e., the media ID 242 of media information that has concept to 234 as the dominant concept 243 or sub-concept 244) from the analysis information database 240. When the search recommendation unit 103 identifies the analysis information data 240A of the extracted media ID 242 from the analysis information database 240 (step S91). As a result, the search recommendation unit 103 can select media (media identified by media ID 242) (next-hop node) that have a predetermined conceptual relationship with the media data 210A (starting node) indicated by the card 410 selected by the user, and further select media (final generation node) that have a predetermined conceptual relationship with the media information of the next-hop node.

[0102] As a process to broaden the search scope for selected media information, information (sub-concepts) that the media information does not directly possess can be added using concepts from 233 and to 234. This process can be performed not just once, but recursively any number of times. For example, a second concept (sub-concept) can be derived from a first concept (dominant concept), and further, a third concept (sub-concept) can be derived from the second concept (sub-concept). That is, the search recommendation unit 103 may add at least one dominant concept 243 and / or sub-concept 244 that has a predetermined conceptual relationship with at least some of the dominant concepts 243 and / or sub-concepts 244 of the media information concepts selected by the user to the concept of the media information. This allows the search recommendation unit 103 to further select media that have concepts that do not have a direct conceptual relationship with the concepts of the media information selected by the user.

[0103] The search recommendation unit 103 extracts preference features 253 associated with the logged-in user's account ID 251 from the account database 250. The search recommendation unit 103 filters the analysis information data 240A based on the extracted preference features 253 (step S92).

[0104] During a search, the recommendation unit 103 extracts multiple media data 210A with the same media ID 211 as the media ID 242 of the filtered multiple analysis information data 240A from the media information database 210. The data generation unit 180 generates multiple cards 410 based on the title 212, text 213, and image identifier 214 of the extracted multiple media data 210A (step S10). The data generation unit 180 causes the display device 41 of the electronic device 40 to display a top screen 400 containing multiple cards 410 based on the multiple media data 210A via the web server 30.

[0105] Figure 22 shows a modified configuration of the search recommendation section.

[0106] Furthermore, the search recommendation unit 103 may not only extract analysis information data 240A from the analysis information database 240, but also generate media information data that matches the user's preference features 253 (step S93). In this case, the data generation unit 180 only needs to generate a card 410 for displaying the generated media information data.

[0107] 8. Examples of prompts and new media information

[0108] Figure 23 shows an example of new media information generated by the prompt.

[0109] Let's take the example where the input to the prompt generation unit 106 consists of a starting node 501 "Yayoi Kusama", next hop nodes 502A "Louis Vuitton", 502B "Vagil Abloh", and ending node 503 "Kanye West", all located sequentially on a single route, and the modality of the new media information is an article 522 and a still image 524 (Figure 6(C)).

[0110] The prompts generated by the prompt generation unit 106 may include, for example, the following: Write article 600 (New Media Information) as a journalist. Write a punchy, fact-checked article. Trace the relationships between all adjacent pairs of nodes (i.e., from "Yayoi Kusama" to "Louis Vuitton" 601, from "Louis Vuitton" to "Vagil Abloh" 602, from "Vagil Abloh" to "Kanye West" 603) between the starting node 501 "Yayoi Kusama", the next hop nodes 502A "Louis Vuitton", 502B "Vagil Abloh", and the ending node 503 "Kanye West" 603). Then, explain the broader relationships between the starting node 501 "Yayoi Kusama" and the ending node 503 "Kanye West" on one or more routes as a summary 604. Include numbered citations in the text. The style of the new media information (article writing style) will match that of, for example, starting node 501 "Yayoi Kusama". Extract 1 to 4 publicly available images 605. The thought process will absolutely not be disclosed. Enter the URL 606 of the website for each node.

[0111] The prompts generated by the prompt generation unit 106 may have tasks such as the following: Link the starting node 501 "Yayoi Kusama" and the ending node 503 "Kanye West". Investigate at least three specific and verifiable facts. For each fact, include one real-world anecdote or product example. Summarize the information in a Japanese article of 500-700 characters in the specified style. Propose up to four images 605, each with a URL and a short alt text. At the end of the article 600, which is new media information, create a Markdown list with the heading "Reference Links" and list clickable URLs 607 of sources that have not yet been provided.

[0112] As another example, if the modality is map 527, the prompt generation unit 106 may generate a prompt to plot their footprints on the map based on "Yayoi Kusama" to "Kanye West" and design a journey tracing their footprints from "Yayoi Kusama" to "Kanye West" in order of proximity. If the modality is music 525, the prompt generation unit 106 may generate a prompt to generate music based on the relationship connecting "Yayoi Kusama" to "Kanye West".

[0113] 9. Conclusion

[0114] Traditionally, information retrieval systems have relied on user input, leading to disparities in the information obtained depending on the user's information literacy. Specifically, conventional multimedia search systems, when given a search query, output multimedia directly related to that query as search results. For example, if the query is a keyword, they output multimedia containing that keyword; if the query is an image, they output multimedia containing images with similar image features or information about the subject. Furthermore, these conventional multimedia search systems primarily require user input of the search query and aim to produce search results that match the explicitly described search needs.

[0115] In such multimedia search systems, it is impossible for the system to provide users with information other than what they expect from their search queries. In other words, the system can only provide information within the range that the user expects. Existing recommendations, through information extraction influenced by filter bubbles and majority bias, fixate on a range of interests and reduce serendipitous discoveries. This situation is thought to encompass the following problems:

[0116] Traditional search systems assume that users are aware of their search needs and can spontaneously describe them in language. (Image search and multimedia search merely show results that match the subject of the search; in other words, they are the same in that they aim for subject matching.) As a result, the system cannot provide a range of information beyond the explicitly stated search needs, and therefore cannot satisfy the user's unconscious and inexpressible intuitive search needs.

[0117] Therefore, according to this embodiment, all media information is connected regardless of modality (single modality or multimodality), and the relationships between all media information, regardless of modality (single modality or multimodality), are presented to the user as a root GUI. Furthermore, searches are performed based on the relationships between media information, regardless of the user's literacy, and new media information is created based on the relationships between media information. Specifically, new media information is created by using two or more media information pieces composed of one or more modalities (single modality or multimodality) and performing simple operations based on the relationships between each piece of media information. By taking additional actions based on the given information and the connections between concepts, it becomes possible to perform searches with a higher degree of freedom. After prompting the user to discover that media information pieces have relationships with each other, input information (prompts) for creating new media information that shows the relationships between media information can be constructed without the user consciously inputting anything.

[0118] Even if a system provides users with the opportunity to discover new connections, it can be difficult for them to determine the appropriate search query to find the relationship between the media information they are searching for and the media information presented. According to this embodiment, after providing users with the opportunity to discover new connections, they can easily search for the relationships between multiple media pieces of information using a card GUI, and the relationships between multiple media pieces of information can be presented to the user using a root GUI, or new media information based on those relationships can be generated. According to this embodiment, the system can stimulate the user's intellectual curiosity, provide them with the opportunity to discover and explore new connections, and then provide them with the basis for those new connections and deeper knowledge. As a result, users can gain a deeper sense of satisfaction and have their intellectual curiosity further stimulated.

[0119] Although various embodiments and modifications of this technology have been described above, this technology is not limited to the embodiments described above, and various modifications can be made without departing from the gist of this technology. [Explanation of symbols]

[0120] 1: Information processing system, 10: Information processing device, 20: Storage device, 30: Web server, 40: Electronic device, 41: Display device, 410: Card, 101: Card GUI control unit, 102: Starting node determination unit, 103: Search recommendation unit, 104: Route search unit, 105: Route GUI control unit, 106: Prompt generation unit, 107: New media information generation unit, 108: New media information output unit, 109: Ending node determination unit, 110: Collection unit, 120: Media information analysis unit, 130: Conceptual relationship calculation unit, 140: Account creation unit, 150: Account usage history analysis unit, 160: Startup recommendation unit, 170: Search recommendation unit, 210: Media information database, 220: Conceptual database, 230: Conceptual relationship database, 240: Analysis information database, 250: Account database.

Claims

1. A starting node determination unit that determines certain media information selected by the user on the display device of an electronic device as the starting node, An endpoint node determination unit that determines another media information selected by the user on the display device as the endpoint node, A route GUI control unit that displays a route GUI on the display device indicating that the starting node to the ending node are connected along the route, A prompt generation unit generates a prompt for generating new media information based on the relationship between the starting node and the ending node, A new media information generation unit that executes the aforementioned prompt to generate the new media information, A new media information output unit that outputs the new media information to the electronic device in order to present the new media information to the user. An information processing system equipped with the following features.

2. The information processing system according to claim 1, A card GUI control unit displays multiple cards, each representing different media information, on the display device for the user to select from, and determines the first and second cards selected by the user from the multiple cards. Furthermore, it is equipped with, The starting node determination unit determines the media information represented by the first card as the starting node, The endpoint node determination unit determines the media information represented by the second card as the endpoint node. Information processing system.

3. The information processing system according to claim 1, Route search unit determines that one or more media information points that exist sequentially along one or more routes from the media information of the starting node to the media information of the ending node are one or more next-hop nodes. Furthermore, it is equipped with, The aforementioned prompt is, The relationship between each of the one or more next-hop nodes and the start node and the end node, and / or The relationship between every pair of adjacent nodes from the starting node through one or more next-hop nodes to the ending node. This is a prompt for generating the new media information based on the above. Information processing system.

4. The information processing system according to claim 2, Among the multiple cards displayed on the display device, there are no cards that represent media information for one or more next-hop nodes, which are one or more media pieces of information that exist sequentially along one or more routes from the media information of the starting node to the media information of the ending node. Information processing system.

5. An information processing system according to claim 1 or 2, The prompt generation unit generates the prompt when the user performs a predetermined action on the root GUI. Information processing system.

6. An information processing system according to claim 1 or 2, The prompt generation unit generates prompts for generating styled content based on the media information of the start node and / or the end node. Information processing system.

7. The information processing system according to claim 3, The modalities of the media information of the starting node, the media information of the ending node, and the modalities of the media information of the one or more next-hop nodes, and the modalities of the new media information are all the same, all different, or partially the same and partially different. Information processing system.

8. The information processing system according to claim 2, The card GUI control unit determines that a predetermined operation has been performed on the first card, which is one of the plurality of cards displayed on the display device. The starting node determination unit determines the first card as the starting node, Multiple media information pieces, each having a predetermined relationship to the starting node which is media information represented by the first card, are determined to be multiple next-hop nodes. The process of determining multiple media information that has a predetermined relationship to each of the multiple next-hop nodes that have been identified is repeated a predetermined number of times to determine the multiple media information of the final generation node reached. The search function is further equipped with a recommendation section. The aforementioned card GUI control unit is Except for the first card, the display is switched to show a plurality of new cards, each representing a plurality of media information for the final generation node, instead of the plurality of cards displayed on the display device. It is determined that a predetermined operation has been performed on the second card, which is one of the multiple cards displayed on the display device. The endpoint node determination unit determines the second card to be the endpoint node. Information processing system.

9. The information processing system according to claim 8, The aforementioned card GUI control unit is The display device displays a functional object and the plurality of cards around the functional object. When the first card is moved toward the functional object, Switch the display so that the aforementioned multiple new cards are displayed around the functional object. Information processing system.

10. An information processing system according to claim 8 or 9, For multiple media pieces of information, extract multiple concepts that each piece of media piece of information possesses, and classify the extracted multiple concepts into dominant concepts and sub-concepts. Concepts having a predetermined conceptual relationship with the dominant concept possessed by each piece of media information are additionally determined as sub-concepts possessed by the media information. The set of media information includes all of the dominant concepts and all of the sub-concepts that the media information possesses, and this set of media information is defined as the conceptual set of media information. Media Information Analysis Department, A conceptual relationship calculation unit calculates the conceptual relationships between individual concepts included in a set of concepts of one media information and individual concepts included in a set of concepts of another media information. Furthermore, it is equipped with, The aforementioned search recommendation unit is: Multiple media information pieces, each containing a concept having a predetermined conceptual relationship with an individual concept included in the set of media information concepts representing the selected first card, are determined to be multiple next-hop nodes. The process of determining multiple media information having individual concepts that have a predetermined conceptual relationship with each concept included in the set of multiple media information concepts of the multiple next-hop nodes that have been determined is repeated a predetermined number of times to determine the multiple media information of the final generation node reached. Information processing system.

11. The information processing system according to claim 2, The card GUI control unit causes the user to display the card representing the new media information generated based on the prompt on the display device of the electronic device, so that the user can select it as either the first card or the second card. Information processing system.

12. The information processing system according to claim 2, The starting node determination unit can determine the media information input by the user as the starting node, instead of the media information represented by the first card selected by the user. Information processing system.

13. The information processing system according to claim 3, When the user performs a predetermined operation on any two of the following nodes: the starting node, the one or more next-hop nodes, and the ending node, the prompt generation unit generates a prompt for generating new media information based on the relationship between the new starting node and the new ending node, with the node on the starting node side of the two operated nodes designated as the new starting node and the node on the ending node side designated as the new ending node. Information processing system.

14. A starting node determination unit that determines certain media information selected by the user on the display device of an electronic device as the starting node, An endpoint node determination unit that determines another media information selected by the user on the display device as the endpoint node, A route GUI control unit that displays a route GUI on the display device indicating that the starting node to the ending node are connected along the route, A prompt generation unit generates a prompt for generating new media information based on the relationship between the starting node and the ending node, A new media information generation unit that executes the aforementioned prompt to generate the new media information, A new media information output unit that outputs the new media information to the electronic device in order to present the new media information to the user. An information processing device equipped with the following.

15. The control circuit of the information processing device executes the information processing program, A certain media information selected by the user on the display device of an electronic device is determined as the starting node. The user selects another media information on the display device, which is then determined as the endpoint node. A route GUI indicating that the starting node to the ending node are connected along the route is displayed on the display device. A prompt is generated to generate new media information based on the relationship between the starting node and the ending node. The aforementioned prompt is executed to generate the new media information, The new media information is output to the electronic device in order to present it to the user. Information processing methods.

16. The control circuit of the information processing device A starting node determination unit that determines another media information selected by the user on the display device of an electronic device as the starting node, An endpoint node determination unit that determines another media information selected by the user on the display device as the endpoint node, A route GUI control unit that displays a route GUI on the display device indicating that the starting node to the ending node are connected along the route, A prompt generation unit generates a prompt for generating new media information based on the relationship between the starting node and the ending node, A new media information generation unit that executes the aforementioned prompt to generate the new media information, New media information output unit that outputs to the electronic device in order to present the new media information to the user. An information processing program that operates as such.

Images