Information processing device, information processing method, and information processing program

Abstract

To provide appropriate information.SOLUTION: An information processor of the present application includes an estimation unit and a provision unit. The estimation unit estimates an interest target as a target which interested a user on the basis of first content in which predetermined operation which shows an interest of the user is done for the user. The provision unit provides the user with second content related to the interest target estimated by the estimation unit.SELECTED DRAWING: Figure 6

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, an information processing method, and an information processing program.

Background Art

[0002] Conventionally, techniques for providing various information to users have been known. For example, there is a technique for providing a product recommendation page in response to the input of a search query (query) (for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, there is room for improvement in the above conventional technology. In the above conventional technology, only a product recommendation page corresponding to a query is provided, and it is difficult to provide information to a user when the user has not input a query. Therefore, it is desired to provide more appropriate information without depending on the user's query input or the like.

[0005] The present application has been made in view of the above, and an object thereof is to provide an information processing apparatus, an information processing method, and an information processing program capable of providing appropriate information.

Means for Solving the Problems

[0006] The information processing device according to the present application is characterized by comprising: an estimation unit that estimates an object of interest that the user has shown interest in, based on a first content in which the user has performed a predetermined operation that indicates the user's interests; and a provision unit that provides the user with a second content related to the object of interest estimated by the estimation unit. [Effects of the Invention]

[0007] According to one embodiment, the effect is achieved that appropriate information can be provided. [Brief explanation of the drawing]

[0008] [Figure 1] Figure 1 shows an example of information processing according to the embodiment. [Figure 2] Figure 2 shows an example of a predetermined operation that demonstrates the user's interests. [Figure 3] Figure 3 shows an example of information processing performed by an information processing system. [Figure 4] Figure 4 shows an example of information processing performed by an information processing system. [Figure 5] Figure 5 shows an example of information processing performed by an information processing system. [Figure 6] Figure 6 shows an example of the configuration of an information processing system according to the embodiment. [Figure 7] Figure 7 shows an example of a user information storage unit according to the embodiment. [Figure 8] Figure 8 shows an example of a target information storage unit according to the embodiment. [Figure 9] Figure 9 shows an example of a content storage unit according to an embodiment. [Figure 10] Figure 10 shows an example of a model information storage unit according to an embodiment. [Figure 11] Figure 11 is a flowchart showing an example of an information processing flow. [Figure 12] Figure 12 is a flowchart showing an example of an information processing flow. [Figure 13] Figure 13 is a flowchart illustrating an example of an information processing flow. [Figure 14] Figure 14 is a flowchart showing an example of an information processing flow. [Figure 15] Figure 15 is a flowchart showing an example of an information processing flow. [Figure 16] Figure 16 is a hardware configuration diagram showing an example of a computer that implements the functions of an information processing device. [Modes for carrying out the invention]

[0009] The following describes in detail, with reference to the drawings, the embodiments for implementing the information processing device, information processing method, and information processing program according to the present application (hereinafter referred to as "embodiments"). Note that these embodiments do not limit the information processing device, information processing method, and information processing program according to the present application. Furthermore, each embodiment can be appropriately combined as long as the processing content is not inconsistent. Also, the same parts are denoted by the same reference numerals in each of the following embodiments, and redundant explanations are omitted.

[0010] (Embodiment) [1. An example of information processing] First, an example of information processing performed by the information processing system 1 will be explained using Figure 1. Figure 1 is a diagram showing an example of information processing according to the embodiment. The information processing device 100 estimates the objects of interest of the user (also called "objects of interest") and performs information processing that includes providing information about the estimated objects of interest. For example, objects of interest include various objects of transaction such as stores and other facilities, and goods. Transaction objects are not limited to goods, but may also include the use of services, etc. Note that the above is merely an example, and objects of interest can be any object that is of interest to the user.

[0011] [1-1. Overall Overview of Processing in Information Processing Systems] Next, the general flow of information processing in the information processing system 1 will be described using FIG. 1. In the following, the case where the information processing apparatus 100 of the information processing system 1 performs processing such as estimation of the user's area of interest will be described as an example. However, each of the following processes may be performed not only by the information processing apparatus 100 but also by any apparatus included in the information processing system 1, such as the terminal apparatus 10, according to the apparatus configuration and functions included in the information processing system 1. For example, the terminal apparatus 10 may perform processing such as estimation of the user's area of interest, which will be described later.

[0012] The information processing apparatus 100 acquires information regarding a model (also referred to as "model information") from an external apparatus 20 (step S1). For example, the information processing apparatus 100 acquires various models such as a model M1, which is a domain estimation model used for domain estimation. Note that the information processing apparatus 100 may learn various models such as the model M1 by itself, which will be described later.

[0013] Also, the information processing apparatus 100 acquires information regarding a target (also referred to as "target information") from the external apparatus 20 (step S2). The information processing apparatus 100 acquires information regarding a facility such as a store (also referred to as "facility information") as target information. For example, the information processing apparatus 100 acquires facility information of a facility such as an actual store existing at the time of processing from the external apparatus 20. The facility information includes various information regarding the facility. For example, the facility information includes various information used for information provision of the facility. For example, the facility information includes various information regarding the facility such as the location (position) of the facility, the business hours of the facility, the category, an image of the facility, a transaction target provided by the facility, and an image of the transaction target provided by the facility.

[0014] Furthermore, the information processing device 100 acquires information about the items traded in the online marketplace (also called "item information") as target information. For example, the information processing device 100 acquires item information about items currently for sale (listed) in the online marketplace from the external device 20. The item information includes various information about the items. For example, the item information includes various information used to provide information about the items. For example, the item information includes various information about the items, such as the seller, selling price, image of the item, and category.

[0015] Furthermore, the target information may include information from various services. For example, the target information may include information about posts on SNS (Social Networking Services). For example, the target information may include posted information (content) posted by users on SNS. For example, the posted information may be text information or image information such as images and videos posted by users on Instagram®, TikTok®, YouTube®, Twitter®, Facebook®, etc.

[0016] Furthermore, the information processing device 100 acquires information about the user (also called "user information") from the terminal device 10 used by the user (step S3). The terminal device 10 transmits user information about the user to the information processing device 100. For example, the information processing device 100 acquires user information including information about the user's actions from the terminal device 10 used by the user. For example, the information processing device 100 acquires user information including the operation history of the user's operation of the terminal device 10.

[0017] For example, user information may include a history of the user's actions on the internet. For example, user information may include a history of the user's actions, such as browsing history and purchase history of information about items traded in electronic commerce (EC) services such as online shopping malls. The items traded in electronic commerce services include items traded in various forms such as online shopping malls, online shopping sites, auction sites, and flea market sites. Furthermore, user information may include a history of the user's actions in the real world, such as the user's location information.

[0018] Furthermore, user information includes sensor information detected regarding the user. For example, user information includes sensor information detected by the terminal device 10. User information may also include location information detected by the terminal device 10. User information may also include user attribute information. For example, attribute information may include various types of information such as demographic attributes like age, gender, residential area, and occupation, or psychographic attributes like interests and lifestyle. Note that attribute information is merely an example, and user information may include any information relating to the user. The information processing device 100 may also acquire user information from the external device 20.

[0019] The information processing device 100 then performs processing to provide information to the user based on various types of information. The information processing device 100 estimates the user's interests based on the user's user information (step S4). For example, the information processing device 100 may estimate the user's interests based on various types of information such as images generated by the user operating the terminal device 10, but this point will be discussed later.

[0020] Then, the information processing device 100 provides information to the user based on the estimated user's interests (step S5). For example, the information processing device 100 provides the user with information indicating the user's interests at a predetermined time. For example, the information processing device 100 transmits content corresponding to the user's interests to the terminal device 10 used by the user. In this way, the information processing device 100 can provide appropriate information by estimating the user's interests and providing information to the user about the estimated interests.

[0021] [1-2. Details of various processes] Based on the information processing described above, the details of various processes will be explained. Points similar to those explained above will be omitted as appropriate. Furthermore, the various processes described below may be combined as appropriate. For example, the map content described later may be provided (displayed) in the messenger application described later. Note that the above is merely an example, and the various processes may be combined in any way within the scope of possible combined processing. In the following explanation, the information processing system 1 is described as the processing entity, but depending on the device configuration included in the information processing system 1, any device such as the information processing device 100 or the terminal device 10 may perform the processing. Also, even in the following explanation where the information processing device 100 is described as the processing entity, the terminal device 10 may be the processing entity. Furthermore, in the following explanation, user U1 is described as an example of a user.

[0022] [1-2-1. Information provision] First, an example of information provision tailored to the user's interests in the information processing system 1 will be explained using Figure 2. Figure 2 is a diagram illustrating an example of a predetermined operation that indicates the user's interests. Figure 2 explains the process using an image (also called a "captured image") generated in response to a capture (screenshot) operation by the user. However, the information used to estimate the user's interests is not limited to captured images; any information may be used. For example, the information used to estimate the user's interests may include various types of information such as images taken by the user using the terminal device 10 and sensor information, but this will be discussed later.

[0023] In Figure 2, the information processing system 1 estimates whether the objects contained in the captured image C1 are of interest to user U1, based on user U1's actions on the captured image C1 displayed by application AP11, which is installed on terminal device 10. In the example shown in Figure 2, the information processing system 1 generates captured image C1, which is an image capturing content representing the restaurant, Steakhouse SH, when user U1 operates terminal device 10.

[0024] For example, application AP11 may be an application for users to manage captured images. For example, the user can use application AP11 to display captured images on terminal device 10 that have not yet been processed for registration as stock (favorites) up to that point (also called "unprocessed captured images"), and then perform an operation to indicate whether the displayed captured images should be registered as favorites. Information processing system 1 manages the captured images that the user has registered as favorites as captured images that indicate the user's interests.

[0025] In Figure 2, user U1 controls whether or not to register the captured image C1 as a favorite by moving the captured image C1 displayed on the terminal device 10 to the left or right using operations such as swiping or flicking.

[0026] For example, if user U1 performs operation OP1 to move the captured image C1 to the left, terminal device 10 accepts operation OP1 as an operation that does not require registration of the captured image C1 as a favorite. In other words, if user U1 performs operation OP1 to move the captured image C1 to the left, information processing system 1 assumes that user U1 has no interest in the subject of the captured image C1 and does not register the captured image C1 as a favorite for user U1.

[0027] Furthermore, for example, if user U1 performs operation OP2 to move the captured image C1 to the right, terminal device 10 accepts operation OP2 as an operation to register the captured image C1 as a favorite. In other words, if user U1 performs operation OP2 to move the captured image C1 to the right, information processing system 1 assumes that user U1 is interested in the subject of the captured image C1 and registers the captured image C1 as a favorite for user U1.

[0028] Note that the left-right direction of operations OP1 and OP2 described above is merely an example, and the left-right direction of operations OP1 and OP2 may be reversed. In this case, the terminal device 10 accepts an operation by the user to move the captured image to the left as an operation requiring registration of the captured image as a favorite. The terminal device 10 also accepts an operation by the user to move the captured image to the right as an operation that does not require registration of the captured image as a favorite.

[0029] Furthermore, the operations OP1 and OP2 described above are merely examples of operations indicating whether or not to register an item as a favorite, and the terminal device 10 may accept various operations from the user as operations indicating whether or not to register an item as a favorite. In Figure 2, if user U1 specifies button BT1, the terminal device 10 may accept that specification as an operation indicating that the captured image C1 does not need to be registered as a favorite. Also, if user U1 specifies button BT2, the terminal device 10 may accept that specification as an operation indicating that the captured image C1 needs to be registered as a favorite.

[0030] Furthermore, although only captured image C1 is shown in Figure 2, if there are multiple unprocessed captured images, the terminal device 10 may sequentially display the unprocessed captured images in response to the operation of whether or not to register the unprocessed captured images as favorites. In Figure 2, after receiving the operation of whether or not to register captured image C1 as a favorite, the terminal device 10 may display another unprocessed captured image (for example, captured image C2) and accept the operation of whether or not to register captured image C2 as a favorite. Also, after receiving the operation of whether or not to register captured image C2 as a favorite, if there are other unprocessed captured images, the terminal device 10 may display those unprocessed captured images (for example, captured image C3) and accept the operation of whether or not to register captured image C3 as a favorite. In this way, if there are multiple unprocessed captured images, the terminal device 10 sequentially displays captured images C1, C2, C3, etc. in response to the operation of whether or not to register them as favorites, and sequentially accepts the operation of whether or not to register each unprocessed captured image as a favorite.

[0031] For example, if user U1 registers captured image C1 as a favorite, the information processing system 1 uses captured image C1 to estimate user U1's interests. For example, the information processing device 100 estimates user U1's interests based on the results of an analysis process that analyzes captured image C1. The information processing device 100 estimates user U1's interests based on the objects included in captured image C1. In this case, the information processing device 100 estimates Steakhouse SH, a store included in captured image C1, as user U1's interest. The information processing system 1 may also use a model to estimate the user's interests, but this will be discussed later.

[0032] Information processing system 1 provides users with information about their areas of interest at predetermined timings. In Figure 2, information processing system 1 provides user U1 with information about steakhouse SH at predetermined timings. For example, information processing system 1 provides information about steakhouse SH to the terminal device 10 used by user U1 when user U1 is close to steakhouse SH. For example, information processing system 1 compares user U1's location information with the location of steakhouse SH and displays information about steakhouse SH on the terminal device 10 used by user U1 when user U1 is within a predetermined range from steakhouse SH. If the user's area of ​​interest is a product, information processing system 1 may provide information about that product when the user can purchase it. For example, if the user's area of ​​interest is a product, information processing system 1 may provide information about that product if it is in stock.

[0033] In the example described above, the user's operation of registering a captured image as a favorite was explained as an example of a predetermined operation that indicates the user's interests. However, the predetermined operation that indicates the user's interests is not limited to the user's operation of registering a captured image as a favorite, but may be any operation. For example, the predetermined operation that indicates the user's interests may be the operation of the user to generate a captured image. In this case, in Figure 2, if the operation of generating a captured image C1 is performed by user U1, the information processing system 1 may infer that Steakhouse SH, which is included in the captured image C1, is an object of interest to user U1.

[0034] As described above, the information processing system 1 can provide appropriate information by estimating the user's interests in response to a predetermined operation by the user and providing the user with information about the estimated interests. In this way, when the user performs a favorites registration operation, the information processing system 1 estimates the user's interests at the time of the registration operation based on information obtainable from the terminal device. Then, at a predetermined timing, the information processing system 1 provides the user with information about the estimated interests.

[0035] Information processing system 1 analyzes captured images to estimate the user's interests. For example, information processing system 1 may use a domain estimation model to identify domains and estimate targets according to those domains, but this will be discussed later. In addition, information processing system 1 may analyze sensor information detected by predetermined sensors, not limited to captured images, to estimate the user's interests. For example, information processing system 1 may estimate stores, etc., that are the target of interest (also called "interest facilities") based on orientation and location, such as sound, location, and browsing history. Furthermore, information processing system 1 may use the user's attribute information to estimate the user's interests. For example, information processing system 1 may estimate the user's interests as those of other users with similar attributes.

[0036] Information processing system 1 may send reminders or other informational notifications regarding the user's interests at any time, depending on the user's interests. For example, if the user's interests are products, information processing system 1 may send push notifications regarding those products when certain conditions are met. Alternatively, if the user's interests are events, information processing system 1 may send push notifications when tickets are sold. For example, information processing system 1 may send reminders when providing content in a domain corresponding to the user's interests.

[0037] For example, if the user's interest is in stores, the information processing system 1 may issue a reminder while displaying content related to gourmet food, maps, etc. Also, if the user's interest is in products, the information processing system 1 may issue a reminder while displaying content related to e-commerce, etc. For example, the information processing system 1 may issue a reminder when a corresponding message is displayed in a messenger application. For example, the information processing system 1 may estimate the user's interests based on information provided on television, radio, etc.

[0038] [1-2-2. Domain-based processing] From here, we will explain an example of how information processing system 1 provides information based on domains, using Figure 3. Figure 3 is a diagram showing an example of information processing performed by the information processing system. Note that explanations of points that are the same as those explained in Figure 2, etc., will be omitted as appropriate. A domain is used to classify the source of information, such as content. For example, a domain indicates the classification of various services that serve as information sources, such as EC service #1, restaurant introduction service #1, SNS service #1, SNS service #2, etc.

[0039] Figure 3 shows how information processing system 1 uses model M1, a domain estimation model, to estimate the domain of content corresponding to captured image C2 (also called "content CO2"), and processes the data based on the estimated domain. In Figure 3, the string "XXXXXX" represents the name of the product in content CO2 (e.g., hair dryer name). For example, let's assume that captured image C2 is a captured image generated by user U1 and is a captured image that user U1 has added to their favorites. Model M1 outputs information to estimate the domain to which the content corresponding to the input captured image belongs. For example, model M1 is a model that outputs a score indicating the likelihood that the content corresponding to the input captured image belongs to each domain. Model M1 can output any information as long as it is possible to estimate the domain for the input data.

[0040] The information processing device 100 may obtain model M1 from a model provision server or the like that provides learning models, or it may learn model M1 itself. For example, the information processing device 100 may obtain model M1 from a model provision server or the like that provides learning models. An example of when the information processing device 100 learns a domain estimation model such as model M1 will be described later.

[0041] In Figure 3, the information processing device 100 estimates the domain of the content corresponding to the captured image by inputting the captured image into model M1. The information processing device 100 inputs the captured image C2 of content CO2 into model M1 (step S11). Model M1, having received the captured image C2, outputs information indicating the estimated domain for content CO2 (step S12). In the example in Figure 3, model M1, having received the captured image C2, outputs scores corresponding to each of multiple domains such as EC service #1, restaurant introduction service #1, SNS service #1, and SNS service #2.

[0042] The information processing device 100 uses the scores corresponding to each of the multiple domains output by Model M1 to estimate that the domain of content CO2 is domain DM1, as shown in the estimated domain information OT1. For example, the information processing device 100 estimates that the domain with the highest score among the multiple domains output by Model M1 is the domain to which content CO2 belongs. For example, if the domain with the highest score among the multiple domains is EC Service #1, the information processing device 100 estimates that content CO2 is content of EC Service #1 (also called "Domain A"). Also, if the domain with the highest score among the multiple domains is Restaurant Introduction Service #1 (also called "Domain B"), the information processing device 100 estimates that content CO2 belongs to EC Restaurant Introduction Service #1.

[0043] Furthermore, Information Processing System 1 estimates the user's interests using a query estimation model for each domain. In this case, for example, Information Processing System 1 estimates that the user's interests are those related to the queries extracted using the query estimation model. Information Processing System 1 estimates the user's interests using a query estimation model (Model M11) corresponding to EC service #1 (Domain A), a query estimation model (Model M12) corresponding to restaurant introduction service #1 (Domain B), a query estimation model (Model M13) corresponding to SNS service #1 (Domain C), etc.

[0044] Information processing system 1 estimates the user's interests using a query estimation model corresponding to the domain estimated for the captured image, from among multiple query estimation models corresponding to each of the multiple domains. For example, query estimation models such as models M11 to M13 take an image such as a captured image as input and extract and output information to be used as a query (e.g., the name of the object) from the input image. Details of query estimation models such as models M11 to M13 will be described later, but any query estimation model can output any information as long as it can estimate the information to be used as a query from the input data. For example, the query estimation model may be a single model common to all domains.

[0045] The information processing device 100 may obtain query estimation models corresponding to each domain, such as models M11, M12, and M13, from a model provision server or the like that provides learning models, or it may learn query estimation models corresponding to each domain, such as models M11, M12, and M13, within its own device. For example, the information processing device 100 obtains query estimation models corresponding to each domain, such as models M11, M12, and M13, from a model provision server or the like that provides learning models. An example of the case where the information processing device 100 learns query estimation models corresponding to each domain, such as models M11, M12, and M13, will be described later.

[0046] In Figure 3, if the domain corresponding to the captured image C2 is domain A, the information processing device 100 uses the model M11 of domain A to estimate the user U1's interests. For example, if the information processing device 100 estimates the domain of content CO2 to be domain A based on the output of model M1 to which the captured image C2 is input, it uses the model M11 of domain A to estimate the user U1's interests based on the captured image C2, as shown in query estimation EX1.

[0047] For example, the information processing device 100 inputs the captured image C2 into the model M11, causes the model M11 to output information to be used as a query, and estimates the user U1's interests based on the information output by the model M11. In Figure 3, when the model M11, which has received the captured image C2 as input, outputs the string "XXXXXX", the information processing device 100 estimates that the user U1's interests are the object indicated by the string "XXXXXX" (also called "product X").

[0048] Furthermore, if the model M11, which has received the captured image C2 as input, outputs the string "XXXXXX", the information processing device 100 extracts the string "XXXXXX" as a query to be used to provide information to user U1. Using the string "XXXXXX" extracted using the model M11, the information processing device 100 performs a search for the domain corresponding to domain A as the target domain, as shown in the information retrieval PS1. In Figure 3, the information processing device 100 determines a domain similar to domain A as the target domain. For example, the information processing device 100 determines a domain related to EC services, similar to domain A (EC service #1), as the target domain. The information processing device 100 determines EC service #2, which is a domain similar to domain A (EC service #1), as the target domain.

[0049] The information processing device 100 then searches for the target domain, EC service #2, using the string "XXXXXX" as the query. The information processing device 100 extracts the content of the target (product X) indicated by the string "XXXXXX" (also called "content CO5") from the content of EC service #2, the target domain, as the second content. In this way, the information processing device 100 extracts content CO5, which introduces product X, from the content of EC service #2, which is a different target domain (second domain) from domain A (first domain) of content CO2 (first content), as the second content.

[0050] The information processing device 100 then provides content CO5 to user U1. The information processing device 100 transmits content CO5 to terminal device 10 used by user U1, and terminal device 10, upon receiving content CO5, displays content CO5. In this way, the information processing device 100 causes content CO5 to be displayed on terminal device 10 used by user U1. As a result, the information processing device 100 provides content CO5 (second content) of EC service #2 (second domain), which is different from EC service #1 (first domain) of content CO2 (first content). In this way, the information processing device 100 can provide appropriate information by estimating the user's interests based on the first content and providing the user with second content related to the estimated interests.

[0051] The above is merely an example, and the information processing device 100 may determine the target domain using various information. In Figure 3, if there are multiple domains similar to domain A, the information processing device 100 may determine the domain used by user U1 as the target domain from among the multiple domains similar to domain A. For example, among EC service #2 and EC service #3, which are domains similar to domain A (EC service #1), the information processing device 100 may determine EC service #2, which is used by user U1, as the target domain.

[0052] For example, the information processing device 100 may determine the target domain to be EC Service #2, which is the domain where user U1 holds points, among EC Service #2 and EC Service #3, which are domains similar to Domain A (EC Service #1). For example, the information processing device 100 may determine the target domain to be EC Service #2, which is the domain where user U1 holds the most points, among EC Service #2 and EC Service #3, which are domains similar to Domain A (EC Service #1). For example, the information processing device 100 may determine the target domain to be the domain with the lowest price for the item of interest among EC Service #1, EC Service #2, and EC Service #3.

[0053] Furthermore, if the domain corresponding to the captured image (referred to as "captured image C3") is domain B, the information processing device 100 uses the model M12 for domain B to estimate the user U1's interests. For example, if the information processing device 100 estimates that the domain of content CO3 is domain B based on the output of model M1 to which captured image C3 is input, it uses the model M12 for domain B to estimate the user U1's interests based on captured image C3, as shown in query estimation EX2.

[0054] For example, the information processing device 100 inputs the captured image C3 into the model M12, causes the model M12 to output information to be used as a query, and estimates the user U1's interests based on the information output by the model M12. In Figure 3, when the model M12, which has received the captured image C3 as input, outputs the string "YYY", the information processing device 100 estimates that the user U1's interests are the object indicated by the string "YYY" (also called "restaurant Y").

[0055] Furthermore, if the model M12, which has received the captured image C3 as input, outputs the string "YYY", the information processing device 100 extracts the string "YYY" as a query to be used to provide information to user U1. Using the string "YYY" extracted using the model M12, the information processing device 100 performs a search for the domain corresponding to domain B as the target domain, as shown in the information search PS2. In Figure 3, the information processing device 100 determines a domain similar to domain B as the target domain. For example, the information processing device 100 determines a domain related to restaurant referral services, similar to domain B (restaurant referral service #1), as the target domain. The information processing device 100 determines restaurant referral service #2, which is a domain similar to domain B (restaurant referral service #1), as the target domain.

[0056] The information processing device 100 then searches the target domain, Restaurant Introduction Service #2, using the string "YYY" as the query. The information processing device 100 extracts the content of Restaurant Introduction Service #2, which is the target domain, that corresponds to the target (restaurant Y) indicated by the string "YYY" (also called "Content CO6") as the second content. In this way, the information processing device 100 extracts Content CO6, which introduces restaurant Y, as the second content from the content of Restaurant Introduction Service #2, which is a different target domain (second domain) from Domain B (first domain) of Content CO3 (first content). The information processing device 100 then provides Content CO6 to user U1. The information processing device 100 transmits Content CO6 to the terminal device 10 used by user U1.

[0057] The above processing is merely an example, and the information processing system 1 may perform processing using various types of information. The information processing system 1 may also estimate the domain of the content without using a domain estimation model. In this case, the information processing system 1 may estimate whether the domain of the content corresponding to the captured image is a product domain or a store domain, depending on whether the content is related to e-commerce or gourmet food. For example, if the content corresponding to the captured image is related to e-commerce, the information processing system 1 may estimate that the domain of the content is a product domain. For example, if the content corresponding to the captured image is related to gourmet food, the information processing system 1 may estimate that the domain of the content is a store domain. The above is merely an example, and the information processing system 1 may estimate the domain of the content using various types of information as appropriate.

[0058] As described above, when providing information on the target corresponding to the captured image, the information processing system 1 replaces it with the destination domain and proposes it to the user. In this way, the information processing system 1 provides the user with content from a predetermined domain related to the user's interests. For example, if the domain corresponding to the captured image is not a predetermined domain, the information processing system 1 provides content from a predetermined domain.

[0059] Furthermore, the information processing system 1 performs domain estimation using a domain estimation model. For example, the information processing system 1 generates queries using different query estimation models for each domain and provides content that represents search results for a given domain. For example, the information processing system 1 may use a query extraction model that has learned where queries are located within an image. For example, the information processing system 1 provides information for a given domain when certain conditions are met. If there is a given domain that offers items of interest at a lower price than the estimated domain, the information processing system 1 provides the user with content that indicates items of interest in that given domain.

[0060] [1-2-3. Examples of information provision using map display] Next, an example of information provision through map display regarding the user's interests in Information Processing System 1 will be explained using Figure 4. Figure 4 is a diagram illustrating an example of information processing performed by the Information Processing System. In Figure 4, the processing is explained using a store called Cafe XXXX (also called "Target Store CX") as an example of an object of interest, but information provision through map display may be similarly applied to transactional objects such as goods, not just stores and other facilities.

[0061] In Figure 4, map content is displayed by application AP12, which is an application installed on terminal device 10. In the example shown in Figure 4, the information processing system 1 displays information INF11 regarding Cafe XXXX (target store CX) superimposed on the map content MP1 when the user U1's location is close to the target store CX. For example, application AP12 is a map application installed on terminal device 10 used by the user. Terminal device 10 displays various information related to application AP12.

[0062] For example, when user U1 operates terminal device 10, the information processing system 1 generates a captured image (also called "captured image C11") which is an image of content representing the target store CX, which is a store. For example, when user U1 operates terminal device 10, terminal device 10 generates a captured image C11 of the target store CX on SNS service #1 in response to user U1's capture operation on content related to the target store CX posted by other users on SNS service #1. In Figure 4, when user U1 registers captured image C11 as a favorite, the information processing system 1 estimates user U1's interests using captured image C11. For example, the information processing device 100 estimates user U1's interests based on the results of an analysis process that analyzes captured image C1. The information processing device 100 estimates user U1's interests based on the objects included in captured image C11. In this case, the information processing device 100 estimates the target store CX, which is a store, included in captured image C1, as user U1's interests.

[0063] Information processing system 1 provides the user with map content that overlays information indicating the location of the object of interest, which corresponds to the object of interest indicated by the object of interest information, onto a map. Figure 4 shows a case where, if user U1 is located within a predetermined range from the target store CX, information processing system 1 displays the information INF11 provided for the target store CX overlaid on the map content MP1. For example, information processing system 1 compares the location information of user U1 with the location of the target store CX, and if user U1 is located within a predetermined range from the target store CX, it displays the information INF11 provided for the target store CX overlaid on the map content MP1 on the terminal device 10.

[0064] The map content MP1 in Figure 4 is content in which information provided by user U1 is superimposed on a map and placed at a location corresponding to the user's interests within the displayed area. For example, the map content MP1 is content in which information provided by user U1 is superimposed on a map showing a predetermined area from the location of user U1, and information provided by user U1 is in the shape of a pin icon indicating the location of the target store CX which is of interest to user U1. Information provided by user U11 includes an image IM1 showing the target store CX which is of interest. Information provided by user U11 also includes text information TX1 such as "Cafe XX○○ Store" which shows the target store CX which is of interest. Information provided by user U11 also includes an icon IC1 indicating that the domain corresponding to the captured image C11 used to estimate that user U1 is interested in the target store CX is SNS service #1.

[0065] In Figure 4, the information processing system 1 provides user U1 with map content MP1 that overlays the information to be provided INF11 onto a map. The information processing system 1 displays the map content MP1 that overlays the information to be provided INF11 onto a map on the terminal device 10 used by user U1. The information processing system 1 displays the map content MP1 on the terminal device 10 in which the information to be provided INF11, which is placed at a location corresponding to the user U1's interests within the display area, is superimposed on the map. For example, the information processing system 1 displays the map content MP1 in which the information to be provided INF11, in the shape of a pin icon indicating the location of the target store CX, which is the user U1's interest, is superimposed on a map that shows a predetermined range from the location where user U1 is located.

[0066] For example, the information processing system 1 displays map content MP1 on the terminal device 10, which superimposes provision information INF11, including an image IM1 showing the target store CX of interest, onto a map. Alternatively, the information processing system 1 displays map content MP1 on the terminal device 10, which superimposes provision information INF11, including text information TX1 such as "Cafe XXXX," showing the target store CX of interest, onto a map. Alternatively, the information processing system 1 displays map content MP1 on the terminal device 10, which superimposes provision information INF11, including an icon IC1 indicating that the captured image C11 used to estimate that the user U1's interest is the target store CX, is a captured image from SNS service #1.

[0067] In this way, the information processing system 1 can provide appropriate information to the user by overlaying information about the user's estimated interests onto a map. As mentioned above, the information processing system 1 may also display the user's interests as reminder items on the map. For example, the information processing system 1 acquires object information indicating the user's interests, identifies the location of the object indicated by the object information, and displays the identified location and content indicating the object overlaid on the map when the user displays the map.

[0068] For example, if the object of interest is a store, the information processing system 1 displays the location of the store. For example, if the object of interest is a product, the information processing system 1 displays the location of the store that sells the product. For example, if the object of interest is a product, the information processing system 1 displays the location of the nearest store that sells the product. For example, if the object of interest is a product, the information processing system 1 overlays an image of the product, rather than an image of the store, onto the store location.

[0069] As described above, the information processing system 1 may provide reminders for stores corresponding to information posted on SNS services. For example, the information processing system 1 may acquire a screenshot of the posted information on SNS, analyze the screenshot to identify the store indicated in the posted image, and provide the user with information about the identified store. For example, the information processing system 1 may analyze a screenshot of a post on an SNS service and provide the user with information indicating a store such as a cafe identified through the analysis. For example, the information processing system 1 may provide information to the user using environmental information such as weather and congestion. For example, the information processing system 1 may provide information to the user when certain conditions are met, such as when it is sunny, when it is not crowded, or when the user is nearby. For example, the information processing system 1 may estimate that a facility near the user's location when the user performs a predetermined operation, such as specifying a button to stock (register), is of interest to the user.

[0070] [1-2-4. Examples of information provision via message] Next, an example of information provision via messages regarding the user's interests in Information Processing System 1 will be explained using Figure 5. Figure 5 is a diagram illustrating an example of information processing performed by the Information Processing System. In Figure 5, the processing is explained using stores such as Izakaya BA Shibuya (also referred to as "Target Store CY") and Izakaya BB Shibuya (also referred to as "Target Store CZ") as examples of items of interest, but information provision via messages may be similar for items of transaction, such as goods, not just stores and other facilities.

[0071] Figure 5 shows an example of terminal device 10 displaying the message exchange of a group GP1, in which three users U1, U2, and U3 participate, in chronological order. Figure 5 shows an example of how terminal device 10 displays the message when user U1 inputs message MS1, "A bar in Shibuya...", and the information processing system 1 provides message MS2 and information to be provided, such as INF21 and INF22. Note that message MS1 can be any information as long as it is clear that the request is for information about a bar in Shibuya. For example, message MS1 could be a string of instructions such as "Tell me about a bar in Shibuya", or it could be a command to the information processing system 1 requesting information to be provided, such as "(command) Shibuya bar". Furthermore, since the display of the message exchange in group GP1 shown in Figure 5 is the same as a typical group chat, a detailed explanation is omitted.

[0072] In Figure 5, the application AP13, installed on the terminal device 10, displays various information such as messages entered by each user and information related to their interests (information for provision). In the example shown in Figure 5, the information processing system 1 displays information for provision along with the message when a user, such as user U1, enters a message related to their interests. For example, application AP13 is a messenger application installed on the terminal device 10 used by the user. The terminal device 10 displays various information related to application AP13.

[0073] For example, when user U1 operates terminal device 10, the information processing system 1 generates a captured image (also called "captured image C21") which is an image capturing content representing a target store CY. In Figure 5, when user U1 registers captured image C21 as a favorite, the information processing system 1 uses captured image C21 to estimate user U1's interests. For example, the information processing device 100 estimates user U1's interests based on the results of an analysis process that analyzes captured image C1. The information processing device 100 estimates user U1's interests based on the objects included in captured image C21. In this case, the information processing device 100 estimates the target store CY, which is a store included in captured image C1, as user U1's interest.

[0074] Furthermore, when user U2 operates the terminal device 10, the information processing system 1 generates a captured image (also called "captured image C22") which is an image of content representing the target store CZ, which is a store. In Figure 5, when user U2 registers captured image C22 as a favorite, the information processing system 1 estimates the target store CZ, which is a store included in captured image C22, as an object of interest for user U2.

[0075] Information processing system 1 provides information about the user's interests that are displayed together with the message in application AP13, based on the interest information and message information. For example, if the message in application AP13 corresponds to the user's interests, information processing system 1 provides information about those interests. In Figure 5, information processing system 1 provides information as available information about the interests of users U1, U2, and U3, who are included in group GP1 which includes user U1, and whose interests correspond to the message "Shibuya izakaya".

[0076] Information processing system 1 acquires information indicating the interests of user U1, user U2, and user U3, and determines that the interests of users U1, U2, and U3 that correspond to "Shibuya" and "Izakaya" will be the targets for information provision (also called "information provision targets"). In Figure 5, information processing system 1 determines that target store CY, which is an interest of user U1, and target store CZ, which is an interest of user U2, will be the targets for information provision to group GP1, which includes users U1, U2, and U3. Information processing system 1 provides information about the targets determined to be information provision targets to each of users U1, U2, and U3.

[0077] In Figure 5, the information processing system 1 provides users U1, U2, and U3 with the message MS2, which reads "It looks like there's a good store nearby!", the information INF21 for the target store CY, and the information INF22 for the target store CZ, respectively. Following the message MS1 entered by user U1, the information processing system 1 displays the message MS2, the information INF21 for the target store CY, and the information INF22 for the target store CZ on the terminal device 10.

[0078] In this way, the information processing system 1 can provide appropriate information to each user of the messaging app by providing them with information about their estimated interests. As mentioned above, the information processing system 1 may also display the user's interests along with the message. For example, the information processing system 1 may acquire information indicating the user's interests, identify the interests of the user that are relevant to the message posted in the messaging app, and then display information indicating the identified interests in the messaging app following the message.

[0079] For example, when a user posts, the information processing system 1 may select from the user's own interests or from the interests of other users. For example, if another user's message is directed at a particular user, the information processing system 1 may display the interests of that user. For example, the information processing system 1 may display information indicating the interests of each user belonging to the message group that are relevant to the message. For example, the information processing system 1 may determine whether a message is directed at everyone, and if so, determine the recipients of the information from among all recipients. For example, if the object of interest is a store, the information processing system 1 may provide information with a link to the store reservation page. For example, if the object of interest is a product, the information processing system 1 may provide information with a link to the product purchase page.

[0080] [1-3. Summary] As described above, when the user makes an operation to instruct the information processing system 1 to generate a captured image, the information processing system 1 captures the information displayed on the screen of the terminal device 10. The information processing system 1 sequentially provides the captured information to the user and accepts the user's operation to indicate whether or not to register the information. For example, the information processing system 1 accepts an operation to swipe the image to one side (e.g., right) as an operation that requires registration, and an operation to swipe the image to the other side (e.g., left) as an operation that does not require registration.

[0081] Information processing system 1 performs image analysis and estimates the user's interests (objects of interest) when the image is captured. For example, information processing system 1 estimates the objects of interest that the user wants to be reminded about.

[0082] Information processing system 1 uses a model that has learned the characteristics of each domain's screen to estimate the domain of the image contained in the captured image. For each estimated domain, information processing system 1 estimates the portion of the image that contains important information. For example, information processing system 1 estimates the title range, image range, description range, etc. For example, information processing system 1 performs character analysis using OCR (Optical Character Recognition) etc. for each range and generates queries. For example, information processing system 1 generates arbitrary queries such as image queries and text queries.

[0083] Information processing system 1 uses queries to estimate corresponding products from products listed on other domains (redirection domains) that correspond to the estimated domain. For example, information processing system 1 identifies products that are of interest to the user and are sold on a different domain than the one the user was browsing at the time of capture.

[0084] Information processing system 1 provides users with identified objects of interest in a manner appropriate to the category to which the object belongs. For example, if the object is food or a product, information processing system 1 provides information about the page where it is sold on the destination domain. For example, if information processing system 1 is providing information during an event on the destination domain, it may automatically post content including a link to the destination domain when the user enters the product name or category in a messaging app.

[0085] If the object of interest is a store, the information processing system 1 may continue to display the location of the store on the map. For example, the information processing system 1 may add a callout to the map and overlay the captured image.

[0086] The information used by the information processing system 1 to estimate the user's interests is not limited to captured images. The information processing system 1 may also sense music and information around the user and use the sensed information to estimate the user's interests. For example, the information processing system 1 may analyze sensed audio information and estimate the user's interests based on the sounds contained in the audio information. In this case, if the sounds contained in the audio information include a product name, the information processing system 1 may estimate that product to be the user's interest. If the sounds contained in the audio information include a store name, the information processing system 1 may estimate that store to be the user's interest. Furthermore, if the sounds contained in the audio information are songs, the information processing system 1 may estimate that songs or the artists associated with those songs to be the user's interests. For example, the information processing system 1 may estimate stores or other facilities of interest based on the user's location and direction of travel. In this case, the information processing system 1 may estimate stores or other facilities located in the direction of travel from the user's location to be the user's interest. The information processing system 1 may estimate that the items sold at stores located in the direction of travel from the user's current location are of interest to that user. For example, the information processing system 1 may estimate the user's interests based on captured images of items purchased at the time of purchase. Through the various processes described above, the information processing device 100 can provide appropriate information.

[0087] [2. Configuration of the Information Processing System] As shown in Figure 6, the information processing system 1 includes a terminal device 10, an external device 20, and an information processing device 100. Multiple terminal devices 10, external devices 20, and information processing devices 100 are connected via a network N, enabling communication by wired or wireless means. Note that the information processing system 1 shown in Figure 6 may include multiple external devices 20 and multiple information processing devices 100.

[0088] The terminal device 10 in this embodiment is a device (mobile terminal) such as a smartphone used by a user. The terminal device 10 is an information processing device used by a user to access content such as web pages displayed in a browser or content for applications. For example, the terminal device 10 may be a desktop PC (Personal Computer), a notebook PC, a tablet terminal, a mobile phone, a PDA (Personal Digital Assistant), etc. The terminal device 10 is not limited to the above examples, and may be, for example, a smartwatch or a wearable device.

[0089] The terminal device 10 accepts various operations from the user. For example, the terminal device 10 accepts various operations from the user via the display surface using a touch panel function. The terminal device 10 may also accept various operations from buttons provided on the terminal device 10 or from a keyboard or mouse connected to the terminal device 10. The terminal device 10 collects various information by storing various information in a storage unit, etc. For example, the terminal device 10 collects the user's activity history and sensor information detected by sensors. The terminal device 10 may transmit the collected information to external devices such as the external device 20 or the information processing device 100. The terminal device 10 provides the information processing device 100 with the user's activity history, such as images captured by the user's operations (captured images).

[0090] Terminal device 10 displays various information. Terminal device 10 displays various information provided by information processing device 100. Terminal device 10 displays various information via application AP11. For example, terminal device 10 accepts an operation indicating the user's evaluation of the captured image displayed by application AP11. Terminal device 10 displays map-related information via application AP12. Terminal device 10 displays message-related information via application AP13.

[0091] Terminal device 10 displays information about the user's interests, estimated using information obtained when the user performs a predetermined operation that indicates the user's interests, at a predetermined time. Terminal device 10 displays second content related to the user's interests, estimated based on first content in which the user performed a predetermined operation that indicates the user's interests. Terminal device 10 displays information about facilities of interest to the user, estimated based on content in which the user performed a predetermined operation that indicates the user's interests.

[0092] The terminal device 10 displays map content that overlays information indicating the user's interests, such as the location of the interest object and the interest object, which are the locations corresponding to the interest object indicated by the interest object information indicating the interest object that the user has shown interest in. Based on the interest object information indicating the interest object that the user has shown interest in and the message information indicating the message in the messenger application, the terminal device 10 displays information about the interest object that is displayed together with the message in the messenger application.

[0093] The external device 20 according to this embodiment is an information processing device that provides various types of information, and is implemented, for example, by a server device or a cloud system. For example, the external device 20 provides user information as various types of information. Another example is that the external device 20 provides information on transaction targets in internet shopping, e-commerce sites, flea market sites, auction sites, travel or restaurant reservation sites, credit card contract sites, financial product provision sites, etc.

[0094] The information processing device 100 according to this embodiment is an information processing device that can communicate with various devices via a predetermined network N such as the Internet, and is implemented, for example, by a server device or a cloud system. For example, the information processing device 100 is connected to other various devices via the network N in a way that allows communication.

[0095] The information processing device 100 uses information obtained when the user performs a predetermined operation that indicates the user's interests to estimate the object of interest that the user has shown interest in, and provides the user with information about the estimated object of interest at a predetermined time. Based on the first content in which the user performs a predetermined operation that indicates the user's interests, the information processing device 100 estimates the object of interest that the user has shown interest in, and provides the user with second content related to the estimated object of interest. Based on the content in which the user performs a predetermined operation that indicates the user's interests, the information processing device 100 estimates the facility of interest that the user has shown interest in, and provides the user with information about the estimated facility of interest.

[0096] The information processing device 100 acquires interest object information that indicates the user's interests, and provides the user with map content that overlays information indicating the interest location, which corresponds to the interest object indicated by the acquired interest object information, and the interest object itself, onto a map. Based on the interest object information that indicates the user's interests, and message information that indicates a message in the messenger application, the information processing device 100 provides the user with information about the interest object that is displayed together with the message in the messenger application.

[0097] As described above, the information processing device 100 may function as a search device that searches for information such as content in a predetermined domain based on a query. For example, the information processing device 100 may function as a search engine that performs search processing on transaction items traded in a predetermined domain. For example, the information processing device 100 has a database (also called the "transaction item database") in which transaction items that are the target of the search processing using queries are indexed and stored, and performs search processing on the information in the transaction item database. For example, the information in the transaction item database is stored in the storage unit 120 (see Figure 6).

[0098] As shown in Figure 1, the information processing device 100 also functions as a search device, meaning that the information processing device 100 and the search device are integrated. However, the information processing device 100 and the search device may be separate entities. In this case, the information processing system 1 (see Figure 6) includes a search device that performs search processing on queries and provides a search service that provides search results. For example, the information processing device 100 sends queries obtained using a query estimation model, etc., to the search device, receives search results from the search device, and sends content generated using those search results to a terminal device 10 used by a user. For example, if the external device 20 is a search device, the information processing device 100 sends queries obtained using a query estimation model, etc., to the external device 20, receives search results from the external device 20, and sends content generated using those search results to a terminal device 10 used by a user.

[0099] The system configuration and processing entity are not limited to the information processing system 1 described above and may take various forms. In the example described above, the system comprises an information processing device 100 as a server that estimates the user's interests and a terminal device 10 as a client. The information processing device 100 and the terminal device 10 were separate entities (separate devices), but the information processing device 100 and the terminal device 10 may be integrated. For example, the terminal device 10 may function as an information processing device that estimates the user's interests. In this case, the information processing system 1 may have a terminal device 10 that also functions as an information processing device that estimates the user's interests and a server device that provides the information used by the terminal device 10 for processing. For example, the server device transmits various information displayed by various applications such as applications AP11, AP12, AP13, etc., to the terminal device 10. The terminal device 10 provides various information to the user by displaying the various information. The system configuration described above is merely an example, and the information processing system 1 may have any device configuration as long as it can provide the user with the desired information.

[0100] [3. Configuration of the Information Processing Device] The following describes an example of the functional configuration of the information processing device 100 described above. Figure 6 is a diagram showing an example of the configuration of the information processing device 100 according to the embodiment. As shown in Figure 6, the information processing device 100 has a communication unit 110, a storage unit 120, and a control unit 130.

[0101] (Communications Department 110) The communication unit 110 is implemented, for example, by a NIC (Network Interface Card). The communication unit 110 is connected to the network N by wire or wireless connection and transmits and receives information with various other devices.

[0102] (Storage unit 120) The storage unit 120 is implemented by, for example, semiconductor memory elements such as RAM (Random Access Memory) and flash memory, or by storage devices such as hard disks and optical discs. The storage unit 120 also includes a user information storage unit 121, a target information storage unit 122, a content storage unit 123, and a model information storage unit 124. The storage unit 120 is not limited to storing the above and stores various types of information used for information provision.

[0103] For example, the memory unit 120 stores information (application programs, etc.) related to various applications such as app AP11, app AP12, and app AP13. For example, the memory unit 120 stores information (image information, etc.) related to a map (e.g., map MP1, etc.) displayed by app AP12.

[0104] (User information storage unit 121) The user information storage unit 121 stores various user information about the user. Figure 7 shows an example of the user information storage unit according to the embodiment. In the example shown in Figure 7, the user information storage unit 121 has items such as "User ID" and "User Information". For example, "User Information" includes items such as "Attribute Information", "Behavioral History", and "Points of Interest".

[0105] The "User ID" is an identifier that identifies a user. "Attribute Information" is attribute information relating to the user's attributes, which is associated with the "User ID". For example, attribute information may include the user's age, gender, telephone number, address, etc. Note that the attribute information described above is merely an example, and may include various other types of information, such as demographic attributes like age and gender, or psychographic attributes like interests and lifestyle.

[0106] The "activity history" is the activity history of a user associated with a "user ID". The activity history includes an operation history that associates captured images of content displayed on the terminal device 10 through user operation with information such as the content corresponding to the captured image and the date and time the captured image was created. The activity history may also include various sensor information detected regarding the user. The activity history may also include a history of location information detected by the terminal device 10.

[0107] Furthermore, the activity history includes various types of information such as purchase history and browsing history of content viewed by the user. For example, purchase history includes information about the items purchased by the user, the type of item, the number of times the item was purchased, and the date and time the item was purchased. For example, browsing history may be a history of content displayed on the terminal device 10 used by the user. For example, browsing history includes information about the content viewed (displayed) by the user, the type of content, the number of times the content was displayed, and the date and time the content was displayed.

[0108] A "target of interest" is an object of interest (target of interest) associated with a "user ID". For example, a list of the user's interests is stored in the "target of interest" section. For example, the list of interests includes information to identify each of the user's targets of interest, and information regarding the estimated date and time when that target became of interest to the user.

[0109] For example, in Figure 7, "U1," identified by user ID, has attribute information as "CH1," behavioral history as "PH1," and interest as "WA1." In the example shown in Figure 7, attribute information is represented by abstract codes such as "CH1," but attribute information may also be specific numerical values, specific strings, or file formats containing various types of information.

[0110] Furthermore, the user information storage unit 121 may store various types of information depending on the purpose, not limited to those mentioned above. The user information storage unit 121 may also store information for each user indicating the number of times the information has been displayed for each target, such as the number of times the user has viewed it.

[0111] (Target information storage unit 122) The target information storage unit 122 stores various types of target information relating to various objects, including trading targets such as goods and facilities such as stores. Figure 8 shows an example of the target information storage unit according to the embodiment. In the example shown in Figure 8, the target information storage unit 122 includes items such as "target ID," "target," "target information," "category," and "corresponding location information." In Figure 8, "category" and "corresponding location information" are explained as separate items from "target information," but category information and location information may be included in the target information.

[0112] "Target ID" is an identifier that identifies the target. "Target" refers to the target associated with the "Target ID". "Target Information" is the target information of the target identified by the "Target ID".

[0113] "Category" refers to information about the category to which the target identified by "Target ID" belongs. "Corresponding Location Information" refers to information about the product to which the target identified by "Target ID" belongs.

[0114] For example, in Figure 8, "M1," identified by the target ID, represents the target "MA1." While the example in Figure 8 uses an abstract code like "MA1," the target is information that can identify an object of interest to the user. For example, the target could be a specific string indicating a facility such as a store (e.g., store). For example, the target could be a specific string indicating a transaction target such as a product (e.g., product name).

[0115] Furthermore, in Figure 8, "M1," identified by the target ID, represents target information as "MD1." Note that in the example shown in Figure 8, the target information is represented by an abstract code such as "MD1," but the target information may also be a file format containing various information about the target.

[0116] Furthermore, in Figure 8, "M1," identified by the target ID, has the category "CT1." Note that in the example shown in Figure 8, the category is represented by an abstract code such as "CT1," but the category is information that indicates the category of the target.

[0117] In Figure 8, "M1," identified by the target ID, corresponds to location information "PD1." While the example in Figure 8 uses an abstract code like "PD1," location information can also represent specific locations such as latitude and longitude (latitude / longitude information) or addresses like "X Prefecture, Y City, Z Town, XX." For example, if the target is a store or other facility, the location information indicates the location of that facility. Similarly, if the target is a product or other transaction, the location information indicates the location of the store or other facility where that product is sold (provided).

[0118] The target information storage unit 122 is not limited to the above and may store various types of information depending on the purpose. For example, the target information storage unit 122 may store information such as the score and the number of times each target is displayed.

[0119] (Content storage unit 123) The content storage unit 123 stores information about the content. Figure 9 shows an example of a content storage unit according to this embodiment. In the example shown in Figure 9, the content storage unit 123 has items such as "Content ID" and "Content".

[0120] A "Content ID" is an identifier that identifies the content. "Content" refers to information about the content associated with the "Content ID." Specifically, "Content" may refer to information about the content itself.

[0121] For example, the content could be content related to a portal site. Other examples include content related to news sites, auction sites, weather forecast sites, shopping sites, finance (stock price) sites, etc.

[0122] Furthermore, the content may also be related to route search websites, map provision websites, travel websites, restaurant review websites, web blog sites, social networking sites, etc.

[0123] For example, in Figure 9, "CN1," identified by its content ID, represents content "CO1." Note that in the example shown in Figure 9, content is represented by abstract codes such as "CO1," but content may also be specific numerical values, specific strings, or file formats containing various information.

[0124] The content storage unit 123 may store various types of information depending on the purpose, not limited to those described above.

[0125] (Model information storage unit 124) The model information storage unit 124 stores information about the model. For example, the model information storage unit 124 stores information (model data) of a trained model (model) that has been trained (generated) through the training process. The model information storage unit 124 stores the data used for training (training data) in association with the trained model (model). Figure 10 is a diagram showing an example of the model information storage unit according to the embodiment. In the example shown in Figure 10, the model information storage unit 124 includes items such as "model ID", "purpose", "model data", and "training data". In the example in Figure 10, the model information storage unit 124 stores the data used for training (training data) in association with the trained model (model).

[0126] The "Model ID" indicates identification information for identifying the model. The "Purpose" indicates the purpose of the corresponding model. The "Model Data" indicates the data of the model. Figure 10 shows an example where conceptual information such as "MDT1" is stored in "Model Data," but in reality, it includes various information that constitutes the model, such as information about the model's configuration (network configuration) and parameters. For example, "Model Data" includes information such as the nodes in each layer of the network, the functions adopted by each node, the connection relationships between nodes, and the connection coefficients set for the connections between nodes.

[0127] "Training data" refers to the data used to train a trained model (model). "Training data" stores information indicating the dataset used to train the corresponding model. For example, "training data" stores data (input information) and the corresponding correct answer information (output information) as training data (also called "training data"). Figure 10 shows an example where conceptual information such as "LDT1" is stored in "training data," but in reality, it includes various information about the data used to train the corresponding model, such as data (input information) and the corresponding correct answer information (output information).

[0128] Figure 10 shows that the model identified by model ID "M1" (model M1) is intended for "domain estimation." Model M1 is a model for estimating the domain of content. For example, model M1 outputs information for classifying which domain the content corresponding to the input data belongs to (e.g., a score corresponding to each domain). It also shows that the model data for model M1 is model data MDT1. Furthermore, it shows that the training data used to train model M1 is training data LDT1.

[0129] The model identified by model ID "M11" (model M11) indicates that its purpose is "Query Estimation (Domain A)". Model M11 is a model that accurately extracts information corresponding to Domain A. For example, model M11 is a model that outputs information showing the results of image analysis after analyzing an input image. Model M11 extracts and outputs information used as a query (e.g., the name of the target) from images corresponding to the content of Domain A. For example, model M11 indicates that it is a model that outputs information used for queries extracted from an input image. Furthermore, it indicates that the model data for model M11 is model data MDT11. Furthermore, it indicates that the training data used to train model M11 is training data LDT11.

[0130] The model identified by model ID "M12" (model M12) indicates that its purpose is "Query Estimation (Domain B)". Model M12 is a model that accurately extracts information corresponding to Domain B. For example, model M12 is a model that outputs information showing the results of image analysis after analyzing an input image. Model M12 extracts and outputs information used as a query (e.g., the name of the target) from images corresponding to the content of Domain B. For example, model M12 indicates that it is a model that outputs information used for queries extracted from an input image. Furthermore, it indicates that the model data for model M12 is model data MDT12. Furthermore, it indicates that the training data used to train model M12 is training data LDT12.

[0131] The model identified by model ID "M13" (model M13) indicates that its purpose is "query estimation (domain C)". Model M13 is a model that accurately extracts information corresponding to domain C. For example, model M13 is a model that outputs information showing the results of image analysis after analyzing an input image. Model M13 extracts and outputs information used as a query (e.g., the name of the target) from images corresponding to the content of domain C. For example, model M13 indicates that it is a model that outputs information used for queries extracted from an input image. Furthermore, it indicates that the model data for model M13 is model data MDT13. Furthermore, it indicates that the training data used to train model M13 is training data LDT13.

[0132] Although Figure 10 illustrates up to Model M13, the model information storage unit 124 may store models that estimate (predict) information corresponding to various domains, such as Model M14 which targets Domain D.

[0133] The model information storage unit 124 is not limited to the above and may store various types of information depending on the purpose. For example, the model information storage unit 124 may contain a model (object recognition model) used to recognize objects, etc., included in the input image. For example, the object recognition model may output information indicating objects in the input image (object name, etc.) and information indicating the location of objects (coordinate information, etc.).

[0134] (Control unit 130) The control unit 130 is a controller, and is implemented, for example, by a CPU (Central Processing Unit) or MPU (Micro Processing Unit) executing various programs (an example of an information processing program) stored in the memory device inside the information processing device 100 using RAM as the working area. Alternatively, the control unit 130 is a controller and can be implemented, for example, by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array).

[0135] As shown in Figure 6, the control unit 130 includes an acquisition unit 131, a reception unit 132, a determination unit 133, an estimation unit 134, a generation unit 135, and a provision unit 136, and realizes or executes the information processing functions and operations described below. Note that the internal configuration of the control unit 130 is not limited to the configuration shown in Figure 6, and other configurations are also acceptable as long as they perform the information processing described later.

[0136] (Acquisition part 131) The acquisition unit 131 acquires various types of information. The acquisition unit 131 acquires various types of information from the storage unit 120. The acquisition unit 131 acquires various types of information from the user information storage unit 121, the target information storage unit 122, the content storage unit 123, and the model information storage unit 124, etc.

[0137] The acquisition unit 131 receives various information from an external information processing device via the communication unit 110. The acquisition unit 131 also receives various information from the terminal device 10 or the external device 20. For example, the acquisition unit 131 acquires user information about the user from the external device 20.

[0138] The acquisition unit 131 acquires information when the user performs a predetermined operation that indicates the user's interests. The acquisition unit 131 acquires information indicating the content for which the predetermined operation indicating the user's interests was performed. The acquisition unit 131 acquires a captured image of the content for which the predetermined operation indicating the user's interests was performed. The acquisition unit 131 acquires a captured image generated by the user's operation.

[0139] The acquisition unit 131 acquires interest object information that indicates the object of interest that the user has shown interest in. The acquisition unit 131 acquires message information that indicates messages in a messenger application. The acquisition unit 131 acquires interest object information that indicates the transaction object or facility that the user has shown interest in. The acquisition unit 131 acquires interest object information that indicates the product or store that the user has shown interest in.

[0140] The acquisition unit 131 acquires sensor information when the user performs a predetermined operation. The acquisition unit 131 also acquires the user's location information when the user performs a predetermined operation.

[0141] The acquisition unit 131 may also acquire user information from the external device 20, such as user attribute information, purchase history on internet shopping sites, e-commerce sites, auction sites, or flea market sites, and behavioral history such as content browsing history. The acquisition unit 131 then stores such user information in the user information storage unit 121. For example, the acquisition unit 131 acquires user information from the external device 20 at predetermined intervals and updates the user information stored in the user information storage unit 121.

[0142] Furthermore, the acquisition unit 131 may acquire information about various objects such as trading items and facilities from the external device 20. For example, the acquisition unit 131 acquires information about trading items traded on an auction site or a flea market site from the external device 20. The acquisition unit 131 then stores such information about trading items in the object information storage unit 122. For example, the acquisition unit 131 acquires information about facilities such as stores from the external device 20. The acquisition unit 131 then stores such information about facilities such as stores in the object information storage unit 122.

[0143] (Reception desk 132) The reception unit 132 receives various requests. The reception unit 132 receives various requests from external information processing devices. The reception unit 132 receives information indicating various requests from external information processing devices via the communication unit 110. For example, the reception unit 132 receives requests from terminal devices 10 or external devices 20. The reception unit 132 receives information requests from terminal devices 10. For example, the reception unit 132 receives information requests from terminal devices 10 for information displayed in various applications such as apps AP11, AP12, AP13, etc.

[0144] (Decision Section 133) The decision unit 133 performs decision processing to determine various information. For example, the decision unit 133 performs decision processing based on various information acquired by the acquisition unit 131. The decision unit 133 performs decision processing based on various information stored in the storage unit 120. For example, the decision unit 133 performs decision processing based on various information received from an external information processing device. The decision unit 133 performs extraction processing to extract various information. For example, the decision unit 133 performs extraction processing based on various information acquired by the acquisition unit 131. The decision unit 133 performs extraction processing based on various information stored in the storage unit 120. For example, the decision unit 133 performs extraction processing based on various information received from an external information processing device.

[0145] For example, the decision unit 133 performs a decision process based on the information estimated by the estimation unit 134. For example, the decision unit 133 performs a decision process based on the information generated by the generation unit 135.

[0146] The decision unit 133 determines the timing of information provision. The decision unit 133 determines the timing according to the object of interest. If the object of interest is a product, the decision unit 133 determines that the timing of information provision is when the user becomes able to purchase the product. The decision unit 133 determines that it is time to provide information when the user becomes able to purchase the product. The decision unit 133 also determines that it is time to provide information when the amount of electronic money or points held by the user exceeds a predetermined value.

[0147] The decision unit 133 determines that the timing for providing information is when the user approaches a store, if the object of interest is a store. The decision unit 133 determines that it is time to provide information when the user approaches a store. The decision unit 133 determines that it is time to provide information when the user's location is within a predetermined range from the store.

[0148] The decision unit 133 determines the domain to be the target of information provision. The decision unit 133 determines the second content to be provided. The decision unit 133 determines the second domain based on predetermined criteria. The decision unit 133 determines the second domain to be a domain similar to the first domain of the first content. The decision unit 133 determines the second content based on predetermined criteria. The decision unit 133 determines the second content to be content that shows the target corresponding to the target of the first content. The decision unit 133 determines the second content to be content that shows the same product as the product of the first content. The decision unit 133 determines the second domain according to the user. The decision unit 133 determines the second domain to be a domain corresponding to the service the user is using.

[0149] The decision unit 133 determines whether or not to provide information about the user's interests based on the message indicated by the message information and the user's interests. The decision unit 133 determines whether or not to provide information about the user's interests based on a comparison between the message indicated by the message information and the user's interests. If the message indicated by the message information includes the user's interests, the decision unit 133 decides to provide information about the user's interests. The decision unit 133 determines information about the user's interests corresponding to the message from the user. The decision unit 133 determines information about the user's interests corresponding to messages from other users among multiple users, including the user.

[0150] (Estimation part 134) The estimation unit 134 performs estimation processing to estimate various types of information. For example, the estimation unit 134 performs estimation processing based on various types of information acquired by the acquisition unit 131. The estimation unit 134 performs estimation processing based on various types of information stored in the storage unit 120. The estimation unit 134 performs estimation processing based on various types of information received from an external information processing device.

[0151] For example, the estimation unit 134 performs estimation processing based on the information determined by the determination unit 133. For example, the estimation unit 134 performs estimation processing based on the information generated by the generation unit 135.

[0152] The estimation unit 134 estimates the object of interest of the user, which is the object that the user has shown interest in, using information from when the user performs a predetermined operation that indicates the user's interests. The estimation unit 134 estimates the object of interest of the user based on the content in which the predetermined operation that indicates the user's interests was performed. The estimation unit 134 estimates the object of interest of the user based on the object included in the content.

[0153] The estimation unit 134 estimates the user's interests using the captured image generated by the user's operation. The estimation unit 134 estimates the user's interests based on the objects included in the captured image. The estimation unit 134 estimates the user's interests based on the results of the analysis process that analyzes the captured image.

[0154] The estimation unit 134 estimates the user's interests based on the domain of the captured image. The estimation unit 134 estimates the user's interests based on the domain estimated using a domain estimation model that estimates the domain of the captured image. The estimation unit 134 estimates the domain of the captured image using a domain estimation model that takes the screen as input and outputs information indicating the domain of the input image.

[0155] The estimation unit 134 estimates the user's interests using sensor information obtained when the user performs a predetermined operation. The estimation unit 134 estimates the user's interests using the user's location information obtained when the user performs a predetermined operation. The estimation unit 134 estimates transaction targets or facilities as the user's interests. The estimation unit 134 estimates products or stores as the user's interests.

[0156] The estimation unit 134 estimates facilities of interest that the user has shown interest in, based on content in which the user has performed a predetermined operation indicating the user's interests. The estimation unit 134 estimates facilities of interest that the user has shown interest in, based on stores that the user has shown interest in. The estimation unit 134 estimates facilities of interest that the user has shown interest in, based on content in a predetermined service. The estimation unit 134 estimates facilities of interest that the user has shown interest in, based on content posted on social networking services (SNS). The estimation unit 134 estimates facilities of interest that the user has shown interest in, based on content posted by other users.

[0157] The estimation unit 134 estimates the user's interests based on the objects included in the content. The estimation unit 134 estimates the user's interests using the captured image generated by the user's operation. The estimation unit 134 estimates the user's interests based on the objects included in the captured image. The estimation unit 134 estimates the user's interests based on the results of the analysis process that analyzes the captured image.

[0158] The estimation unit 134 estimates the user's interests and facilities based on the domain of the captured image. The estimation unit 134 estimates the user's interests and facilities based on the domain estimated using a domain estimation model that estimates the domain of the captured image. The estimation unit 134 estimates the domain of the captured image using a domain estimation model that takes the screen as input and outputs information indicating the domain of the input image. The estimation unit 134 estimates the user's interests and facilities using sensor information when the user performs a predetermined operation. The estimation unit 134 estimates the user's interests and facilities using the user's location information when the user performs a predetermined operation.

[0159] The estimation unit 134 estimates the object of interest that the user has shown interest in, based on the first content in which the user has performed a predetermined operation that indicates the user's interests. The estimation unit 134 estimates the user's object of interest based on the object included in the first content. The estimation unit 134 estimates the user's object of interest using a captured image of the first content generated by the user's operation.

[0160] (Generation unit 135) The generation unit 135 performs generation processing to estimate various types of information. For example, the generation unit 135 performs generation processing based on various types of information acquired by the acquisition unit 131. The generation unit 135 performs generation processing based on various types of information stored in the storage unit 120. The generation unit 135 performs generation processing based on various types of information received from an external information processing device.

[0161] For example, the generation unit 135 executes a generation process based on the information determined by the determination unit 133. For example, the generation unit 135 executes a generation process based on the information generated by the generation unit 135.

[0162] The generation unit 135 generates information to be provided to the user. The generation unit 135 generates map content in which information indicating the interest location, which is the location corresponding to the interest object indicated by the interest object information, and the interest object itself are superimposed on a map. The generation unit 135 generates map content including an icon indicating the interest location. The generation unit 135 generates map content including an image indicating the interest object. The generation unit 135 generates map content including text information indicating the interest object.

[0163] The generation unit 135 generates map content that includes information indicating the objects of interest estimated by the estimation unit 134. The generation unit 135 generates map content in which information indicating the location of interest corresponding to the object of interest, which is a transaction target or facility, is superimposed on the map. The generation unit 135 generates map content in which information indicating the location of interest corresponding to the object of interest, which is a product or store, is superimposed on the map.

[0164] The generation unit 135 generates information about the transaction subject or facility that will be displayed along with the message in the messenger application. The generation unit 135 also generates information about the product or store that will be displayed along with the message in the messenger application.

[0165] The generation unit 135 generates various types of information, such as content (map content), to be provided to an external information processing device, using various technologies as appropriate. The generation unit 135 generates information such as content to be provided to the terminal device 10. For example, the generation unit 135 generates information such as content to be provided to the terminal device 10 based on the information stored in the storage unit 120. The generation unit 135 may generate information such as content by any processing method, as long as it is possible to generate information such as content to be provided to an external information processing device. For example, the generation unit 135 generates information such as content to be provided to the terminal device 10 by using various technologies such as image generation and image processing as appropriate. For example, the generation unit 135 generates information such as content to be provided to the terminal device 10 by using various technologies such as Java (registered trademark) as appropriate. The generation unit 135 may also generate information such as content to be provided to the terminal device 10 based on CSS, JavaScript (registered trademark), or HTML format. Furthermore, for example, the generation unit 135 may generate content and other information in various formats such as JPEG (Joint Photographic Experts Group), GIF (Graphics Interchange Format), and PNG (Portable Network Graphics).

[0166] The generation unit 135 generates various types of information to be provided to the user. The generation unit 135 generates content such as map content MP1. The generation unit 135 generates various types of information such as information for provision INF11, INF21, and INF22.

[0167] Furthermore, the generation unit 135 may function as a learning unit that executes a learning process to learn a learning model (model). In this case, the generation unit 135 executes the learning process based on various information acquired by the acquisition unit 131. The generation unit 135 executes the learning process based on information from an external information processing device or information stored in the storage unit 120. The generation unit 135 executes the learning process based on information stored in the model information storage unit 124. The generation unit 135 stores the model generated by learning in the model information storage unit 124.

[0168] The generation unit 135 learns a domain estimation model for estimating the domain of the content. The generation unit 135 learns a query estimation model for estimating (extracting) information to be used as queries within the content.

[0169] The generation unit 135 performs learning processing. The generation unit 135 performs various types of learning. The generation unit 135 learns various types of information based on the information acquired by the acquisition unit 131. The generation unit 135 learns (generates) a model. The generation unit 135 learns various types of information such as the model. The generation unit 135 generates a model through learning. The generation unit 135 learns the model using various machine learning techniques. For example, the generation unit 135 learns the parameters of the model (network). The generation unit 135 learns the model using various machine learning techniques.

[0170] The generation unit 135 generates various learning models such as models M1, M11, M12, and M13. The generation unit 135 learns the network parameters. For example, the generation unit 135 learns the network parameters of various learning models such as models M1, M11, M12, and M13. The generation unit 135 generates various learning models such as models M1, M11, M12, and M13 by performing a learning process using the learning data stored in the model information storage unit 124. For example, the generation unit 135 generates a model used for speech recognition. The generation unit 135 generates various learning models such as models M1, M11, M12, and M13 by learning the network parameters of various learning models such as models M1, M11, M12, and M13.

[0171] The generation unit 135 performs learning processing based on the training data (teacher data) stored in the model information storage unit 124. The generation unit 135 generates various learning models such as models M1, M11, M12, and M13 by performing learning processing using the training data stored in the model information storage unit 124.

[0172] For example, the generation unit 135 generates a model used to estimate the probability (possibility) of a store leaving the site. For example, when store information is input, the generation unit 135 generates a model that outputs a score indicating the probability (possibility) that the store corresponding to that store information will leave the e-commerce site. The generation unit 135 generates model M1 by learning the network parameters of model M1.

[0173] The learning method used by the generation unit 135 is not particularly limited, but for example, training data may be prepared by linking data (input information) with correct answer information (output information), and this training data may be input into a computational model based on a multilayer neural network for learning.

[0174] For example, when the generation unit 135 trains model M1, it uses the training data LDT1 stored in the model information storage unit 124 to train model M1. For example, the training data LDT1 includes data that associates captured images of content with information indicating the domain to which that content belongs (ground truth information).

[0175] For example, when the generation unit 135 inputs a captured image to model M1, it performs a learning process so that model M1 outputs the domain to which the captured image is associated. When the generation unit 135 inputs a captured image C1 to model M1, it performs a learning process so that among the scores of each domain output by model M1, the score of the domain to which the captured image C1 is associated (for example, domain A) becomes larger. When the generation unit 135 inputs a captured image C1 to model M1, it performs a learning process so that among the scores of each domain output by model M1, the scores of domains other than domain A become smaller. For example, when the generation unit 135 inputs a captured image C1 to model M1, it performs a learning process so that the score of domain A output by model M1 approaches 1. For example, when the generation unit 135 inputs a captured image C1 to model M1, it performs a learning process so that the scores of domains other than domain A output by model M1 approach 0.

[0176] Furthermore, the generation unit 135 may learn query estimation models corresponding to each domain, such as models M11, M12, and M13. For example, when the generation unit 135 learns model M11, it learns model M11 using the learning data LDT11 stored in the model information storage unit 124. For example, the learning data LDT11 includes data that associates captured images of content corresponding to domain A with information indicating the target in the captured image (ground truth information). The ground truth information is information that indicates the target used in the query from the captured image. The ground truth information is information about strings indicating the transaction target, such as product names, and strings indicating facilities, such as store names. For example, the ground truth information may be strings used as queries in the captured image and the region where those strings are located.

[0177] For example, when a captured image is input to the model M11, the generation unit 135 performs a learning process using an arbitrary learning method so that the information output by the model M11 becomes the correct information (label) associated with that captured image. For example, the generation unit 135 learns a model M11 that is suited to domain A by learning using the learning data LDT11, which associates captured images of content corresponding to domain A with their correct information.

[0178] For example, when the generation unit 135 receives a captured image of content corresponding to domain A as input to model M11, it performs a learning process so that model M11 outputs information indicating the target indicated by the correct answer information associated with that captured image. Note that any learning process can be used for model M11 as long as it is possible to train a model that extracts desired information from an image. This may be the same as conventional learning methods such as image recognition and optical character recognition (OCR), and a detailed explanation is omitted.

[0179] Furthermore, the generation unit 135 may also train models M12, M13, etc., in the same way as model M11. For example, when training model M12, the generation unit 135 trains model M12 using the training data LDT12 stored in the model information storage unit 124. For example, the training data LDT12 includes data that associates captured images of content corresponding to domain B with information indicating the target in the captured image (ground truth information). For example, the generation unit 135 trains model M12 that is suitable for domain B by training using the training data LDT12 which associates captured images of content corresponding to domain B with their ground truth information.

[0180] For example, when the generation unit 135 trains model M13, it uses the training data LDT13 stored in the model information storage unit 124 to train model M13. For example, the training data LDT13 includes data that associates captured images of content corresponding to domain C with information indicating the target in the captured images (ground truth information). For example, the generation unit 135 trains model M13 that is suited to domain C by training using the training data LDT13 which associates captured images of content corresponding to domain C with their ground truth information.

[0181] Furthermore, the learning methods for each model are not limited to those described above, and any publicly known techniques can be applied. The generation of each model may be carried out using various conventional machine learning techniques as appropriate. For example, the model may be generated using supervised learning machine learning techniques such as SVM (Support Vector Machine). Alternatively, the model may be generated using unsupervised learning machine learning techniques. For example, the model may be generated using deep learning techniques. For example, the model may be generated using various deep learning techniques such as DNN (Deep Neural Network), RNN (Recurrent Neural Network), and CNN (Convolutional Neural Network) as appropriate. The above descriptions regarding model generation are illustrative examples, and the model may be generated using a learning method appropriately selected according to the available information. For example, the generation unit 135 may generate models M1, M11, M12, M13, etc. by any method, as long as it is possible to train models M1, M11, M12, M13, etc. to output information corresponding to the correct answer when an image included in the training data is input.

[0182] (Providing Department 136) The information provision unit 136 provides various information. The information provision unit 136 transmits various information to an external information processing device via the communication unit 110. The information provision unit 136 transmits various information to the terminal device 10 or the external device 20. The information provision unit 136 transmits content to the terminal device 10. The information provision unit 136 transmits information related to the transaction subject to the terminal device 10.

[0183] The provisioning unit 136 provides the user with information about the object of interest estimated by the estimation unit 134 at a predetermined time. The provisioning unit 136 provides information according to the decision made by the decision unit 133. The provisioning unit 136 provides information if the decision unit 133 decides to provide it. The provisioning unit 136 provides information at the timing determined by the decision unit 133.

[0184] The information provider 136 provides information about the subject of interest at a timing appropriate to the subject of interest. If the subject of interest is a product, the information provider 136 provides information about the subject of interest when the product is available for purchase. If the subject of interest is a store, the information provider 136 provides information about the subject of interest when the user is near the store. The information provider 136 displays the information about the subject of interest on the terminal device 10 used by the user. The information provider 136 transmits the information about the subject of interest to the terminal device 10 used by the user.

[0185] The provision unit 136 provides the user with information about facilities of interest estimated by the estimation unit 134. The provision unit 136 displays the information about facilities of interest on the terminal device 10 used by the user. The provision unit 136 transmits the information about facilities of interest to the terminal device 10 used by the user.

[0186] The provisioning unit 136 provides the user with second content related to the interests estimated by the estimation unit 134. The provisioning unit 136 provides second content in a predetermined domain. The provisioning unit 136 provides second content in a second domain which is a predetermined domain different from the first domain of the first content. The provisioning unit 136 provides second content in a second domain determined based on predetermined criteria. The provisioning unit 136 provides second content in a second domain determined according to the user. The provisioning unit 136 displays the second content on the terminal device 10 used by the user. The provisioning unit 136 transmits the second content to the terminal device 10 used by the user.

[0187] The providing unit 136 provides map content generated by the generating unit 135. The providing unit 136 provides the user with map content that overlays information indicating the object of interest and the location of interest, which is the location corresponding to the object of interest indicated by the object of interest information, onto the map. The providing unit 136 provides map content that includes an icon indicating the location of interest. The providing unit 136 provides map content that includes an image indicating the object of interest. The providing unit 136 provides map content that includes text information indicating the object of interest.

[0188] The provisioning unit 136 provides map content that includes information indicating the objects of interest estimated by the estimation unit 134. The provisioning unit 136 provides map content that overlays information indicating the location of interest corresponding to the object of interest, which is a transaction target or facility, onto the map. The provisioning unit 136 provides map content that overlays information indicating the location of interest corresponding to the object of interest, which is a product or store, onto the map. The provisioning unit 136 displays the map content on the terminal device 10 used by the user. The provisioning unit 136 transmits the map content to the terminal device 10 used by the user.

[0189] If the decision unit 133 determines that it is appropriate to provide information about the object of interest, the provision unit 136 provides the user with information about the object of interest that will be displayed along with the message in the messenger application. Based on the information about the object of interest and the message information, the provision unit 136 provides the user with information about the object of interest that will be displayed along with the message in the messenger application.

[0190] The providing unit 136 provides information about the transaction object or facility generated by the generating unit 135. The providing unit 136 provides information about the product or store generated by the generating unit 135. The providing unit 136 provides information about the object of interest corresponding to the message indicated by the message information. The providing unit 136 provides information about the object of interest corresponding to the message from the user. The providing unit 136 provides information about the object of interest corresponding to the message from other users among multiple users, including the user.

[0191] The provisioning unit 136 provides information about the transaction subject or facility that is displayed along with the message in the messenger application. The provisioning unit 136 provides information about the product or store that is displayed along with the message in the messenger application. The provisioning unit 136 displays information about the subject of interest along with the message on the terminal device used by the user. The provisioning unit 136 transmits the information about the subject of interest displayed along with the message to the terminal device used by the user.

[0192] The providing unit 136 provides information generated by the generating unit 135. The providing unit 136 provides content containing the information generated by the generating unit 135. The providing unit 136 transmits the content to the terminal device 10 used by the user. The providing unit 136 transmits information to be displayed by the application AP 11 to the terminal device 10.

[0193] The service provider 136 transmits information to be displayed by the application AP12 to the terminal device 10. The service provider 136 transmits map content MP1 to the terminal device 10. The service provider 136 transmits map content MP1 with information about Cafe XX, which is of interest to the user, to the terminal device 10 used by that user. The service provider 136 transmits map content MP1 with content CO11, which includes images and text of Cafe XX, superimposed to the terminal device 10.

[0194] The provisioning unit 136 transmits information to be displayed in the application AP13 to the terminal device 10. The provisioning unit 136 transmits content CO21, CO22, etc., which will be displayed in the application AP13 along with message MS1, etc., to the terminal device 10 used by that user. The provisioning unit 136 transmits content CO21, CO22, etc., which includes information about izakayas in Shibuya that are of interest to the user, to the terminal device 10 used by that user.

[0195] [4. Processing Procedure] Next, the information processing procedure performed by the information processing device 100 according to the embodiment will be described using Figures 11 to 15. Figures 11 to 15 are flowcharts illustrating an example of the information processing flow. For example, Figures 11 to 15 are flowcharts illustrating an example of the information processing flow performed by the information processing device.

[0196] First, let's explain Figure 11. For example, Figure 11 shows an example of a processing flow related to information provision based on the estimation of the user's interests performed by the information processing device 100.

[0197] In Figure 11, the information processing device 100 uses information obtained when the user performs a predetermined operation that indicates the user's interests to estimate the object of interest that the user has shown interest in (step S101). Then, the information processing device 100 provides the user with information about the object of interest at a predetermined timing (step S102).

[0198] Next, Figure 12 will be explained. For example, Figure 12 shows an example of a processing flow for providing second content in accordance with the estimation results based on the first content performed by the information processing device 100.

[0199] In Figure 12, the information processing device 100 estimates the object of interest that the user has shown interest in, based on the first content in which the user has performed a predetermined operation that indicates the user's interests (step S201). Then, the information processing device 100 provides the user with second content related to the object of interest (step S202).

[0200] Next, Figure 13 will be explained. For example, Figure 13 shows an example of a processing flow related to information provision based on the estimation of the user's interests and preferred facilities performed by the information processing device 100.

[0201] In Figure 13, the information processing device 100 estimates the facilities of interest of the user based on the content in which the user has performed a predetermined operation that indicates the user's interests (step S301). Then, the information processing device 100 provides the user with information regarding the facilities of interest (step S302).

[0202] Next, Figure 14 will be explained. For example, Figure 14 shows an example of a processing flow related to the provision of map content based on the user's interests by the information processing device 100.

[0203] In Figure 14, the information processing device 100 acquires interest object information, which indicates the object of interest that the user has shown interest in (step S401). The information processing device 100 then provides the user with map content that overlays information indicating the interest location, which is the location corresponding to the interest object indicated by the interest object information, and the interest object itself, onto a map (step S402).

[0204] Next, Figure 15 will be explained. For example, Figure 15 shows an example of a processing flow related to the provision of information content based on the user's interests and messages in the messenger application, performed by the information processing device 100.

[0205] In Figure 15, the information processing device 100 acquires interest object information, which indicates the object of interest that the user has shown interest in, and message information, which indicates a message in the messenger application (step S501). Then, based on the interest object information and message information, the information processing device 100 provides the user with information about the interest object that will be displayed together with the message in the messenger application (step S502).

[0206] [5. Variations] The information processing device 100 described above may be implemented in various other forms besides those described above. Therefore, other embodiments of the information processing device 100 will be described below.

[0207] [5-1. User] The users to whom information is provided can be any user whose interests can be estimated. For example, the users to whom information is provided can be users who are registered as members of the specified service, or users who are not registered as members of the specified service.

[0208] [5-2. Target] In the example above, products and stores were used as examples of objects of interest for users, but the estimated objects of interest are not limited to products and stores; they can be any object. For example, the objects of interest could be services used by the user, or facilities such as stadiums that serve as event venues. The above is merely an example, and the objects of interest can be anything that is of interest to the user.

[0209] [5-3. Provide information regarding the subject of the transaction] In the above embodiment, an example of information processing in which the information processing device 100 provides various types of information such as content to the terminal device 10 was described, but the embodiment is not limited to the above example. For example, an external server may provide various types of information such as content to the terminal device 10. In this case, the information processing device 100 may provide the external server with information related to the user's interests.

[0210] For example, an external server provides a service that sends reminders (notifications) about the user's interests. In this case, the information processing device 100 may provide the external server with information about the user's interests, such as images of the interests and information specifying the timing for sending reminders to the user about the interests.

[0211] [5-4. Program] Furthermore, the information processing device 100 according to the above-described embodiment is realized by a computer 1000 having a configuration as shown in Figure 16. Figure 16 is a diagram showing an example of a hardware configuration. The computer 1000 is connected to an output device 1010 and an input device 1020, and has a configuration in which an arithmetic unit 1030, a primary storage device 1040, a secondary storage device 1050, an output IF (Interface) 1060, an input IF 1070, and a network IF 1080 are connected by a bus 1090.

[0212] The arithmetic unit 1030 operates based on programs stored in the primary storage device 1040 and the secondary storage device 1050, as well as programs read from the input device 1020, and executes various processes. The primary storage device 1040 is a memory device, such as RAM, that temporarily stores data used by the arithmetic unit 1030 for various calculations. The secondary storage device 1050 is a storage device where data used by the arithmetic unit 1030 for various calculations and various databases are registered, and is implemented using ROM (Read Only Memory), HDD (Hard Disk Drive), flash memory, etc.

[0213] Output IF1060 is an interface for transmitting information to be output to output devices 1010, which output various types of information such as monitors and printers. It is implemented using connectors of standards such as USB (Universal Serial Bus), DVI (Digital Visual Interface), and HDMI (High Definition Multimedia Interface). Input IF1070 is an interface for receiving information from various input devices 1020, such as mice, keyboards, and scanners. It is implemented using, for example, USB.

[0214] The input device 1020 may also be a device that reads information from, for example, an optical recording medium such as a CD (Compact Disc), DVD (Digital Versatile Disc), or PD (Phase Change Rewritable Disk), a magneto-optical recording medium such as an MO (Magneto-Optical disk), tape media, magnetic recording media, or semiconductor memory. Furthermore, the input device 1020 may also be an external storage medium such as a USB memory stick.

[0215] Network IF1080 receives data from other devices via network N and sends it to the arithmetic unit 1030, and also transmits data generated by the arithmetic unit 1030 to other devices via network N.

[0216] The arithmetic unit 1030 controls the output device 1010 and the input device 1020 via the output IF 1060 and the input IF 1070. For example, the arithmetic unit 1030 loads a program from the input device 1020 or the secondary storage device 1050 onto the primary storage device 1040 and executes the loaded program.

[0217] For example, when computer 1000 functions as an information processing device 100, the arithmetic unit 1030 of computer 1000 realizes the functions of the control unit 130 by executing a program loaded onto the primary storage device 1040.

[0218] [5-5. Others] Furthermore, among the processes described in the above embodiments and modifications, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically by known methods. In addition, the processing procedures, specific names, and information including various data and parameters shown in the above document and drawings can be changed at will unless otherwise specified. For example, the various information shown in each figure is not limited to the information shown.

[0219] Furthermore, the components of each illustrated device are functionally conceptual and do not necessarily need to be physically configured as shown. In other words, the specific forms of distribution and integration of each device are not limited to those shown, and all or part of them can be functionally or physically distributed and integrated in any unit according to various loads and usage conditions.

[0220] Furthermore, the embodiments and modifications described above can be combined as appropriate, provided that the processing content is not inconsistent.

[0221] Furthermore, the terms "section, module, unit" mentioned above can be replaced with "means" or "circuit," etc. For example, the acquisition unit can be replaced with acquisition means or acquisition circuit.

[0222] [6. Effects] As described above, the information processing device 100 according to this embodiment includes an estimation unit 134 and a provision unit 136. The estimation unit 134 estimates the object of interest that the user has shown interest in, based on a first content in which the user has performed a predetermined operation that indicates the user's interests. The provision unit 136 provides the user with a second content related to the object of interest estimated by the estimation unit 134.

[0223] Thus, the information processing device 100 according to this embodiment can provide appropriate information to the user by providing the user with second content related to an object of interest estimated based on the first content in which the user has performed a predetermined operation indicating the user's interests.

[0224] Furthermore, in the information processing device 100 according to this embodiment, the providing unit 136 provides second content of a predetermined domain.

[0225] Thus, the information processing device 100 according to this embodiment can provide appropriate information by providing second content in a predetermined domain.

[0226] Furthermore, in the information processing device 100 according to the embodiment, the providing unit 136 provides second content in a second domain which is a predetermined domain different from the first domain of the first content.

[0227] Thus, the information processing device 100 according to this embodiment can provide appropriate information by providing a second content in a second domain, which is a predetermined domain different from the first domain of the first content.

[0228] Furthermore, in the information processing device 100 according to the embodiment, the providing unit 136 provides second content of a second domain determined based on predetermined criteria.

[0229] Thus, the information processing device 100 according to this embodiment can provide appropriate information by providing a second content of a second domain determined based on predetermined criteria.

[0230] Furthermore, in the information processing device 100 according to the embodiment, the provisioning unit 136 provides second content of a second domain determined according to the user.

[0231] Thus, the information processing device 100 according to this embodiment can provide appropriate information by providing a second content in a second domain determined according to the user.

[0232] Furthermore, in the information processing device 100 according to this embodiment, the estimation unit 134 estimates the user's interests based on the objects included in the first content.

[0233] Thus, the information processing device 100 according to this embodiment can provide appropriate information by estimating the user's interests based on the objects included in the first content.

[0234] Furthermore, in the information processing device 100 according to this embodiment, the estimation unit 134 estimates the user's interests using a captured image of the first content generated by the user's operation.

[0235] Thus, the information processing device 100 according to this embodiment can provide appropriate information by estimating the user's interests using a captured image of the first content generated by the user's operation.

[0236] Furthermore, in the information processing device 100 according to this embodiment, the estimation unit 134 estimates the user's interests based on the objects included in the captured image.

[0237] Thus, the information processing device 100 according to this embodiment can provide appropriate information by estimating the user's interests based on the objects included in the captured image.

[0238] Furthermore, in the information processing device 100 according to this embodiment, the estimation unit 134 estimates the user's interests based on the results of the analysis process that analyzes the captured image.

[0239] Thus, the information processing device 100 according to this embodiment can provide appropriate information by estimating the user's interests based on the results of an analysis process that analyzes the captured image.

[0240] Furthermore, in the information processing device 100 according to this embodiment, the estimation unit 134 estimates the user's interests based on the domain of the captured image.

[0241] Thus, the information processing device 100 according to this embodiment can provide appropriate information by estimating the user's interests based on the domain of the captured image.

[0242] Furthermore, in the information processing device 100 according to this embodiment, the estimation unit 134 estimates the user's interests based on the domains estimated using a domain estimation model that estimates the domains of the captured image.

[0243] Thus, the information processing device 100 according to this embodiment can provide appropriate information by estimating the user's interests based on the domains estimated using a domain estimation model that estimates the domains of captured images.

[0244] Furthermore, in the information processing device 100 according to the embodiment, the estimation unit 134 takes the screen as input and estimates the domain of the captured image using a domain estimation model that outputs information indicating the domain of the input image.

[0245] Thus, the information processing device 100 according to this embodiment can provide appropriate information by estimating the domain of a captured image using a domain estimation model that takes a screen as input and outputs information indicating the domain of the input image.

[0246] Furthermore, in the information processing device 100 according to the embodiment, the estimation unit 134 estimates the user's interests using sensor information obtained when the user performs a predetermined operation.

[0247] Thus, the information processing device 100 according to this embodiment can provide appropriate information by estimating the user's interests and concerns using sensor information obtained when the user performs a predetermined operation.

[0248] Furthermore, in the information processing device 100 according to the embodiment, the estimation unit 134 estimates the user's interests using the user's location information when the user performs a predetermined operation.

[0249] Thus, the information processing device 100 according to this embodiment can provide appropriate information by estimating the user's interests using the user's location information when the user performs a predetermined operation.

[0250] Furthermore, in the information processing device 100 according to this embodiment, the estimation unit 134 estimates the transaction target or facility as an object of interest to the user.

[0251] Thus, the information processing device 100 according to this embodiment can provide appropriate information by estimating the transaction target or facility as an object of interest to the user.

[0252] Furthermore, in the information processing device 100 according to this embodiment, the estimation unit 134 estimates products or stores as objects of interest to the user.

[0253] Thus, the information processing device 100 according to this embodiment can provide appropriate information by estimating products or stores as objects of interest to the user.

[0254] Furthermore, in the information processing device 100 according to this embodiment, the provisioning unit 136 displays the second content on the terminal device 10 used by the user.

[0255] Thus, the information processing device 100 according to this embodiment can provide appropriate information by displaying the second content on the terminal device 10 used by the user.

[0256] Furthermore, in the information processing device 100 according to this embodiment, the provisioning unit 136 transmits the second content to the terminal device 10 used by the user.

[0257] Thus, the information processing device 100 according to this embodiment can provide appropriate information by transmitting the second content to the terminal device 10 used by the user.

[0258] Although some embodiments of the present invention have been described in detail above with reference to the drawings, these are illustrative examples, and the present invention can be implemented in various other forms with modifications and improvements based on the knowledge of those skilled in the art, starting with the embodiments described in the disclosure section of the invention. [Explanation of Symbols]

[0259] N Network 1. Information Processing System 10 Terminal devices 20 External device 100 Information Processing Devices 110 Communications Department 120 Storage section 121 User information storage unit 122 Target Information Storage Unit 123 Content Storage Unit 124 Model Information Storage Unit 130 Control Unit 131 Acquisition Department 132 Reception Department 133 Decision Section 134 Estimation Department 135 Generation part 136 Provision Department

Claims

1. An estimation unit that estimates an interest target, which is an interest target, based on a first content which is content provided in a first domain that falls under any of the service classifications among a plurality of service classifications including EC (Electronic Commerce) services, restaurant introduction services and SNS (Social Networking Service) services, and which contains information about the subject, and which is content on which a predetermined operation has been performed to indicate the user's interest in the subject and to store information about the subject; A providing unit that provides the user with second content related to the subject of interest estimated by the estimation unit, which is a second content from a second domain that is a different domain from the first domain, and which is a second content related to the subject of interest estimated by the estimation unit, and which is a domain that belongs to one of the classifications of the plurality of services. An information processing device characterized by comprising:

2. The aforementioned supply unit is, The second content of the second domain is provided, which is determined based on predetermined criteria. The information processing apparatus according to feature 1.

3. The aforementioned supply unit is, The second content of the second domain is provided, which is determined according to the user. The information processing apparatus according to feature 2.

4. The estimation unit, Based on the subject matter included in the first content, the user's interests are estimated. The information processing apparatus according to claim 3.

5. The estimation unit, Using the captured image of the first content generated by the user's actions, the user's interests are estimated. The information processing apparatus according to feature 1.

6. The estimation unit, Based on the objects included in the captured image, the user's interests are estimated. The information processing apparatus according to feature 5.

7. The estimation unit, Based on the results of the analysis process that analyzes the captured image, the user's interests are estimated. The information processing apparatus according to feature 6.

8. The estimation unit, Based on the domain of the captured image, the user's interests are estimated. The information processing apparatus according to feature 7.

9. The estimation unit, Based on the domain estimated using a domain estimation model that estimates the domain of the captured image, the user's interests are estimated. The information processing apparatus according to feature 8.

10. The estimation unit, The domain of the captured image is estimated using the domain estimation model, which takes the screen as input and outputs information indicating the domain of the input image. The information processing apparatus according to feature 9.

11. The estimation unit, The transaction target or facility is estimated to be the user's area of ​​interest. The information processing apparatus according to feature 1.

12. The estimation unit, The product or store is estimated to be the object of interest of the user. The information processing apparatus according to feature 11.

13. The aforementioned supply unit is, The second content is displayed on the terminal device used by the user. The information processing apparatus according to feature 1.

14. The aforementioned supply unit is, The second content is transmitted to the terminal device used by the user. The information processing apparatus according to feature 1.

15. A method of information processing performed by a computer, An estimation step to estimate the object of interest, which is the object that the user has shown interest in, based on a first content which is content provided in a first domain that falls under one of several service classifications, including EC (Electronic Commerce) services, restaurant introduction services, and SNS (Social Networking Service) services, and which contains information about the object, and which is content on which a predetermined operation has been performed to indicate the user's interest in the object and to store information about the object; A provisioning step of providing the user with second content related to the subject of interest estimated by the estimation step, which is a domain in which a service belongs to one of the classifications of the plurality of services, and which is a domain different from the first domain. An information processing method characterized by including

16. An estimation procedure for estimating an interest target, which is an interest target, based on a first content which is content provided in a first domain that falls under any of the service classifications among a plurality of service classifications including EC (Electronic Commerce) services, restaurant introduction services and SNS (Social Networking Service) services, and which contains information about the subject, and which is content on which a predetermined operation has been performed to indicate the user's interest in the subject and to store information about the subject; A provision procedure for providing to the user a second content related to the subject of interest estimated by the estimation procedure, which is a second content from a second domain that is a domain belonging to one of the classifications of the plurality of services, and is a domain different from the first domain. An information processing program characterized by causing a computer to execute it.

Images