Information processing device, method, and program

Abstract

To provide an information processing device capable of more accurately estimating a user's preference, and a method and a program.SOLUTION: An information processing device includes a news information acquisition unit for acquiring news information representing contents of news distributed by a news site, an image information acquisition unit for acquiring images associated with a user and incidental information including information of at least a photographing date of the images, an image retrieval unit for retrieving an image having high relevance to the news information on the basis of image contents and the photographing date of the image from among the images, and an estimation unit for estimating a user's preference from the image hit by retrieval by the image retrieval unit and the news information used in the retrieval.SELECTED DRAWING: Figure 7

Description

[Technical field]

[0001] The present invention relates to an information processing device, method, and program, and more particularly to an information processing technique for estimating a user's preferences from images held by the user. [Background technology]

[0002] In Patent Document 1, an image posted by a user and the associated poster code are stored on a server that provides a social networking service (SNS). The information processing device described in the publication analyzes the tastes and tendencies of contributors by downloading content data including comments from a server.

[0003] Patent Document 2 describes a technique for estimating a user's hobbies and tastes based on the user's behavioral information, shooting information, shot images, text data posted on a social networking site, and the like.

[0004] Patent Document 3 describes a technology for searching for images from among a group of images stored in a storage site using tag information associated with the image, such as the shooting date and time, the shooting location, and the name of the subject, as candidates for image search keywords. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2019-28793 A [Patent Document 2] JP 2014-110001 A [Patent Document 3] JP 2010-20719 A Summary of the Invention [Problem to be solved by the invention]

[0006] Recently, recommendation systems that recommend various products and / or services have been used in e-commerce sites or SNS advertisements. Such recommendation systems can realize useful recommendations by correctly grasping the preferences of users. Although the techniques described in Patent Documents 1 and 2 can roughly estimate the preferences of users, this is not necessarily sufficient. For example, it is difficult to evaluate the degree of a user's preferences, such as whether the degree of the user's preferences (preference level) is extremely high or enthusiastic, by simply analyzing a group of images. In order to provide more appropriate information to each user, it is necessary to estimate the preferences of the user more accurately.

[0007] The present invention has been made in consideration of the above circumstances, and has an object to provide an information processing device, method, and program capable of more accurately estimating a user's preferences. [Means for solving the problem]

[0008] An information processing device according to one aspect of the present disclosure includes an image information acquisition unit that acquires an image associated with a user and auxiliary information including at least information on the shooting date of the image, a news information acquisition unit that acquires news information indicating the content of the news distributed by a news site, an image analysis unit that analyzes the image content from the image, and an estimation unit that estimates the user's preferences based on the image content grasped by processing of the image analysis unit and the news information for the period corresponding to the shooting date.

[0009] The news information may be news articles distributed from a news site, or may be information such as matters specified from the contents of a news article or keywords extracted. A user is an actual "person," and typically, an individual user is identified by unique identification information such as a user ID (Identification). The term "user preferences" is not limited to the object of a preference, but also includes concepts such as the degree of preference, things or matters that a user values, and things or matters that are important to a user.

[0010] According to this aspect, information that cannot be understood only from image analysis and accompanying information is obtained from news sites, and the image analysis results and news information are combined to estimate user preferences. This makes it possible to estimate user preferences more accurately and provide appropriate recommendations.

[0011] An information processing device according to another aspect of the present disclosure may further include a related information generating unit that generates information related to the user's preferences estimated by the estimation unit.

[0012] For example, the information related to the user's preferences may include information on products or services recommended to the user. According to this aspect, appropriate suggestions can be made to the user.

[0013] In still another aspect of the present disclosure, the estimation unit can be configured to estimate a degree of preference of the user from the news information.

[0014] An information processing device according to yet another aspect of the present disclosure can be configured to further include a news search unit that extracts news related to an image from articles distributed on a plurality of pre-specified news sites based on information about the shooting date.

[0015] In still another aspect of the present disclosure, the incidental information may include information about a location where the image was taken, and the news search unit may be configured to extract news related to the image by using the information about the location where the image was taken.

[0016] By using information about the location where the image was taken, it becomes easier to extract news related to the image.

[0017] In yet another aspect of the present disclosure, the image analysis unit may include a word generation unit that generates words related to the image content, and the news search unit may be configured to extract news related to the image using the generated words.

[0018] The words related to the image content may be, for example, names of objects shown in the image, details of an event, or locations identified by landmark buildings. The "words" may be referred to as "keywords" or "phrases." The words generated by the word generation unit may be added to the supplementary information of the image.

[0019] In still another aspect of the present disclosure, the news search unit can be configured to search for news articles containing a predetermined specific keyword and extract news related to the image.

[0020] The predetermined specific keywords may include at least one of the following: popular, rush, expensive, high price, anniversary, valuable, and rare. These words suggest a high degree of preference or a high importance of the matter.

[0021] An information processing device according to yet another aspect of the present disclosure further includes a storage device that stores a plurality of images associated with a user, and an image search unit that searches for images highly relevant to news information from among the group of images stored in the storage device, and the estimation unit can be configured to estimate the user's preferences from images found by the search by the image search unit and the news information used in the search.

[0022] An information processing device according to yet another aspect of the present disclosure can be configured to further include a news information list generation unit that collects news articles from a plurality of pre-specified news sites via a news information acquisition unit, and generates a news information list that organizes news information including dates, locations, and related keywords for each subject of the collected news articles.

[0023] In yet another aspect of the present disclosure, the image search unit can be configured to search for images that are highly relevant to the date, location, and related keywords of the news information from among a group of images stored in the storage device, and the estimation unit can be configured to estimate the user's preferences based on the images found by the search by the image search unit and the information used in the search.

[0024] In yet another aspect of the present disclosure, the news information list generation unit can be configured to add identification information indicating that the news information of a news article containing a predetermined specific keyword is related to a news article containing the specific keyword when listing the news information in the news information list.

[0025] In yet another aspect of the present disclosure, the estimation unit can be configured to, when a search returns an image that is highly relevant to news information to which identification information is attached, determine the degree of user preference for the subject of the news information to which the identification information is attached from the identification information.

[0026] In yet another aspect of the present disclosure, the storage device may be configured to store a plurality of images associated with each of a plurality of users.

[0027] According to this embodiment, it is possible to utilize information in a variety of ways, such as analyzing the preferences of each user, analyzing the preference trends of multiple users through statistical processing, and classifying multiple users based on the similarity of their preferences.

[0028] In yet another aspect of the present disclosure, at least a portion of the image analysis unit and the estimation unit may be configured using a trained model using a neural network.

[0029] For example, some or all of the processes such as image object recognition, object-related word generation, and inference processing for estimating preferences can be realized using a trained model trained using deep learning.

[0030] An information processing method according to yet another aspect of the present disclosure includes an information processing device configured using a computer acquiring an image associated with a user and auxiliary information including at least information on the shooting date of the image, acquiring news information indicating the content of the news distributed by a news site, analyzing the image content from the image, and estimating the user's preferences based on the image content understood by the analysis process and the news information for the period corresponding to the shooting date.

[0031] An information processing method according to yet another aspect of the present disclosure may further include the information processing device generating information related to an estimated user preference.

[0032] A program relating to yet another aspect of the present disclosure is a program for enabling a computer to realize the following functions: acquiring an image associated with a user and additional information including at least information on the date the image was taken; acquiring news information indicating the content of the news distributed by a news site; analyzing the image content from the image; and estimating the user's preferences based on the image content understood by the analysis process and the news information for the period corresponding to the date the image was taken.

[0033] An information processing device according to yet another aspect of the present disclosure includes a processor and a non-transitory computer-readable medium having instructions stored thereon to be executed by the processor, and the processor executes the instructions to perform processes including obtaining an image associated with a user and ancillary information including at least information on the date the image was taken, obtaining news information indicating the content of the news distributed by a news site, analyzing the image content from the image, and estimating the user's preferences based on the image content grasped by the analysis process and the news information for the period corresponding to the date the image was taken. Effect of the Invention

[0034] According to the present invention, the user's preferences are estimated by combining the image analysis results with news information distributed from a news site, so that the user's preferences can be estimated more accurately. [Brief description of the drawings]

[0035] [Figure 1] FIG. 1 is a diagram showing an overall configuration of an example of a computer system including an information processing device according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a functional block diagram showing an example of the configuration of the image storage server. [Diagram 3] FIG. 3 is a functional block diagram illustrating an example of the configuration of the information processing device according to the first embodiment. [Figure 4] FIG. 4 is a flowchart illustrating an example of a procedure of an information processing method according to an embodiment of the present invention. [Diagram 5] FIG. 5 is a flowchart showing an example of processing by the information processing device according to the first embodiment. [Figure 6] FIG. 6 is a diagram showing an example of a group of images taken by a user. [Figure 7] FIG. 7 is a functional block diagram showing an example of the configuration of an information processing device according to the second embodiment. [Figure 8] FIG. 8 is a chart showing an example of a news information list that compiles news information collected from a plurality of news sites. [Figure 9] FIG. 9 is a block diagram showing an example of a hardware configuration of a computer. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0036] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[0037] Overall configuration of the computer system Fig. 1 is an overall configuration diagram showing an example of a computer system including an information processing device according to an embodiment of the present invention. The computer system 10 shown in Fig. 1 is a system that provides a cloud storage service for storing image data, and includes an image storage server 20 and an information processing device 30. Note that Fig. 1 illustrates an example in which the image storage server 20 and the information processing device 30 are configured as separate devices, but these functions may be realized by one computer, or may be realized by sharing the processing functions among two or more computers.

[0038] The image storage server 20 and the information processing device 30 are connected to an electric communication line 70. The electric communication line 70 may be, for example, a wide area network such as the Internet. The term "connection" is not limited to a wired connection and also includes the concept of a wireless connection.

[0039] A user who uses the cloud storage service of this example is required to agree to a predetermined set of terms of use and register as a user before using the service. After completing user registration, the user can upload image data to the image storage server 20 using an information terminal such as the user terminal 72 or the storefront terminal 74.

[0040] Each of the user terminal 72 and the storefront terminal 74 is a device equipped with a communication function that can be connected to the telecommunication line 70. The user terminal 72 may be, for example, a smartphone, a tablet terminal, or a personal computer owned by the user. The user terminal 72 is not limited to being owned by the user, and may be a device shared by multiple people. The storefront terminal 74 is an information terminal installed in various stores such as a store that provides a photo print service or a convenience store. The storefront terminal 74 is equipped with a media interface for importing image data from an external storage device such as a memory card and / or a communication interface that can be connected to an external device. Although one user terminal 72 and one storefront terminal 74 are shown in FIG. 1, multiple user terminals 72 and multiple storefront terminals 74 may be connected to the telecommunication line 70.

[0041] The image storage server 20 organizes, stores, and manages image data received from the user terminal 72 or the storefront terminal 74 for each user.

[0042] The information processing device 30 performs various information processing such as analyzing images stored in the image storage server 20, generating tag information according to image contents such as objects or scenes of the images, and analyzing user preferences. "Image contents" may be rephrased as "photographed contents." The processing function of the information processing device 30 may be incorporated in the image storage server 20.

[0043] A plurality of news sites NS1, NS2, . . . NSn are connected to the telecommunications line 70. Hereinafter, the plurality of news sites NS1, NS2, . . . NSn will be collectively referred to as "news site NS". The news site NS includes a web server that distributes news articles. The information processing device 30 collects information from a plurality of pre-specified news sites NS. The pre-specified news site NS is preferably a site with highly reliable articles, for example, a news site provided by a national newspaper company, a local newspaper company, a news agency, a television station, or a similar news organization. Some of the plurality of news sites NS may be, for example, a news distribution service site that aggregates articles provided by a plurality of news providers and distributes news.

[0044] The information processing device 30 estimates the user's preferences using images stored in the image storage server 20 and news information obtained from the news site NS, and proposes various products and / or services that match the user's preferences.

[0045] Example of the configuration of the image storage server 20 2 is a functional block diagram showing an example of the configuration of the image storage server 20. The image storage server 20 includes a communication unit 22, a control unit 24, and an image storage 26. The communication unit 22 is a communication interface for connecting to an electric communication line 70. The control unit 24 controls the transfer of data via the communication unit 22. The control unit 24 also includes a user authentication unit 28, which controls the writing of data to the image storage 26 and the reading of data from the image storage 26. The user authentication unit 28 performs user authentication processing.

[0046] The image storage 26 is a large-capacity storage device in which images uploaded by each user are organized and stored for each user. If an index that identifies each of a plurality of users is i, the images held by user Ui are associated with information about user Ui and stored in the image storage 26. For example, the images held by user U1 are associated with information about user U1 and stored in the image storage 26. Similarly, the images held by user U2 are associated with information about user U2 and stored in the image storage 26. The images held by user Ui may be classified according to keywords such as the date of shooting or the location of shooting and stored in the image storage 26.

[0047] The images stored in the image storage 26 may be digital photographs taken with a photographing device such as a digital camera or a smartphone, or may be images obtained by converting analog photographs into digital data. The files of the images stored in the image storage 26 may include additional information related to the images. The images stored in the image storage 26 may also be videos.

[0048] The incidental information includes at least one of information such as information on the shooting date and time, information on the shooting location, information on the name identifying the subject, information on the scene, information on the event at which the image was shot, information indicating the name of the object in the image, and information on keywords used to classify or search for images. The incidental information preferably includes at least information on the shooting date. More preferably, the incidental information includes information on the shooting date and time and information on the shooting location. The incidental information includes the concepts of tag information, metadata, and annotation.

[0049] The information on the photographing date and time may be, for example, date and time information obtained from a built-in clock of the photographing device used for photographing, such as a digital camera or a smartphone. The information on the photographing location may be, for example, location information obtained from a GPS (Global Positioning System) device built into the photographing device. The information may be the date and time of shooting. When an image is taken using a photographing device capable of recording the date and time of shooting and location information, the additional information including the date and time of shooting and location information is automatically added, and an image file including the additional information is generated. Note that, if the photographing device is set to disable the use of location information, the location information is not recorded in the image file, and the information on the date and time of shooting is recorded as the additional information.

[0050] The incidental information is not limited to being automatically added by a photographing device or the like, but may be at least one of the photographing date, photographing time, and photographing location identified by processing image data, or may be information input or edited by a user performing an input operation using an appropriate input interface as necessary. For example, the photographing date information can be obtained from the date information recorded in an analog photograph. Also, for example, the photographing location can be identified from a landmark building or the like detected using an object recognition technology based on image analysis. Part of the incidental information may be written by the information processing device 30.

[0051] Example of configuration of information processing device 30 3 is a functional block diagram showing an example of the configuration of an information processing device 30 according to the first embodiment. The functions of the information processing device 30 can be realized by a combination of computer hardware and software. The information processing device 30 includes a communication unit 32, a calculation processing unit 34, a storage device 35, an input device 36, and a display device 38.

[0052] The communication unit 32 is a communication interface for connecting to the electric communication line 70. The calculation processing unit 34 is configured to include, for example, a CPU (Central Processing Unit). The calculation processing unit 34 includes an image information acquisition unit 40, an image analysis unit 42, an incidental information analysis unit 44, a news search unit 46, a news information acquisition unit 48, and a preference estimation unit 50. The calculation processing unit 34 performs various processes by utilizing the storage area of ​​the storage device 35.

[0053] The image information acquisition unit 40 includes an interface for acquiring data of images and incidental information. The image information acquisition unit 40 may be configured to include a data input terminal for acquiring data of images and incidental information from an external signal processing unit or another signal processing unit within the device. The image information acquisition unit 40 may be configured integrally with the communication unit 32. The image information acquisition unit 40 acquires images and incidental information from the image storage server 20 via the communication unit 32. The image information acquisition unit 40 may acquire images and incidental information from a user terminal 72 or a storefront terminal 74.

[0054] The images acquired via the image information acquisition unit 40 are sent to the image analysis unit 42. The image analysis unit 42 performs processing such as scene analysis and object recognition on the input images. The image analysis unit 42 includes a word generation unit 43. The word generation unit 43 generates words related to the image content, such as the names of objects or events appearing in the image. The words generated by the word generation unit 43 may be added to the supplementary information as tag data for the image. Based on the words generated by the word generation unit 43, it becomes possible to automatically classify a group of images. The analysis results of the image analysis unit 42 are sent to the news search unit 46 and the preference estimation unit 50.

[0055] The incidental information acquired via the image information acquisition unit 40 is sent to the incidental information analysis unit 44. The incidental information analysis unit 44 extracts information to be used for searching news articles from the contents of the incidental information. The incidental information analysis unit 44 extracts information such as the shooting date, shooting time, and shooting location.

[0056] The news search unit 46 extracts news related to the image from articles distributed by a number of pre-specified news sites NS, based on at least the information on the shooting date. Since there is a time difference between the date and time when a news item occurs and the date and time when a news article about that item is distributed, when searching or collecting news article information, it is preferable to determine relevance by taking into account such a time difference and allowing a time range of at least one day, preferably several days.

[0057] It is preferable that the news search unit 46 extracts news related to the image using information on the shooting location in addition to information on the shooting date. It is also preferable that the news search unit 46 extracts news related to the image using words generated by the word generation unit 43. Furthermore, the news search unit 46 may extract news related to the image by searching for news articles containing a predetermined specific keyword.

[0058] The news information acquisition unit 48 acquires news information that represents the contents of the news distributed by the news site NS. The news information acquisition unit 48 includes an interface for acquiring news article data from the news site NS. The news information acquisition unit 48 may be configured to include a data input terminal that acquires image and incidental information data from an external signal processing unit or another signal processing unit within the device. The news information acquisition unit 48 may be configured integrally with the communication unit 32. The news information acquisition unit 48 collects information from the news site NS via the communication unit 32.

[0059] The preference estimation unit 50 performs processing to estimate the user's preferences based on the image content grasped by the processing of the image analysis unit 42 and news information for the period corresponding to the shooting date. Here, "user preferences" includes concepts such as the user's tendency to like things, the degree of preference, things or matters that the user values, and things or matters that are important to the user. The degree of preference includes, for example, the degree of preference as to whether the user is significantly more enthusiastic (fanatic), that is, a core fan, compared to the general public. The degree of preference may be called "preference degree" or "core degree."

[0060] The preference estimation unit 50 further includes a related information generation unit 51 that generates information related to the estimated user preference. The information related to the preference includes, for example, recommendation information that proposes a product or service related to the preference. The related information generation unit 51 in this example generates recommendation information that notifies the user of information on a recommended product or service that is recommended to the user in relation to the user's preference. The recommendation information generated by the preference estimation unit 50 is provided to the user terminal 72, etc. via the communication unit 32. The preference estimation unit 50 is an example of an "estimation unit" in this disclosure.

[0061] At least a part of the image analysis unit 42 and the preference estimation unit 50 is configured by a trained model in which a model using a neural network is trained by machine learning. In the present example, the image analysis unit 42 and the preference estimation unit 50 use a trained model trained by deep learning.

[0062] The storage device 35 includes a semiconductor memory in the CPU, a main memory, and an auxiliary memory. Images and accompanying information acquired from the image storage server 20 are stored in the storage device 35. The storage device 35 may be used as a part or all of the image storage 26. The image storage 26 or the storage device 35, or a combination thereof, is an example of a "storage device" in this disclosure.

[0063] The input device 36 is, for example, a keyboard, a mouse, a touch panel, or other pointing devices, or a voice input device, or a suitable combination of these. The display device 38 is, for example, a liquid crystal display, an organic electro-luminescence (OEL) display, a projector, or a suitable combination of these. It is composed of a combination of

[0064] Overview of information processing method The information processing device 30 estimates the preferences of a user based on the shooting details and accompanying information of an image held by the user and the news information corresponding thereto. Among the accompanying information of an image, the information on the shooting date and the shooting location can be used when extracting news information corresponding to the user's image from a plurality of news articles distributed by a news site. In addition, the accompanying information of an image can be used when extracting an image corresponding to a certain news information from a group of images.

[0065] News information can be information that includes facts or matters that are difficult to grasp through image analysis. In other words, news information is information that is useful for evaluating the degree of a user's preference for matters that can be grasped from an image, and is also information that is useful for evaluating the importance of the image or the importance of the matters depicted in the image.

[0066] The information processing device 30 estimates the user's preferences using news information corresponding to the image in addition to information indicating the content of the image (the content of the shot) grasped by image analysis, thereby enabling the information processing device 30 to estimate the user's preferences more accurately than if the news information is not used.

[0067] 4 is a flowchart illustrating an example of the procedure of an information processing method according to an embodiment of the present invention. Each step in FIG. 4 is realized by a computer functioning as the information processing device 30 executing a program.

[0068] The information processing method of this embodiment includes the information processing device 30 acquiring images and associated information (step S1), the information processing device 30 acquiring news information (step S2), the information processing device 30 performing image analysis (step S3), the information processing device 30 estimating user preferences (step S4), and the information processing device 30 generating recommendation information (step S5).

[0069] In step S1, the information processing device 30 acquires images and their associated information owned by a specific user from the image storage server 20. The "specific user" here refers to a person whose preferences are to be estimated.

[0070] In step S2, the information processing device 30 acquires news information from a news site. For example, the information processing device 30 acquires information on news articles distributed at a time corresponding to the shooting date based on the supplementary information of the image. The "time period corresponding to the shooting date" may be the same day as the shooting date, or may be a range of several days before and after the shooting date including the shooting date. Note that although the "shooting date" is used as the basis here, information on news articles may also be collected based on the shooting date and time, including time information.

[0071] In step S3, the information processing device 30 analyzes the image acquired in step S1. This image analysis step includes, for example, a process of detecting a subject object by object recognition and a process of generating keywords related to the detected object. The image analysis algorithm may be a trained neural network model trained using machine learning.

[0072] The information processing device 30 analyzes at least one, preferably a plurality of images, and more preferably all images, among a group of images held by the user.

[0073] In step S4, the information processing device 30 estimates the user's preferences based on the image analysis result obtained in step S3 and the news information obtained in step S2. The preference estimation algorithm may be a trained neural network model trained using machine learning.

[0074] In step S5, the information processing device 30 generates recommendation information according to the user preferences estimated in step S4. The recommendation information generated in step S5 is output from the information processing device 30 and displayed, for example, on the display screen of the user terminal 72. After step S5, the information processing device 30 ends the flowchart of FIG.

[0075] The information processing device 30 can provide appropriate recommendation information that matches the preferences of each user by executing the flowchart of FIG. 4 for each user.

[0076] Example of Processing Flow by Information Processing Device 30 According to First Embodiment A more detailed example will be described with reference to Fig. 5. Fig. 5 is a flowchart showing an example of processing by the information processing device 30 according to the first embodiment.

[0077] In step S11, the information processing device 30 acquires a group of images owned by the user. The information processing device 30 may acquire the group of images from the image storage server 20, or may acquire the group of images from the user terminal 72 or the in-store terminal 74. The acquired group of images is stored in the storage device 35.

[0078] In step S12, the information processing device 30 analyzes the image content of each image included in the acquired image group. The process of step S12 is performed by the image analysis unit .

[0079] In step S13, the information processing device 30 analyzes the auxiliary information of the image. The process of step S13 is performed by the auxiliary information analysis unit 44. Note that the order of steps S12 and S13 may be reversed, or steps S12 and S13 may be processed in parallel.

[0080] In step S14, the arithmetic processing unit 34 of the information processing device 30 determines whether or not there are any unanalyzed images. If there are any images among the images acquired in step S11 that have not been subjected to the analysis processing in steps S12 and S13, the arithmetic processing unit 34 returns to step S12. If the analyses in steps S12 and S13 are performed on all images and the determination result in step S14 is No, the arithmetic processing unit 34 proceeds to step S16.

[0081] In step S16, the calculation processing unit 34 searches for related news based on the image content, date and time, and location grasped in steps S12 and S13, and determines whether or not news information related to the image has been extracted.

[0082] If the determination result in step S16 is a Yes determination, that is, if news information related to the image has been extracted, the calculation processing unit 34 proceeds to step S20. If the determination result in step S16 is a No determination, that is, if news information related to the image has not been extracted, the calculation processing unit 34 proceeds to step S17. In step S17, the calculation processing unit 34 searches for local news based on the position information of the image, and determines whether or not news information related to the image has been collected.

[0083] If the determination result in step S17 is a Yes determination, the calculation processing unit 34 proceeds to step S20. If the determination result in step S17 is a No determination, the calculation processing unit 34 proceeds to step S18. In step S18, the calculation processing unit 34 further changes the search conditions to search for related news and determines whether or not news information related to the image has been collected. In step S18, for example, the shooting date information is ignored and a search is performed using only the image content or location information. If the determination result in step S18 is a Yes determination, the calculation processing unit 34 proceeds to step S20. If the determination result in step S18 is a No determination, the calculation processing unit 34 proceeds to step S21.

[0084] In step S20, the calculation processing unit 34 estimates the user's preference level based on the contents of the news article extracted in any of steps S16 to S 18. If there is a news article corresponding to the image, it is possible to evaluate the user's preference level that cannot be understood from the image contents.

[0085] In step S21, the calculation processing unit 34 estimates the user's preference level from the image content without using news information. The processes of steps S20 and S21 are performed by the preference estimation unit 50. After step S20 or step S21, the calculation processing unit 34 proceeds to step S22.

[0086] In step S22, the calculation processing unit 34 generates recommendation information according to the estimated user preference. The process of step S22 is performed by the related information generating unit 51. The recommendation information generated in step S22 is output from the information processing device 30 and provided to the user terminal 72, etc. After step S22, the information processing device 30 ends the flowchart of FIG.

[0087] The information processing device 30 can provide appropriate recommendation information that matches the preferences of each user by executing the flowchart of FIG. 5 for each user.

[0088] Example 1 The operation of the information processing device 30 will be described below with a specific example. As a result of analyzing the photographed content of an image held by a user U, the keywords "Leisure facility T", "Character M", and "Parade" were automatically generated. "Leisure facility T" and "Character M" are real names. In addition, from the supplementary information of the image, the photographed date was "November 18th", and The location was "Leisure Facility T."

[0089] When we searched for articles on news sites using these keywords, the following news articles were extracted:

[0090] [News article] The hugely popular character M celebrated its 90th anniversary on November 18th. At leisure facility T, customers rushed to the attractions to celebrate M's birthday, resulting in an abnormal situation where people had to wait up to 11 hours. The strange scene in the "Land of Dreams" has caused complaints from customers. When the user's preferences are analyzed in consideration of the contents of this news article, it is estimated that this user U is a core fan of leisure facility T and / or character M. That is, according to the contents of the news article, this user U visits leisure facility T on the special anniversary of its 90th birth, despite the disadvantage of being heavily crowded with people waiting up to 11 hours, which would make most people hesitate to visit. Such behavior of user U can be evaluated as indicating an extremely high degree of preference for leisure facility T and / or character M. In addition, the image in the photo is a precious scene of the 90th birth anniversary, and is likely to be something of special importance to user U.

[0091] Therefore, for this user U, it is possible to recommend products related to leisure facility T and / or character M that only a core fan would want to purchase, or to recommend products and / or services related to special anniversaries.

[0092] Example 2 As a result of analyzing the contents of images owned by a certain user U, the keyword "watching soccer" was automatically generated. In addition, the additional information of the image indicated that the date of the photo was "October 31st" and the location of the photo was "Shinjuku." When searching for articles on a news site using the words contained in these keywords, the following news article was extracted.

[0093] [News article] The Japanese national team beat Australia in the final Asian qualifying round for the FIFA World Cup on the 31st, securing their place in the finals, sending the whole country into a frenzy! Immediately after the end of the match, a large number of excited supporters, mostly young people, rushed to the scramble crossing in front of Shibuya Station in Tokyo, causing a moment of chaos. The Metropolitan Police Department was in charge of security to prevent any further trouble. As a result of the news search, no news related to the location information of the shooting location "Shinjuku" was extracted, but a news article related to "soccer" was extracted. When the user's preferences are analyzed in consideration of the content of this news article, it is estimated that this user U is a soccer fan. In other words, from the content of the news article, it is considered that the image in the photo is a scene of watching an important match in the Asian final qualifying round that determined the user's participation in the main tournament, and it is highly likely that this is something of special importance to user U. Therefore, it is possible to recommend soccer-related products to user U, or products related to the match watched and / or products related to the tournament.

[0094] Example 1 of using news information By using object recognition technology through image analysis, it is possible to recognize what objects appear in each image. For example, it is possible to recognize what characters appear in each image. Here, let us assume that three types of characters, Character A, Character B, and Character C, are recognized from a group of images owned by a certain user. Each of Character A, Character B, and Character C actually has a proper noun.

[0095] However, it is not possible to evaluate which character is more important to the user based on the results of this image analysis alone. Note that the "user" in the case of being important or significant to the user may include people close to the user, such as the user's family.

[0096] Therefore, in this embodiment, online news articles are searched for using image object recognition, supplementary information, and the like as search items, and the contents of the news articles are used to evaluate the degree of preference.

[0097] FIG. 6 shows an example of a group of images owned by a certain user. The date of shooting is identified from the additional information. Character A, character B, and character C appearing in the images are identified by object recognition. The shooting location is identified, for example, from GPS information included in the additional information. If GPS information is not included in the additional information, the location can be identified by object recognition of landmark buildings or the like, or from information on mobile phone base stations, and the identified location information may be used.

[0098] The news search unit 46 searches for the keywords "photo date," "character name," and "photo location" using "AND conditions" for articles on multiple pre-specified news sites NS. For example, in the case of the example in FIG. 6, the search is performed using the following search formula.

[0099] Search formula 1: "April 7" * "Character A" * "Minato Mirai" Search 2: "April 14" * "Character B" * "Shin-Yokohama" Search formula 3: "April 21" * "Character C" * "Shinjuku" As a result, for example, there are no relevant articles in "Search Formula 3" and no search results are returned, but there are relevant articles in "Search Formula 1" and "Search Formula 2" and search results are returned. In such a case, it can be presumed that the user or his / her family members, etc., taking an image of character A in Minato Mirai and taking an image of character B in Shin-Yokohama were more intentional than taking an image of character C on another day (April 21st). This makes it possible to extract characters A and B as things that the user holds dear. Note that although "month and date" are used here as the shooting date, "year, month, and date" including "year" may also be used.

[0100] <Example 2 of using news information> When searching using the above-mentioned search formulas 1 to 3, an "AND condition" is further used to search for articles containing specific words in each of search formulas 1 to 3. The specific words are called "specific keywords." Examples of specific keywords are the following words:

[0101] Specific keywords: {large, rush, expensive, high price, anniversary, valuable, rare} These specific keywords suggest that the degree of the user's preference is extremely high. The specific keywords are determined in advance. For news articles containing the words "large number" or "inundated", the user's active intention of "want to see even if it's crowded" can be inferred. For news articles containing the words "expensive" or "high price", the user's active intention of "want to see even if it's expensive, or want to buy even if it's expensive" can be inferred. For news articles containing the words "anniversary" or "anniversary", the user's active intention of "as a core fan, I definitely want to go to a special commemorative event to celebrate" can be inferred. For news articles containing the words "valuable" or "rare", the user's active intention of "as a core fan, I definitely want to see or get it" can be inferred.

[0102] Other examples of information that can help infer preferences: When estimating a user's preferences, the preference estimation unit 50 may use at least one of the information on the frequency of photographing and the interval between photographing, in addition to the information on the image content, the photographing date and time, and the photographing location. For example, if a large number of images are photographed at short intervals, the degree of interest in the photographed content is considered to be high. Also, if the frequency of photographing a certain object is high, the degree of interest is considered to be high.

[0103] <Examples of providing recommendation information> The information processing device 30 identifies products and / or services related to the estimated user preferences, and recommends the products and / or services to the user. The recommendation period is a fixed period (for example, one year) from the shooting date when the number of images was large. The recommendation may be terminated after the fixed period has elapsed. It is preferable to appropriately adjust the recommendation period depending on the type of product and / or service to be proposed.

[0104] When recommending a product and / or a service, the related information generating unit 51 may attach information indicating a discount or a price reduction.

[0105] In addition, when the same event occurs consecutively, the information processing device 30 may store the number of times it has occurred, and when it detects that the same event has not occurred after a predetermined period of time has passed, it may determine the discount rate or discount amount based on the number of times it has occurred.

[0106] Second Embodiment FIG. 7 is a functional block diagram showing a configuration example of an information processing device 130 according to a second embodiment. Instead of the information processing device 30 described in FIG. 3, the information processing device 130 shown in FIG. 7 may be adopted. In FIG. 7, elements that are the same as or similar to those in the configuration shown in FIG. 3 are given the same reference numerals, and their description will be omitted. Regarding the information processing device 130 shown in FIG. 7, differences from the information processing device 30 according to the first embodiment will be described. The information processing device 30 according to the first embodiment shown in FIG. 3 is configured to collect information from a news site using supplementary information of an image and / or an analysis result of the image. In contrast, the information processing device 130 according to the second embodiment shown in FIG. 7 is configured to first collect news information from a news site NS, and then search for images that are highly related to the date, time, place, and keyword of the listed news.

[0107] The information processing device 130 includes a calculation processing unit 134 replacing the calculation processing unit 34. As shown in FIG.

[0108] The news information list generating unit 54 generates a news information list from news articles acquired via the news information acquiring unit 48. The news information list is a list in which the date, time, location, keywords, etc. are organized for each content of a news article. The news information used for preference estimation is not limited to the news article itself, but may be information that has been processed (edited) based on a news article, such as information listed in the news information list.

[0109] The image search unit 56 searches the group of images stored in the image storage server 20 for images that are highly related to the date, time, place, and keywords listed in the news information list. When performing an image search, it is preferable that tag data such as keywords related to the image content is added to each image. This tag data can be generated by the word generation unit 43. The search results by the image search unit 56 are sent to the preference estimation unit 50.

[0110] The preference estimation unit 50 estimates the user's preference from the images extracted by the image search unit 56, and generates recommendation information related to the estimated user's preference. Note that the function of the image search unit 56 may be incorporated into the preference estimation unit 50. A specific example of processing by the information processing device 130 will be described.

[0111] <Example 3 of using news information> Since the number of news sites NS is finite, the information processing device 130 collects all information about events occurring in Japan, such as information about events, the release of new products or services, etc., from multiple news sites NS, for example, each day. Note that while the explanation here is given using news from "Japan" as an example, information may be collected from news sites in multiple countries, or from news sites around the world. The range of countries or regions from which news information is collected may be specified in advance.

[0112] Regarding the timing of collecting information from the news site NS, for example, it is considered that events that occurred on Sundays are often distributed as news on the same day or the following Monday, so it is assumed that information about events that occurred on Sundays is collected on Tuesday. The information processing device 130 collects the date, time of occurrence (time zone), location, and related keywords for each news item.

[0113] Fig. 8 is a chart showing an example of a news information list. The news information list generating unit 54 generates, for example, a news information list as shown in Fig. 8. Although news reporting the release of a new product such as "No. 2001" in Fig. 8 is not related to a "location", it is conceivable that a user will take a photo of the newly released product with the product in hand.

[0114] It may be difficult to imagine related images for news such as "No. 2002" announcing the start of a service, but since it may be technically difficult to exclude news articles that are poorly related to the image, when collecting information, the news may be listed without exclusion processing. If there is no image related to No. 2002 among the images stored online, the image search result will be "no match", and there is no problem from the system perspective, so the information processing device 130 may mechanically collect news articles.

[0115] Information for classifying the type of article may be added to the news information list. The news information list generating unit 54 may generate words for classifying the type of article from the content of the news.

[0116] The information processing device 130 searches for images that are highly related to the time, place, and keywords listed above, for example, from all images saved online by all users of the system on the day of information collection. For images found through this image search, the items indicated by the keywords used in the search indicate what is important to the user who owns the image.

[0117] <Example 4 of using news information> When listing news information in the above "Example 3 of using news information," news articles that contain specific keywords that have been determined in advance are flagged. If an image is found related to an article with this flag, it is clear that the user who owns the image is a core fan of the keyword related to the article.

[0118] As with "Example 2 of Use of News Information," the specific keywords are phrases that suggest an extremely high degree of preference for the user, and may be, for example, {large, flooded, expensive, high price, anniversary, valuable, rare}.

[0119] The news information list generating unit 54 performs a process of determining whether a news article contains a specific phrase and assigning a flag according to the determination result. The news information list includes flag information. The flag is an example of "identification information" in this disclosure.

[0120] <<Method for Providing Appropriate Recommendations to Users>> As specifically described in the above "Examples 1 to 4 of Use of News Information," according to the embodiment of the present invention, it is possible to evaluate the importance of objects in an image to a user. That is, each object identified by image analysis can be classified into the following [1] to [3]. That is, it can be classified into [1] objects that simply appear in the image many times, [2] objects that the user considers important, and [3] objects for which the user is a core fan.

[0121] These classifications correspond to the degree of a user's preference for the object. When providing recommendations of products and / or services related to an object classified into any of [1] to [3], it is preferable to vary the content, frequency, and number of recommendations provided according to the classification into [1] to [3].

[0122] For example, the greater the importance of the object, the higher the frequency of recommendations for that object. The greater the importance of the object, the more events held in more distant locations are recommended. The greater the importance of the object, the more expensive products and / or services are recommended.

[0123] 《Protection of User Personal Information》 <1> In an embodiment of the present invention, a system administrator obtains consent from a user to analyze the user's images and to send recommendations based on the analysis results.

[0124] <2> The entity that sends to a user recommendations for products and / or services that a provider of a product and / or service etc. wishes to recommend to the user may be a system administrator or the provider of the product and / or service etc.

[0125] <3> When a provider of goods and / or services, etc. is the entity that sends recommendations to users, the provider shall obtain consent from the user to hand over information necessary to send the recommendations to the provider of goods and / or services, etc. It is preferable that the information necessary to send recommendations be the bare minimum, such as an email address.

[0126] <4> When analyzing images of multiple users and sending a large number of photographed subjects to partner companies, user information and information that could identify the user will not be provided. In addition, consent will be obtained from the user to anonymize the information before providing it.

[0127] <<Example of computer hardware configuration>> 9 is a block diagram showing an example of a hardware configuration of a computer. The computer 800 may be a personal computer, a workstation, or a server computer. The computer 800 can be used as a device that realizes the functions of the image storage server 20, the information processing device 30, the user terminal 72, and the in-store terminal 74 already described.

[0128] The computer 800 includes a CPU (Central Processing Unit) 802, a RAM (Random Access Memory) 804, a ROM (Read Only Memory) 806, a GPU (Graphics Processing Unit) 808, a storage 810, a communication unit 812, an input device 814, a display device 816, and a bus 818. The GPU 808 may be provided as needed, and may be omitted if the computational load is not large.

[0129] The CPU 802 reads out various programs stored in the ROM 806 or the storage 810, etc., and executes various processes. The RAM 804 is used as a working area for the CPU 802. The RAM 804 is also used as a storage unit that temporarily stores the read out programs and various data.

[0130] The storage 810 includes a storage device configured using, for example, a hard disk device, an optical disk, a magneto-optical disk, or a semiconductor memory, or an appropriate combination of these. The storage 810 stores various programs and data required for learning processing, image analysis processing, and / or preference estimation processing and other various processing. The programs stored in the storage 810 are loaded into the RAM 804, and executed by the CPU 802, so that the computer functions as a means for performing various processing specified by the programs.

[0131] The communication unit 812 is an interface that performs communication processing with an external device via wired or wireless communication and exchanges information with the external device.

[0132] The input device 814 is an input interface that accepts various operational inputs to the computer 800. The input device 814 may be, for example, a keyboard, a mouse, a touch panel, or other pointing devices, or a voice input device, or an appropriate combination of these.

[0133] The display device 816 is an output interface that displays various information. The display device 816 may be, for example, a liquid crystal display, an organic electro-luminescence (OEL) display, a projector, or an appropriate combination of these.

[0134] About the programs that run computers A program that causes a computer to realize part or all of the processing functions of at least one of the image storage server 20, information processing device 30, and information processing device 130 described in each of the above embodiments can be recorded on a computer-readable medium such as an optical disk, a magnetic disk, a semiconductor memory, or other tangible, non-transitory information storage medium, and the program can be provided through this information storage medium.

[0135] In addition, instead of providing the program by storing it on such a tangible, non-transitory information storage medium, it is also possible to provide the program signal as a download service using a telecommunication line such as the Internet.

[0136] It is also possible to provide part or all of at least one of the processing functions of the image analysis function, preference estimation function, and recommendation providing function described in each of the above embodiments as an application server, and provide a service that provides the processing functions via a telecommunications line.

[0137] <Hardware configuration of each processing unit> The hardware structure of the processing unit that performs various processes such as the control unit 24, user authentication unit 28, image information acquisition unit 40, image analysis unit 42, word generation unit 43, incidental information analysis unit 44, news search unit 46, news information acquisition unit 48, preference estimation unit 50, related information generation unit 51, news information list generation unit 54, and image search unit 56 described in Figures 2, 3, and 7 is, for example, various processors as shown below.

[0138] Various types of processors include CPUs, which are general-purpose processors that execute programs and function as various processing units, GPUs, which are processors specialized for image processing, programmable logic devices (PLDs), such as field programmable gate arrays (FPGAs), whose circuit configuration can be changed after manufacture, and dedicated electrical circuits, such as application specific integrated circuits (ASICs), which are processors with a circuit configuration designed specifically to execute specific processes.

[0139] A processing unit may be composed of one of these various processors, or may be composed of two or more processors of the same type or different types. For example, a processing unit may be composed of multiple FPGAs, or a combination of a CPU and an FPGA, or a combination of a CPU and a GPU. Also, multiple processing units may be composed of one processor. As an example of multiple processing units being composed of one processor, first, as represented by a computer such as a client or a server, there is a form in which one processor is composed of a combination of one or more CPUs and software, and this processor functions as multiple processing units. Second, as represented by a system on chip (SoC), there is a form in which a processor is used that realizes the functions of the entire system including multiple processing units in one IC (Integrated Circuit) chip. In this way, the various processing units are composed of one or more of the above various processors as a hardware structure.

[0140] Furthermore, the hardware structure of these various processors is, more specifically, an electric circuit that combines circuit elements such as semiconductor elements.

[0141] Variation 1 The storage service using the image storage server 20 and the recommendation service using the information processing device 30 may be managed and operated by different system administrators (for example, different corporate entities).

[0142] Variation 2 The image storage server 20 may be provided with the function of the image analysis unit 42 in the information processing device 30 or 130 .

[0143] Variation 3 The images associated with the user are not limited to images owned by the user and stored in the image storage server 20, but may be posted images posted to an SNS server, etc.

[0144] "others" The configurations described in the above-described embodiment and the features described in the modified examples may be used in appropriate combinations, and some features may be replaced with each other.

[0145] The above-described embodiments of the present invention may be modified, added, or deleted as appropriate without departing from the spirit of the present invention. The present invention is not limited to the above-described embodiments, and many modifications may be made by those having ordinary skill in the relevant field within the technical concept of the present invention. [Explanation of symbols]

[0146] 10. Computer Systems 20 Image storage server 22 Communications Department 24 Control Unit 26 Image Storage 28 User authentication section 30 Information processing device 32 Communications Department 34 Processing unit 35 Storage device 36 Input Devices 38 Display device 40 Image information acquisition unit 42 Image Analysis Unit 43 Word Generation Unit 44 Additional Information Analysis Department 46 News Search Department 48 News Information Acquisition Department 50 Preference Estimation Department 51 Related Information Generation Unit 54 News information list generation unit 56 Image Search Section 70 Telecommunications Lines 72 User terminal 74 Point of sale terminal 130 Information processing equipment 134 Processing unit 800 Computers 810 Storage 812 Communications Department 814 Input Device 816 Display device 818 Bus S1-S5 Steps of information processing method S11 to S22: Processing steps by the information processing device according to the first embodiment

Claims

1. a news information acquisition unit that acquires news information representing the contents of news distributed by a news site; an image information acquisition unit that acquires an image associated with a user and supplementary information including at least information on a shooting date of the image; an image search unit that searches for images that are highly relevant to the news information from among the images based on the image contents and the shooting dates of the images; an estimation unit that estimates the user's preferences from images found by a search performed by the image search unit and the news information used in the search; An information processing device comprising:

2. The information processing device according to claim 1 , further comprising a related information generating unit that generates information related to the user's preference estimated by the estimation unit.

3. The information processing apparatus according to claim 2 , wherein the information related to the user's preferences includes information on a product or service recommended to the user.

4. The information processing device according to claim 1 , wherein the estimation unit estimates a degree of preference of the user from the news information.

5. The supplementary information includes information on the location where the image was taken, The information processing device according to claim 1 , wherein the image search unit extracts an image related to the news by using information on the shooting location.

6. a word generating unit that generates words related to the image content; The information processing device according to claim 1 , wherein the image search unit extracts images related to the news by using the generated words.

7. The information processing device of claim 1, further comprising a news information list generation unit that collects news articles from a plurality of pre-specified news sites via the news information acquisition unit, and generates a news information list that organizes the news information including dates, locations, and related keywords for each subject of the collected news articles.

8. the image search unit searches for an image highly related to a date, a location, and a related keyword of the news information from among a group of images stored in a storage device that stores the plurality of images associated with the user; The information processing device according to claim 7 , wherein the estimation unit estimates the user's preference based on images found by a search performed by the image search unit and information used in the search.

9. An information processing device as described in claim 7 or 8, wherein the news information list generation unit, when listing the news information of a news article containing a predetermined specific keyword in the news information list, adds identification information indicating that the news information is about a news article containing the specific keyword.

10. An information processing device as described in Claim 9, wherein the predetermined specific keywords include at least one of the following: large number, rush, expensive, high price, anniversary, valuable, and rare.

11. The information processing device of claim 9, wherein the estimation unit determines the degree of the user's preference for the subject of the news information to which the identification information is attached from the identification information when the search returns an image that is highly relevant to the news information to which the identification information is attached.

12. The information processing apparatus according to claim 1 , further comprising a storage device that stores the plurality of images associated with the user.

13. The information processing apparatus according to claim 8 or 12, wherein the storage device stores a plurality of images associated with a plurality of users, respectively.

14. The information processing device according to claim 1 , wherein at least a part of the image search unit and the estimation unit is configured by a trained model using a neural network.

15. An information processing device configured using a computer, Acquiring news information representing the contents of news distributed by a news site; Acquiring an image associated with a user and supplementary information including at least information on a shooting date of the image; searching for an image highly related to the news information from among the images based on the image content and the shooting date of the image; estimating the user's preferences from images found by the search and the news information used in the search; An information processing method comprising:

16. The information processing device, The method of claim 15 , further comprising generating information related to the estimated preferences of the user.

17. On the computer, A function to obtain news information that represents the contents of news distributed by news sites, A function of acquiring images associated with a user and supplementary information including at least information on the shooting date of the images; A function of searching for an image highly related to the news information from among the images based on the image content and the shooting date of the image; A function of estimating the user's preferences from images found by the search and the news information used in the search; A program to achieve this.

Images