Information processing device, information processing method, and information processing program
The information processing device addresses the lack of context in conventional image evaluation by estimating and displaying user information, enhancing user experience by reminding them of the image's context.
Patent Information
- Application Number
- JP2021132704
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2021-08-17
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2041-08-17
AI Technical Summary
Conventional image evaluation technologies fail to provide appropriate information regarding user-taken images, as they rely solely on the number of screening cycles without considering the context or user information.
An information processing device that acquires user information, estimates the context of image capture based on this information, and displays context information alongside the image to provide relevant details.
Enhances user experience by helping users remember the context of their images through displayed context information, thereby providing appropriate information about the images taken.
Smart Images

Figure 0007590285000001 
Figure 0007590285000002 
Figure 0007590285000003
Abstract
Description
[Technical field]
[0001] The present invention relates to an information processing device, an information processing method, and an information processing program. [Background technology]
[0002] In recent years, with the advancement of cameras and computer peripheral technologies, the number of captured images stored by users has increased enormously. Therefore, there is a demand for technology to evaluate and organize such a huge number of captured images according to some criteria.
[0003] For example, a selection cycle is repeated in which images are judged as "use", "not use" or "reserved", and when the judgment for all images is completed, the "reserved" images are judged again as "reserved". Then, for the image finally judged as "use", the number of selection cycles in which the judgment was made is applied as an evaluation. [Prior art documents] [Patent documents]
[0004] [Patent Document 1] JP 2011-234078 A Summary of the Invention [Problem to be solved by the invention]
[0005] However, the above-mentioned conventional technology does not necessarily provide appropriate information regarding the image captured by the user. For example, the above-mentioned conventional technology only applies the number of selection cycles in which the judgment was made to the image judged as "usable" as an evaluation, so it cannot be said that the above-mentioned conventional technology can provide appropriate information regarding the image captured by the user.
[0006] The present application has been made in consideration of the above, and has an object to provide an information processing device, an information processing method, and an information processing program that can provide appropriate information regarding an image captured by a user. [Means for solving the problem]
[0007] The information processing device of the present application is characterized by comprising an acquisition unit that acquires user information regarding a user who captured an image, an estimation unit that estimates the context when the user captured the image based on the user information, and a display control unit that, when displaying the image, displays context information indicating the context together with the image. Effect of the Invention
[0008] According to one aspect of the embodiment, it is possible to provide appropriate information regarding an image captured by a user. [Brief description of the drawings]
[0009] [Figure 1] FIG. 1 is a diagram showing an overview of information processing according to an embodiment. [Diagram 2] FIG. 2 is a diagram illustrating an example of the configuration of a terminal device according to the embodiment. [Diagram 3] FIG. 3 is a diagram illustrating an example of a configuration of an information providing device according to an embodiment. [Figure 4] FIG. 4 is a diagram illustrating an example of the user information database. [Diagram 5] FIG. 5 is a diagram illustrating an example of the history information database. [Figure 6] FIG. 6 is a diagram showing an example of the image information database. [Figure 7] FIG. 7 is a diagram illustrating an example of a hardware configuration. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0010] Hereinafter, the information processing device, the information processing method, and the information processing program according to the present application will be described in detail with reference to the drawings. Note that the information processing device, the information processing method, and the information processing program according to the present application are not limited to the embodiments. In addition, the same components in the following embodiments are denoted by the same reference numerals, and duplicated descriptions are omitted.
[0011] [1. Overview of information processing method] Fig. 1 is a diagram showing an overview of information processing according to an embodiment. The information processing shown in Fig. 1 is realized by an information processing system 1. In Fig. 1, it is assumed that the information processing according to the embodiment is realized by an information providing device 100, which is an example of an information processing device according to the present application.
[0012] 1, an information processing system 1 according to an embodiment includes a terminal device 10 and an information providing device 100. The terminal device 10 and the information providing device 100 are connected to each other via a network N (not shown) in a wired or wireless manner so as to be able to communicate with each other. The network N is, for example, a local area network (LAN) or a wide area network (WAN) such as the Internet.
[0013] Furthermore, the number of devices included in the information processing system 1 shown in Fig. 1 is not limited to that shown in the figure. For example, in Fig. 1, for the sake of simplicity, only one terminal device 10 is shown, but this is merely an example and is not limiting, and two or more devices may be included.
[0014] The terminal device 10 shown in FIG. 1 is an information processing device used by a user U (hereinafter, may be abbreviated as user U) of a service provided by the information providing device 100. For example, the terminal device 10 is realized by a smartphone, a tablet terminal, a notebook PC (Personal Computer), a desktop PC, a mobile phone, a PDA (Personal Digital Assistant), or the like. The terminal device 10 displays information distributed by the information providing device 100 by a web browser or an application. The terminal device 10 is also equipped with a camera. The example shown in FIG. 1 illustrates a case where the terminal device 10 is a smartphone.
[0015] The information providing device 100 is an information processing device that provides a service (hereinafter also referred to as an image providing service) that accepts uploads of images taken by users and delivers images in response to user requests, and is realized by, for example, a server device or a cloud system. Specifically, the information providing device 100 provides, as an example of the image providing service, a service related to SNS (Social Networking Service) such as Facebook (registered trademark) and Instagram (registered trademark), or a message service such as LINE (registered trademark). For example, when the information providing device 100 accepts posting of various information such as text entered by a user or images (still images, moving images) taken by the user using the terminal device 10, the information providing device 100 makes the accepted information public, thereby allowing the information to be shared among users of the image providing service. Note that FIG. 1 will explain a case where the information providing device 100 provides a message service such as LINE (registered trademark).
[0016] In FIG. 1, on June 10, 2021, a user U1 of an image providing service provided by the information providing device 100 takes an image G1 using a terminal device 10 and posts the taken image G1 to a message service. The terminal device 10 transmits the image G1 to the information providing device 100 in response to an operation by the user U1. The information providing device 100 acquires the image G1 from the terminal device 10 (step S1). Note that, although FIG. 1 describes a case where the image G1 is a still image, the image G1 may also be a moving image.
[0017] When the information providing device 100 accepts the posting of the image G1, it makes the accepted image G1 public, thereby allowing the image G1 to be shared among users of the message service. Furthermore, when the information providing device 100 acquires the image G1, it acquires user information about the user U1 who captured the image G1 (step S2).
[0018] Next, when the information providing device 100 acquires the user information, it estimates the context when the user U1 captured the image G1 based on the acquired user information (step S3). For example, the information providing device 100 estimates that the family of the user U1 moved together around the time when the image G1 was captured, based on the history of the location information of the family of the user U1 around the time when the image G1 was captured. Next, since the family of the user U1 moved together around the time when the image G1 was captured, the information providing device 100 estimates that the user U1 was on a family trip, as the context when the user U1 captured the image G1. Next, when the information providing device 100 estimates the context, it stores the context information CT1 indicating the estimated context in association with the image G1 (step S3).
[0019] After that, on August 2, 2021, some time after June 10, 2021, user U1 views image G1 posted to the message service. In response to an operation by user U1, terminal device 10 transmits a display request for image G1 to information providing device 100. Information providing device 100 accepts the display request for image G1 from terminal device 10 (step S4).
[0020] Next, when the information providing device 100 accepts a display request for the image G1, it provides the context information CT1 together with the image G1 to the terminal device 10 (step S5). For example, the information providing device 100 generates content in which the context information CT1 (in the example shown in FIG. 1 , text information "traveling with family") is displayed as a caption for the image G1 together with the image G1.
[0021] For example, the information providing device 100 generates content that displays a list of thumbnails of a plurality of images taken by the user U1, as well as context information indicating the context of the user U1 when each of the images was taken as a caption for each of the plurality of images. After generating the content, the information providing device 100 transmits the generated content to the terminal device 10. Upon receiving the content, the terminal device 10 displays the received content on a screen.
[0022] As described above, the information providing device 100 according to the embodiment estimates the context when the user U1 took the image G1 based on the user information about the user U1 who took the image G1. Furthermore, when the information providing device 100 displays the image G1, it causes the terminal device 10 to display the context information CT1 indicating the context when the user U1 took the image G1 together with the image G1. In this way, when the information providing device 100 displays the image G1, it causes the user U1 to visually recognize the context information CT1 (in the example shown in FIG. 1, text information "traveling with family") together with the image G1, thereby helping the user U1 to recall experiences related to the image G1. Therefore, the information providing device 100 can provide appropriate information about the image taken by the user.
[0023] [2. Example of terminal device configuration] Fig. 2 is a diagram showing an example of the configuration of the terminal device 10. As shown in Fig. 2, the terminal device 10 includes a communication unit 11, a display unit 12, an input unit 13, a positioning unit 14, a sensor unit 20, a control unit 30 (controller), and a storage unit 40.
[0024] (Communications Department 11) The communication unit 11 is connected to a network N (not shown) by wire or wirelessly, and transmits and receives information to and from the information providing device 100 via the network N. For example, the communication unit 11 is realized by a NIC (Network Interface Card), an antenna, etc.
[0025] (Display section 12) The display unit 12 is a display device that displays various information such as position information. For example, the display unit 12 is a liquid crystal display (LCD) or an organic electro-luminescent display (OLED). The display unit 12 is a touch panel display, but is not limited to this.
[0026] (Input section 13) The input unit 13 is an input device that accepts various operations from the user U. The input unit 13 also has, for example, buttons for inputting characters, numbers, and the like. If the display unit 12 is a touch panel display, a part of the display unit 12 functions as the input unit 13. The input unit 13 may be a microphone that accepts voice input from the user U. The microphone may be wireless.
[0027] (Positioning unit 14) The positioning unit 14 receives a signal (radio wave) transmitted from a satellite of a GPS (Global Positioning System), and acquires position information (e.g., latitude and longitude) indicating the current position of the terminal device 10, which is the device itself, based on the received signal. That is, the positioning unit 14 measures the position of the terminal device 10. Note that the GPS is merely an example of a GNSS (Global Navigation Satellite System).
[0028] The positioning unit 14 can measure the position by various methods other than GPS. For example, the positioning unit 14 may measure the position by using various communication functions of the terminal device 10 as described below as an auxiliary positioning means for position correction or the like.
[0029] (Wi-Fi positioning) For example, the positioning unit 14 uses a Wi-Fi (registered trademark) communication function of the terminal device 10 or a communication network provided by each communication company to measure the position of the terminal device 10. Specifically, the positioning unit 14 performs Wi-Fi communication or the like and measures the distance to a nearby base station or access point to measure the position of the terminal device 10.
[0030] (Beacon positioning) The positioning unit 14 may measure the position by using a Bluetooth (registered trademark) function of the terminal device 10. For example, the positioning unit 14 measures the position of the terminal device 10 by connecting to a beacon transmitter connected by the Bluetooth (registered trademark) function.
[0031] (geomagnetic positioning) The positioning unit 14 also measures the position of the terminal device 10 based on a geomagnetic pattern of a structure measured in advance and a geomagnetic sensor included in the terminal device 10.
[0032] (RFID positioning) Furthermore, for example, if the terminal device 10 has a function of an RFID (Radio Frequency Identification) tag equivalent to a contactless IC card used at station ticket gates, stores, etc., or has a function of reading an RFID tag, the location of use is recorded together with information on a payment or the like made by the terminal device 10. The positioning unit 14 may obtain such information to measure the location of the terminal device 10. Furthermore, the location may be measured by an optical sensor, an infrared sensor, or the like provided in the terminal device 10.
[0033] The positioning unit 14 may, if necessary, measure the position of the terminal device 10 using one or a combination of the positioning means described above.
[0034] (Sensor unit 20) The sensor unit 20 includes various sensors mounted on or connected to the terminal device 10. The connection may be wired or wireless. For example, the sensors may be detection devices other than the terminal device 10, such as wearable devices and wireless devices. In the example shown in FIG. 2, the sensor unit 20 includes an acceleration sensor 21, a gyro sensor 22, an air pressure sensor 23, a temperature sensor 24, a sound sensor 25, a light sensor 26, a magnetic sensor 27, and an image sensor (camera) 28.
[0035] The above-mentioned sensors 21-28 are merely examples and are not limiting. That is, the sensor unit 20 may be configured to include some of the sensors 21-28, or may include other sensors such as a humidity sensor in addition to or instead of the sensors 21-28. For example, the sensor unit 20 may include a biosensor capable of measuring biometric information of the user (for example, body temperature, heartbeat, blood pressure, heart rate, etc.). The terminal device 10 acquires the biometric information of the user detected by the biosensor included in the sensor unit 20. The terminal device 10 transmits the acquired biometric information of the user to the information providing device 100.
[0036] The acceleration sensor 21 is, for example, a three-axis acceleration sensor, and detects physical movements of the terminal device 10, such as the moving direction, speed, and acceleration of the terminal device 10. The gyro sensor 22 detects physical movements of the terminal device 10, such as tilts in three axial directions, based on the angular velocity, etc. of the terminal device 10. The air pressure sensor 23 detects, for example, the air pressure around the terminal device 10.
[0037] Since the terminal device 10 includes the above-mentioned acceleration sensor 21, gyro sensor 22, and air pressure sensor 23, it is possible to measure the position of the terminal device 10 using a technique such as Pedestrian Dead-Reckoning (PDR) that uses these sensors 21 to 23. This makes it possible to obtain indoor position information that is difficult to obtain using a positioning system such as GPS.
[0038] For example, the number of steps, walking speed, and distance walked can be calculated by a pedometer using the acceleration sensor 21. In addition, the moving direction, line of sight direction, and body inclination of the user U can be known by using the gyro sensor 22. In addition, the altitude and floor number on which the terminal device 10 of the user U is located can be known from the air pressure detected by the air pressure sensor 23.
[0039] The air temperature sensor 24 detects, for example, the air temperature around the terminal device 10. The sound sensor 25 detects, for example, sounds around the terminal device 10. The light sensor 26 detects the illuminance around the terminal device 10. The magnetic sensor 27 detects, for example, the geomagnetism around the terminal device 10. The image sensor 28 (for example, a camera) captures an image around the terminal device 10.
[0040] The above-mentioned air pressure sensor 23, temperature sensor 24, sound sensor 25, light sensor 26, and image sensor 28 can detect the air pressure, temperature, sound, and illuminance, and capture images of the surroundings, thereby detecting the environment and situation around the terminal device 10. In addition, the accuracy of the location information of the terminal device 10 can be improved based on the environment and situation around the terminal device 10.
[0041] (Control unit 30) The control unit 30 includes, for example, a microcomputer having a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM, an input / output port, and various other circuits. The control unit 30 may also be configured with hardware such as an integrated circuit, for example, an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array). The control unit 30 includes a transmitting unit 31, a receiving unit 32, and a processing unit 33.
[0042] (Transmitter 31) The transmission unit 31 can transmit, for example, various pieces of information input by the user U using the input unit 13, various pieces of information detected by the sensors 21-28 mounted on or connected to the terminal device 10, and position information of the terminal device 10 measured by the positioning unit 14 to the information providing device 100 via the communication unit 11. For example, the transmission unit 31 transmits an image captured by the image sensor 28 to the information providing device 100. In addition, the transmission unit 31 transmits a request to display the image to the information providing device 100.
[0043] (Receiving unit 32) The receiving unit 32 can receive, via the communication unit 11, various pieces of information provided by the information providing device 100 and requests for various pieces of information from the information providing device 100. For example, the receiving unit 32 receives content including an image and context information from the information providing device 100 via the communication unit 11.
[0044] (Processing section 33) The processing unit 33 controls the entire terminal device 10 including the display unit 12 etc. For example, the processing unit 33 can output various information transmitted by the transmission unit 31 and various information received by the reception unit 32 from the information providing device 100 to the display unit 12 for display. For example, the processing unit 33 can output content including an image and context information received by the reception unit 32 to the display unit 12 for display.
[0045] (Storage unit 40) The storage unit 40 is realized by, for example, a semiconductor memory element such as a random access memory (RAM) or a flash memory, or a storage device such as a hard disk drive (HDD), a solid state drive (SSD), an optical disk, etc. Various programs and various data are stored in the storage unit 40.
[0046] [3. Example of the configuration of the information providing device] 3 is a diagram showing an example of a configuration of the information providing device 100 according to the embodiment. As shown in FIG.
[0047] (Communication unit 110) The communication unit 110 is realized by, for example, a network interface card (NIC), etc. The communication unit 110 is also connected to a network N (not shown) in a wired or wireless manner.
[0048] (Storage unit 120) The storage unit 120 is realized by, for example, a semiconductor memory element such as a random access memory (RAM) or a flash memory, or a storage device such as a hard disk or an optical disk. As shown in FIG. 3, the storage unit 120 has a user information database 121, a history information database 122, and an image information database 123.
[0049] (User Information Database 121) The user information database 121 stores various information related to a user U of the service provided by the information providing device 100. For example, the user information database 121 stores various information such as attributes of the user U. Fig. 4 is a diagram showing an example of the user information database 121. In the example shown in Fig. 4, the user information database 121 has items such as "User ID (Identifier)", "Age", "Gender", "Home", "Workplace", and "Interests".
[0050] "User ID" indicates identification information for identifying user U. Furthermore, "age" indicates the age of user U identified by the user ID. Note that "age" may be the specific age of user U identified by the user ID, such as 35 years old. Furthermore, "gender" indicates the gender of user U identified by the user ID.
[0051] Furthermore, "home" indicates the location information of the home of user U identified by the user ID. In the example shown in Fig. 4, "home" is illustrated as an abstract code such as "LC11", but it may also be latitude and longitude information or the like. Furthermore, for example, "home" may also be a region name or an address.
[0052] Furthermore, "workplace" indicates location information of the workplace (school in the case of a student) of user U identified by the user ID. Note that in the example shown in Fig. 4, "workplace" is illustrated as an abstract code such as "LC12", but it may also be latitude and longitude information, etc. Furthermore, for example, "workplace" may also be the name of a region or an address.
[0053] Furthermore, "interests" indicate the interests of a user U identified by a user ID. In other words, "interests" indicate subjects in which a user U identified by a user ID is highly interested. For example, an "interest" may be a search query (keyword) that a user U inputs into a search engine. Note that, although one "interest" is illustrated for each user U in the example shown in FIG. 4, there may be multiple "interests."
[0054] For example, in the example shown in FIG. 4, the age of user U identified by user ID "U1" is "20s" and the gender is "male." Furthermore, for example, the user U identified by user ID "U1" indicates that his / her home address is "LC11." Furthermore, for example, the user U identified by user ID "U1" indicates that his / her workplace is "LC12." Furthermore, for example, the user U identified by user ID "U1" indicates that he / she is interested in "sports."
[0055] Here, in the example shown in Figure 4, abstract values such as "U1", "LC11", and "LC12" are used for illustration, but "U1", "LC11", and "LC12" are assumed to store information such as specific character strings and numerical values. In the following figures relating to other information, abstract values may also be illustrated.
[0056] The user information database 121 may store various information according to the purpose, not limited to the above. For example, the user information database 121 may store various information related to the terminal device 10 of the user U. The user information database 121 may also store information related to the attributes of the user U, such as demographic attributes, psychographic attributes, geographic attributes, and behavioral attributes. For example, the user information database 121 may store information such as name, family structure, place of origin (hometown), occupation, job title, income, qualifications, type of residence (detached house, apartment, etc.), whether or not the user has a car, commuting time, commuting route, commuter pass section (station, line, etc.), frequently used station (other than the nearest station to home or workplace), extracurricular activities (location, time zone, etc.), hobbies, interests, and lifestyle.
[0057] (Historical Information Database 122) The history information database 122 stores various information related to history information (log data) indicating the behavior of the user U. Fig. 5 is a diagram showing an example of the history information database 122. In the example shown in Fig. 5, the history information database 122 has items such as "user ID", "location history", "search history", "browsing history", "purchase history", and "posting history".
[0058] "User ID" indicates identification information for identifying user U. "Location history" indicates location history, which is a history of user U's location and movements. "Search history" indicates search history, which is a history of search queries entered by user U. "Browser history" indicates browsing history, which is a history of content viewed by user U. "Purchase history" indicates purchase history, which is a history of purchases made by user U. "Post history" indicates posting history, which is a history of posts made by user U. "Post history" may include questions regarding user U's possessions.
[0059] For example, in the example shown in Figure 5, user U identified by user ID "U1" moved as shown in "Location History #1," searched as shown in "Search History #1," viewed content as shown in "Browse History #1," purchased specified products etc. at specified stores etc. as shown in "Purchase History #1," and posted as shown in "Post History #1."
[0060] Here, in the example shown in Figure 5, abstract values such as "U1," "Location History #1," "Search History #1," "Browse History #1," "Purchase History #1," and "Post History #1" are used to illustrate the data; however, specific information such as character strings and numbers is stored in "U1," "Location History #1," "Search History #1," "Browse History #1," "Purchase History #1," and "Post History #1."
[0061] The history information database 122 may store various information according to the purpose, not limited to the above. For example, the history information database 122 stores message information related to the history of messages exchanged between the user U and other users in the past in a messaging service such as LINE (registered trademark).
[0062] The history information database 122 may also store the user U's visit history to a physical store or facility visit history. The history information database 122 may also store the user U's payment history in electronic payments.
[0063] (Image Information Database 123) The image information database 123 stores various information related to images captured by the user U. Fig. 6 is a diagram showing an example of the image information database 123. In the example shown in Fig. 6, the image information database 123 has items such as "image ID", "photographer", "shooting date and time", "shooting location", "person information included in the image", and "context information".
[0064] "Image ID" indicates identification information for identifying an image. "Photographer" indicates identification information for identifying the user who photographed the image identified by the image ID.
[0065] Furthermore, the "shooting date and time" indicates the date and time when the image identified by the image ID was shot. For example, if the image is a video, the "shooting date and time" may be a set of information on the start date and time of shooting the video and the end date and time of shooting the video. Note that in the example shown in FIG. 6, the "shooting date and time" is illustrated using an abstract value such as "shooting date and time #1", but the "shooting date and time" stores information such as a specific character string or a numerical value.
[0066] Furthermore, the "photography location" indicates the location information of the location where the image identified by the image ID was photographed. In the example shown in FIG. 6, the "photography location" is illustrated using an abstract value such as "photography location #1", but it may also be latitude and longitude information, etc. Furthermore, for example, the "photography location" may also be a region name or an address.
[0067] Furthermore, the "personal information included in the image" indicates information about a person appearing in the image identified by the image ID. In the example shown in FIG. 6, the "personal information included in the image" is illustrated using an abstract value such as "personal information #1", but the "personal information included in the image" stores information such as a specific character string or a numerical value. For example, the "personal information included in the image" may be information indicating the relationship between the person appearing in the image identified by the image ID and the photographer (e.g., "child", "workplace colleague", etc.). Furthermore, the "personal information included in the image" may be a user ID that identifies the person appearing in the image, or may be the name of the person appearing in the image, etc.
[0068] Furthermore, the "context information" indicates the context of the user (photographer) who captured the image identified by the image ID. Note that in the example shown in FIG. 6, the "context information" is illustrated using an abstract value such as "CT1", but the "context information" stores information such as specific character strings and numerical values. For example, the "context information" may be character information indicating the context (e.g., "traveling with family," "child's birthday," etc.).
[0069] (Control unit 130) Returning to Fig. 3, the description will be continued. The control unit 130 is a controller, and is realized by, for example, a central processing unit (CPU), a micro processing unit (MPU), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or the like, executing various programs (corresponding to an example of an information processing program) stored in a storage device inside the information providing device 100 using a storage area such as a RAM as a working area. In the example shown in Fig. 3, the control unit 130 has an acquisition unit 131, a generation unit 132, an estimation unit 133, a reception unit 134, and a display control unit 135.
[0070] (Acquisition part 131) The acquisition unit 131 acquires various information related to users of the services provided by the information providing device 100 from the terminal device 10, various servers, etc., via the communication unit 110. For example, the acquisition unit 131 acquires information indicating the user (such as a user ID), location information of the user, attribute information of the user, etc. Then, the acquisition unit 131 registers the information in the user information database 121 of the storage unit 120. The acquisition unit 131 may also acquire biometric information of the user from the terminal device 10. When acquiring the biometric information of the user, the acquisition unit 131 may register the biometric information in the user information database 121 in association with the acquisition date and time when the terminal device 10 acquired the biometric information.
[0071] Furthermore, the acquisition unit 131 acquires various types of history information (log data) indicating the actions of the user from various servers and the like via the communication unit 110. Then, the acquisition unit 131 registers the acquired history information in the history information database 122 of the storage unit 120.
[0072] The acquisition unit 131 also acquires an image taken by a user from the terminal device 10 of the user via the communication unit 110. For example, the acquisition unit 131 acquires an image taken by a user together with a user ID that identifies the user (photographer) who took the image. After acquiring the image, the acquisition unit 131 acquires image information related to the acquired image. For example, the acquisition unit 131 acquires information indicating the shooting date and time and the shooting location, which are metadata linked to the acquired image, as an example of the image information. Furthermore, the acquisition unit 131 acquires subject user information related to a subject user who is a person included in the image, as an example of the image information. For example, the acquisition unit 131 identifies a person (subject user) included in the image using a known image analysis technique. Then, the acquisition unit 131 acquires subject user information by referring to the user information database 121 and the history information database 122 based on the user ID of the subject user, for example. After acquiring the image information, the acquisition unit 131 stores the acquired image information in the image information database 123 in association with the user ID of the photographer.
[0073] (Generation unit 132) The generating unit 132 generates a machine learning model that estimates user information related to a person included in an image taken by a user from the image. Specifically, the generating unit 132 trains the machine learning model by using a pair of an image taken by a user and user information related to a person included in the image as correct answer data. That is, the generating unit 132 trains the machine learning model using a technique such as backpropagation so that when an image taken by a user is input to the machine learning model, user information related to a person included in the image or information corresponding to the user information related to a person included in the image is output. For example, the generating unit 132 specifies information related to an exchange with a person included in the image as user information related to a person included in the image from among the purchase information, search information, browsing information, location information, reservation information, posting information, and message information of the user before and after the shooting time of the image. For example, the generating unit 132 specifies information related to a history of message exchanges between the user and the person shown in the image (for example, a chat destination, etc.) from among the message information before and after the shooting time of the image as user information related to a person included in the image.
[0074] Furthermore, the generating unit 132 estimates user information related to a person included in the image using a trained machine learning model. For example, the generating unit 132 inputs an acquired image to the trained machine learning model. Then, the generating unit 132 acquires user information related to a person included in an image output by the machine learning model when the image is input (or user information related to a person included in an image indicated by information output by the machine learning model) as an estimation result. For example, the generating unit 132 estimates purchase information, search information, browsing information, location information, reservation information, posting information, and message information output by the machine learning model when an image taken by a user is input. For example, the generating unit 132 acquires information on a history of message exchanges between the user and the person appearing in the image (for example, a chat destination, etc.) as an estimation result.
[0075] (Estimation part 133) When the acquisition unit 131 acquires an image, the estimation unit 133 acquires user information on the user who captured the image (hereinafter, also referred to as the photographer). More specifically, the estimation unit 133 acquires user information on the photographer before and after the capture date and time of the image (for example, a predetermined period including the capture date and time of the image). For example, the acquisition unit 131 acquires user information on the photographer before and after the capture date and time of the image by referring to the user information database 121 and the history information database 122 based on the user ID acquired together with the image. For example, the acquisition unit 131 acquires, as examples of user information, purchase information on the purchase history of the photographer, search information on the search history of the photographer, browsing information on the browsing history of the photographer, location information on the location history of the photographer, reservation information on the reservation history of the photographer, posting information on the posting history of the photographer, and message information on the contents of the message of the photographer.
[0076] Next, when the estimation unit 133 acquires the user information of the photographer around the shooting date and time of the image, the estimation unit 133 estimates the context when the user (photographer) captured the image based on the acquired user information. More specifically, the estimation unit 133 estimates the purpose of the user's behavior when the user captured the image as an example of the context. For example, it is assumed that the user's family uses the service provided by the information providing device 100. At this time, based on the location information of the user's family around the shooting date and time of the image, if the location history of the user's family moves together around the shooting date and time of the image, the estimation unit 133 estimates that the purpose of the user's behavior when the user captured the image was a family trip.
[0077] Furthermore, the estimation unit 133 estimates user information related to a person included in the image, as an example of context, using the machine learning model generated by the generation unit 132. For example, the estimation unit 133 estimates a history of message exchanges (e.g., chat destinations) between the user and the person appearing in the image around the time the image was taken, as an example of context. When the estimation unit 133 estimates the context when the user took the image, it stores context information indicating the estimated context in the image information database 123 in association with the image.
[0078] (Reception Department 134) The receiving unit 134 receives an image display request from the terminal device 10 via the communication unit 110. For example, the receiving unit 134 receives a display request for an image designated by a user from among images photographed by the user.
[0079] (Display control unit 135) When displaying an image, the display control unit 135 displays context information indicating a context together with the image. Specifically, when the reception unit 134 receives a request to display an image, the display control unit 135 refers to the image information database 123 and acquires the image designated by the user and the context information associated with the image designated by the user. Then, the display control unit 135 generates content in which the image designated by the user and the context information associated with the image are displayed together with the image designated by the user.
[0080] For example, the display control unit 135 generates content in which a list of thumbnails of a plurality of images taken by a user is displayed together with context information indicating the context of the user when each of the images was taken as a caption for each of the plurality of images. After generating the content, the display control unit 135 then transmits the generated content to the terminal device 10.
[0081] Furthermore, the display control unit 135 generates content in which a history of message exchanges between the user and the person appearing in the image before and after the image was captured (for example, chat destinations) is displayed together with the image as context information. After generating the content, the display control unit 135 transmits the generated content to the terminal device 10.
[0082] 4. Modifications The above-described terminal device 10 and information providing device 100 may be implemented in various different forms other than the above-described embodiment. Therefore, modifications of the embodiment will be described below.
[0083] In the above embodiment, the estimation unit 133 estimates the purpose of the user's behavior when the user took an image (e.g., a family trip) as an example of the context, but the estimation unit 133 may estimate a context other than the purpose of the user's behavior. Specifically, the estimation unit 133 may estimate the state of the user, the emotion of the user, or an event occurring around the user when the user took the image as the context.
[0084] For example, the estimation unit 133 estimates the emotion of the user when the user took the image based on the biometric information of the user around the time of taking the image. For example, if the user's body temperature, blood pressure, heart rate, etc. around the time of taking the image are within a range of values indicating a predetermined emotion (for example, anger, joy, sadness, etc.), the estimation unit 133 may estimate that the user had a predetermined emotion (for example, anger, joy, sadness, etc.) when taking the image.
[0085] Furthermore, the estimation unit 133 estimates the context based on the relationship between the person included in the image and the user. For example, it is assumed that the person included in the image uses a service provided by the information providing device 100. At this time, for example, if the workplace of the person included in the image is the same as the workplace of the user who took the image, the estimation unit 133 may estimate that an event related to the user's workplace (e.g., an internal meeting, a company trip, etc.) was taking place when the user took the image. Furthermore, if the person included in the image is a family member of the user, the estimation unit 133 may estimate that an event related to the user's family (e.g., a family member's birthday, a family trip, etc.) was taking place when the user took the image.
[0086] In addition, based on the location information of the user around the date and time when the image was taken, for example, if the user stayed at the workplace for a long time around the date and time when the image was taken, the estimation unit 133 may estimate that the user's condition at the time the image was taken was that he or she had been at the workplace for a long time and the work was difficult.
[0087] In addition, the estimation unit 133 may perform semantic analysis of text information contained in the user's posted information around the time the image was taken using a publicly known natural language processing technique, thereby estimating the purpose of the user's actions at the time the user took the image, the user's state, the user's emotions, or events that occurred around the user.
[0088] In addition, in the above embodiment, a case has been described in which the estimation unit 133 estimates the context of the photographer who took the image based on the user information of the photographer who took the image, but the estimation unit 133 may also estimate the context of a user other than the photographer based on user information of the user other than the photographer.
[0089] Specifically, the estimation unit 133 estimates the context of the subject user at the time the image was captured, based on subject user information related to the subject user who is a person included in the image. When the estimation unit 133 estimates the context of the subject user, the estimation unit 133 stores subject context information indicating the context of the subject user in association with the image in the image information database 123. When the display control unit 135 displays an image on the terminal device 10 of the subject user, it displays the subject context information indicating the context of the subject user together with the image.
[0090] In the above embodiment, the processing executed by the information providing device 100 may actually be executed by the terminal device 10. That is, the processing may be completed in a standalone manner (by the terminal device 10 alone). In this case, the terminal device 10 is assumed to have the functions of the information providing device 100 in the above embodiment.
[0091] 5. Effects As described above, the information processing device (information providing device 100) according to the present application includes an acquisition unit 131, an estimation unit 133, and a display control unit 135. The acquisition unit 131 acquires user information related to a user who captured an image. The estimation unit 133 estimates the context when the user captured the image based on the user information. When displaying an image, the display control unit 135 displays context information indicating the context together with the image.
[0092] In this way, when the information processing device displays an image, the information processing device can help the user to recall the experience related to the image when the image was taken by displaying the context information together with the image and allowing the user to visually recognize the context information. Therefore, the information processing device can provide appropriate information regarding the image taken by the user.
[0093] Furthermore, the estimation unit 133 estimates, as the context, the purpose of the user's action when the user captured the image, the user's state, the user's emotion, or an event that occurred around the user.
[0094] As a result, when the information processing device displays an image, it can help the user recall experiences related to the image when the image was taken by displaying and allowing the user to visually recognize the purpose of the user's actions at the time the image was taken, the user's state, the user's emotions, or events that occurred around the user.
[0095] Furthermore, the estimation unit 133 estimates the context based on the user information before and after the shooting date and time of the image.
[0096] This allows the information processing device to appropriately estimate the context in which the user captured the image, based on the user information before and after the capture date and time of the image.
[0097] Furthermore, the estimation unit 133 estimates a context based on the relationship between a person included in an image and the user.
[0098] This allows the information processing device to appropriately estimate the context in which the user captured the image, based on the relationship between the person included in the image and the user.
[0099] In addition, the acquisition unit 131 acquires, as user information, purchase information related to the user's purchasing history, search information related to the user's search history, browsing information related to the user's browsing history, location information related to the user's location history, reservation information related to the user's reservation history, posting information related to the user's posting history, and message information related to the content of the user's messages.
[0100] This allows the information processing device to appropriately estimate the context when the user captured an image, based on behavioral information related to the user's behavior history.
[0101] The information processing device (information providing device 100) further includes a generating unit 132. The generating unit 132 generates a machine learning model that estimates user information related to a person included in an image from the image. The display control unit 135 displays the user information related to the person included in the image estimated using the machine learning model as context information.
[0102] As a result, when an information processing device displays an image, it can help the user recall experiences related to the image when the image was captured by, for example, displaying a history of messages exchanged between the user and a person included in the image before and after the user captured the image together with the image, allowing the user to visually check the history of messages exchanged between the user and a person included in the image before and after the user captured the image.
[0103] Furthermore, the acquisition unit 131 acquires subject user information related to the subject user who is a person included in the image. The estimation unit 133 estimates the context of the subject user when the image was captured based on the subject user information. When displaying the image, the display control unit 135 displays subject context information indicating the context of the subject user together with the image.
[0104] As a result, when the information processing device displays an image on the subject user's terminal device, it can display context information along with the image so that the subject user can visually confirm it, thereby helping the subject user recall the experience related to the image when the image was taken.
[0105] [6. Hardware Configuration] Moreover, the information providing device 100 according to each embodiment described above is realized, for example, by a computer 1000 configured as shown in Fig. 7. The information providing device 100 will be described below as an example. Fig. 7 is a hardware configuration diagram showing an example of a computer that realizes the functions of the information providing device. The computer 1000 has a CPU 1100, a ROM 1200, a RAM 1300, a HDD 1400, a communication interface (I / F) 1500, an input / output interface (I / F) 1600, and a media interface (I / F) 1700.
[0106] The CPU 1100 operates and controls each unit based on a program stored in the ROM 1200 or the HDD 1400. The ROM 1200 stores a boot program executed by the CPU 1100 when the computer 1000 is started up, programs that depend on the hardware of the computer 1000, and the like.
[0107] The HDD 1400 stores programs executed by the CPU 1100, data used by such programs, etc. The communication interface 1500 receives data from other devices via a communication network 500 (corresponding to the network N in the embodiment) and sends the data to the CPU 1100, and also transmits data generated by the CPU 1100 to other devices via the communication network 500.
[0108] The CPU 1100 controls output devices such as a display and a printer, and input devices such as a keyboard and a mouse, via the input / output interface 1600. The CPU 1100 acquires data from the input devices via the input / output interface 1600. The CPU 1100 also outputs data generated via the input / output interface 1600 to the output devices.
[0109] Media interface 1700 reads a program or data stored in recording medium 1800 and provides it to CPU 1100 via RAM 1300. CPU 1100 loads the program from recording medium 1800 onto RAM 1300 via media interface 1700 and executes the loaded program. Recording medium 1800 is, for example, an optical recording medium such as a DVD (Digital Versatile Disc) or a PD (Phase change rewritable Disc), a magneto-optical recording medium such as an MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory.
[0110] For example, when the computer 1000 functions as the information providing device 100, the CPU 1100 of the computer 1000 executes a program loaded onto the RAM 1300 to realize the function of the control unit 130. In addition, the HDD 1400 stores each piece of data in the storage device of the information providing device 100. The CPU 1100 of the computer 1000 reads and executes these programs from the recording medium 1800, but as another example, these programs may be obtained from another device via a predetermined communication network.
[0111] [7. Other] Although some of the embodiments of the present application have been described in detail above with reference to the drawings, these are merely examples, and the present invention can be embodied in other forms that incorporate various modifications and improvements based on the knowledge of those skilled in the art, including the forms described in the Disclosure of the Invention section.
[0112] Further, the information providing device 100 described above can be flexibly modified in configuration, for example, by calling an external platform or the like using an API (Application Programming Interface) or network computing to realize some functions.
[0113] Furthermore, the term "unit" in the claims may be read as "means" or "circuit," etc. For example, an acquisition unit may be read as an acquisition means or an acquisition circuit. [Explanation of symbols]
[0114] 1. Information Processing Systems 10 Terminal Equipment 100 Information provision device 110 Communications Department 120 Storage section 121 User Information Database 122 History Information Database 123 Image Information Database 130 Control section 131 Acquisition Department 132 Generation part 133 Estimation Department 134 Reception Department 135 Display control unit
Claims
1. An acquisition unit that acquires relationship information generated from an image taken by a user using a machine learning model that has been trained to output relationship information indicating a relationship between the subject user, who is a person included in an image taken by the user, and the user when the image is input; an estimation unit that identifies a relationship between the subject user and the user based on the relationship information acquired by the acquisition unit, and estimates, based on the relationship, that an event based on the relationship was taking place as a context when the user took the image; a display control unit that, when displaying the image, displays context information indicating the context together with the image; An information processing device comprising:
2. 1. A computer-implemented information processing method, comprising: an acquisition step of acquiring relationship information generated from an image taken by a user using a machine learning model that has been trained to output relationship information indicating a relationship between the subject user, who is a person included in an image taken by the user, and the user when the image is input; an estimation step of identifying a relationship between the subject user and the user based on the relationship information acquired by the acquisition step, and estimating, based on the relationship, that an event based on the relationship was taking place as a context when the user took the image; a display control step of displaying context information indicating the context together with the image when the image is displayed; 13. An information processing method comprising:
3. An acquisition step of acquiring relationship information generated from an image taken by a user using a machine learning model trained to output relationship information indicating a relationship between the subject user, who is a person included in an image taken by the user, and the user when the image is input; an estimation step of identifying a relationship between the subject user and the user based on the relationship information acquired by the acquisition step, and estimating, based on the relationship, that an event based on the relationship was taking place as a context when the user captured the image; a display control step of displaying, when the image is displayed, context information indicating the context together with the image; An information processing program that causes a computer to execute the above.
Citation Information
Patent Citations
Image managing system and method for managing image as well as storage medium
JP2002010178A
Image processor and control method thereof
JP2011234078A
Image processing device and program
JP2014160963A
