Image processing device, method for operating image processing device, and program for operating image processing device

Abstract

An image processing device comprising a processor, wherein the processor receives inputs of emotions of a user about an image at multiple time points, and stores emotion information at the multiple time points in association with the image.

Description

Image processing device, operation method of image processing device, and operation program of image processing device

[0001] The technology of the present disclosure relates to an image processing device, an operating method for an image processing device, and an operating program for an image processing device.

[0002] Japanese Patent Application Laid-Open No. 2017-092528 describes an imaging device including an imaging unit, a display unit, a selection receiving unit, an imaging instruction unit, and an associating unit. The display unit displays a plurality of image elements representing different moods. The selection receiving unit receives a user's selection operation from among the image elements displayed by the display unit. The imaging instruction unit causes the imaging unit to capture an image in response to the user's operation. When the selection receiving unit receives a selection operation for an image element within a predetermined period based on the timing at which the imaging instruction unit causes the image to be captured, the associating unit associates mood information representing a mood associated with the selected image element with image data captured by the imaging unit.

[0003] One embodiment of the technique of the present disclosure provides an image processing device, an operating method for the image processing device, and an operating program for the image processing device that are capable of more accurately grasping a user's feelings regarding an image.

[0004] The image processing device of the present disclosure includes a processor, and the processor receives input of a user's emotions toward images at multiple points in time, and stores the information on emotions at the multiple points in time in association with the images.

[0005] The image is preferably a printed image printed out on instant film, and the multiple points in time are a combination selected from the points in time when the printed image is photographed, the points in time when the instant film is stored as a digital image, and the points in time when the digital image of the instant film is posted to a social networking service.

[0006] When displaying the image, the processor preferably applies a display effect according to the emotional information.

[0007] It is preferable that images can be searched using emotional information as search keywords.

[0008] Preferably, the processor estimates the emotion using a machine learning model that outputs emotion information in response to an input image, and displays the estimated emotion to the user.

[0009] It is preferable that the emotion information and images stored in association with each other are used as training data for a machine learning model.

[0010] Preferably, the image is a printed image printed out on instant film, the multiple points in time include the point in time when the printed image is photographed, and the processor acquires a digital image of the instant film, performs image analysis of the digital image to read text written by the user on the instant film, performs natural language analysis of the text, and, based on the results of the natural language analysis, estimates the emotion at the time when the printed image is photographed, and displays the estimated emotion to the user.

[0011] Preferably, the images are printed images printed out on instant film, the multiple points in time include points in time when the digital images on the instant film are posted to a social networking service, and the processor acquires text entered by the user when posting the digital image to the social networking service, performs natural language analysis of the text, and, based on the results of the natural language analysis, estimates the emotion at the time when the digital image is posted to the social networking service, and displays the estimated emotion to the user.

[0012] Preferably, the processor acquires state information of the user from a wearable device worn by the user, estimates an emotion based on the state information, and displays the estimated emotion to the user.

[0013] Preferably, the processor detects facial expressions of people appearing in the image, estimates emotions based on the detected facial expressions, and displays the estimated emotions to the user.

[0014] The method of operating an image processing device according to the present disclosure includes accepting input of a user's emotions regarding images at multiple points in time, and storing the information on the emotions at the multiple points in time in association with the images.

[0015] The operating program of the image processing device disclosed herein causes a computer to perform processing including accepting input of a user's emotions regarding images at multiple points in time, and storing the information on emotions at multiple points in time in association with the images.

[0016] 9 is a diagram showing an instant camera, a user terminal, and an image management server. FIG. 1 is a block diagram showing computers constituting the user terminal and the image management server. FIG. 2 is a block diagram showing a processing unit of a CPU of a user terminal. FIG. 3 is a diagram showing a storage instruction screen. FIG. 4 is a diagram showing a storage instruction screen after a storage instruction button has been pressed. FIG. 5 is a diagram showing an emotion input menu for when shooting. FIG. 6 is a diagram showing processing by a browser control unit when the OK button is pressed on the storage instruction screen of FIG. 5. FIG. 7 is a diagram showing an image playback display screen. FIG. 8 is a diagram showing an image playback display screen after a post button has been pressed. FIG. 9 is a diagram showing processing by a browser control unit when the OK button is pressed on the image playback display screen of FIG. 1 is a diagram illustrating the processing of each processing unit of the image management server when a search request is sent from a user terminal. FIG. 2 is a diagram illustrating an image list display screen on which search results are displayed. FIG. 3 is a flowchart illustrating the processing procedure of a user terminal. FIG. 4 is a flowchart illustrating the processing procedure of the image management server. FIG. 5 is a flowchart illustrating the processing procedure of a user terminal. FIG. 6 is a flowchart illustrating the processing procedure of the image management server. FIG. 7 is a diagram illustrating the processing of each processing unit of the image management server when an emotion estimation request is sent from a user terminal. FIG. 8 is a diagram illustrating the processing of the emotion estimation unit. FIG. 9 is a diagram illustrating an emotion input menu at the time of shooting in which emotion estimation results are displayed in the form of a speech bubble message. FIG. 10 is a diagram illustrating the processing in the learning phase of the emotion estimation model. FIG. 11 is a diagram illustrating another example of the processing in the learning phase of the emotion estimation model. FIG. 12 is a diagram illustrating a policy for adopting training data for the emotion estimation model. FIG. 13 is a diagram illustrating the processing of each processing unit of the image management server when an emotion estimation request including an instant film with text written on it is sent from a user terminal.32 is a diagram showing a detailed configuration of the feeling estimation unit in the aspect shown in FIG. 32. FIG. 33 is a diagram showing the processing of each processing unit of the image management server when an emotion estimation request including text information entered by a user when posting an instant film is transmitted from a user terminal. FIG. 34 is a diagram showing a detailed configuration of the feeling estimation unit in the aspect shown in FIG. 34. FIG. 35 is a diagram showing a smartwatch and status information. FIG. 36 is a graph showing body temperature fluctuation data. FIG. 37 is a diagram showing the processing of each processing unit of the image management server when an emotion estimation request including status information is transmitted from a user terminal. FIG. 38 is a diagram showing the processing of each processing unit of the image management server when an emotion estimation request including a printed image with a face in it is transmitted from a user terminal. FIG. 39 is a diagram showing a detailed configuration of the feeling estimation unit in the aspect shown in FIG. 39. FIG. 39 is a diagram showing an emotion estimation model that outputs an emotion estimation result in response to input of a printed image, a text reading result, and status information.

[0017] As an example, as shown in Figure 1, a user U photographs a subject with an instant camera 10 and prints out the image of the subject on an instant film 11. The instant film 11 may be either a silver halide film or a thermal film. In Figure 1, an ice cream parfait is shown as the subject. Hereinafter, the image printed out on the instant film 11 will be referred to as a printed image 12.

[0018] The printed image 12 is located approximately in the center of the instant film 11. The size of the printed image 12 is slightly smaller than the instant film 11. Therefore, a margin is provided between the edge of the instant film 11 and the edge of the printed image 12. In particular, a relatively large margin 13 is provided at the bottom of the instant film 11. In the margin 13, the user U can write text 14 with an oil-based pen or the like. FIG. 1 shows an example in which "delicious!" is written as the text 14.

[0019] A user U takes a picture on an instant film 11 using the camera function of a user terminal 15 and stores the instant film 11 as a digital image. The user terminal 15 is a device having a camera function, an image playback / display function, an image transmission / reception function, etc. Specifically, the user terminal 15 is a smartphone, a tablet terminal, a compact digital camera, a mirrorless single-lens camera, a notebook personal computer, etc. The user terminal 15 is an example of an "image processing device" according to the technology of the present disclosure.

[0020] The user terminal 15 is connected to the image management server 17 via a network 16 so as to be able to communicate with each other. The network 16 is, for example, a WAN (Wide Area Network) such as the Internet or a public communication network. The user terminal 15 transmits (uploads) print images 12 to the image management server 17. The user terminal 15 also receives (downloads) print images 12 from the image management server 17.

[0021] The image management server 17 is, for example, a server computer, a workstation, or the like, and together with the user terminal 15 is an example of an "image processing device" according to the technology of the present disclosure. In this way, the "image processing device" according to the technology of the present disclosure may be realized across multiple devices. Multiple user terminals 15 of multiple users U are connected to the image management server 17 via the network 16.

[0022] 2, the computers that make up the user terminal 15 and the image management server 17 basically have the same configuration, and include a storage 20, a memory 21, a CPU (Central Processing Unit) 22, a communication unit 23, a display 24, and an input device 25. These are interconnected via a bus line 26.

[0023] The storage 20 is a hard disk drive built into the computer that constitutes the user terminal 15 and the image management server 17, or connected via a cable or network. Alternatively, the storage 20 is a disk array consisting of multiple hard disk drives. The storage 20 stores control programs such as an operating system, various application programs (hereinafter abbreviated as APs (Application Programs)), and various data associated with these programs. Note that a solid state drive may be used instead of a hard disk drive.

[0024] The memory 21 is a work memory for the CPU 22 to execute processing. The CPU 22 loads a program stored in the storage 20 into the memory 21 and executes processing in accordance with the program. In this way, the CPU 22 comprehensively controls each part of the computer. The CPU 22 is an example of a "processor" according to the technology of the present disclosure. The memory 21 may be built into the CPU 22.

[0025] The communication unit 23 is a network interface that controls the transmission of various information via the network 16, etc. The display 24 displays various screens. Each screen is equipped with an operation function using a GUI (Graphical User Interface). The computer that constitutes the user terminal 15 and the image management server 17 accepts input of operation instructions from an input device 25 via each screen. The input device 25 is a keyboard, mouse, touch panel, microphone for voice input, etc.

[0026] In the following explanation, the parts of the computer that make up the user terminal 15 (storage 20, CPU 22, display 24, and input device 25) are distinguished by adding the suffix "A" to their symbols, and the parts of the computer that make up the image management server 17 (storage 20 and CPU 22) are distinguished by adding the suffix "B" to their symbols.

[0027] As an example, as shown in FIG. 3 , a print image AP 30 is stored in the storage 20A of the user terminal 15. The print image AP 30 is installed in the user terminal 15 by the user U. The print image AP 30 is an AP that causes the computer constituting the user terminal 15 to function as an "image processing device" according to the technology of the present disclosure. In other words, the print image AP 30 is an example of an "operation program for an image processing device" according to the technology of the present disclosure. When the print image AP 30 is launched, the CPU 22A of the user terminal 15 functions as a browser control unit 32 in cooperation with the memory 21, etc. The browser control unit 32 controls the operation of a web browser dedicated to the print image AP 30.

[0028] The browser control unit 32 generates various screens. The browser control unit 32 displays the generated various screens on the display 24A. The browser control unit 32 also accepts various operation instructions input by the user U from the input device 25A via the various screens. The browser control unit 32 transmits various requests to the image management server 17 in response to the operation instructions.

[0029] As an example, as shown in FIG. 4 , the browser control unit 32 displays a storage instruction screen 35 on the display 24A in response to an instruction from the user U. The storage instruction screen 35 is a screen for issuing a storage instruction to store the instant film 11, and ultimately the printed image 12, as a digital image. The storage instruction screen 35 displays a frame 36 for placing the instant film 11 to be stored as a digital image, a message 37, and a storage instruction button 38. The message 37 urges the user U to place the instant film 11 to be stored as a digital image within the frame 36 and press the storage instruction button 38. The storage instruction button 38 functions as a shutter button, so to speak.

[0030] The user U follows the message 37 to place the instant film 11 that he / she wishes to store as a digital image within the frame 36 and presses the storage instruction button 38. When the storage instruction button 38 is pressed, the browser control unit 32 changes the display of the storage instruction screen 35 to that shown in FIG.

[0031] 5, the storage instruction screen 35 displayed after the storage instruction button 38 is pressed includes a message 40, an emotion input menu 41A, an emotion input menu 41B, and an OK button 42. The message 40 prompts the user U to input emotions toward the print image 12 at the time the print image 12 is photographed (hereinafter referred to as "at the time of photographing") and at the time the instant film 11 (print image 12) is stored as a digital image (hereinafter referred to as "at the time of storage"), and to press the OK button 42. The emotion input menu 41A is a GUI for inputting the user U's emotions toward the print image 12 at the time of photographing. The emotion input menu 41B is a GUI for inputting the user U's emotions toward the print image 12 at the time of storage. Hereinafter, when there is no need to particularly distinguish between the emotion input menus 41A and 41B, they may be referred to as the emotion input menu 41.

[0032] 6, the emotion input menu 41A has a face shape 43A that represents the emotion "joy," a face shape 43B that represents the emotion "anger," a face shape 43C that represents the emotion "sad," and a face shape 43D that represents the emotion "fun." Above the face shapes 43A to 43D, characters indicating the emotion represented by each of the face shapes 43A to 43D are displayed. Hereinafter, when there is no need to particularly distinguish between the face shapes 43A to 43D, they may be referred to as face shape 43.

[0033] Face types 43A to 43D are displayed when emotion input menu 41A is selected in the state shown in Fig. 5. Any one of face types 43A to 43D can be selected. User U selects one face type 43 from face types 43A to 43D that is appropriate for the emotion felt toward print image 12 at the time of photographing. Fig. 6 shows an example in which face type 43A, which expresses "joy" as the emotion, is selected, as indicated by hatching. Note that emotion input menu 41B has the same configuration as emotion input menu 41A, and therefore is not shown or described here.

[0034] As an example, as shown in FIG. 7 , user U inputs the desired emotion in emotion input menus 41A and 41B in accordance with message 40, and then presses OK button 42. When OK button 42 is pressed, browser control unit 32 generates emotion information 45A. Emotion information 45A includes the emotions at the time of capture and storage that were input in emotion input menus 41A and 41B. Emotion information 45A also includes image ID (Identification Data) for uniquely identifying print image 12. FIG. 7 shows an example in which face shape 43A representing "joy" is selected in both emotion input menus 41A and 41B, and emotion information 45A is generated in which "joy" is registered as the emotion at the time of capture and storage.

[0035] As an example, as shown in FIG. 8 , the browser control unit 32 displays an image playback display screen 50 on the display 24A in response to an instruction from the user U. The image playback display screen 50 is a screen that plays back and displays the instant film 11 stored as a digital image, and ultimately the printed image 12. The browser control unit 32 applies a display effect to the printed image 12 according to the emotion information 45A. FIG. 8 shows an example of emotion information 45A in which "joy" is registered as the emotion at the time of shooting and storage shown in FIG. 7. In this case, the browser control unit 32 displays multiple star marks 51 on the printed image 12 as a display effect.

[0036] The image playback display screen 50 is provided with a posting button 52 for posting the instant film 11 stored as a digital image to a social networking service (hereinafter referred to as SNS) through an application program. When the posting button 52 is pressed, the browser control unit 32 changes the display of the image playback display screen 50 to that shown in FIG.

[0037] 9 , after the post button 52 is pressed, the image playback display screen 50 displays icons 55 for various SNS application programs, a message 56, an emotion input menu 41C, and an OK button 57. The message 56 prompts the user U to input emotions about the printed image 12 at the time of posting the digital image on the instant film 11 to the SNS (hereinafter referred to as "posting") and to press the OK button 57. The emotion input menu 41C is a GUI for inputting the emotions of the user U about the printed image 12 at the time of posting. The emotion input menu 41C has the same configuration as the emotion input menu 41A, etc., and therefore is not shown or described here.

[0038] As an example, as shown in Figure 10, user U selects icon 55, inputs the desired emotion in emotion input menu 41C according to message 56, and then presses OK button 57. When OK button 57 is pressed, browser control unit 32 generates emotion information 45B. Emotion information 45B includes the emotion at the time of posting that was input in emotion input menu 41C and an image ID. Figure 10 shows an example in which face shape 43D representing "fun" is selected in emotion input menu 41C, and emotion information 45B is generated in which "fun" is registered as the emotion at the time of posting. Hereinafter, when there is no need to particularly distinguish between emotion information 45A and 45B, they may be referred to as emotion information 45.

[0039] 8 and 9 show the display effect when the emotion is "joy," but examples of display effects for other emotions are shown in FIGS. 11 to 13. That is, when the emotion is "anger," as shown in FIG. 11, the browser control unit 32 displays, as a display effect, multiple anger marks 60 that resemble blood vessels in the temples, commonly known as veins, on the printed image 12. When the emotion is "sad," as shown in FIG. 12, the browser control unit 32 displays, as a display effect, multiple teardrop marks 61 that resemble teardrops on the printed image 12. When the emotion is "joy," as shown in FIG. 13, the browser control unit 32 displays, as a display effect, multiple musical note marks 62 on the printed image 12.

[0040] The browser control unit 32 also applies display effects to the printed image 12 when the printed image 12 is reproduced and displayed on the image reproduction display screen 50 after posting a digital image from the instant film 11 to a social networking service. The browser control unit 32 applies display effects according to the emotion most frequently selected at the time of shooting, storage, and posting. For example, if the emotion at the time of shooting and storage was "joy" and the emotion at the time of posting was "fun," the browser control unit 32 applies a display effect using star marks 51 that matches "joy." If the emotion at the time of shooting and storage is different, or if the emotion at the time of shooting, storage, and posting is different, the browser control unit 32 applies a display effect according to the emotion at the most recent point in time. Animations such as blinking the star marks 51, changing the size of the anger marks 60, moving the tear marks 61 up and down, or changing the angle of the musical note marks 62 may also be used.

[0041] As an example, as shown in FIG. 14 , an operating program 70 is stored in storage 20B of image management server 17. Operating program 70 is an AP for causing a computer constituting image management server 17 to function as an "image processing device" according to the technology of the present disclosure. In other words, like print image AP 30, operating program 70 is an example of an "operating program for an image processing device" according to the technology of the present disclosure. Storage 20B also stores an image database (hereinafter referred to as DB (Data Base)) 71. Although not shown, storage 20B also stores a user ID for uniquely identifying user U, a password set by user U, and a terminal ID for uniquely identifying user terminal 15 as account information for user U.

[0042] When the operating program 70 is started, the CPU 22B of the image management server 17 cooperates with the memory 21 and the like to function as a reception unit 75, a read / write (hereinafter abbreviated as RW (Read Write)) control unit 76, and a distribution control unit 77.

[0043] The reception unit 75 receives various requests from the user terminal 15. The reception unit 75 outputs the various requests to the RW control unit 76 and the distribution control unit 77. The RW control unit 76 controls the storage of various data in the storage 20B and the reading of various data from the storage 20B. The RW control unit 76 particularly controls the storage of print images 12, etc. in the image DB 71 and the reading of print images 12, etc. from the image DB 71. The distribution control unit 77 controls the distribution of various data to the user terminal 15.

[0044] 15, the image DB 71 has a storage area 80 for each user U. A user ID is registered in the storage area 80. The storage area 80 stores instant film 11 stored as digital images, and thus printed images 12, and emotion information 45, associated with each image ID. For printed images 12 that have not been posted to SNS, such as the printed image 12 with the image ID "P00002," emotions at the time of posting are not registered.

[0045] Each printed image 12 has registered tag information. The tag information is a word that succinctly describes the subject appearing in the printed image 12. The tag information may be manually entered by the user U or derived using image analysis software. Although not shown, each printed image 12 also has registered therein the instant camera 10 settings at the time of shooting (macro, selfie, flash photography, etc.), the scene at the time of shooting (daytime, nighttime, outdoors, indoors, etc.), and an image quality evaluation score.

[0046] 16, when the OK button 42 is pressed on the storage instruction screen 35, the browser control unit 32 generates emotion information 45A as shown in Fig. 7 and then sends a first storage request 85A to the image management server 17. The first storage request 85A includes the user ID, the instant film 11 stored as a digital image, and in turn the printed image 12, and the emotion information 45A.

[0047] The reception unit 75 receives the first storage request 85A and outputs the first storage request 85A to the RW control unit 76. In response to the first storage request 85A, the RW control unit 76 stores the printed image 12 and the emotion information 45A in association with each other in the storage area 80 of the image DB 71 corresponding to the user ID. FIG. 16 shows an example in which the printed image 12 depicting a car and the emotion "joy" at the time of shooting and the emotion "fun" at the time of storage in the emotion information 45A are stored in association with each other in the storage area 80 of the user U with the user ID "U00001." Note that tag information and the like are omitted from FIG. 16 . The same applies to subsequent FIG. 17 and the like.

[0048] 17, when the OK button 57 is pressed on the image playback display screen 50, the browser control unit 32 generates emotion information 45B as shown in Fig. 10 and then sends a second storage request 85B to the image management server 17. The second storage request 85B includes the user ID and emotion information 45B.

[0049] The reception unit 75 receives the second storage request 85B and outputs the second storage request 85B to the RW control unit 76. In response to the second storage request 85B, the RW control unit 76 stores emotion information 45B in the storage area 80 of the image DB 71 corresponding to the user ID. FIG. 17 shows an example in which the printed image 12 is a photograph of a car, as in the case of FIG. 16 , and the emotion "fun" at the time of posting of the emotion information 45B is stored in the storage area 80 of user U with user ID "U00001." In this way, the RW control unit 76 associates and stores the emotion information 45 and the printed image 12 at three points in time: when the image was taken, when it was stored, and when it was posted, in the image DB 71.

[0050] 18, the browser control unit 32 displays an image list display screen 90 on the display 24A in response to an instruction from the user U. The image list display screen 90 displays a list of thumbnail images 12S of print images 12 on instant film 11 stored as digital images. When the user U selects one thumbnail image 12S on this image list display screen 90, the display transitions to the image playback display screen 50 shown in FIG.

[0051] The image list display screen 90 is provided with a search bar 91. The user U inputs search keywords into the search bar 91 to search for a desired print image 12. Any word and the words for each emotion in the emotion information 45, such as "joy," "anger," "sad," and "fun," can be input as search keywords.

[0052] 19, when a search keyword is entered in a search bar 91, the browser control unit 32 generates a search request 95. The search request 95 includes a user ID and the search keyword entered in the search bar 91. In FIG. 19, an example is shown in which the arbitrary word "family" and the emotional word "fun" are entered as search keywords.

[0053] As an example, as shown in FIG. 20 , the browser control unit 32 transmits a search request 95 to the image management server 17. The reception unit 75 receives the search request 95 and outputs the search request 95 to the RW control unit 76. The RW control unit 76 searches for print images 12 whose emotion information 45 and tag information match the search keyword among the print images 12 stored in the storage area 80 of the image DB 71 corresponding to the user ID. The RW control unit 76 outputs the image ID of the searched print image 12 to the delivery control unit 77. The delivery control unit 77 delivers the image ID from the RW control unit 76 to the user terminal 15 that has requested the search request 95. The delivery control unit 77 identifies the user terminal 15 that has requested the search request 95 based on the user ID included in the search request 95. FIG. 20 shows an example in which the search keywords are "family" and "fun," the same as in FIG. 19 . In this example, three print images 12 with image IDs "P00200," "P00201," and "P00202" are searched for, each having "fun" registered in the emotion information 45 and "family" registered in the tag information.

[0054] 21 , the browser control unit 32 displays, on the image list display screen 90, only thumbnail images 12S of the print images 12 corresponding to the image IDs searched for by the RW control unit 76 and distributed by the distribution control unit 77 as search results. In this way, print images 12 can be searched for using emotion information 45 as a search keyword. While an AND search of an arbitrary word and an emotion word has been exemplified here, this is not limiting. It is also possible to search for print images 12 using only an arbitrary word as a search keyword, using only one emotion word as a search keyword, or using two or more emotion words as search keywords.

[0055] Next, the operation of the above configuration will be described with reference to the flowcharts shown in Figures 22, 23, 24, and 25. As shown in Figure 3, the CPU 22A of the user terminal 15 functions as a browser control unit 32 when the print image AP 30 is activated. Also, as shown in Figure 14, the CPU 22B of the image management server 17 functions as a reception unit 75, a RW control unit 76, and a delivery control unit 77 when the operating program 70 is activated.

[0056] To store the desired instant film 11 as a digital image, the user U displays the storage instruction screen 35 shown in Fig. 4 on the display 24A. Then, the user places the instant film 11 to be stored as a digital image within the frame 36 and presses the storage instruction button 38. This causes the browser control unit 32 to accept the instruction to store the instant film 11 (YES in step ST100 of Fig. 22). The browser control unit 32 then transitions the display of the storage instruction screen 35 to the screen shown in Fig. 5.

[0057] User U operates emotion input menus 41A and 41B to input emotions at the time of shooting and at the time of storage, and then presses OK button 42. This causes browser control unit 32 to accept the input of emotions for print image 12 at the time of shooting and storage (step ST110). As shown in FIG. 7, emotion information 45A at the time of shooting and storage is generated by browser control unit 32 (step ST120). Then, as shown in FIG. 16, under the control of browser control unit 32, a first storage request 85A including emotion information 45A is sent to image management server 17 (step ST130).

[0058] In the image management server 17, the first storage request 85A is received by the reception unit 75 (YES in step ST150 in FIG. 23 ). The first storage request 85A is output from the reception unit 75 to the RW control unit 76. Then, under the control of the RW control unit 76, emotion information 45A at the time of shooting and storage and the print image 12 are associated with each other and stored in the image DB 71 (step ST160).

[0059] In order to post a desired instant film 11 to the SNS, the user U displays the image playback display screen 50 shown in Fig. 8 on the display 24A. Then, the user presses the post button 52. This causes the browser control unit 32 to accept an instruction to post the instant film 11 (YES in step ST200 of Fig. 24). The browser control unit 32 causes the display on the image playback display screen 50 to transition to the screen shown in Fig. 9.

[0060] User U operates emotion input menu 41C to input the emotion at the time of posting, and then presses OK button 57. This causes browser control unit 32 to accept the emotion input for print image 12 at the time of posting (step ST210). As shown in FIG. 10, emotion information 45B at the time of posting is generated by browser control unit 32 (step ST220). Then, as shown in FIG. 17, under the control of browser control unit 32, a second storage request 85B including emotion information 45B is sent to image management server 17 (step ST230).

[0061] In the image management server 17, the second storage request 85B is received by the reception unit 75 (YES in step ST250 in FIG. 25 ). The first storage request 85A is output from the reception unit 75 to the RW control unit 76. Then, under the control of the RW control unit 76, the emotion information 45B at the time of posting and the printed image 12 are associated with each other and stored in the image DB 71 (step ST260).

[0062] As described above, the browser control unit 32 of the CPU 22A of the user terminal 15 accepts input of the user U's emotions regarding the printed image 12 at multiple points in time. The RW control unit 76 of the CPU 22B of the image management server 17 stores the emotion information 45 and the printed image 12 at multiple points in time in the image DB 71 in association with each other.

[0063] In JP 2017-092528 A, the timing for inputting the user U's feelings toward the printed image 12 is limited to the time of capture. Therefore, it is not possible to accurately grasp the user U's feelings toward the printed image 12, which change over time. In contrast, as described above, the technology disclosed herein accepts input of the user U's feelings toward the printed image 12 at multiple points in time, and associates the emotion information 45 at the multiple points in time with the printed image 12 and stores it. Therefore, it is possible to more accurately grasp the user U's feelings toward the printed image 12.

[0064] The image is a printed image 12 printed out on an instant film 11. The multiple points in time are the time when the printed image 12 is photographed (time of photographing), the time when the instant film 11 is stored as a digital image (time of storage), and the time when the digital image of the instant film 11 is posted to a social networking site (time of posting). This makes it possible to accurately grasp the user U's feelings toward the printed image 12 at the time of photographing, storage, and posting. Note that the multiple points in time are not limited to all of the time of photographing, storage, and posting, but may be any combination selected from these (for example, the time of photographing and storage, or the time of storage and posting).

[0065] As shown in Figures 8 and 11 to 13, the browser control unit 32 applies a display effect corresponding to the emotion information 45 when displaying the print image 12. This allows the user U's emotion toward the print image 12 to be understood at a glance. Furthermore, the display of the print image 12, which tends to be dull, can be made more tasteful. In addition to the display effect, music corresponding to the emotion information 45 may be played.

[0066] 18 to 21, the print image 12 can be searched for using the emotion information 45 as a search keyword. This makes it possible to search for a print image 12 that better matches the intention of the user U.

[0067] [Second Embodiment] As shown in Fig. 26 as an example, in this embodiment, when the storage instruction button 38 is pressed on the storage instruction screen 35 shown in Fig. 4 or when the post button 52 is pressed on the image playback display screen 50 shown in Fig. 8, the browser control unit 32 sends an emotion estimation request 100 to the image management server 17. The emotion estimation request 100 includes a user ID, an instant film 11 stored as a digital image, and a printed image 12. The printed image 12 is a photograph of the automobile shown in Fig. 16.

[0068] The CPU 22B of the image management server 17 of this embodiment functions as a feeling estimation unit 101 in addition to the processing units 75 to 77 (the RW control unit 76 is not shown in FIG. 26) of the first embodiment.

[0069] The receiving unit 75 receives the emotion estimation request 100 and outputs the emotion estimation request 100 to the emotion estimation unit 101. The emotion estimation unit 101 estimates the emotion of the user U regarding the printed image 12 in response to the emotion estimation request 100. The emotion estimation unit 101 outputs an emotion estimation result 103, which is a result of estimating the emotion of the user U regarding the printed image 12, to the distribution control unit 77. The distribution control unit 77 distributes the emotion estimation result 103 to the user terminal 15 that is the source of the emotion estimation request 100.

[0070] As an example, as shown in FIG. 27 , the feeling estimation unit 101 estimates the feeling of a user U toward a print image 12 using a feeling estimation model 105. The feeling estimation model 105 is stored in a storage 20B. The feeling estimation model 105 is read from the storage 20B by the RW control unit 76 and output to the feeling estimation unit 101. The feeling estimation model 105 is configured by, for example, a convolutional neural network. The feeling estimation model 105 is an example of a "machine learning model" related to the technology of the present disclosure.

[0071] The feeling estimation unit 101 inputs the printed image 12 of the feeling estimation request 100 to the feeling estimation model 105, and causes the feeling estimation model 105 to output a feeling estimation result 103 of the user U with respect to the printed image 12. Fig. 27 shows an example in which the feeling of the user U with respect to the printed image 12 is estimated to be "joy."

[0072] When receiving the emotion estimation result 103 from the distribution control unit 77, the browser control unit 32 displays a speech bubble message 108 above the face shape 43 corresponding to the emotion in the emotion estimation result 103 in the emotion input menu 41, as shown in FIG. 28 as an example. The speech bubble message 108 indicates to the user U the emotion estimated by the emotion estimation model 105, such as "Did you feel this way?" FIG. 28 shows an example in which the speech bubble message 108 is displayed above the face shape 43A expressing "joy" as the emotion corresponding to the emotion in the emotion estimation result 103 in the emotion input menu 41A at the time of shooting. Although not shown in the drawings, the browser control unit 32 also displays the speech bubble message 108 above the face shape 43 corresponding to the emotion in the emotion estimation result 103 in the emotion input menu 41B at the time of storage and the emotion input menu 41C at the time of posting.

[0073] 29, emotion estimation model 105 is trained by being given training data 110. Training data 110 is a set of training print image 12L and correct emotion information 45CA.

[0074] In the learning phase, training print images 12L are input to emotion estimation model 105. In response to input of training print images 12L, emotion estimation model 105 outputs training emotion estimation results 103L. A loss calculation is performed for emotion estimation model 105 using a loss function based on training emotion estimation results 103L and ground truth emotion information 45CA. Then, update settings for various coefficients of emotion estimation model 105 are performed in accordance with the results of the loss calculation, and emotion estimation model 105 is updated in accordance with the update settings.

[0075] In the learning phase, the above-mentioned series of processes, including input of learning print image 12L to emotion estimation model 105, output of learning emotion estimation result 103L from emotion estimation model 105, loss calculation, update setting, and updating of emotion estimation model 105, are repeated while teacher data 110 is exchanged. The repetition of the above-mentioned series of processes is terminated when the estimation accuracy of learning emotion estimation result 103L for correct emotion information 45CA reaches a predetermined set level. Emotion estimation model 105 whose estimation accuracy has thus reached the set level is stored in storage 20B and used by emotion estimation unit 101. Note that learning may be terminated when the above-mentioned series of processes has been repeated a set number of times, regardless of the estimation accuracy of learning emotion estimation result 103L for correct emotion information 45CA.

[0076] As described above, in the second embodiment, the feeling estimation unit 101 estimates a feeling using the feeling estimation model 105 that outputs the feeling estimation result 103 in response to the input of the print image 12. The browser control unit 32 displays the feeling estimated by the feeling estimation unit 101 to the user U by displaying a speech bubble message 108. Machine learning models such as the feeling estimation model 105 have recently been widely put into practical use, and their estimation accuracy has also improved. This makes it possible to support the input of more appropriate feelings.

[0077] As an example, as shown in FIG. 30 , the print image 12 and emotion information 45 stored in association with the image DB 71 may be used as the training data 110 for the emotion estimation model 105. Specifically, the print image 12 stored in the image DB 71 is used as the training print image 12L, and the emotion information 45 at the time of capture stored in the image DB 71 is used as the correct emotion information 45CA. The reason for using the emotion information 45 at the time of capture as the correct emotion information 45CA is that it is believed to more honestly represent the user U's emotion toward the print image 12 than the information at the time of storage or posting. This allows for effective use of the print image 12 and emotion information 45 stored in the image DB 71. This eliminates the need to prepare new training print images 12L and correct emotion information 45CA for the training data 110. Furthermore, this reduces the risk of suffering from a lack of training data 110, thereby suppressing the risk of overtraining the emotion estimation model 105 due to a lack of training data 110.

[0078] When the printed image 12 and emotion information 45 stored in association with the image DB 71 are used as training data 110 for the emotion estimation model 105, as shown in FIG. 31 as an example, emotion estimation results 103 are also stored in association with the image DB 71 along with the printed image 12 and emotion information 45. It is preferable to actively employ data in which the emotion information 45 at the time of shooting differs from the emotion estimation result 103 (expressed as "emotion information at the time of shooting ≠ emotion estimation result" in FIG. 31) as training data 110 for the emotion estimation model 105. Data in which the emotion information 45 at the time of shooting differs from the emotion estimation result 103 is data in which the emotion estimation model 105 has incorrectly estimated an emotion. For this reason, actively employing data in which the emotion information 45 at the time of shooting differs from the emotion estimation result 103 as training data 110 for the emotion estimation model 105 can improve the estimation accuracy of the emotion estimation model 105.

[0079] The emotion estimation model 105 may be trained by the image management server 17, or by a device other than the image management server 17. Furthermore, the emotion estimation model 105 may be trained continuously after being stored in the storage 20B.

[0080] 32 as an example, in this embodiment, when the storage instruction button 38 is pressed on the storage instruction screen 35 shown in FIG. 4, the browser control unit 32 sends an emotion estimation request 115 to the image management server 17. The emotion estimation request 115 includes a user ID, an instant film 11 stored as a digital image, and a printed image 12. Note that the printed image 12, like the case of FIG. 16, is of a car, and the user U has written text 14 saying "I bought a new car!!" in the margin 13.

[0081] The CPU 22B of the image management server 17 of this embodiment functions as a feeling estimation unit 116 in addition to the processing units 75 to 77 (the RW control unit 76 is not shown in FIG. 32) of the first embodiment.

[0082] The receiving unit 75 receives the emotion estimation request 115 and outputs the emotion estimation request 115 to the emotion estimation unit 116. In response to the emotion estimation request 115, the emotion estimation unit 116 estimates the emotion of the user U toward the printed image 12 at the time of capture. The emotion estimation unit 116 outputs an emotion estimation result 117, which is a result of estimating the emotion of the user U toward the printed image 12 at the time of capture, to the distribution control unit 77. The distribution control unit 77 distributes the emotion estimation result 117 to the user terminal 15 that is the source of the emotion estimation request 115.

[0083] As an example, as shown in FIG. 33 , the emotion estimation unit 116 has a text reading unit 120 and a natural language analysis unit 121. The instant film 11 containing the emotion estimation request 115 is input to the text reading unit 120. The text reading unit 120 performs image analysis on the instant film 11 stored as a digital image to read text 14 written by the user U in the margins 13 or the like of the instant film 11. The text reading unit 120 outputs a text reading result 122, which is the result of reading the text 14, to the natural language analysis unit 121.

[0084] The natural language analysis unit 121 performs natural language analysis on the text 14 of the text reading result 122. The natural language analysis unit 121 estimates the emotion of the user U toward the printed image 12 at the time of shooting based on the results of the natural language analysis, and outputs the emotion estimation result 117. The natural language analysis unit 121 estimates the emotion of the user U toward the printed image 12 at the time of shooting, for example, using an emotion estimation model that outputs the emotion estimation result 117 in response to input results of the natural language analysis. Here, the natural language analysis performed by the natural language analysis unit 121 includes morphological analysis, syntactic analysis, semantic analysis, context analysis, and the like. Figure 33 shows an example in which the text reading unit 120 reads the text 14, "I bought a new car!!", and the emotion of the user U toward the printed image 12 at the time of shooting is estimated to be "joy."

[0085] Although not shown in the figure, when the emotion estimation result 117 is received from the delivery control unit 77, the browser control unit 32 displays a speech bubble message 108 above the face shape 43 corresponding to the emotion of the emotion estimation result 117 in the emotion input menu 41A, as in the case shown in FIG. 28 .

[0086] As described above, in the third embodiment, the text reading unit 120 of the emotion deduction unit 116 reads the text 14 written by the user U on the instant film 11 by performing image analysis on the instant film 11 stored as a digital image. The natural language analysis unit 121 of the emotion deduction unit 116 performs natural language analysis on the text 14 and, based on the results of the natural language analysis, deduces the emotion of the user U regarding the printed image 12 at the time of capture. The browser control unit 32 displays the emotion estimated by the emotion deduction unit 116 to the user U by displaying the speech bubble message 108. This makes it possible to support input of an emotion appropriate for the text 14 written by the user U on the instant film 11.

[0087] [Fourth Embodiment] In this embodiment, when the post button 52 is pressed on the image playback display screen 50 shown in Fig. 8, the browser control unit 32 accepts input of text 125 by the user U. The text 125 is an explanatory text or the like to be attached to the instant film 11 when the instant film 11 is posted on an SNS. After accepting the input of the text 125, the browser control unit 32 changes the display of the image playback display screen 50 to that shown in Fig. 9.

[0088] 34 as an example, in this embodiment, when input of text 125 by user U is accepted, the browser control unit 32 sends an emotion estimation request 126 to the image management server 17. The emotion estimation request 126 includes a user ID and input text information 127. The input text information 127 includes text 125. Note that the printed image 12 shows a mother and daughter playing in a park, and the text 125 written by user U reads, "Family outing to XX Park, daughter having fun."

[0089] The CPU 22B of the image management server 17 of this embodiment functions as a feeling estimation unit 128 in addition to the processing units 75 to 77 (the RW control unit 76 is not shown in FIG. 34) of the first embodiment.

[0090] The receiving unit 75 receives the emotion estimation request 126 and outputs the emotion estimation request 126 to the emotion estimation unit 128. The emotion estimation unit 128 estimates the emotion of the user U toward the printed image 12 at the time of posting in response to the emotion estimation request 126. The emotion estimation unit 128 outputs an emotion estimation result 129, which is a result of estimating the emotion of the user U toward the printed image 12 at the time of posting, to the delivery control unit 77. The delivery control unit 77 delivers the emotion estimation result 129 to the user terminal 15 that requested the emotion estimation request 126.

[0091] 35 , the feeling estimation unit 128 includes a text acquisition unit 130 and a natural language analysis unit 131. Input text information 127 of the feeling estimation request 126 is input to the text acquisition unit 130. The text acquisition unit 130 outputs the input text information 127 to the natural language analysis unit 131.

[0092] The natural language analysis unit 131 performs natural language analysis on the text 125. Based on the results of the natural language analysis, the natural language analysis unit 131 estimates the user U's feelings toward the printed image 12 at the time of posting and outputs a feeling estimation result 129. Like the natural language analysis unit 121, the natural language analysis unit 131 estimates the user U's feelings toward the printed image 12 at the time of posting using, for example, a feeling estimation model that outputs the feeling estimation result 129 in response to input results of the natural language analysis. Like the natural language analysis unit 121, the natural language analysis performed by the natural language analysis unit 131 includes morphological analysis, syntactic analysis, semantic analysis, contextual analysis, and the like. Figure 35 shows an example in which the user U's feelings toward the printed image 12 at the time of posting are estimated to be "fun" from the text 125, "Family outing to XX Park, daughter having fun."

[0093] Although not shown in the figure, when the browser control unit 32 receives the emotion estimation result 129 from the delivery control unit 77, the browser control unit 32 displays a speech bubble message 108 above the face shape 43 corresponding to the emotion in the emotion estimation result 129 in the emotion input menu 41C, as in the case shown in FIG. 28 .

[0094] As described above, in the fourth embodiment, the text acquisition unit 130 of the emotion deduction unit 128 acquires the input text information 127 to acquire the text 125 input by the user U when posting the instant film 11 stored as a digital image on the SNS. The natural language analysis unit 131 of the emotion deduction unit 128 performs natural language analysis on the text 125 and estimates the emotion of the user U regarding the printed image 12 at the time of posting based on the results of the natural language analysis. The browser control unit 32 displays the emotion estimated by the emotion deduction unit 128 to the user U by displaying the speech bubble message 108. This makes it possible to support the user U in inputting an emotion that is appropriate for the text 125 input by the user U when posting the instant film 11 on the SNS.

[0095] 36 as an example, in this embodiment, state information 136 of a user U output from a smart watch 135 worn on the wrist of the user U is used to estimate the emotion of the user U. The smart watch 135 is an example of a "wearable device" according to the technology of the present disclosure.

[0096] The smartwatch 135 is connected to the instant camera 10 and the user terminal 15 via short-range wireless communication such as Bluetooth (registered trademark) so that they can communicate with each other. The smartwatch 135 works in conjunction with the instant camera 10 and the user terminal 15. More specifically, the smartwatch 135 transmits status information 136 to the user terminal 15 in conjunction with an instruction to the instant camera 10 to capture the print image 12, an instruction to the user terminal 15 to store the print image 12 by pressing the store instruction button 38, and an instruction to post the print image 12 by pressing the post button 52.

[0097] The status information 136 includes body temperature fluctuation data, pulse fluctuation data, blood pressure fluctuation data, and angular velocity fluctuation data. As shown in FIG. 37 as an example, the body temperature fluctuation data is time-series data indicating changes in the user U's body temperature for 30 seconds before and after the instruction to capture, store, or post the print image 12 (0 seconds). Although not shown, like the body temperature fluctuation data, the pulse fluctuation data, blood pressure fluctuation data, and angular velocity fluctuation data are also time-series data indicating changes in the user U's pulse, blood pressure, and angular velocity for 30 seconds before and after the instruction to capture, store, or post the print image 12. The angular velocity fluctuation data allows the user U's hand shake state to be ascertained when the instruction to capture, store, or post the print image 12 is issued.

[0098] 38 as an example, in this embodiment, when state information 136 is received from a smartwatch 135, the browser control unit 32 sends an emotion estimation request 140 to the image management server 17. The emotion estimation request 140 includes the user ID and the state information 136.

[0099] The CPU 22B of the image management server 17 of this embodiment functions as a feeling estimation unit 141 in addition to the processing units 75 to 77 (the RW control unit 76 is not shown in FIG. 38) of the first embodiment.

[0100] The receiving unit 75 receives an emotion estimation request 140 and outputs the emotion estimation request 140 to the emotion estimation unit 141. In response to the emotion estimation request 140, the emotion estimation unit 141 estimates the emotion of the user U toward the printed image 12 at the time of shooting, storing, or posting. The emotion estimation unit 141 outputs an emotion estimation result 142, which is a result of estimating the emotion of the user U toward the printed image 12, to the distribution control unit 77. The distribution control unit 77 distributes the emotion estimation result 142 to the user terminal 15 that has requested the emotion estimation request 140. The emotion estimation unit 141 estimates the emotion of the user U toward the printed image 12 using, for example, an emotion estimation model that outputs the emotion estimation result 142 in response to input of state information 136.

[0101] Although not shown in the figure, when the browser control unit 32 receives the emotion estimation result 142 from the delivery control unit 77, the browser control unit 32 displays a speech bubble message 108 above the face shape 43 corresponding to the emotion in the emotion estimation result 142 in the emotion input menu 41, as in the case shown in FIG. 28 .

[0102] As described above, in the fifth embodiment, the feeling deduction unit 141 acquires the state information 136 of the user U from the smart watch 135 worn by the user U, and deduces the feeling of the user U regarding the print image 12 based on the state information 136. The browser control unit 32 displays the feeling estimated by the feeling deduction unit 141 to the user U by displaying the speech bubble message 108. This makes it possible to support input of a feeling adapted to the state information 136 of the user U.

[0103] The wearable device is not limited to the exemplified smart watch 135. It may be a device worn by wrapping around the head of the user U, or may be a device built into the clothing worn by the user U. Furthermore, in addition to or instead of the body temperature variation data, etc., respiration variation data, etc. may be used as the status information 136.

[0104] 39 as an example, in this embodiment, when the storage instruction button 38 is pressed on the storage instruction screen 35 shown in FIG. 4, the browser control unit 32 sends an emotion estimation request 145 to the image management server 17. The emotion estimation request 145 includes a user ID, an instant film 11 stored as a digital image, and a printed image 12. Note that the printed image 12 shown is an image of a smiling couple.

[0105] The CPU 22B of the image management server 17 of this embodiment functions as a feeling estimation unit 146 in addition to the processing units 75 to 77 (the RW control unit 76 is not shown in FIG. 39) of the first embodiment.

[0106] The reception unit 75 receives the emotion estimation request 145 and outputs the emotion estimation request 145 to the emotion estimation unit 146. The emotion estimation unit 146 estimates the emotion of the user U toward the printed image 12 at the time of capture in response to the emotion estimation request 145. The emotion estimation unit 146 outputs an emotion estimation result 147, which is a result of estimating the emotion of the user U toward the printed image 12 at the time of capture, to the distribution control unit 77. The distribution control unit 77 distributes the emotion estimation result 147 to the user terminal 15 that requested the emotion estimation request 145.

[0107] As an example, as shown in FIG. 40 , the emotion estimation unit 146 has a face extraction unit 150 and a facial expression detection unit 151. The printed image 12 of the emotion estimation request 145 is input to the face extraction unit 150. The face extraction unit 150 uses a well-known face extraction technique to extract the faces of people appearing in the printed image 12. The face extraction unit 150 outputs a face extraction result 152, which is the result of the face extraction, to the facial expression detection unit 151.

[0108] The facial expression detection unit 151 uses well-known image recognition technology to detect facial expressions of people in the face extraction result 152. The facial expression detection unit 151 estimates the emotion of the user U toward the printed image 12 at the time of capture based on the facial expression detection result, and outputs the emotion estimation result 147. The facial expression detection unit 151 estimates the emotion of the user U toward the printed image 12 at the time of capture using, for example, an emotion estimation model that outputs the emotion estimation result 147 in response to input of the facial expression detection result. Figure 40 shows an example in which the emotion of the user U toward the printed image 12 at the time of capture is estimated to be "fun" from the face extraction result 152 of a smiling couple.

[0109] Although not shown in the figure, when the browser control unit 32 receives the emotion estimation result 147 from the delivery control unit 77, the browser control unit 32 displays a speech bubble message 108 above the face shape 43 corresponding to the emotion of the emotion estimation result 147 in the emotion input menu 41A, as in the case shown in FIG. 28 .

[0110] As described above, in the sixth embodiment, the facial expression detection unit 151 of the emotion estimation unit 146 detects the facial expression of a person appearing in the printed image 12 and estimates the emotion of the user U regarding the printed image 12 based on the facial expression detection result. The browser control unit 32 displays the emotion estimated by the emotion estimation unit 146 to the user U by displaying the speech bubble message 108. This makes it possible to support input of an emotion suited to the facial expression of the person appearing in the printed image 12.

[0111] The second to sixth embodiments may be implemented independently or in combination. For example, when the second, third, and fifth embodiments are implemented in combination, an emotion estimation model 155 shown in FIG. 41 may be used as an example. The emotion estimation model 155 outputs an emotion estimation result 156 in response to inputs of the printed image 12, the text reading result 122, and the state information 136. The emotion estimation model 155 is an example of a "machine learning model" according to the technology of the present disclosure. This increases the amount of material for estimating the emotion of the user U toward the printed image 12 compared to when the second, third, and fifth embodiments are implemented independently, thereby further improving the estimation accuracy of the emotion estimation result 156.

[0112] The emotions estimated by the emotion estimation units 101, 116, 128, 141, and 146 in the second to sixth embodiments may be stored as emotion information 45. Then, the stored emotion information 45 may be presented to the user U, and an instruction to modify the emotion information 45 from the user U may be accepted.

[0113] In each of the above embodiments, the user U is almost forced to input emotions for the print image 12 when photographing, storing, and posting the image, but this is not limiting. Any configuration is sufficient as long as it is possible to accept input of emotions for the print image 12 when photographing, storing, and posting the image, and a configuration that forces the user U to input emotions is not necessary. Furthermore, there is no need to reserve a storage area for emotion information 45 in the image DB 71 in advance. It is sufficient to reserve a storage area for emotion information 45 in the image DB 71 each time emotion input is accepted.

[0114] Emotions are not limited to the examples of "joy," "anger," "sadness," and "fun." They may also be "nostalgic," "love," "fear," or "happiness." Furthermore, the image is not limited to the printed image 12, but may also be a digital image taken with a device having a camera function.

[0115] The time points are not limited to the time of shooting, storage, and posting as exemplified above, but may be regular time points such as one year, two years, five years, or ten years after shooting.

[0116] The image management server 17 may be responsible for all or part of the functions of the browser control unit 32 of the user terminal 15. Specifically, various screens such as the storage instruction screen 35 are generated in the image management server 17 and distributed to the user terminal 15 in the form of screen data for web distribution created using a markup language such as XML (Extensible Markup Language). In this case, the browser control unit 32 of the user terminal 15 reproduces various screens to be displayed on the web browser based on the screen data and displays them on the display 24A. Note that other data description languages ​​such as JSON (Javascript (registered trademark) Object Notation) may be used instead of XML.

[0117] The hardware configuration of the computer that constitutes the image management server 17 can be modified in various ways. For example, the image management server 17 can be configured with multiple computers separated as hardware in order to improve processing power and reliability. For example, the functions of the reception unit 75 and RW control unit 76 and the function of the distribution control unit 77 can be distributed and performed by two computers. In this case, the image management server 17 is configured with two computers. In addition, all or part of the functions of the image management server 17 may be performed by the user terminal 15.

[0118] In this way, the hardware configuration of the computers of the user terminal 15 and the image management server 17 can be changed as appropriate depending on the required performance, such as processing power, safety, reliability, etc. Furthermore, in addition to the hardware, APs such as the print image AP 30 and the operating program 70 can also be duplicated or stored in multiple storage devices in order to ensure safety and reliability.

[0119] In each of the above embodiments, for example, the hardware structure of processing units that perform various processes, such as the browser control unit 32, the reception unit 75, the RW control unit 76, the delivery control unit 77, the emotion estimation units 101, 116, 128, 141, and 146, the text reading unit 120, the natural language analysis units 121 and 131, the text acquisition unit 130, the face extraction unit 150, and the facial expression detection unit 151, can be various processors shown below. The various processors include CPUs 22A and 22B, which are general-purpose processors that execute software (print image AP 30 and operating program 70) and function as various processing units, as well as programmable logic devices (PLDs), which are processors whose circuit configuration can be changed after manufacture, such as FPGAs (Field Programmable Gate Arrays), and / or dedicated electrical circuits, such as ASICs (Application Specific Integrated Circuits), which are processors with circuit configurations designed specifically for executing specific processes.

[0120] A single processing unit may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (e.g., a combination of multiple FPGAs and / or a combination of a CPU and an FPGA).Furthermore, multiple processing units may be configured with a single processor.

[0121] Examples of configuring multiple processing units with a single processor include, first, a form in which one processor is configured with a combination of one or more CPUs and software, as typified by computers such as client and server, and this processor functions as multiple processing units. Second, a form in which a processor is used to realize the functions of the entire system including multiple processing units with a single IC (Integrated Circuit) chip, as typified by systems on chips (SoCs). In this way, various processing units are configured using one or more of the above-mentioned various processors as a hardware structure.

[0122] Furthermore, more specifically, the hardware structure of these various processors can be an electric circuit (circuitry) that combines circuit elements such as semiconductor elements.

[0123] From the above description, the technology described in the following supplementary paragraphs can be understood.

[0124] [Supplementary Item 1] An image processing device including a processor, which accepts input of a user's emotions regarding images at a plurality of time points, and stores the images in association with the emotion information at the plurality of time points. [Supplementary Item 2] The image processing device of Supplementary Item 1, wherein the images are printed images printed out on instant film, and the plurality of time points are a combination selected from the time points when the printed images are captured, the time points when the instant film is stored as a digital image, and the time points when the digital image of the instant film is posted to a social networking service. [Supplementary Item 3] The image processing device of Supplementary Item 1 or Supplementary Item 2, wherein the processor applies a display effect corresponding to the emotion information when displaying the images. [Supplementary Item 4] The image processing device of any one of Supplementary Item 1 to Supplementary Item 3, wherein the images can be searched using the emotion information as a search keyword. [Supplementary Item 5] The image processing device of any one of Supplementary Item 1 to Supplementary Item 4, wherein the images can be searched using the emotion information as a search keyword. [Supplementary Item 6] The image processing device according to Supplementary Item 5, wherein the associated and stored emotion information and the image are used as training data for the machine learning model. [Supplementary Item 7] The image processing device according to any one of Supplementary Items 1 to 6, wherein the image is a printed image printed out on instant film, the plurality of time points includes a time point at which the printed image is captured, the processor: acquires a digital image of the instant film, performs image analysis of the digital image to read text written on the instant film by the user, performs natural language analysis of the text, estimates the emotion at the time the printed image was captured according to the results of the natural language analysis, and displays the estimated emotion to the user.[Supplementary Item 8] The image processing device of any one of Supplementary Items 1 to 7, wherein the image is a printed image printed out on instant film, the plurality of time points includes a time point when the digital image on the instant film is posted to a social networking service, and the processor acquires text entered by the user when posting the digital image to the social networking service, performs natural language analysis of the text, estimates the emotion at the time when the digital image is posted to the social networking service according to a result of the natural language analysis, and displays the estimated emotion to the user. [Supplementary Item 9] The image processing device of any one of Supplementary Items 1 to 8, wherein the processor acquires status information of the user from a wearable device worn by the user, estimates the emotion based on the status information, and displays the estimated emotion to the user. [Supplementary Item 10] The image processing device according to any one of Supplementary Items 1 to 9, wherein the processor detects a facial expression of a person appearing in the image, estimates the emotion based on the detected facial expression, and displays the estimated emotion to the user.

[0125] The technology of the present disclosure can be appropriately combined with the various embodiments and / or various modified examples described above. Furthermore, it is not limited to the above-described embodiments, and various configurations can be adopted without departing from the spirit of the present disclosure. Furthermore, the technology of the present disclosure extends not only to programs but also to storage media that non-temporarily store programs.

[0126] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[0127] In this specification, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed by connecting them with "and / or."

[0128] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

Claims

1. An image processing device comprising a processor, which receives input of a user's emotions regarding images at multiple points in time, and stores information about the emotions at the multiple points in time in association with the images.

2. The image processing device according to claim 1, wherein the image is a printed image printed out on instant film, and the plurality of points in time are combinations selected from the points in time when the printed image is photographed, the points in time when the instant film is stored as a digital image, and the points in time when the digital image of the instant film is posted to a social networking service.

3. The image processing device according to claim 1, wherein the processor applies a display effect according to the emotional information when displaying the image.

4. The image processing device according to claim 1, wherein the image can be searched using the emotion information as a search keyword.

5. The image processing device according to claim 1, wherein the processor estimates the emotion using a machine learning model that outputs information about the emotion in response to the input of the image, and displays the estimated emotion to the user.

6. The image processing device according to claim 5, wherein the emotion information and the image stored in association with each other are used as training data for the machine learning model.

7. The image processing device of claim 1, wherein the image is a printed image printed out on instant film, the plurality of points in time include the point in time at which the printed image is photographed, and the processor: acquires a digital image of the instant film; performs image analysis of the digital image to read text written by the user on the instant film; performs natural language analysis of the text; estimates the emotion at the time the printed image is photographed according to the results of the natural language analysis; and displays the estimated emotion to the user.

8. The image processing device of claim 1, wherein the image is a printed image printed out on instant film, the plurality of points in time include a point in time when the digital image on the instant film is posted to a social networking service, and the processor: acquires text entered by the user when posting the digital image to the social networking service; performs natural language analysis of the text; estimates the emotion at the time when the digital image is posted to the social networking service according to the results of the natural language analysis; and displays the estimated emotion to the user.

9. The image processing device according to claim 1, wherein the processor acquires state information of the user from a wearable device worn by the user, estimates the emotion based on the state information, and displays the estimated emotion to the user.

10. An image processing device according to claim 1, wherein the processor detects facial expressions of people appearing in the image, estimates the emotion based on the detected facial expressions, and displays the estimated emotion to the user.

11. A method for operating an image processing device, comprising: accepting input of a user's emotions regarding images at a plurality of points in time; and storing the emotion information at the plurality of points in time in association with the images.

12. An operating program for an image processing device that causes a computer to execute a process including: accepting input of a user's emotions regarding an image at multiple points in time; and storing the information on the emotions at the multiple points in time in association with the image.