Image processing device, method for operating image processing device, and program for operating image processing device

Abstract

An image processing device includes a processor that receives input of emotion of a user with respect to an image at a plurality of points in time, associates information of the emotion at the plurality of points in time with the image, and stores the information.

Description

Technical Field

[0001] The technology of the present invention relates to an image processing device, an operating method of the image processing device, and an operating program of the image processing device. Background Art

[0002] Japanese Patent Publication No. 2017-092528 describes a photographic device comprising a photographic unit, a display unit, a selection receiving unit, a photographic instruction unit, and an association establishing unit. The display unit displays a plurality of image elements representing different moods. The selection receiving unit receives a user's selection operation from the image elements displayed by the display unit. The photographic instruction unit causes the photographic unit to capture an image in accordance with the user's operation. The association establishing unit uses the time when the photographic instruction unit captures the image as a reference, and when the selection operation of the image element is received by the selection receiving unit within a predetermined period, the mood information representing the mood associated with the selected image element is associated with the image data captured by the photographic unit. Summary of the invention

[0003] One embodiment of the technology according to the present invention provides an image processing device, an operating method of the image processing device, and an operating program of the image processing device that can more accurately grasp a user's emotion regarding an image.

[0004] Means for solving technical problems

[0005] The image processing device of the present invention includes a processor that receives input of user's emotions regarding an image at multiple time points, and associates and stores information on the emotions at the multiple time points with the image.

[0006] The preferred image is a printed image printed onto instant film, and the multiple time points are a combination selected from the time point of capturing the printed image, the time point of storing the instant film as a digital image, and the time point of posting the digital image of the instant film to a social networking service.

[0007] Preferably, the processor applies a display effect corresponding to the emotional information when displaying the image.

[0008] The preferred image can be searched using emotional information as a search keyword.

[0009] Preferably, the processor estimates the emotion using a machine learning model that outputs information of the emotion based on the input of the image, and displays the estimated emotion to the user.

[0010] The information and images of emotions stored preferably in an associated manner are used as teacher data for the machine learning model.

[0011] The preferred image is a printed image output onto instant film, and the multiple time points include the time point when the printed image was captured, and the processor performs the following processing: acquiring a digital image of the instant film; reading the text recorded by the user on the instant film by performing image analysis on the digital image; performing natural language analysis of the text; estimating the emotion at the time point when the printed image was captured based on the results of the natural language analysis; and displaying the estimated emotion to the user.

[0012] Preferably, the image is a printed image printed onto instant film, and the multiple time points include a time point when the digital image of the instant film is posted to a social networking service, and the processor performs the following processing: obtains text entered by the user when the digital image is posted to the social networking service; performs natural language analysis of the text; estimates the emotion at the time point when the digital image is posted to the social networking service based on the results of the natural language analysis; and displays the estimated emotion to the user.

[0013] Preferably, the processor obtains the user's state information from the wearable device worn by the user, estimates emotions based on the state information, and displays the estimated emotions to the user.

[0014] Preferably, the processor detects the expression of a person shown in the image, estimates the emotion based on the detection result of the expression, and displays the estimated emotion to the user.

[0015] The working method of the image processing device of the present invention comprises the following steps: accepting the input of the user's emotions about the image at multiple time points; and associating the information of the emotions at multiple time points with the image and storing them.

[0016] The operating program of the image processing device of the present invention causes the computer to execute processing including the following steps: accepting input of user's emotions for images at multiple time points; and associating and storing the information of the emotions at multiple time points with the images. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a diagram showing an instant camera, a user terminal, and an image management server.

[0018] Figure 2 This is a block diagram showing computers constituting a user terminal and an image management server.

[0019] Figure 3 This is a block diagram showing a processing unit of a CPU of a user terminal.

[0020] Figure 4 A diagram showing a storage instruction screen.

[0021] Figure 5 FIG. 1 is a diagram showing a storage instruction screen after the storage instruction button is pressed.

[0022] Figure 6 This figure shows the emotion input menu during shooting.

[0023] Figure 7 It means in Figure 5 FIG. 1 is a diagram showing the processing of the browser control unit when the OK button is pressed in the storage instruction screen.

[0024] Figure 8 A diagram showing an image playback display screen.

[0025] Fig. 9 This is a diagram showing an image playback display screen after the publish button is pressed.

[0026] Fig.10 It means in Fig. 9 This figure shows the processing of the browser control unit when the OK button is pressed on the image playback display screen.

[0027] Fig.11 This diagram shows the display effect when the emotion is "anger".

[0028] Fig.12 This diagram shows the display effect when the emotion is "sad".

[0029] Fig.13 This diagram shows the display effect when the emotion is "happy".

[0030] Fig.14 This is a block diagram showing a processing unit of a CPU of an image management server.

[0031] Fig.15 It is a diagram showing the details of the data stored in the image DB.

[0032] Fig.16 This is a diagram showing the processing of each processing unit of the image management server when the first storage request is transmitted from the user terminal.

[0033] Fig.17 This is a diagram showing the processing of each processing unit of the image management server when the second storage request is transmitted from the user terminal.

[0034] Fig.18 This is a diagram showing an image list display screen.

[0035] Fig.19 This is a diagram showing the processing of the browser control unit when a search keyword is input in the search bar.

[0036] Fig. 20 This is a diagram showing the processing of each processing unit of the image management server when a search request is transmitted from a user terminal.

[0037] Fig.21 FIG. 1 is a diagram showing an image list display screen displaying search results.

[0038] Fig. 22 This is a flowchart showing the processing procedure of the user terminal.

[0039] Fig.23 This is a flowchart showing the processing procedure of the image management server.

[0040] Fig.24 This is a flowchart showing the processing procedure of the user terminal.

[0041] Fig.25 This is a flowchart showing the processing procedure of the image management server.

[0042] Fig.26 This is a diagram showing the processing of each processing unit of the image management server when an emotion estimation request is transmitted from a user terminal.

[0043] Fig. 27 It is a diagram showing the processing of the emotion estimation unit.

[0044] Fig.28 This is a diagram showing an emotion input menu during shooting, in which the emotion estimation result is displayed in the form of a bubble message.

[0045] Fig.29 2 is a diagram showing the process in the learning phase of the emotion estimation model.

[0046] Fig.30 This is a diagram showing another example of processing in the learning phase of the emotion estimation model.

[0047] Fig.31 This is a diagram showing a policy for adopting teacher data of an emotion estimation model.

[0048] Fig.32 This is a diagram showing the processing of each processing unit of the image management server when a request for estimating an emotion including an instant film having text recorded thereon is transmitted from a user terminal.

[0049] Fig.33 Yes means Fig.32 A diagram showing a detailed configuration of an emotion estimation unit in the embodiment shown.

[0050] Fig.34 This is a diagram showing the processing of each processing unit of the image management server when a mood estimation request including text information input by a user when distributing instant films is transmitted from a user terminal.

[0051] Fig.35 Yes means Fig.34 A diagram showing a detailed configuration of an emotion estimation unit in the embodiment shown.

[0052] Fig.36 A diagram showing a smart watch and status information.

[0053] Fig.37 This is a graph showing body temperature change data.

[0054] Fig.38 This is a diagram showing the processing of each processing unit of the image management server when an emotion estimation request including state information is transmitted from a user terminal.

[0055] Fig.39 This is a diagram showing the processing of each processing unit of the image management server when a request for estimating an emotion of a printed image including a human face is transmitted from a user terminal.

[0056] Fig.40 Yes means Fig.39 A diagram showing a detailed configuration of an emotion estimation unit in the embodiment shown.

[0057] Fig.41 This is a diagram showing an emotion estimation model that outputs an emotion estimation result based on inputs of a printed image, a text reading result, and status information. DETAILED DESCRIPTION

[0058] As an example, Figure 1 As shown, a user U uses an instant camera 10 to photograph a subject, and prints out an image of the subject on an instant film 11. The instant film 11 may be either a silver salt type or a thermal sensitive type. Figure 1 In FIG. 1 , an ice cream sundae is shown as an example of a subject. Hereinafter, an image printed and output to the instant film 11 is referred to as a printed image 12 .

[0059] The printed image 12 is arranged at the approximate center of the instant film 11. Moreover, the size of the printed image 12 is slightly smaller than that of the instant film 11. Therefore, a blank space is provided between the edge of the instant film 11 and the edge of the printed image 12. In particular, a relatively large blank space 13 is provided at the bottom of the instant film 11. The user U can write text 14 in the blank space 13 with a marker pen or the like. Figure 1 , an example in which “delicious!” is written as the text 14 is shown.

[0060] The user U uses the camera function of the user terminal 15 to shoot the instant film 11 and stores the instant film 11 as a digital image. The user terminal 15 is a device having a camera function, an image playback and display function, and an image transmission and reception function. Specifically, the user terminal 15 is a smart phone, a tablet terminal, a small digital camera, a mirrorless single-lens camera, or a notebook personal computer. The user terminal 15 is an example of an "image processing device" involved in the technology of the present invention.

[0061] The user terminal 15 is connected to the image management server 17 via the network 16 so as to be able to communicate with each other. The network 16 is, for example, a WAN (Wide Area Network) such as the Internet or a public communication network. The user terminal 15 sends (uploads) the print image 12 to the image management server 17. In addition, the user terminal 15 receives (downloads) the print image 12 from the image management server 17.

[0062] The image management server 17 is, for example, a server computer, a workstation, etc., and is an example of the "image processing device" involved in the technology of the present invention together with the user terminal 15. In this way, the "image processing device" involved in the technology of the present invention can be implemented across multiple devices. Multiple user terminals 15 of multiple users U are connected to the image management server 17 via the network 16.

[0063] As an example, Figure 2 As shown, the computers constituting the user terminal 15 and the image management server 17 have basically the same structure and include a storage device 20, a memory 21, a CPU (Central Processing Unit) 22, a communication unit 23, a display 24, and an input device 25. These are connected to each other via a bus 26.

[0064] The storage device 20 is a hard disk drive built into the computer constituting the user terminal 15 and the image management server 17, or connected via a cable or a network. Alternatively, the storage device 20 is a disk array formed by connecting multiple hard disk drives. The storage device 20 stores control programs such as an operating system, various application programs (hereinafter referred to as AP (Application Program)), and various data accompanying these programs. In addition, a solid-state drive can also be used instead of a hard disk drive.

[0065] The memory 21 is a working memory for the CPU 22 to perform processing. The CPU 22 loads the program stored in the storage device 20 into the memory 21 and performs processing according to the program. Thus, the CPU 22 centrally controls each part of the computer. The CPU 22 is an example of a "processor" involved in the technology of the present invention. In addition, the memory 21 can be built into the CPU 22.

[0066] The communication unit 23 is a network interface that controls the transmission of various information via the network 16 or the like. The display 24 displays various screens. Various screens have operation functions based on GUI (Graphical User Interface). The computer constituting the user terminal 15 and the image management server 17 receives input of operation instructions from the input device 25 through various screens. The input device 25 is a keyboard, a mouse, a touch panel, a microphone for voice input, etc.

[0067] In addition, in the following description, symbols are assigned to distinguish them as follows: each part of the computer that constitutes the user terminal 15 (storage device 20, CPU22, display 24 and input device 25) is assigned an additional mark "A", and each part of the computer that constitutes the image management server 17 (storage device 20 and CPU22) is assigned an additional mark "B".

[0068] As an example, Figure 3 As shown, the printing image AP30 is stored in the storage device 20A of the user terminal 15. The printing image AP30 is installed in the user terminal 15 by the user U. The printing image AP30 is an AP for making the computer constituting the user terminal 15 function as the "image processing device" involved in the technology of the present invention. That is, the printing image AP30 is an example of the "image processing device operation program" involved in the technology of the present invention. When the printing image AP30 is activated, the CPU 22A of the user terminal 15 cooperates with the memory 21 and the like to function as the browser control unit 32. The browser control unit 32 controls the operation of the dedicated web browser of the printing image AP30.

[0069] The browser control unit 32 generates various screens. The browser control unit 32 displays the generated various screens on the display 24A. Furthermore, the browser control unit 32 receives various operation instructions input from the user U through the input device 25A through the various screens. The browser control unit 32 sends various requests corresponding to the operation instructions to the image management server 17.

[0070] As an example, Figure 4 As shown, the browser control unit 32 displays a storage instruction screen 35 on the display 24A according to the instruction of the user U. The storage instruction screen 35 is a screen for performing a storage instruction to store the instant film 11 and the printed image 12 as a digital image. A frame 36 for storing the instant film 11 to be stored as a digital image, a message 37, and a storage instruction button 38 are displayed on the storage instruction screen 35. The content of the message 37 is to urge the user U to store the instant film 11 to be stored as a digital image in the frame 36 and press the storage instruction button 38. The storage instruction button 38 functions as a so-called shutter button.

[0071] The user U stores the instant film 11 to be stored as a digital image in the frame 36 according to the message 37, and presses the storage instruction button 38. When the storage instruction button 38 is pressed, the browser control unit 32 displays the storage instruction screen 35 as shown in FIG. Figure 5 Transition as shown.

[0072] As an example, Figure 5 As shown, a message 40, an emotion input menu 41A, an emotion input menu 41B, and an OK button 42 are provided on the storage instruction screen 35 after the storage instruction button 38 is pressed. The content of the message 40 is to urge the user U to input the emotion for the print image 12 at the time of photographing the print image 12 (hereinafter referred to as the time of photographing) and the time of storing the instant film 11 (print image 12) as a digital image (hereinafter referred to as the time of storing), and to press the OK button 42. The emotion input menu 41A is a GUI for inputting the emotion of the user U for the print image 12 at the time of photographing. The emotion input menu 41B is a GUI for inputting the emotion of the user U for the print image 12 at the time of storing. In the following, when there is no need to distinguish between the emotion input menus 41A and 41B, it may be referred to as the emotion input menu 41.

[0073] As an example, Figure 6 As shown, the emotion input menu 41A has a face type 43A representing the emotion "joy", a face type 43B representing the emotion "anger", a face type 43C representing the emotion "sadness", and a face type 43D representing the emotion "happiness". Characters representing the emotions expressed by each face type 43A to 43D are displayed above the face types 43A to 43D. In the following, when there is no need to distinguish between the face types 43A to 43D, they are sometimes recorded as face type 43.

[0074] exist Figure 5 When the emotion input menu 41A is selected in the state shown, face shapes 43A to 43D are displayed. Any one of the face shapes 43A to 43D can be selected. The user U selects a face shape 43 that is suitable for the emotion of the printed image 12 when photographing from among the face shapes 43A to 43D. Figure 6 In FIG. 4 , as shown by hatching, an example is shown in which a face shape 43A representing “joy” is selected as an emotion. In addition, the emotion input menu 41B has the same structure as the emotion input menu 41A, so illustration and description are omitted.

[0075] As an example, Figure 7As shown, the user U inputs the desired emotion in the emotion input menus 41A and 41B according to the message 40, and then presses the OK button 42. When the OK button 42 is pressed, the browser control unit 32 generates emotion information 45A. The emotion information 45A includes the emotions at the time of shooting and storage input through the emotion input menus 41A and 41B. In addition, the emotion information 45A includes an image ID (Identification Data) for uniquely identifying the printed image 12. Figure 7 , an example is shown in which a face shape 43A indicating “joy” is selected in both the emotion input menus 41A and 41B, and emotion information 45A in which “joy” is registered as the emotion at the time of photographing and storing is generated.

[0076] As an example, Figure 8 As shown, the browser control unit 32 displays the image playback display screen 50 on the display 24A according to the instruction of the user U. The image playback display screen 50 is a screen for playing back and displaying the instant film 11 and the printed image 12 stored as digital images. The browser control unit 32 applies the display effect corresponding to the emotion information 45A to the printed image 12. Figure 8 In Figure 7 The example shown is an example in which the emotion information 45A of "joy" is registered as the emotion at the time of shooting and storing. In this case, the browser control unit 32 displays a plurality of star marks 51 on the print image 12 as a display effect.

[0077] A post button 52 is provided on the image playback display screen 50 for posting the instant film 11 stored as a digital image to a social network service (hereinafter referred to as SNS (Social Networking Service)) through an application. When the post button 52 is pressed, the browser control unit 32 makes the image playback display screen 50 display as shown in FIG. Fig. 9 Transition as shown.

[0078] As an example, Fig. 9 As shown, after the post button 52 is pressed, icons 55 of various SNS applications, a message 56, an emotion input menu 41C, and an OK button 57 are provided on the image playback display screen 50. The content of the message 56 is to urge the user U to input the emotion for the printed image 12 at the time point when the digital image of the instant film 11 is posted to the SNS (hereinafter referred to as the time of posting) and to press the OK button 57. The emotion input menu 41C is a GUI for inputting the emotion for the printed image 12 of the user U at the time of posting. The emotion input menu 41C has the same structure as the emotion input menu 41A, etc., and therefore, illustration and description thereof are omitted.

[0079] As an example, Fig.10 As shown, the user U selects icon 55, and inputs the desired emotion in emotion input menu 41C according to message 56, and then presses OK button 57. When OK button 57 is pressed, browser control unit 32 generates emotion information 45B. Emotion information 45B includes the emotion and image ID at the time of posting input through emotion input menu 41C. Fig.10 , an example is shown in which a face shape 43D indicating "happiness" is selected in the emotion input menu 41C, and emotion information 45B is generated in which "happiness" is registered as the emotion at the time of posting. In the following, when there is no need to particularly distinguish between emotion information 45A and 45B, they are sometimes described as emotion information 45.

[0080] exist Figure 8 and Fig. 9 In the figure, the display effect when the emotion is "joy" is shown, and the display effect when the emotion is other emotions is shown as an example. Figure 11 to Figure 13 That is, when the emotion is "anger", Fig.11 As shown, as a display effect, the browser control unit 32 displays a plurality of anger marks 60 simulating the blood vessels of the temples, i.e., the so-called blue veins, in the printed image 12. When the emotion is "sadness", as shown in FIG. Fig.12 As shown, the browser control unit 32 displays a plurality of tear marks 61 simulating teardrops in the printed image 12 as a display effect. Fig.13 As shown, the browser control unit 32 displays a plurality of musical note marks 62 in the print image 12 as a display effect.

[0081] Even after the digital image of the instant film 11 is posted to the SNS, when the printed image 12 is played back and displayed in the image playback display screen 50, the browser control unit 32 performs a display effect on the printed image 12. The browser control unit 32 performs a display effect based on the emotion that is selected in large numbers at the time of shooting, storage, and posting. For example, when the emotion at the time of shooting and storage is "joy", and the emotion at the time of posting is "happiness", the browser control unit 32 performs a display effect of a star mark 51 that matches "joy". In the case where the emotions at the time of shooting and storage are different or the emotions at the time of shooting, storage, and posting are different, the browser control unit 32 performs a display effect based on the emotion at the update time point. In addition, the star mark 51 can be made to flash, the size of the anger mark 60 can be changed, the tear mark 61 can be made to flow up and down, or the angle of the note mark 62 can be changed, etc. can be displayed with animation.

[0082] As an example, Fig.14As shown, a work program 70 is stored in the storage device 20B of the image management server 17. The work program 70 is an AP for making the computer constituting the image management server 17 function as an "image processing device" involved in the technology of the present invention. That is, the work program 70 is an example of a "work program of an image processing device" involved in the technology of the present invention, similarly to the print image AP30. An image database (hereinafter referred to as DB (Data Base)) 71 is also stored in the storage device 20B. In addition, although not shown in the figure, a user ID for uniquely identifying the user U, a password set by the user U, and a terminal ID for uniquely identifying the user terminal 15 are stored in the storage device 20B as the account information of the user U.

[0083] When the operation program 70 is activated, the CPU 22B of the image management server 17 cooperates with the memory 21 and the like to function as a receiving unit 75 , a read / write (hereinafter referred to as RW (Read Write)) control unit 76 , and a distribution control unit 77 .

[0084] The receiving unit 75 receives various requests from the user terminal 15. The receiving unit 75 outputs the various requests to the RW control unit 76 and the distribution control unit 77. The RW control unit 76 controls the storage of various data in the storage device 20B and the reading of various data from the storage device 20B. In particular, the RW control unit 76 controls the storage of the print image 12 and the like in the image DB 71 and the reading of the print image 12 and the like from the image DB 71. The distribution control unit 77 controls the distribution of various data to the user terminal 15.

[0085] As an example, Fig.15 As shown in FIG. 1 , a storage area 80 is provided for each user U in the image DB 71. A user ID is registered in the storage area 80. In the storage area 80, the instant film 11 stored as a digital image, the printed image 12, and the emotion information 45 are associated and stored by the image ID. For example, in the printed image 12 shown by the image ID "P00002", the emotion at the time of posting is not registered in the printed image 12 that has not been posted to the SNS.

[0086] Tag information is registered in each printed image 12. Tag information is a word that simply indicates the subject shown in the printed image 12. Tag information includes information manually input by the user U or information derived using image analysis software. In addition, although not shown in the figure, the settings of the instant camera 10 when shooting (macro, Selfie, flash shooting, etc.), the scene when shooting (daytime, nighttime, outdoor, indoor, etc.), and the image quality evaluation score are also registered in each printed image 12.

[0087] As an example, Fig.16As shown in FIG. 1 , when the OK button 42 is pressed in the storage instruction screen 35, the browser control unit 32 Figure 7 After the emotion information 45A is generated, the first storage request 85A is sent to the image management server 17. The first storage request 85A includes the user ID, the instant film 11 stored as a digital image, the printed image 12, and the emotion information 45A.

[0088] The receiving unit 75 receives the first storage request 85A and outputs the first storage request 85A to the RW control unit 76. The RW control unit 76 associates the print image 12 and the emotion information 45A with each other according to the first storage request 85A and stores them in the storage area 80 of the image DB 71 corresponding to the user ID. Fig.16 , an example is shown in which the emotion "joy" when shooting and the emotion "happiness" when storing the printed image 12 of the existing car and the emotion information 45A are associated and stored in the storage area 80 of the user U with the user ID "U00001". Fig.16 In the figure, the label information and the like are omitted. Fig.17 The same goes for etc.

[0089] As an example, Fig.17 As shown in FIG. 1 , when the OK button 57 is pressed on the image playback display screen 50, the browser control unit 32 Fig.10 After the emotion information 45B is generated, the second storage request 85B is sent to the image management server 17. The second storage request 85B includes the user ID and the emotion information 45B.

[0090] The receiving unit 75 receives the second storage request 85B and outputs the second storage request 85B to the RW control unit 76. The RW control unit 76 stores the emotion information 45B in the storage area 80 of the image DB 71 corresponding to the user ID in accordance with the second storage request 85B. Fig.17 In FIG. 1 , the printed image 12 is shown with Fig.16 In the same manner as in the case of the image of the existing car, the emotion “happy” of the emotion information 45B at the time of posting is stored in the storage area 80 of the user U with the user ID “U00001”. In this way, the RW control unit 76 associates the emotion information 45 at the three time points of shooting, storage, and posting with the printed image 12 and stores them in the image DB 71.

[0091] As an example, Fig.18As shown, the browser control unit 32 displays an image list display screen 90 on the display 24A according to the instruction of the user U. The image list display screen 90 displays thumbnail images 12S of the printed images 12 of the instant film 11 stored as digital images. In the image list display screen 90, when the user U selects one of the thumbnail images 12S, the display transitions to Figure 8 An image playback display screen 50 is shown.

[0092] A search bar 91 is provided on the image list display screen 90. In the search bar 91, the user U inputs a search keyword for searching for a desired print image 12. As the search keyword, any word and the words of each emotion of the emotion information 45 such as "joy", "anger", "sadness" and "happiness" can be input.

[0093] As an example, Fig.19 As shown in FIG. 1 , when a search keyword is input in the search field 91, the browser control unit 32 generates a search request 95. The search request 95 includes the user ID and the search keyword input in the search field 91. Fig.19 , an example in which an arbitrary word "family" and an emotional word "happy" are input as search keywords is shown.

[0094] As an example, Fig. 20 As shown, the browser control unit 32 sends the search request 95 to the image management server 17. The receiving unit 75 receives the search request 95 and outputs the search request 95 to the RW control unit 76. The RW control unit 76 searches for the printed images 12 whose emotion information 45 and tag information are consistent with the search keyword among the printed images 12 stored in the storage area 80 of the image DB 71 corresponding to the user ID. The RW control unit 76 outputs the image ID of the searched printed image 12 to the distribution control unit 77. The distribution control unit 77 distributes the image ID from the RW control unit 76 to the user terminal 15 that is the source of the search request 95. The distribution control unit 77 determines the user terminal 15 that is the source of the search request 95 based on the user ID included in the search request 95. Fig. 20 In the figure, the search keyword is shown as Fig.19 The same example of "family" and "happiness". Next, an example of searching for three printed images 12 having image IDs "P00200", "P00201" and "P00202" in which "happiness" is registered in the emotion information 45 and "family" is registered in the tag information is shown.

[0095] As an example, Fig.21As shown, the browser control unit 32 displays only the thumbnail images 12S of the printed images 12 of the image ID searched by the RW control unit 76 and distributed by the distribution control unit 77 in the image list display screen 90 as search results. In this way, the printed images 12 can be searched using the emotion information 45 as a search keyword. In addition, the AND search of arbitrary words and emotion words is illustrated here, but it is not limited to this. Of course, it is also possible to search for printed images 12 by setting only arbitrary words as search keywords, or only one emotion word as search keywords, or two or more emotion words as search keywords.

[0096] Next, as an example, refer to Fig. 22 , Fig.23 , Fig.24 and Fig.25 The flowchart shown in the figure explains the operation based on the above structure. Figure 3 As shown in FIG. 1 , the CPU 22A of the user terminal 15 functions as the browser control unit 32 by starting the print image AP 30. Fig.14 As shown, the CPU 22B of the image management server 17 functions as a receiving unit 75 , an RW control unit 76 , and a distribution control unit 77 by activating the operation program 70 .

[0097] In order to store the desired instant film 11 as a digital image, the user U uses Figure 4 The storage instruction screen 35 shown is displayed on the display 24A. Then, the instant film 11 to be stored as a digital image is placed in the frame 36, and the storage instruction button 38 is pressed. As a result, the browser control unit 32 receives the storage instruction of the instant film 11 ( Fig. 22 The browser control unit 32 stores the display of the instruction screen 35. Figure 5 Transition as shown.

[0098] The user U operates the emotion input menus 41A and 41B, inputs the emotion at the time of shooting and the emotion at the time of storage, and then presses the OK button 42. Thus, the browser control unit 32 receives the input of the emotion for the print image 12 at the time of shooting and the time of storage (step ST110). Figure 7 As shown in FIG. 1 , the browser control unit 32 generates the emotion information 45A at the time of shooting and storage (step ST120). Fig.16 As shown, under the control of the browser control unit 32, the first storage request 85A including the emotion information 45A is transmitted to the image management server 17 (step ST130).

[0099] In the image management server 17, the receiving unit 75 receives the first storage request 85A ( Fig.23The first storage request 85A is output from the receiving unit 75 to the RW control unit 76. Then, under the control of the RW control unit 76, the emotion information 45A at the time of shooting and storage is associated with the print image 12 and stored in the image DB 71 (step ST160).

[0100] In order to post the desired instant film 11 on the SNS, the user U Figure 8 The image playback display screen 50 shown is displayed on the display 24A. Then, the publish button 52 is pressed. As a result, the browser control unit 32 receives the publish instruction of the instant film 11 ( Fig.24 The browser control unit 32 displays the image playback display screen 50 as shown in FIG. Fig. 9 Transition as shown.

[0101] The user U operates the emotion input menu 41C, inputs the emotion for publishing, and then presses the OK button 57. Thus, the browser control unit 32 receives the input of the emotion for the print image 12 for publishing (step ST210). Fig.10 As shown in FIG. 1 , the browser control unit 32 generates the emotion information 45B at the time of publishing (step ST220). Fig.17 As shown, under the control of the browser control unit 32, the second storage request 85B including the emotion information 45B is transmitted to the image management server 17 (step ST230).

[0102] In the image management server 17, the receiving unit 75 receives the second storage request 85B ( Fig.25 The first storage request 85A is output from the receiving unit 75 to the RW control unit 76. Then, under the control of the RW control unit 76, the emotion information 45B at the time of publishing is associated with the print image 12 and stored in the image DB 71 (step ST260).

[0103] As described above, the browser control unit 32 of the CPU 22A of the user terminal 15 receives input of emotions of the user U regarding the print image 12 at multiple time points. The RW control unit 76 of the CPU 22B of the image management server 17 associates the emotion information 45 at multiple time points with the print image 12 and stores them in the image DB 71.

[0104] In Japanese Patent Application Laid-Open No. 2017-092528, the time when the user U inputs the emotion of the printed image 12 is limited to the time of shooting. Therefore, it is impossible to accurately grasp the emotion of the user U for the printed image 12 that changes over time. In contrast, as described above, the technology of the present invention accepts the input of the emotion of the user U for the printed image 12 at multiple time points, and associates and stores the emotion information 45 at multiple time points with the printed image 12. Therefore, the emotion of the user U for the printed image 12 can be grasped more accurately.

[0105] The image is a printed image 12 printed and output to the instant film 11. Then, the multiple time points are the time point of photographing the printed image 12 (photographing), the time point of storing the instant film 11 as a digital image (storing), and the time point of posting the digital image of the instant film 11 to the SNS (posting). Therefore, the emotions of the user U regarding the printed image 12 at the time of photographing, storing, and posting can be accurately grasped. In addition, the multiple time points are not limited to all of the time of photographing, storing, and posting, and any combination selected from these (for example, the time of photographing and storing, or the time of storing and posting, etc.) may be used.

[0106] like Figure 8 and Figure 11 to Figure 13 As shown, the browser control unit 32 applies a display effect corresponding to the emotion information 45 when displaying the printed image 12. Therefore, the emotion of the user U regarding the printed image 12 can be clearly seen at a glance. In addition, the display of the printed image 12, which tends to become boring, can be made interesting. In addition, music corresponding to the emotion information 45 can also be played together with the display effect.

[0107] like Figure 18 to Figure 21 As shown, the printed image 12 can be searched using the emotion information 45 as a search keyword. Therefore, the printed image 12 that better matches the intention of the user U can be searched.

[0108] [Second embodiment]

[0109] As an example, Fig.26 As shown, in this embodiment, Figure 4 In the storage instruction screen 35 shown in FIG. 1 , the storage instruction button 38 is pressed and in the Figure 8 When the publish button 52 is pressed in the image playback display screen 50 shown in FIG. 1 , the browser control unit 32 sends the emotion estimation request 100 to the image management server 17. The emotion estimation request 100 includes the user ID, the instant film 11 stored as a digital image, and the printed image 12. The printed image 12 is a Fig.16 An image of a car is shown in FIG.

[0110] The CPU 22B of the image management server 17 of this embodiment includes the processing units 75 to 77 (in Fig.26 In addition to the RW control unit 76 (not shown), it also functions as an emotion estimation unit 101.

[0111] The receiving unit 75 receives the emotion estimation request 100 and outputs the emotion estimation request 100 to the emotion estimation unit 101. The emotion estimation unit 101 estimates the emotion of the user U with respect to the printed image 12 based on the emotion estimation request 100. The emotion estimation unit 101 outputs the emotion estimation result 103, which is the result of estimating the emotion of the user U with respect to the printed image 12, to the distribution control unit 77. The distribution control unit 77 distributes the emotion estimation result 103 to the user terminal 15 from which the emotion estimation request 100 was requested.

[0112] As an example, Fig. 27 As shown, the emotion estimation unit 101 estimates the emotion of the user U with respect to the printed image 12 using the emotion estimation model 105. The emotion estimation model 105 is stored in the storage device 20B. The emotion estimation model 105 is read from the storage device 20B by the RW control unit 76 and output to the emotion estimation unit 101. The emotion estimation model 105 is composed of, for example, a convolutional neural network. The emotion estimation model 105 is an example of a "machine learning model" involved in the technology of the present invention.

[0113] The emotion estimation unit 101 inputs the printed image 12 of the emotion estimation request 100 into the emotion estimation model 105, and outputs the emotion estimation result 103 of the user U with respect to the printed image 12 from the emotion estimation model 105. Fig. 27 2 shows an example in which the emotion of the user U with respect to the printed image 12 is estimated to be “joy”.

[0114] When the emotion estimation result 103 is received from the distribution control unit 77, as an example, Fig.28 As shown, the browser control unit 32 displays a bubble message 108 above the face 43 corresponding to the emotion of the emotion estimation result 103 in the emotion input menu 41. The bubble message 108 is a message to the user U, "Is this the mood?" The content is the emotion estimated by the emotion estimation model 105. Fig.28 , an example is shown in which a bubble message 108 is displayed above a face shape 43A representing "joy" as an emotion corresponding to the emotion of the emotion estimation result 103 in the emotion input menu 41A during shooting. In addition, although not shown in the figure, the browser control unit 32 also displays a bubble message 108 above a face shape 43 corresponding to the emotion of the emotion estimation result 103 in the emotion input menu 41B during storage and the emotion input menu 41C during publishing.

[0115] As an example, Fig.29 As shown, the emotion estimation model 105 is learned by the provided teacher data 110. The teacher data 110 is a set of the learning printed image 12L and the correct answer emotion information 45CA.

[0116] In the learning phase, the learning printed image 12L is input to the emotion estimation model 105. The emotion estimation model 105 outputs a learning emotion estimation result 103L based on the input of the learning printed image 12L. Based on the learning emotion estimation result 103L and the correct emotion information 45CA, a loss calculation of the emotion estimation model 105 using a loss function is performed. Then, based on the result of the loss calculation, various coefficients of the emotion estimation model 105 are updated and set, and the emotion estimation model 105 is updated according to the updated setting.

[0117] In the learning phase, the above series of processes including input of the learning printed image 12L to the emotion estimation model 105, output of the learning emotion estimation result 103L from the emotion estimation model 105, loss calculation, update setting, and update of the emotion estimation model 105 are repeated while the teacher data 110 is replaced. When the estimation accuracy of the learning emotion estimation result 103L relative to the correct emotion information 45CA reaches a preset level, the repetition of the above series of processes ends. In this way, the emotion estimation model 105 whose estimation accuracy reaches the set level is stored in the storage device 20B and used in the emotion estimation unit 101. In addition, the learning may be ended when the above series of processes are repeated a set number of times regardless of the estimation accuracy of the learning emotion estimation result 103L relative to the correct emotion information 45CA.

[0118] Thus, in the second embodiment, the emotion estimation unit 101 estimates the emotion using the emotion estimation model 105 that outputs the emotion estimation result 103 based on the input of the printed image 12. The browser control unit 32 displays the emotion estimated by the emotion estimation unit 101 to the user U by displaying a bubble message 108. Machine learning models such as the emotion estimation model 105 have been widely put into practical use recently, and their estimation accuracy has also been improved. Therefore, it is possible to more appropriately assist the input of emotions.

[0119] In addition, as an example, Fig.30As shown, the printed image 12 and the emotion information 45 associated with and stored in the image DB 71 may be used as the teacher data 110 of the emotion estimation model 105. Specifically, the printed image 12 stored in the image DB 71 is used as the printed image 12L for learning, and the emotion information 45 at the time of shooting stored in the image DB 71 is used as the correct emotion information 45CA. Here, the emotion information 45 at the time of shooting is used as the correct emotion information 45CA because it is believed that the emotion of the user U with respect to the printed image 12 is more realistically expressed than when it is stored and published. In this way, the printed image 12 and the emotion information 45 stored in the image DB 71 can be effectively utilized. There is no need to prepare the printed image 12L for learning and the correct emotion information 45CA for the teacher data 110. In addition, the trouble caused by the insufficiency of the teacher data 110 can be reduced, and the risk of over-learning of the emotion estimation model 105 caused by the insufficiency of the teacher data 110 can be suppressed.

[0120] When the printed image 12 and the emotion information 45 associated with and stored in the image DB 71 are used as the teacher data 110 of the emotion estimation model 105, as an example, Fig.31 As shown, the emotion estimation result 103 is also associated with the image DB 71 together with the printed image 12 and the emotion information 45 and stored. Then, it is preferred that the emotion information 45 at the time of shooting and the emotion estimation result 103 are different (in Fig.31 In the example, the data recorded as (emotion information at the time of shooting ≠ emotion estimation result) is actively adopted as the teacher data 110 of the emotion estimation model 105. The data in which the emotion information 45 at the time of shooting is different from the emotion estimation result 103 is data in which the emotion estimation model 105 incorrectly estimates the emotion. Therefore, if the data in which the emotion information 45 at the time of shooting is different from the emotion estimation result 103 is actively adopted as the teacher data 110 of the emotion estimation model 105, the estimation accuracy of the emotion estimation model 105 can be improved.

[0121] The emotion estimation model 105 may be learned in the image management server 17 or in another device other than the image management server 17. Furthermore, the emotion estimation model 105 may be learned continuously after being stored in the storage device 20B.

[0122] [Third embodiment]

[0123] As an example, Fig.32 As shown, in this embodiment, Figure 4When the storage instruction button 38 is pressed in the storage instruction screen 35 shown in FIG. 1 , the browser control unit 32 sends an emotion estimation request 115 to the image management server 17. The emotion estimation request 115 includes the user ID, the instant film 11 stored as a digital image, and the printed image 12. Fig.16 Similarly, an image showing an existing car is illustrated in the printed image 12 , and a text 14 of “Bought a new car!!” is written in the blank 13 by the user U.

[0124] The CPU 22B of the image management server 17 of this embodiment includes the processing units 75 to 77 (in Fig.32 In addition to the RW control unit 76 (not shown), it also functions as an emotion estimation unit 116.

[0125] The receiving unit 75 receives the emotion estimation request 115 and outputs the emotion estimation request 115 to the emotion estimation unit 116. The emotion estimation unit 116 estimates the emotion of the user U with respect to the printed image 12 at the time of shooting based on the emotion estimation request 115. The emotion estimation unit 116 outputs the emotion estimation result 117, which is the result of estimating the emotion of the user U with respect to the printed image 12 at the time of shooting, to the distribution control unit 77. The distribution control unit 77 distributes the emotion estimation result 117 to the user terminal 15 that is the source of the request for the emotion estimation request 115.

[0126] As an example, Fig.33 As shown, the emotion estimation unit 116 includes a text reading unit 120 and a natural language analysis unit 121. The instant film 11 of the emotion estimation request 115 is input to the text reading unit 120. The text reading unit 120 reads the text 14 recorded by the user U on the blank 13 of the instant film 11 by performing image analysis on the instant film 11 stored as a digital image. The text reading unit 120 outputs the result of reading the text 14, that is, the text reading result 122, to the natural language analysis unit 121.

[0127] The natural language analysis unit 121 performs a natural language analysis on the text 14 of the text reading result 122. The natural language analysis unit 121 estimates the emotion of the user U regarding the printed image 12 at the time of shooting based on the result of the natural language analysis, and outputs the emotion estimation result 117. The natural language analysis unit 121 estimates the emotion of the user U regarding the printed image 12 at the time of shooting, for example, using an emotion estimation model that outputs the emotion estimation result 117 based on the input of the result of the natural language analysis. Here, the natural language analysis performed by the natural language analysis unit 121 includes morpheme analysis, syntactic analysis, semantic analysis, and context analysis. Fig.332 shows an example in which the text 14 “Bought a new car!!” is read by the text reading unit 120 , and the emotion of the user U with respect to the printed image 12 at the time of shooting is estimated to be “joyful”.

[0128] Although not shown in the figure, upon receiving the emotion estimation result 117 from the distribution control unit 77, the browser control unit 32 and Fig.28 Similarly to the illustrated case, in the emotion input menu 41A, the bubble message 108 is displayed above the face shape 43 corresponding to the emotion of the emotion estimation result 117 .

[0129] Thus, in the third embodiment, the text reading unit 120 of the emotion estimation unit 116 reads the text 14 recorded on the instant film 11 by the user U by performing image analysis on the instant film 11 stored as a digital image. The natural language analysis unit 121 of the emotion estimation unit 116 performs natural language analysis on the text 14, and estimates the emotion of the user U regarding the printed image 12 at the time of shooting based on the result of the natural language analysis. The browser control unit 32 displays the emotion estimated by the emotion estimation unit 116 to the user U by displaying the bubble message 108. Therefore, the user U can be assisted in inputting an emotion suitable for the text 14 recorded on the instant film 11.

[0130] [Fourth embodiment]

[0131] In this embodiment, Figure 8 When the publish button 52 is pressed on the image playback display screen 50 shown in FIG. 1 , the browser control unit 32 receives input of a text 125 from the user U. The text 125 is a description text or the like attached to the instant film 11 when the instant film 11 is published on the SNS. After receiving the input of the text 125, the browser control unit 32 changes the display of the image playback display screen 50 to the following: Fig. 9 Transition as shown.

[0132] As an example, Fig.34 As shown, in this embodiment, when receiving the input based on the text 125 of the user U, the browser control unit 32 sends the emotion estimation request 126 to the image management server 17. The emotion estimation request 126 includes the user ID and the input text information 127. The input text information 127 includes the text 125. In addition, the printed image 12 shows a mother and daughter playing in the park, and the text 125 written by the user U "the family went to XX park to play and the daughter was excited."

[0133] The CPU 22B of the image management server 17 of this embodiment includes the processing units 75 to 77 (in Fig.34 In addition to the RW control unit 76 (not shown), it also functions as an emotion estimation unit 128.

[0134] The receiving unit 75 receives the emotion estimation request 126 and outputs the emotion estimation request 126 to the emotion estimation unit 128. The emotion estimation unit 128 estimates the emotion of the user U regarding the printed image 12 at the time of publication based on the emotion estimation request 126. The emotion estimation unit 128 outputs the emotion estimation result 129, which is the result of the estimated emotion of the user U regarding the printed image 12 at the time of publication, to the distribution control unit 77. The distribution control unit 77 distributes the emotion estimation result 129 to the user terminal 15 that is the source of the request for the emotion estimation request 126.

[0135] As an example, Fig.35 As shown, the emotion estimation unit 128 includes a text acquisition unit 130 and a natural language analysis unit 131. The input text information 127 of the emotion estimation request 126 is input to the text acquisition unit 130. The text acquisition unit 130 outputs the input text information 127 to the natural language analysis unit 131.

[0136] The natural language analysis unit 131 performs a natural language analysis on the text 125. The natural language analysis unit 131 estimates the emotion of the user U regarding the printed image 12 at the time of posting based on the result of the natural language analysis, and outputs the emotion estimation result 129. The natural language analysis unit 131 is similar to the natural language analysis unit 121, for example, using an emotion estimation model that outputs the emotion estimation result 129 based on the input of the result of the natural language analysis to estimate the emotion of the user U regarding the printed image 12 at the time of posting. The natural language analysis performed by the natural language analysis unit 131 is also similar to the natural language analysis unit 121, including morphological analysis, syntactic analysis, semantic analysis, and context analysis. Fig.35 2 shows an example in which the emotion of the user U regarding the printed image 12 at the time of posting is estimated to be “happy” based on the text 125 “My daughter is excited when my family went to XX park to play”.

[0137] Although not shown in the figure, upon receiving the emotion estimation result 129 from the distribution control unit 77, the browser control unit 32 and Fig.28 Similarly to the illustrated case, in the emotion input menu 41C, the bubble message 108 is displayed above the face shape 43 corresponding to the emotion of the emotion estimation result 129 .

[0138] Thus, in the fourth embodiment, the text acquisition unit 130 of the emotion estimation unit 128 acquires the text 125 input by the user U when posting the instant film 11 stored as a digital image to the SNS by acquiring the input text information 127. The natural language analysis unit 131 of the emotion estimation unit 128 performs natural language analysis of the text 125, and estimates the emotion of the user U regarding the printed image 12 at the time of posting based on the result of the natural language analysis. The browser control unit 32 displays the emotion estimated by the emotion estimation unit 128 to the user U by displaying the bubble message 108. Therefore, it is possible to assist the user U in inputting an emotion that matches the text 125 input when posting the instant film 11 to the SNS.

[0139] [Fifth embodiment]

[0140] As an example, Fig.36 As shown, in this embodiment, state information 136 of the user U output from a smart watch 135 worn on the wrist of the user U is used to estimate the mood of the user U. The smart watch 135 is an example of a “wearable device” involved in the technology of the present invention.

[0141] The smart watch 135 is connected to the instant camera 10 and the user terminal 15 via short-range wireless communication such as Bluetooth (registered trademark) so as to be able to communicate with each other. The smart watch 135 is linked with the instant camera 10 and the user terminal 15. More specifically, the smart watch 135 transmits the status information 136 to the user terminal 15 in conjunction with the shooting instruction of the print image 12 of the instant camera 10, the storage instruction of the print image 12 to the user terminal 15 based on the pressing of the storage instruction button 38, and the publishing instruction of the print image 12 to the user terminal 15 based on the pressing of the publishing button 52.

[0142] The status information 136 includes body temperature variation data, pulse variation data, blood pressure variation data, and angular velocity variation data. Fig.37 As shown in FIG. 1 , the body temperature change data is time series data showing changes in the body temperature of the user U for 30 seconds before and after (0 second) when the photographing instruction, storage instruction, or release instruction of the printing image 12 is given. Although not shown in the figure, the pulse change data, blood pressure change data, and angular velocity change data are also time series data showing changes in the pulse, blood pressure, and angular velocity of the user U for 30 seconds before and after the photographing instruction, storage instruction, or release instruction of the printing image 12, similarly to the body temperature change data. Based on the angular velocity change data, the hand shaking state of the user U when the photographing instruction, storage instruction, or release instruction of the printing image 12 is given can be understood.

[0143] As an example, Fig.38As shown, in this embodiment, upon receiving the status information 136 from the smart watch 135 , the browser control unit 32 transmits an emotion estimation request 140 to the image management server 17 . The emotion estimation request 140 includes the user ID and the status information 136 .

[0144] The CPU 22B of the image management server 17 of this embodiment includes the processing units 75 to 77 (in Fig.38 In addition to the RW control unit 76 (not shown), it also functions as an emotion estimation unit 141.

[0145] The receiving unit 75 receives the emotion estimation request 140 and outputs the emotion estimation request 140 to the emotion estimation unit 141. The emotion estimation unit 141 estimates the emotion of the user U regarding the printed image 12 when it is photographed, stored, or published, based on the emotion estimation request 140. The emotion estimation unit 141 outputs the result of estimating the emotion of the user U regarding the printed image 12, that is, the emotion estimation result 142 to the distribution control unit 77. The distribution control unit 77 distributes the emotion estimation result 142 to the user terminal 15 from which the emotion estimation request 140 is requested. The emotion estimation unit 141 estimates the emotion of the user U regarding the printed image 12 using, for example, an emotion estimation model that outputs the emotion estimation result 142 based on the input of the state information 136.

[0146] Although not shown in the figure, upon receiving the emotion estimation result 142 from the distribution control unit 77, the browser control unit 32 and Fig.28 Similarly to the illustrated case, in the emotion input menu 41 , the bubble message 108 is displayed above the face shape 43 corresponding to the emotion of the emotion estimation result 142 .

[0147] Thus, in the fifth embodiment, the emotion estimation unit 141 acquires the state information 136 of the user U from the smart watch 135 worn on the wrist of the user U, and estimates the emotion of the user U with respect to the printed image 12 based on the state information 136. The browser control unit 32 displays the emotion estimated by the emotion estimation unit 141 to the user U by displaying the bubble message 108. Therefore, the user U can be assisted in inputting an emotion that matches the state information 136.

[0148] The wearable device is not limited to the smart watch 135 exemplified. It may be a device attached to the head of the user U or a device built into clothes worn by the user U. Furthermore, in addition to or instead of the body temperature change data, breathing change data may be used as the status information 136.

[0149] [Sixth embodiment]

[0150] As an example, Fig.39 As shown, in this embodiment, Figure 4 When the storage instruction button 38 is pressed in the storage instruction screen 35 shown, the browser control unit 32 sends an emotion estimation request 145 to the image management server 17. The emotion estimation request 145 includes a user ID, the instant film 11 stored as a digital image, and the printed image 12. In addition, the printed image 12 shows an example of an image of a couple with smiling faces.

[0151] The CPU 22B of the image management server 17 of this embodiment includes the processing units 75 to 77 (in Fig.39 In addition to the RW control unit 76 (not shown), it also functions as an emotion estimation unit 146.

[0152] The receiving unit 75 receives the emotion estimation request 145 and outputs the emotion estimation request 145 to the emotion estimation unit 146. The emotion estimation unit 146 estimates the emotion of the user U with respect to the printed image 12 at the time of shooting based on the emotion estimation request 145. The emotion estimation unit 146 outputs the emotion estimation result 147, which is the result of estimating the emotion of the user U with respect to the printed image 12 at the time of shooting, to the distribution control unit 77. The distribution control unit 77 distributes the emotion estimation result 147 to the user terminal 15 that is the source of the request for the emotion estimation request 145.

[0153] As an example, Fig.40 As shown, the emotion estimation unit 146 includes a face extraction unit 150 and an expression detection unit 151. The printed image 12 of the emotion estimation request 145 is input to the face extraction unit 150. The face extraction unit 150 extracts the face of the person shown in the printed image 12 using a well-known face extraction technology. The face extraction unit 150 outputs a face extraction result 152, which is a result of extracting the face, to the expression detection unit 151.

[0154] The expression detection unit 151 detects the facial expression of the person in the face extraction result 152 using a known image recognition technology. The expression detection unit 151 estimates the emotion of the user U regarding the printed image 12 at the time of shooting based on the expression detection result, and outputs the emotion estimation result 147. The expression detection unit 151 estimates the emotion of the user U regarding the printed image 12 at the time of shooting, for example, using an emotion estimation model that outputs the emotion estimation result 147 based on the input of the expression detection result. Fig.40 2 shows an example in which the emotion of the user U regarding the printed image 12 at the time of shooting is estimated to be “happy” based on the face extraction result 152 of a smiling couple.

[0155] Although not shown in the figure, upon receiving the emotion estimation result 147 from the distribution control unit 77, the browser control unit 32 and Fig.28Similarly to the illustrated case, in the emotion input menu 41A, the bubble message 108 is displayed above the face shape 43 corresponding to the emotion of the emotion estimation result 147 .

[0156] Thus, in the sixth embodiment, the expression detection unit 151 of the emotion estimation unit 146 detects the expression of the person shown in the printed image 12, and estimates the emotion of the user U with respect to the printed image 12 based on the expression detection result. The browser control unit 32 displays the bubble message 108 to display the emotion estimated by the emotion estimation unit 146 to the user U. Therefore, the input of the emotion suitable for the expression of the person shown in the printed image 12 can be assisted.

[0157] The second to sixth embodiments described above may be implemented individually or in combination. For example, when the second, third, and fifth embodiments described above are implemented in combination, as an example, Fig.41 The emotion estimation model 155 shown in the figure can be used. The emotion estimation model 155 outputs the emotion estimation result 156 based on the input of the printed image 12, the text reading result 122 and the state information 136. The emotion estimation model 155 is an example of the "machine learning model" involved in the technology of the present invention. In this way, compared with the case where the second, third and fifth embodiments described above are implemented alone, more materials for estimating the emotions of the user U with respect to the printed image 12 are added, so that the estimation accuracy of the emotion estimation result 156 can be further improved.

[0158] In the second to sixth embodiments, the emotions estimated by the emotion estimation units 101, 116, 128, 141, and 146 may be stored as the emotion information 45. The stored emotion information 45 may be presented to the user U, and a correction instruction based on the emotion information 45 from the user U may be received.

[0159] In the above-mentioned embodiments, the user U is semi-forced to input emotions for the printed image 12 when photographing, storing, and publishing, but the present invention is not limited to this. As long as the input of emotions for the printed image 12 when photographing, storing, and publishing can be accepted, there is no need to force the user U to input emotions. In addition, there is no need to reserve a storage area for the emotion information 45 in the image DB 71 in advance. It is sufficient to reserve a storage area for the emotion information 45 in the image DB 71 each time the input of emotions is accepted.

[0160] In addition, the emotions are not limited to the exemplified "joy", "anger", "sadness" and "happiness". They may be "nostalgia", "love", "fear" or "happiness". Furthermore, the image is not limited to the printed image 12, but may be a digital image taken by a device having a camera function.

[0161] The time point is not limited to the exemplified time of shooting, storage and release, but may be a regular time point one year, two years, five years or ten years after shooting.

[0162] The image management server 17 may also assume all or part of the functions of the browser control unit 32 of the user terminal 15. Specifically, various screens such as the storage instruction screen 35 are generated in the image management server 17, and distributed and output to the user terminal 15 in the form of screen data for web page distribution created by a markup language such as XML (Extensible Markup Language). In this case, the browser control unit 32 of the user terminal 15 reproduces various screens displayed on the web browser based on the screen data, and displays them on the display 24A. In addition, other data description languages ​​such as JSON (Javascript (registered trademark) Object Notation) may be used instead of XML.

[0163] The hardware structure of the computer constituting the image management server 17 can be modified in various ways. For example, in order to improve processing capacity and reliability, the image management server 17 may be constituted by a plurality of computers separated as hardware. For example, the functions of the receiving unit 75 and the RW control unit 76 and the function of the distribution control unit 77 may be distributed to two computers. In this case, the image management server 17 is constituted by two computers. Furthermore, the user terminal 15 may assume all or part of the functions of the image management server 17.

[0164] In this way, the hardware structure of the computer of the user terminal 15 and the image management server 17 can be appropriately changed according to the required performance such as processing power, security, reliability, etc. In addition, not limited to hardware, APs such as the print image AP 30 and the work program 70 can of course be duplicated or distributed and stored in multiple storage devices for the purpose of ensuring security and reliability.

[0165] In the above-mentioned embodiments, for example, various processors shown below can be used as the hardware structure of the processing unit (Processing Unit) that executes various processes such as the browser control unit 32, the receiving unit 75, the RW control unit 76, the distribution control unit 77, the emotion estimation units 101, 116, 128, 141 and 146, the text reading unit 120, the natural language analysis units 121 and 131, the text acquisition unit 130, the face extraction unit 150 and the expression detection unit 151. Among the various processors, in addition to general-purpose processors, namely CPUs 22A and 22B, which execute software (print image AP 30 and the operation program 70) and function as various processing units, processors whose circuit structure can be changed after manufacturing, such as FPGAs (Field Programmable Gate Arrays), programmable logic devices (PLDs) and / or ASICs (Application Specific Integrated Circuits) have circuit structures specially designed to execute specific processes, namely dedicated circuits.

[0166] One processing unit may be composed of one of these various processors, or a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs and / or a combination of a CPU and an FPGA). Furthermore, multiple processing units may be composed of one processor.

[0167] As an example of a plurality of processing units constituted by one processor, there is the following method: as represented by computers such as clients and servers, one processor is constituted by a combination of one or more CPUs and software, and the processor functions as a plurality of processing units. Second, there is the following method: as represented by a system on chip (SoC), a processor is used to implement the functions of the entire system including a plurality of processing units through one IC (Integrated Circuit) chip. In this way, various processing units are constituted as a hardware structure using one or more of the above-mentioned various processors.

[0168] Furthermore, as the hardware configuration of these various processors, more specifically, a circuit (Circuitry) formed by combining circuit elements such as semiconductor elements can be used.

[0169] From the above description, the techniques described in the following supplementary notes can be understood.

[0170] [Supplementary Item 1]

[0171] An image processing device comprises a processor,

[0172] The processor performs the following processing:

[0173] Accepting user input of emotions regarding the image at multiple time points; and

[0174] The emotion information at the plurality of time points is associated with the image and stored.

[0175] [Supplementary Item 2]

[0176] The image processing device according to supplementary item 1, wherein:

[0177] The image is a printed image outputted to an instant film.

[0178] The multiple time points are a combination selected from a time point when the printed image is photographed, a time point when the instant film is stored as a digital image, and a time point when the digital image of the instant film is published to a social networking service.

[0179] [Supplementary Item 3]

[0180] The image processing device according to Supplementary item 1 or 2, wherein:

[0181] The processor applies a display effect corresponding to the emotional information when displaying the image.

[0182] [Supplementary Item 4]

[0183] An image processing device according to any one of supplementary items 1 to 3, wherein:

[0184] The image can be searched using the emotion information as a search keyword.

[0185] [Supplementary Item 5]

[0186] An image processing device according to any one of supplementary items 1 to 4, wherein:

[0187] The processor performs the following processing:

[0188] estimating the emotion using a machine learning model that outputs information about the emotion based on an input of the image; and

[0189] The estimated emotion is displayed to the user.

[0190] [Supplementary Item 6]

[0191] The image processing device according to supplementary item 5, wherein:

[0192] The associated and stored information of the emotion and the image are used as teacher data for the machine learning model.

[0193] [Supplementary Note 7]

[0194] An image processing device according to any one of supplementary items 1 to 6, wherein:

[0195] The image is a printed image outputted onto an instant film.

[0196] The plurality of time points include a time point at which the printed image is captured,

[0197] The processor performs the following processing:

[0198] Acquiring a digital image of the instant film;

[0199] Reading the text recorded by the user on the instant film by performing image analysis on the digital image;

[0200] performing a natural language analysis of the text;

[0201] estimating the emotion at the time when the printed image was captured based on the result of the natural language analysis; and

[0202] The estimated emotion is displayed to the user.

[0203] [Supplementary Item 8]

[0204] An image processing device according to any one of supplementary items 1 to 7, wherein:

[0205] The image is a printed image outputted onto an instant film.

[0206] The plurality of time points include a time point when the digital image of the instant film is posted to a social networking service,

[0207] The processor performs the following processing:

[0208] acquiring text input by the user when posting the digital image to a social networking service;

[0209] performing a natural language analysis of the text;

[0210] estimating the emotion at the time when the digital image is posted to a social networking service based on the result of the natural language analysis; and

[0211] The estimated emotion is displayed to the user.

[0212] [Supplementary Item 9]

[0213] An image processing device according to any one of supplementary items 1 to 8, wherein:

[0214] The processor performs the following processing:

[0215] Acquiring status information of the user from a wearable device worn by the user;

[0216] estimating the emotion based on the state information; and

[0217] The estimated emotion is displayed to the user.

[0218] [Supplementary Item 10]

[0219] An image processing device according to any one of supplementary items 1 to 9, wherein:

[0220] The processor performs the following processing:

[0221] Detecting the facial expressions of the persons shown in the image;

[0222] estimating the emotion according to the detection result of the expression; and

[0223] The estimated emotion is displayed to the user.

[0224] The technology of the present invention can also appropriately combine the above-mentioned various embodiments and / or various variations. Moreover, it is not limited to the above-mentioned embodiments, and various structures can be adopted as long as they do not deviate from the main purpose. In addition, the technology of the present invention also relates to a storage medium for non-temporary storage of programs in addition to the program.

[0225] The recorded contents and illustrated contents shown above are detailed descriptions of the parts involved in the technology of the present invention, and are only an example of the technology of the present invention. For example, the descriptions related to the above-mentioned structure, function, action and effect are descriptions related to an example of the structure, function, action and effect of the parts involved in the technology of the present invention. Therefore, it goes without saying that unnecessary parts can be deleted from the recorded contents and illustrated contents shown above, or new elements can be added or replaced without departing from the technical purpose of the present invention. In addition, in order to avoid complicated situations and facilitate understanding of the parts involved in the technology of the present invention, descriptions of technical common sense that does not need to be specifically explained on the basis of being able to implement the technology of the present invention are omitted in the recorded contents and illustrated contents shown above.

[0226] In this specification, "A and / or B" has the same meaning as "at least one of A and B". That is, "A and / or B" means that it can be only A, only B, or a combination of A and B. Furthermore, in this specification, when three or more items are combined and described using "and / or", the same idea as "A and / or B" also applies.

[0227] All documents, patent applications, and technical standards described in this specification are incorporated herein by reference to the same extent as if each document, patent application, or technical standard was specifically and individually described as being incorporated by reference.

Claims

1. An image processing device comprising a processor, The processor performs the following processing: Accepting user input of emotions regarding the image at multiple time points; and The emotion information at the plurality of time points is associated with the image and stored.

2. The image processing device according to claim 1, wherein: The image is a printed image outputted onto an instant film. The multiple time points are a combination selected from a time point when the printed image is photographed, a time point when the instant film is stored as a digital image, and a time point when the digital image of the instant film is published to a social networking service.

3. The image processing device according to claim 1, wherein: The processor applies a display effect corresponding to the emotional information when displaying the image.

4. The image processing device according to claim 1, wherein: The image can be searched using the emotion information as a search keyword.

5. The image processing device according to claim 1, wherein: The processor performs the following processing: estimating the emotion using a machine learning model that outputs information about the emotion based on an input of the image; and The estimated emotion is displayed to the user.

6. The image processing device according to claim 5, wherein: The associated and stored information of the emotion and the image are used as teacher data for the machine learning model.

7. The image processing device according to claim 1, wherein: The image is a printed image outputted onto an instant film. The plurality of time points include a time point at which the printed image is captured, The processor performs the following processing: Acquiring a digital image of the instant film; Reading the text actually recorded by the user on the instant film by performing image analysis on the digital image; performing a natural language analysis of the text; estimating the emotion at the time when the printed image was captured based on the result of the natural language analysis; and The estimated emotion is displayed to the user.

8. The image processing device according to claim 1, wherein: The image is a printed image outputted onto an instant film. The plurality of time points include a time point when the digital image of the instant film is posted to a social networking service, The processor performs the following processing: acquiring text input by the user when posting the digital image to a social networking service; performing a natural language analysis of the text; estimating the emotion at a time when the digital image is posted to a social networking service based on the result of the natural language analysis; and The estimated emotion is displayed to the user.

9. The image processing apparatus according to claim 1, wherein: The processor performs the following processing: Acquiring status information of the user from a wearable device worn by the user; estimating the emotion based on the state information; and The estimated emotion is displayed to the user.

10. The image processing apparatus according to claim 1, wherein: The processor performs the following processing: Detecting the facial expressions of the persons shown in the image; estimating the emotion according to the detection result of the expression; and The estimated emotion is displayed to the user.

11. A working method of an image processing device, comprising the following steps: Accepting user input of emotions about images at multiple time points; and The emotion information at the plurality of time points is associated with the image and stored.

12. A working program of an image processing device, the working program causing a computer to execute a process comprising the following steps: Accepting user input of emotions regarding the image at multiple time points; and The emotion information at the plurality of time points is associated with the image and stored.

Images