Image storage device, method and program

Abstract

The purpose of the present invention is to provide an image storage device, a method, and a non-transitory computer-readable medium that make it possible to extract characteristic video based on an individual's emotions. An image storage device (1) according to the present disclosure is provided with an image acquisition unit (11), a facial expression classification unit (22), and an image storage unit (210). The image acquisition unit (11) acquires image data. The facial expression classification unit (22) classifies face-image data included in the image data into prescribed emotions. The image storage unit (210) stores, so as to allow for delivery to a terminal, image data to which an emotion identifier for identifying the classified emotion is linked.

Description

[Technical field]

[0001] The present invention relates to an image storage device, method and non-transitory computer readable medium. [Background technology]

[0002] Patent Document 1 discloses a technology for recognizing at least one of an action and an expression of a specific individual based on the acquisition result of an image of the specific individual, recognizing a video scene characteristic of the individual based on the recognition result, and extracting the specific image from the acquisition result. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2019-125870 A Summary of the Invention [Problem to be solved by the invention]

[0004] In Patent Document 1, external features such as an individual's behavior and facial expressions are used to extract images that contain characteristic specific images. However, Patent Document 1 has a problem in that it cannot analyze emotions, which are the inner part of an individual, and therefore cannot extract characteristic video scenes based on an individual's emotions.

[0005] In view of such problems, the present disclosure aims to provide an image storage device, method, and non-transitory computer-readable medium capable of extracting characteristic video scenes based on individual emotions. [Means for solving the problem]

[0006] The image storage device of the present disclosure includes an image acquisition unit that acquires image data, an expression classification unit that classifies facial image data contained in the image data into predetermined emotions, and an image storage unit that stores the image data linked to an emotion identifier for identifying the classified emotion so that it can be delivered to a terminal.

[0007] An image storage device according to another aspect of the present disclosure includes an image acquisition means for acquiring image data, an audio acquisition means for acquiring audio data corresponding to the image data, an audio emotion classification means for classifying a person's emotion from the audio data, and an image storage means for storing the image data linked to an emotion identifier for identifying the classified emotion so as to be deliverable to a terminal.

[0008] The method disclosed herein includes acquiring image data, classifying facial image data contained in the image data into a predetermined emotion, and storing the image data associated with an emotion identifier for identifying the classified emotion so as to be deliverable to a terminal.

[0009] The program disclosed herein causes a computer to execute the following processes: acquiring image data; classifying facial image data contained in the image data into a predetermined emotion; and storing the image data, linked to an emotion identifier for identifying the classified emotion, so that the image data can be distributed to a terminal. Effect of the Invention

[0010] The present disclosure provides an image storage device, method, and non-transitory computer-readable medium capable of extracting characteristic video scenes based on personal emotions. [Brief description of the drawings]

[0011] [Figure 1] 1 is a block diagram showing a configuration of an image storage device according to a first embodiment. [Diagram 2] 4 is a flowchart showing the operation of the image storage device according to the first embodiment. [Diagram 3] FIG. 11 is a block diagram showing a configuration of an image storage system according to a second embodiment. [Figure 4] 10 is a flowchart showing the operation of an image storage system according to a second embodiment. [Diagram 5]10 is a flowchart showing the operation of an image storage system according to a second embodiment. [Figure 6] FIG. 11 is a schematic diagram showing an example of output of emotion icons in the image storage system according to the second embodiment. [Figure 7] FIG. 13 is a block diagram showing a configuration of an image storage system according to a third embodiment. [Figure 8] FIG. 13 is a schematic diagram showing a process of an interest level calculation unit in the image storage system according to the third embodiment. [Figure 9] FIG. 2 is a block diagram showing the configuration of a computer according to the present embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In each drawing, the same or corresponding elements are denoted by the same reference numerals, and duplicated descriptions will be omitted as necessary for clarity of explanation. Moreover, the "image" described in the embodiment includes a still image and a moving image.

[0013] (First embodiment) First, the configuration of an image storage device 1 according to the first embodiment will be described with reference to Fig. 1. The image storage device 1 includes an image acquisition unit 11, a facial expression classification unit 22, and an image storage unit 210.

[0014] The image acquisition unit 11 acquires image data. The facial expression classification unit 22 classifies face image data included in the image data into a predetermined emotion. The image storage unit 210 stores the image data associated with an emotion identifier for identifying the classified emotion so as to be deliverable to a terminal.

[0015] Next, the operation of the image storage device 1 according to the first embodiment will be described with reference to FIG. First, the image acquisition unit 11 acquires image data (step S101). Next, the facial expression classification unit 22 classifies the facial image data included in the image data into a predetermined emotion (step S102). Next, the image storage unit 210 stores the image data associated with an emotion identifier for identifying the classified emotion so as to be deliverable to a terminal (step S103).

[0016] Therefore, in the image storage device 1 according to the first embodiment, by analyzing emotions, which are an inner part of an individual, from external characteristics such as an individual's facial expression, it is possible to extract characteristic video scenes using the individual's emotions as a trigger.

[0017] Second embodiment Next, the configuration of an image storage system 200 according to a second embodiment will be described with reference to Fig. 3. The second embodiment specifically describes the first embodiment. Image storage system 200 comprises a camera (image acquisition unit) 11, a microphone (audio acquisition unit) 12, an image storage device 20, and a terminal 30. Image storage system 200 is installed in a kindergarten, for example, and can store video scenes in which children have characteristic emotions from among images taken of daily events at the kindergarten. Parents of the children can then acquire specific images they select from the stored specific images. Note that image storage system 200 is not limited to being installed in a kindergarten, and can be installed anywhere where children can enjoy watching their children.

[0018] Camera 11 is a fixed camera installed in a facility such as a kindergarten, and communicates wirelessly or wired with image storage device 20 via network N. Camera 11 captures images of the facility such as a kindergarten, and transmits the captured image data to image storage device 20. Here, the captured images are still images or videos.

[0019] The microphone 12 is a microphone installed in a facility such as a kindergarten, and communicates wirelessly or wired with the image storage device 20 via the network N. The microphone 12 captures audio from the facility such as a kindergarten, and transmits the captured audio data to the image storage device 20.

[0020] The image storage device 20 is a server that communicates wirelessly or wired with the camera 11, microphone 12, and terminal 30 via the network N. The image storage device 20 includes a face data extraction unit 21, a facial expression classification unit 22, a voice data extraction unit 23, a voice emotion classification unit 24, an emotion determination unit 25, an individual identification unit 26, a face recognition data storage unit 27, a voice recognition data storage unit 28, an individual data storage unit 29, an image storage unit 210, an image data storage unit 211, an image editing unit 212, an icon notification unit 213, and an image distribution unit 214.

[0021] The face data extraction unit 21 extracts face image data of a predetermined person from the image data, using the image data acquired from the camera 11. The face data extraction unit 21 supplies the extracted face image data to the facial expression classification unit 22.

[0022] The facial expression classification unit 22 analyzes the facial image data acquired from the facial data extraction unit 21, and generates emotion data including information classifying what emotion a person is feeling. The aforementioned emotions include, for example, joy, sadness, empathy, surprise, presence, attention, confusion, contempt, disgust, fear, and the like.

[0023] The voice data extraction unit 23 extracts individual voice data from the voice data acquired from the microphone 12. The voice data extraction unit 23 supplies the extracted individual voice data to the voice emotion classification unit 24.

[0024] The voice emotion classification unit 24 analyzes the individual's voice data acquired from the voice data extraction unit 23, and generates emotion data including information classifying what emotion the person is feeling. The above-mentioned emotions are similar to the emotions analyzed by the facial expression classification unit 22, and include, for example, joy, sadness, sympathy, surprise, presence, attention, confusion, contempt, disgust, and fear.

[0025] The emotion determination unit 25 uses emotion data acquired from the facial expression classification unit 22 or the voice emotion classification unit 24 to determine whether the emotion of the person classified by the facial expression classification unit 22 or the voice emotion classification unit 24 is a specific emotion. Here, the specific emotion is preset, such as "happiness" or "sadness." When the emotion determination unit 25 determines that the emotion of the person is a specific emotion, it generates a specific emotion identifier indicating the specific emotion, and supplies the specific emotion identification and the face image data or voice data to the individual identification unit 26.

[0026] The individual identification unit 26 refers to the face recognition data stored in the face recognition data storage unit 27 and the personal data stored in the personal data storage unit 29 to identify a person from the acquired face image data. If the individual identification unit 26 is able to identify a person, it acquires a personal identifier for identifying the person. Furthermore, the individual identification unit 26 refers to the voice recognition data stored in the voice recognition data storage unit 28 and the personal data stored in the personal data storage unit 29 to identify a person from the acquired personal voice data. If the individual identification unit 26 is able to identify a person, it acquires a personal identifier for identifying the person. Then, the individual identification unit 26 supplies the acquired personal identifier and specific emotion identifier to the image accumulation unit 210.

[0027] The image storage unit 210 extracts a specific image corresponding to a specific emotion from the image data acquired from the camera 11. Specifically, the image storage unit 210 extracts a specific image within a predetermined range before and after the point in time when the specific emotion is determined to be present from the image data acquired from the camera 11. For example, the specific image shows a video acquired from the camera 11 for 10 seconds before and after the point in time when the child shows the emotion of "joy". Then, the image storage unit 210 associates the extracted specific image with a personal identifier and a specific emotion identifier and stores them in the image data storage unit 211.

[0028] The image editing unit 212 edits the specific image stored in the image data storage unit 211. For example, the image editing unit 212 performs editing by masking, such as blurring the faces of persons other than the person corresponding to the linked personal identifier, in the specific image stored in the image data storage unit 211.

[0029] The icon notification unit 213 outputs an emotion icon to the terminal 30 via the network N. The emotion icon is an icon showing a specific emotion such as "happiness" or "sadness", and at least one type is generated. Here, the user of the terminal 30 can select which specific image showing which emotion to play by selecting an emotion icon output to the terminal 30. In addition, the icon notification unit 213 may link the difference between the time when the person had the specific emotion and the current time to the emotion icon and output it to the terminal 30.

[0030] The image delivery unit 214 receives an instruction from the terminal 30 via the network N, acquires a specific image corresponding to the emotion icon from the image data storage unit 211, and delivers the acquired specific image to the terminal 30.

[0031] Terminal 30 is, for example, a mobile terminal such as a smartphone or tablet, or a fixed terminal such as a PC (Personal Computer). Terminal 30 is provided with an emotion notification / image playback application 31 that enables a user of terminal 30 to receive images delivered from image storage device 20. Emotion notification / image playback application 31 outputs emotion icons received from icon notification unit 213 of image storage device 20 to a display, and transmits information on the emotion icon selected by the user to image storage device 20. Emotion notification / image playback application 31 also outputs a specific image received from image delivery unit 214 of image storage device 20 to a display.

[0032] Next, an example of the operation of the image storage system 200 according to the second embodiment will be described with reference to Figures 4 to 5. In Figures 4 to 5, an example in which the image storage system 200 is installed in a kindergarten will be described.

[0033] First, the camera 11 captures an image in the kindergarten and transmits the captured image data to the image storage device 20 (step S201). Next, the face data extraction unit 21 of the image storage device 20 extracts face image data of a person from the image data acquired by the camera 11 (step S202). Then, the face data extraction unit 21 supplies the extracted face image data to the facial expression classification unit 22.

[0034] Next, the facial expression classification unit 22 analyzes the facial image data acquired from the facial data extraction unit 21, and generates emotion data including information classifying what emotion the person is feeling (step S203). Here, the above-mentioned emotions are, for example, emotions such as joy, sadness, sympathy, surprise, presence, attention, confusion, contempt, disgust, and fear. Specifically, the facial expression classification unit 22 classifies what emotion the person is feeling by performing a predetermined image processing on the facial image data of the person. The predetermined image processing is, for example, extraction of feature points (or feature amounts), matching of the extracted feature points with reference data, convolution processing of image data and processing using teacher data learned by machine learning, processing using teacher data by deep learning, etc. However, the method by which the facial expression classification unit 22 classifies emotions is not limited to the above-mentioned processing.

[0035] Next, the emotion determination unit 25 uses the emotion data acquired from the facial expression classification unit 22 or the voice emotion classification unit 24 to determine whether the emotion of the person classified by the facial expression classification unit 22 is a specific emotion (step S204). Here, the specific emotion is set in advance. For example, it is assumed that the specific emotion is set as "joy". If the emotion of the person included in the emotion data is "joy", the emotion of the person is determined to be a specific emotion. On the other hand, if the emotion of the person included in the emotion data is "sadness", the emotion of the person is determined to be not a specific emotion. Note that multiple specific emotions such as "joy, sadness, surprise" may be set. Also, the specific emotion may be set by the administrator of the image storage system 200 or the user of the terminal 30.

[0036] If the emotion determination unit 25 determines that the emotion of the person classified by the facial expression classification unit 22 is a specific emotion (YES in step S204), it generates a specific emotion identifier for identifying the specific emotion. Then, the emotion determination unit 25 supplies the specific emotion identifier and face image data to the individual identification unit 26, and proceeds to step S205. On the other hand, if the emotion determination unit 25 determines that the emotion of the person classified by the facial expression classification unit 22 is not a specific emotion (NO in step S204), it returns to the processing of step S201 or step S206 described below.

[0037] Next, the individual identification unit 26 refers to the face recognition data stored in the face recognition data storage unit 27 and the personal data stored in the personal data storage unit 29 to identify a person from the acquired face image data (step S205). If the individual identification unit 26 can identify a person, it acquires a personal identifier for identifying the person. Then, the individual identification unit 26 supplies the acquired personal identifier and specific emotion identifier to the image accumulation unit 210.

[0038] Furthermore, the camera 11 captures images in the kindergarten and transmits the captured image data to the image storage device 20. At the same time, the microphone 12 captures voices in the kindergarten and transmits the captured voice data to the image storage device 20 (step S206).

[0039] Next, the voice data extraction unit 23 extracts voice data of a predetermined person from the voice data acquired from the microphone 12 (step S207). The voice data extraction unit 23 supplies the extracted voice data to the voice emotion classification unit 24.

[0040] Next, the voice emotion classification unit 24 analyzes the voice data acquired from the voice data extraction unit 23, and generates emotion data including information classifying what emotion the person is feeling (step S208). The above-mentioned emotions are similar to the emotions analyzed by the facial expression classification unit 22, such as joy, sadness, sympathy, surprise, presence, attention, confusion, contempt, disgust, and fear.

[0041] Next, the emotion determination unit 25 uses the emotion data acquired from the voice emotion classification unit 24 to determine whether the emotion of the person classified by the voice emotion classification unit 24 is a specific emotion (step S209). The determination method is the same as the method described in step S204. Then, when the emotion determination unit 25 determines that the emotion of the person classified by the voice emotion classification unit 24 is a specific emotion (step S209 YES), it generates a specific emotion identifier for identifying the specific emotion. Then, the emotion determination unit 25 supplies the specific emotion identifier and face image data to the individual identification unit 26 and proceeds to step S210. On the other hand, when the emotion determination unit 25 determines that the emotion of the person classified by the voice emotion classification unit 24 is not a specific emotion (step S209 NO), it returns to the process of step S201 or step S206 described later.

[0042] Next, the individual identification unit 26 refers to the voice recognition data stored in the voice recognition data storage unit 28 and the personal data stored in the personal data storage unit 29, and identifies the person from the acquired individual's voice data (step S210). If the individual identification unit 26 can identify the person, it acquires a personal identifier for identifying the person. Then, the individual identification unit 26 supplies the acquired personal identifier and specific emotion identifier to the image accumulation unit 210.

[0043] Next, the image accumulation unit 210 extracts a specific image corresponding to the specific emotion from the image data acquired from the camera 11 (step S211). For example, the image accumulation unit 210 extracts a specific image within a predetermined range before and after the point in time when the specific emotion is determined to be present from the image data acquired from the camera 11. For example, the specific image shows a video acquired from the camera 11 for 10 seconds before and after the point in time when the child shows the emotion of "joy". Then, the image accumulation unit 210 associates the extracted specific image with the personal identifier and the specific emotion identifier and accumulates them in the image data storage unit 211 (step S212).

[0044] Next, the image editing unit 212 edits the specific image stored in the image data storage unit 211 (step S213). For example, the image editing unit 212 analyzes the specific image stored in the image data storage unit 211, and performs masking such as blurring the faces of persons other than the person corresponding to the personal identifier linked to the specific image in the stored specific image. In addition to blurring, masking includes mosaic processing, transformation processing, mask processing, and processing of superimposing a predetermined icon image.

[0045] Furthermore, after the processing of step S205 or step S210, the icon notification unit 213 causes the emotion notification / image playback application 31 of the terminal 30 to output an emotion icon via the network N (step S214). The emotion icon is an icon indicating a specific emotion such as "happiness" or "sadness", and at least one type of emotion icon is generated. Here, the user of the terminal 30 can select which emotion a specific image with which to play back by selecting an emotion icon output to the terminal 30. In addition, the icon notification unit 213 may cause the terminal 30 to output the time when the specific emotion appeared, linked to the emotion icon.

[0046] For example, the emotion notification / image playback application 31 of the terminal 30 outputs emotion icons on the display as shown in Fig. 6. In Fig. 6, the emotion notification / image playback application 31 outputs an emotion icon I1 indicating "happiness", an emotion icon I2 indicating "sadness", and an emotion icon I3 indicating "surprise". The emotion icon I1 indicating "happiness", the emotion icon I2 indicating "sadness", and the emotion icon I3 indicating "surprise" are linked to information of "5 minutes ago", "15 minutes ago", and "60 minutes ago", respectively. For example, the information of "5 minutes ago" linked to the emotion icon I1 indicating "happiness" indicates that a person showed "happiness" 5 minutes ago from the current time.

[0047] Next, when an emotion icon of the terminal 30 is selected by the user of the terminal 30, the image delivery unit 214 acquires a specific image corresponding to the specific emotion identifier and personal identifier associated with the selected emotion icon from the image data storage unit 211. Then, the image delivery unit 214 transmits the acquired image data to the terminal 30, and causes the emotion notification / image playback application 31 of the terminal 30 to output the specific image (step S215). For example, as shown in FIG. 6, when the user of the terminal 30 selects the emotion icon I1 of "joy", the image delivery unit 214 causes the terminal 30 to output the specific image of "joy".

[0048] The image delivery unit 214 outputs to the terminal 30 a specific image corresponding to the specific emotion identifier and personal identifier linked to the selected emotion icon, but may also output the emotion of at least one person other than the person corresponding to the personal identifier that is included in the specific image. Here, the image accumulation unit 210 associates the extracted specific image, personal identifier, and specific emotion identifier with the personal identifiers and emotion-related information of other people, and accumulates them in the image data storage unit 211. Thus, parents can infer the cause of their child's emotion (such as joy) from the emotions of the people around them.

[0049] Therefore, in the image storage system 200 according to the second embodiment, by analyzing emotions, which are an inner part of an individual, from external characteristics such as an individual's facial expression, it is possible to extract characteristic video scenes using the individual's emotions as a trigger.

[0050] Furthermore, by using the image storage system 200, parents who use the terminal 30 can learn more about their children's kindergarten activities through video than they would have been able to learn through a communication book or an interview with a teacher, and store the information as data to share with the family. Meanwhile, kindergartens can improve trusting relationships with parents by providing videos of children's activities as they really are. Furthermore, kindergartens can evaluate educational content and teachers by understanding the emotions of their children. Furthermore, by using the image storage system 200, the fixed cameras can be used to extract video scenes that are characteristic of people, and therefore the fixed cameras can be effectively used for purposes other than surveillance.

[0051] (Third embodiment) The image storage system 300 is used in a manner different from that of the image storage system 200 in the following respects. The image storage system 300 acquires characteristic video scenes of students in an online lesson, such as a music class. In the following embodiments, an online lesson refers to a lesson held using multiple terminals connected to each other via a communication line so that they can communicate with each other.

[0052] The terminals connected to the online lesson are, for example, personal computers, smartphones, tablet terminals, mobile phones with cameras, etc. In the following example, in the online lesson, the "student" is taking the lesson using a terminal different from the "teacher."

[0053] In addition, the image storage system 300 outputs a report on the concentration and satisfaction of the online lesson and the understanding of the instruction content based on the student's facial expression and voice. Here, the report may be output in association with a specific characteristic image.

[0054] Next, the configuration of an image storage system 300 according to the third embodiment will be described with reference to Fig. 7. The image storage system 300 includes an interest level calculation unit 215 in addition to the configuration of the image storage system 200.

[0055] Camera 11 and microphone 12 are installed in a mobile terminal such as a smartphone or tablet used for online lessons, or a fixed terminal such as a PC. Camera 11 captures an image of a student in an online lesson. Microphone 12 acquires audio associated with the image of the student in an online lesson. Camera 11 may also capture an image of a teacher in an online lesson. Microphone 12 may also acquire audio associated with the image of the teacher in an online lesson.

[0056] The facial expression classification unit 22 according to the third embodiment has the following functions in addition to the functions of the facial expression classification unit 22 according to the second embodiment. The facial expression classification unit 22 classifies the emotions of a person from the face image data acquired from the face data extraction unit 21, and calculates the degree of the classified emotions of the person as a numerical value. For example, the facial expression classification unit 22 calculates the person's attention level, confusion level, contempt level, disgust level, fear level, happiness level, sympathy level, surprise level, and presence level as a numerical value from 0 to 100.

[0057] The voice emotion classifying unit 24 according to the third embodiment has the following functions in addition to the functions of the voice emotion classifying unit 24 according to the second embodiment. In addition to the functions of the voice emotion classifying unit 24 according to the second embodiment, the voice emotion classifying unit 24 classifies the emotions of a person from the individual voice data acquired from the voice data extracting unit 23, and calculates the degree of the classified person's emotion as a numerical value. For example, the voice emotion classifying unit 24 calculates the person's attention level, confusion level, contempt level, disgust, fear, happiness level, sympathy level, surprise level, and presence level as a numerical value from 0 to 100.

[0058] The interest level calculation unit 215 calculates the student's interest level in the lesson (concentration level, satisfaction level, understanding level, etc.) from the classification result of the facial expression classification unit 22 or the voice and emotion classification unit 24. Specifically, the interest level calculation unit 215 receives emotion data as an input data group, as shown in FIG. 8. When the interest level calculation unit 215 receives the above-mentioned input data group, it performs a preset process and generates an output data group using the input data group. The output data group indicates the interest level in the lesson of the user who uses the image storage system 300. The output data group indicates, for example, the concentration level, satisfaction level with the lesson, and understanding level of the teaching content. Note that the interest level indicated as the output data group may be the same as the interest level included in the input data group, or may be different. Similarly, the empathy level indicated as the output data group may be the same as the empathy level included in the input data group, or may be different. Here, the interest level calculation section 215 may calculate, for example, the student's emotions regarding the images during the lesson and the transition over time of the student's interest in the lesson.

[0059] The image accumulation unit 210 extracts, from the image data acquired from the camera 11, a specific image corresponding to a predetermined range from the point in time when the emotion determination unit 25 determines that the emotion of the person is a specific emotion, and stores the extracted specific image in the image data storage unit 211. The image accumulation unit 210 according to the third embodiment has the following functions in addition to the above-mentioned functions of the image accumulation unit 210 according to the second embodiment. The image accumulation unit 210 stores, in the image data storage unit 211, the analysis result of the interest degree calculation unit 215 corresponding to the specific image in association with the specific image.

[0060] When storing a specific image of a student in the image data storage unit 211, the image accumulation unit 210 may store a specific image of a teacher corresponding to the specific image of the student in the image data storage unit 211. When storing a specific image of a student in the image data storage unit 211, the image accumulation unit 210 may store in the image data storage unit 211 the teacher's emotions or the teacher's degree of interest in the lesson in association with the specific image of the student.

[0061] The image delivery unit 214 according to the third embodiment has the following functions in addition to the functions of the image delivery unit 214 according to the second embodiment. The image delivery unit 214 receives an instruction from the terminal 30 via the network N, acquires a specific image corresponding to an emotion icon from the image data storage unit 211, and delivers the acquired specific image to the terminal 30. At that time, the image delivery unit 214 causes the terminal 30 to output, for example, a time transition of the student's interest in the lesson in the specific image, using a graph on a dashboard or the like. The image delivery unit 214 also causes the terminal 30 to output the teacher's emotion in the specific image or the teacher's interest in the lesson, using a graph on a dashboard or the like.

[0062] It should be noted that image storage system 300 does not need to include individual identification unit 26 when only one person is included in the image captured by camera 11, such as in a private lesson, since there is no need to identify the individual.

[0063] Therefore, the image storage system 300 can extract characteristic video scenes by analyzing emotions, which are an inner part of an individual, from external features such as an individual's facial expression, and using the individual's emotions as a trigger. Therefore, a teacher or the classroom to which the teacher belongs can improve the relationship of trust with the student's parents by providing the teacher with images of the student's lessons.

[0064] Furthermore, the image storage system 300 provides the degree of interest of students in the lesson to the teacher, the classroom to which the teacher belongs, parents, students, etc. Therefore, the teacher or the classroom to which the teacher belongs can use the degree of interest of students in the lesson to review the content of instruction and to formulate future instructional policies. By checking the student's degree of interest in the lesson and the video during the lesson, the student's parents can understand what kind of instruction the teacher is giving to their child, and can confirm the child's emotional reaction, attitude, and compatibility with the teacher from the video.

[0065] <Example of hardware configuration> Each functional component of the above-mentioned image storage device 1, camera 11, microphone 12, image storage device 20, and terminal 30 (hereinafter referred to as each device) may be realized by hardware that realizes each functional component (e.g., hardwired electronic circuitry, etc.), or may be realized by a combination of hardware and software (e.g., a combination of an electronic circuitry and a program that controls it, etc.). Below, a further explanation will be given of the case where each functional component of each device is realized by a combination of hardware and software.

[0066] FIG. 9 is a block diagram illustrating a hardware configuration of a computer. Each device can be realized by a computer 500 having the hardware configuration shown in FIG. 9. The computer 500 is a portable computer such as a smartphone or a tablet terminal. Meanwhile, the computer 500 may be a portable computer or a stationary computer such as a PC. The computer 500 may be a dedicated computer designed to realize each device or may be a general-purpose computer. The computer 500 may also be a stationary computer such as a PC (Personal Computer).

[0067] For example, a desired function can be provided to the computer 500 by installing a predetermined application in the computer 500. For example, a system can be provided by installing an application that realizes each function of each device in the computer 500.

[0068] The computer 500 has a bus 502, a processor 504, a memory 506, a storage device 508, an input / output interface (I / F) 510, and a network interface (I / F) 512. The bus 502 is a data transmission path for the processor 504, the memory 506, the storage device 508, the input / output interface 510, and the network interface 512 to transmit and receive data to and from each other. However, the method of connecting the processor 504 and the like to each other is not limited to a bus connection.

[0069] The processor 504 is a processor such as a central processing unit (CPU), a graphics processing unit (GPU), or a field-programmable gate array (FPGA). The memory 506 is a main storage device realized using a random access memory (RAM) or the like. The storage device 508 is an auxiliary storage device realized using a hard disk, a solid state drive (SSD), a memory card, or a read only memory (ROM) or the like.

[0070] The input / output interface 510 is an interface for connecting the computer 500 to an input / output device. For example, the input / output interface 510 is connected to an input device such as a keyboard and an output device such as a display device.

[0071] The network interface 512 is an interface for connecting the computer 500 to a network. This network may be a LAN (Local Area Network) or a WAN (Wide Area Network).

[0072] Programs for implementing desired functions are stored in the storage device 508. The processor 504 reads the programs into the memory 506 and executes them to implement the various functional components of each device.

[0073] The present invention is not limited to the above-described embodiment, and can be modified as appropriate without departing from the spirit and scope of the present invention.

[0074] The above-mentioned program can be stored and supplied to a computer using various types of non-transitory computer-readable media. The non-transitory computer-readable media includes various types of tangible recording media. Examples of the non-transitory computer-readable media include magnetic recording media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical recording media (e.g., magneto-optical disks), CD-ROM (Read Only Memory), CD-R, CD-R / W, and semiconductor memory (e.g., mask ROM, PROM (Programmable ROM), EPROM (Erasable PROM), flash ROM, RAM (Random Access Memory)). The program may also be supplied to a computer by various types of temporary computer-readable media. Examples of the temporary computer-readable media include electric signals, optical signals, and electromagnetic waves. The temporary computer-readable media can supply the program to a computer via a wired communication path such as an electric wire and an optical fiber, or via a wireless communication path.

[0075] The present invention is not limited to the above-described embodiment, and can be modified as appropriate without departing from the spirit and scope of the present invention.

[0076] A part or all of the above-described embodiments can be described as, but is not limited to, the following supplementary notes. (Appendix 1) an image acquisition unit that acquires image data; a facial expression classification unit that classifies facial image data included in the image data into a predetermined emotion; an image storage unit that stores the image data associated with an emotion identifier for identifying the classified emotion so as to be deliverable to a terminal; Image storage device. (Appendix 2) An emotion determination unit that determines whether the classified emotion is a predetermined specific emotion, When the emotion determination unit determines that the classified emotion is the predetermined specific emotion, the image storage unit extracts a specific image corresponding to the specific emotion from the image data, and stores the specific image associated with a specific emotion identifier for identifying the specific emotion in a manner that allows it to be delivered to a terminal. 2. The image storage device of claim 1. (Appendix 3) When the emotion determination unit determines that the classified emotion is the predetermined specific emotion, the image storage unit extracts a specific image included within a predetermined time before and after the time when the classified emotion is determined to be the specific emotion from the image data, and stores the specific image associated with the specific emotion identifier so as to be deliverable to a terminal. 3. An image storage device as described in claim 2. (Appendix 4) The image distribution unit further includes an image distribution unit that distributes the image associated with the emotion identifier to the terminal. 4. An image storage device according to claim 2 or 3. (Appendix 5) an individual identification unit that identifies a person who is determined to have the specific emotion from the face image data; An image editing unit that edits the specific image stored in the image storage unit, the image storage unit stores identification information of the person determined to have the specific emotion in association with the specific image; The image editing unit masks at least one person other than the person determined to have the specific emotion in the specific image. 5. The image storage device of claim 4. (Appendix 6) an icon notification unit that causes a terminal to output an icon indicating the specific emotion of the person determined to have the specific emotion, The icon notification unit causes the terminal to output at least one type of the icon; The image delivery unit delivers the specific image corresponding to the icon selected by the user of the terminal to the terminal. 6. The image storage device of claim 5. (Appendix 7) the image storage unit further associates with the stored specific image an emotion of at least one person other than the person determined to have the specific emotion, and stores the emotion; The image delivery unit links the emotion of at least one person other than the person determined to have the specific emotion with the specific image and delivers the specific image to the terminal. 7. An image storage device according to claim 5 or 6. (Appendix 8) an interest level calculation unit that calculates a person's interest level in lessons based on the emotions classified by the facial expression classification unit; The image storage unit stores the image data in association with a degree of interest in the lesson, The image delivery unit links the image data with the degree of interest in the lesson and delivers the image data to the terminal. 8. An image storage device according to any one of claims 4 to 7. (Appendix 9) a voice acquisition unit that acquires voice data corresponding to the image data; A voice emotion classification unit that classifies the emotion of a person from the voice data. 9. An image storage device according to any one of claims 1 to 8. (Appendix 10) an image acquisition unit that acquires image data; a voice acquisition unit that acquires voice data corresponding to the image data; a voice emotion classification unit for classifying emotions of a person from the voice data; an image storage unit that stores the image data associated with an emotion identifier for identifying the classified emotion so as to be deliverable to a terminal; Image storage device. (Appendix 11) An emotion determination unit that determines whether the classified emotion is a predetermined specific emotion, When the emotion determination unit determines that the classified emotion is the predetermined specific emotion, the image storage unit extracts a specific image corresponding to the specific emotion from the image data, and stores the specific image associated with a specific emotion identifier for identifying the specific emotion in a manner that allows it to be delivered to a terminal. 11. The image storage device of claim 10. (Appendix 12) acquiring image data; classifying facial image data included in the image data into a predetermined emotion; and storing the image data associated with an emotion identifier for identifying the classified emotion in a manner that allows the image data to be delivered to a terminal. method. (Appendix 13) A process of acquiring image data; A process of classifying facial image data included in the image data into a predetermined emotion; and storing the image data associated with an emotion identifier for identifying the classified emotion so as to be deliverable to a terminal. program.

[0077] This application claims priority based on Japanese Patent Application No. 2021-029035, filed on February 25, 2021, the disclosure of which is incorporated herein in its entirety. [Explanation of symbols]

[0078] 1. Image storage device 11 Image acquisition unit (camera) 12 Voice acquisition unit (microphone) 20 Image storage device 21 Face data extraction section 22 Facial expression classification section 23 Audio data extraction unit 24 Voice emotion classification unit 25 Emotion Judgment Department 26 Personal Identification Unit 27 Face recognition data storage unit 28 Voice recognition data storage unit 29 Personal data storage unit 30 Terminals 31 Emotion notification / image playback app 200 Image Storage System 210 Image storage unit 211 Image data storage unit 212 Image Editing Department 213 Icon notification section 214 Image Distribution Department 215 Interest Level Calculation Unit 300 Image Storage System 500 Computers 502 Bus 504 Processor 506 Memory 508 Storage Devices 510 Input / Output Interface (I / F) 510 Input / Output Interface 512 Network Interface (I / F) 512 Network Interface N Network

Claims

1. An image acquisition means for acquiring image data; A facial expression classification means for classifying facial image data included in the image data into a predetermined emotion; an image storage means for storing the image data associated with an emotion identifier for identifying the classified emotion so as to be deliverable to a terminal; and an emotion determination means for determining whether the classified emotion is a predetermined specific emotion, When the emotion determination means determines that the classified emotion is the predetermined specific emotion, the image storage means extracts a specific image corresponding to the specific emotion from the image data, and stores the specific image associated with a specific emotion identifier for identifying the specific emotion so as to be deliverable to a terminal. Image storage device.

2. When the emotion determination means determines that the classified emotion is the predetermined specific emotion, the image storage means extracts a specific image included within a predetermined time before and after the time when the classified emotion is determined to be the specific emotion from the image data, and stores the specific image linked to the specific emotion identifier so as to be deliverable to a terminal.

2. The image storage device according to claim 1.

3. The image distribution device further includes an image distribution unit that distributes the image associated with the emotion identifier to the terminal.

3. The image storage device according to claim 1 or 2.

4. an individual identification means for identifying a person who is determined to have the specific emotion from the face image data; an image editing means for editing the specific image stored in the image storage means, the image storage means stores identification information of the person determined to have the specific emotion in association with the specific image; The image editing means masks at least one person other than the person determined to have the specific emotion in the specific image.

4. The image storage device according to claim 3.

5. an icon notifying unit that outputs, to a terminal, an icon indicating the specific emotion of the person who has been determined to have the specific emotion; The icon notification means causes the terminal to output at least one type of the icon; The image delivery means delivers to the terminal the specific image corresponding to the icon selected by the user of the terminal.

5. The image storage device according to claim 4.

6. An image acquisition means for acquiring image data; a voice acquisition means for acquiring voice data corresponding to the image data; a voice emotion classification means for classifying an emotion of a person from the voice data; an image storage means for storing the image data associated with an emotion identifier for identifying the classified emotion so as to be deliverable to a terminal; and an emotion determination means for determining whether the classified emotion is a predetermined specific emotion, When the emotion determination means determines that the classified emotion is the predetermined specific emotion, the image storage means extracts a specific image corresponding to the specific emotion from the image data, and stores the specific image associated with a specific emotion identifier for identifying the specific emotion so as to be deliverable to a terminal. Image storage device.

7. Obtaining image data; classifying facial image data included in the image data into a predetermined emotion; and storing the image data associated with an emotion identifier for identifying the classified emotion in a manner that allows the image data to be delivered to a terminal; determining whether the classified emotion is a predetermined specific emotion; and when it is determined that the classified emotion is the predetermined specific emotion, extracting a specific image corresponding to the specific emotion from the image data, and storing the specific image associated with a specific emotion identifier for identifying the specific emotion in a manner that allows it to be distributed to a terminal. method.

8. A process of acquiring image data; A process of classifying facial image data included in the image data into a predetermined emotion; and storing the image data associated with an emotion identifier for identifying the classified emotion in a manner that allows the image data to be distributed to a terminal. A process of determining whether the classified emotion is a predetermined specific emotion; and further causing the computer to execute a process of, when it is determined that the classified emotion is the predetermined specific emotion, extracting a specific image corresponding to the specific emotion from the image data, and storing the specific image associated with a specific emotion identifier for identifying the specific emotion so as to be deliverable to a terminal. program.

9. Acquiring image data; acquiring audio data corresponding to the image data; classifying a person's emotion from the voice data; and storing the image data associated with an emotion identifier for identifying the classified emotion in a manner that allows the image data to be delivered to a terminal; determining whether the classified emotion is a predetermined specific emotion; and when it is determined that the classified emotion is the predetermined specific emotion, extracting a specific image corresponding to the specific emotion from the image data, and storing the specific image associated with a specific emotion identifier for identifying the specific emotion in a manner that allows it to be distributed to a terminal. method.

10. A process for acquiring image data; acquiring audio data corresponding to the image data; A process of classifying a person's emotion from the voice data; and storing the image data associated with an emotion identifier for identifying the classified emotion in a manner that allows the image data to be distributed to a terminal. A process of determining whether the classified emotion is a predetermined specific emotion; and further causing the computer to execute a process of, when it is determined that the classified emotion is the predetermined specific emotion, extracting a specific image corresponding to the specific emotion from the image data, and storing the specific image associated with a specific emotion identifier for identifying the specific emotion so as to be deliverable to a terminal. program.

Images