Imaging apparatus system, and server device

The imaging device system and server device address the lack of emotion processing in moving images by using audio and biometric data to determine and apply image effects, effectively expressing user emotions in video content.

JP2024165535AInactive Publication Date: 2024-11-28PANASONIC INTELLECTUAL PROPERTY MANAGEMENT CO LTD
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
JP2023081812
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-05-17
Publication Date
2024-11-28
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

Existing technologies, such as those described in Patent Document 1, primarily focus on estimating and reflecting the emotions of subjects in still images and lack clear methods for processing moving images, failing to express the emotions of a user experiencing an event.

Method used

An imaging device system and server device that utilize an imaging sensor, interface unit, emotion determination unit, and image processing unit to acquire and process audio and biometric data during video capture, determining the user's emotions and applying image effects based on these data to express the user's feelings.

Benefits of technology

The system effectively processes video data to reflect the user's emotions experienced during an event, enhancing the emotional expression and realism of the captured footage.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024165535000001_ABST
    Figure 2024165535000001_ABST
Patent Text Reader

Abstract

To allow a user being a photographer to process dynamic images according to emotions obtained through an experience in an event, thereby expressing the user's emotions.SOLUTION: An imaging apparatus system (1) has: an imaging sensor (114) that generates dynamic image data through photographing; and an interface unit (106) that acquires data. The interface unit acquires at least one of voice data acquired during the photographing and biological data that is a user's biological information acquired during the photographing. The imaging apparatus system further has: a recording unit (202) that records a data set in which at least one data of the voice data and the biological data is associated with the dynamic image data; an emotion determination unit (200) that determines, by using the at least one data, emotions held by the user in relation to an event occurring during the photographing; and an image processing unit (204) that executes processing according to the determined emotions on the dynamic image data.SELECTED DRAWING: Figure 5
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present disclosure relates to an imaging device system and a server device. [Background technology]

[0002] Patent Document 1 discloses a technology that calculates an emotion quotient for each subject and processes the image to be closer to the impression of the photographer. The information processing device described in Patent Document 1 records data related to the photographer's attention level and emotion for each subject shown in the image data together with the captured image data, and performs predetermined image processing based on the recorded data. For example, it becomes possible to reproduce a sense of realism by enlarging and reproducing a specific subject so as to be closer to the way it appears to the photographer's naked eye, i.e., the impression. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2019-047234 A Summary of the Invention [Problem to be solved by the invention]

[0004] Patent Document 1 describes a technology that estimates emotions for each "subject" and reflects those emotions in an image. The description in Patent Document 1 is mostly limited to explanations of still images, and although there is a reference to videos, it is not clear how they are processed. An object of the present disclosure is to express the emotions of a user who is a photographer by processing a video in accordance with the emotions the user felt after experiencing an event. [Means for solving the problem]

[0005] An imaging device system according to an aspect of the present disclosure includes an imaging sensor that generates video data by capturing images, an interface unit that acquires data, a recording unit, an emotion determination unit, and an image processing unit. The interface unit acquires at least one of voice data acquired during capturing images and biometric data that is biometric information of a user acquired during capturing images. The recording unit records a data set in which at least one of the voice data and the biometric data is associated with the video data. The emotion determination unit uses at least one of the video data, the voice data, and the biometric data to determine the emotion felt by the user regarding an event that occurred at the time of capturing images. The image processing unit performs processing on the video data according to the determined emotion.

[0006] A server device according to an embodiment of the present disclosure is used in an imaging device system having an imaging device. The imaging device has an imaging sensor that generates video data by shooting, a microphone that generates audio data, and an interface unit that acquires biometric data. The imaging device further has a transmission circuit that transmits the video data and at least one of the audio data and the biometric data. The server device has a communication circuit that communicates with the imaging device, a recording unit, an emotion determination unit, and a data generation unit. The recording unit records a data set in which at least one of the audio data and the biometric data is associated with the video data received by the communication circuit. The emotion determination unit uses at least one of the audio data and the biometric data to determine the emotion felt by the user regarding an event that occurred at the time of shooting. The data generation unit performs processing on the video data according to the determined emotion, or analyzes an event that indicates the cause of the user's emotion to generate factor analysis data or event analysis data. Effect of the Invention

[0007] According to the present disclosure, the user who filmed the event can process the video in accordance with the emotions he or she felt after experiencing the event, thereby expressing the user's emotions. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a block diagram showing a configuration of an imaging device system according to a first embodiment of the present disclosure. [Diagram 2] A block diagram showing the internal configuration of the smart glasses and the server device. [Diagram 3] A diagram showing an example of the appearance of smart glasses [Figure 4] A diagram showing smart glasses worn by a user [Diagram 5] FIG. 1 is a diagram showing a procedure of processing performed in an imaging device system. [Figure 6] FIG. 1 shows an example of a service that can be provided using an imaging device system. [Figure 7A] FIG. 13 shows an example of image effects added to video data according to determined emotions. [Figure 7B] FIG. 13 shows an example of image effects added to video data according to determined emotions. [Figure 7C] FIG. 13 shows an example of image effects added to video data according to determined emotions. [Figure 7D] FIG. 13 shows an example of image effects added to video data according to determined emotions. [Figure 7E] FIG. 13 shows an example of image effects added to video data according to determined emotions. [Figure 8] Example of a graph showing the frequency of positive events that improved QoL by month [Figure 9] An example of a display of items of interest to a user, extracted using the user's video data, audio data, movement data, biometric data, and gaze data. [Figure 10] Example of event analysis data provided by the monitoring service [Figure 11] FIG. 1 is a diagram showing a procedure in which a selected service is provided by selecting a service menu from a user terminal. [Figure 12] FIG. 1 shows an example of a factor or processing method used to execute video processing according to the emotion of a user who is a video shooter. [Figure 13] A diagram showing how to determine each individual emotion index and examples of its values. [Figure 14] FIG. 1 is a diagram for explaining an example of a process for enhancing a moving image. [Figure 15] A diagram for explaining how to determine the emotion of enjoying dancing and, based on the emotion, display an avatar that shows the state of enjoying dancing in video data. [Figure 16] FIG. 1 shows an example of a CPU having a switcher and an emotion determination unit. [Figure 17] A diagram to explain how, when eating a meal in a live broadcast, the ingredients look delicious and the emotion of deliciousness is judged and then processed into the video. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, the embodiments will be described with reference to the drawings as appropriate. However, more detailed explanation than necessary may be omitted. For example, detailed explanation of already well-known matters or duplicate explanation of substantially the same configuration may be omitted. This is to avoid the following explanation becoming unnecessarily redundant and to facilitate understanding by those skilled in the art.

[0010] The inventor(s) provide the accompanying drawings and the following description so that those skilled in the art can fully understand the present disclosure, and do not intend for them to limit the subject matter described in the claims.

[0011] 1. Embodiment 1 1-1. Example of imaging device system configuration 1 is a block diagram showing a configuration of an imaging device system 1 according to a first embodiment of the present disclosure. The imaging device system 1 includes, for example, smart glasses 10, which are imaging devices, and a server device 20. The smart glasses 10 and the server device 20 can communicate with each other via a telecommunications line 30, for example, the Internet. The imaging device system 1 may further include a smartphone 40a and a PC 40b.

[0012] Fig. 2 is a block diagram showing the internal configuration of the smart glasses 10 and the server device 20. As shown in Fig. 2, the smart glasses 10 have a CPU 100, a memory 102, a communication circuit 104, and an interface device 106. The smart glasses 10 also have various sensors 110, 114, 118, 122, and 126, and circuit groups 112, 116, 120, 124, and 128 that process sensor signals from each sensor. Inside the smart glasses 10, each component can exchange data via a communication bus.

[0013] The gaze sensor 110 detects the gaze of a user wearing the smart glasses 10. The gaze sensor 110 has an infrared light source (infrared LED) 112a, an infrared camera 112b, and a gaze calculation circuit 112c. The infrared light source 112a irradiates the eyeball with infrared light, and the infrared camera 112b captures an image of the eyeball. Then, using the captured image, the gaze calculation circuit 112c detects the position of the corneal reflection image of the light source on the pupil and the corneal surface. The gaze calculation circuit 112c pre-stores the relationship between the position of the corneal reflection image and the viewpoint coordinates on the image captured by the infrared camera 112b. The gaze calculation circuit 112c measures which part of the captured image the user was looking at based on the positional relationship with the corneal reflection image as the reference point and the pupil as the moving point. In this embodiment, the intersection position between the gaze and the image is called gaze data S. In FIG. 1, the gaze of the user in the landscape is shown diagrammatically as gaze data S.

[0014] The imaging sensor 114 is an image sensor. A known CMOS image sensor can be used as the imaging sensor 114, so a detailed description of the imaging sensor 114 will be omitted. A charge signal is output from the imaging sensor 114. The image processing circuit 116 generates still image data from the charge signal. Video data is obtained by continuously generating still image data. The smart glasses 10 of this embodiment have an out-camera 114a and an in-camera 114b. Although an imaging sensor may be provided in each of the out-camera 114a and the in-camera 114b, for convenience in this specification, the imaging sensor of the out-camera 114a and the imaging sensor of the in-camera 114b may be referred to as the imaging sensor 114a and the imaging sensor 114b.

[0015] The smart glasses 10 of this embodiment include an outer camera for capturing images of the outside world and an inner camera for capturing images of the face of a user wearing the smart glasses 10. The outer camera generates video data V, and the inner camera generates face data F capturing an image of the user's face (FIG. 1).

[0016] The microphone 118 is a sound sensor that converts sounds propagating in the surrounding space into electrical signals. The sound processing circuit 120 extracts only the sounds uttered by the user based on, for example, the input volume level, and outputs them as sound data A (FIG. 1).

[0017] The motion sensor 122 is, for example, a so-called nine-axis sensor in which an inertial sensor including a three-axis acceleration sensor and a three-axis gyro sensor, and a three-axis geomagnetic sensor are housed in one housing. The motion data extraction circuit 124 generates motion data detected by the motion sensor 122 from an output signal from the motion sensor 122. A plurality of motion sensors 122 may be attached to the user's wrists, both hands, feet, head, etc. Alternatively, if a motion capture sensor having a band type, clothing type, or the like that is easy to wear on the body is used as the motion sensor 122, motion data indicating the motion of the body can be easily obtained with high accuracy. In these cases, the motion data extraction circuit 124 can also generate motion data using output signals from each motion sensor 122 acquired using wireless communication using Bluetooth (registered trademark) between the smart glasses 10 and each motion sensor 122. If this type of motion sensor 122 is used, the motion of the user's entire body can be obtained with high accuracy.

[0018] The biosensor 126 is a general term for sensors that collect biosignals of a user. In this embodiment, the biosensor 126 is a sensor that detects at least one of body temperature, blood pressure, heart rate, and brain waves, for example. When the biosensor 126 detects body temperature, the biosensor 126 is called a thermometer, when it detects blood pressure, when it detects heart rate, it is called a heart rate monitor, and when it detects brain waves, it is called an electroencephalograph. Various biosignals are obtained from the biosensor 126. The biodata extraction circuit 128 extracts necessary data from among them. For example, when the biosensor 126 is an electroencephalograph, the circuit 128 outputs the obtained brainwave signal itself, a specific frequency component of a pre-specified brainwave, or a peak value after a certain time has elapsed, for example, 300 ms, from the input of a stimulus as biodata E (FIG. 1). In the same manner, when the biosensor 126 is the above-mentioned thermometer, blood pressure monitor, or heart rate monitor, the circuit 128 extracts necessary data as biodata E (FIG. 1).

[0019] The server device 20 also includes a CPU 200 , a storage unit 202 , an image processing circuit 204 , and a communication circuit 206 .

[0020] The server device 20 receives video data, audio data, movement data, biometric data, and gaze data from the smart glasses 10 via the communication circuit 206. The CPU 200 associates them with each other based on the same acquisition time and stores them in the storage unit 202 as a dataset. The storage unit 202 is a general term for a primary storage device such as a random access memory (RAM), a hard disk drive (HDD), a solid state drive (SSD), and other secondary storage devices. In this embodiment, the storage unit 202 stores an event prediction model M in advance. The event prediction model M is read into the CPU 200 and used, for example. A specific description of the event prediction model M will be given later. In this embodiment, the video data, audio data, biometric data, and gaze data are transmitted and received between the smart glasses 10 and the server device 20, and a dataset including these data is stored in the storage unit 202. However, it is not necessary that all of these data are always transmitted and received to configure the dataset. Ultimately, it is sufficient that at least one of audio data and biometric data is associated with the video data as a dataset.

[0021] Fig. 3 shows an example of the appearance of the smart glasses 10. Fig. 3 shows the arrangement of the gaze sensor 110, the out-camera 114a and in-camera 114b which are image sensors, the microphone 118, the electroencephalograph 126a and the thermometer 126b which are biosensors, which are provided in the smart glasses 10. The in-camera 114b has, for example, a wide-angle lens, an ultra-wide-angle lens, or a fisheye lens, and can capture at least a part of the user's face, and preferably the entire face.

[0022] FIG. 4 shows the smart glasses 10 worn by a user. The gaze sensor 110 is arranged so that it can irradiate infrared rays onto the user's eyeballs and capture an image. The electroencephalograph 126a and the thermometer 126b are arranged at positions where the user's brain waves and body temperature can be measured, respectively. Depending on the brain waves to be acquired, it is preferable to provide the electrodes of the electroencephalograph 126a at positions where the brain waves can be easily measured.

[0023] In this embodiment, various sensors are mounted on the smart glasses 10, but this is just an example, and it is not necessary to provide all the sensors and circuits for processing the sensor signals within one housing. A configuration may be adopted in which some or all of the various sensors and circuits are provided outside the housing, and data from the outside is received via the interface 106.

[0024] 1-2. Operation of the imaging device system 1 and services that can be provided using the imaging device system 1 1-2-1. Overview of operation of imaging device system 1 The operation of the imaging device system 1 will be described below with reference to Fig. 5. For ease of understanding, an example will be given.

[0025] Fig. 5 shows the procedure of processing performed in the imaging device system 1. The procedure is shown in the form of a flowchart for ease of understanding. Please note that this flowchart is not executed by a specific arithmetic circuit such as a CPU, but is a process performed in cooperation with the smart glasses 10 and the server device 20 constituting the imaging device system 1. Steps S1 to S3 in Fig. 5 are processes performed by the smart glasses 10, and steps S4 to S6 are processes performed by the server device 20.

[0026] In step S1, the CPU 100 of the smart glasses 10 acquires video data generated by the imaging sensor 114. In step S2, the CPU 100 acquires audio data, movement data, gaze data, and biometric data generated by the microphone 118, the movement sensor 122, the biometric sensor 126, and associated circuits, respectively.

[0027] In step S3, the CPU 100 transmits the acquired video data, audio data, movement data, biological data, and gaze data to the server device 20 via the communication circuit 104 and the electric communication line 30.

[0028] In step S4, the CPU 200 of the server device 20 receives the video data, audio data, biometric data, and gaze data through the communication circuit 206, and records them in the storage unit 202 as a data set in which they are associated with each other. In step S5, the CPU 200 uses at least one of the audio data, biometric data, and gaze data to determine the user's feelings regarding the event that occurred at the time of shooting. In this specification, the term "event" refers to an occurrence or user experience that appeals to at least one of the user's five senses, i.e., sight, hearing, taste, smell, and touch. The user's experience includes both an experience caused by the user's active behavior and a passive experience that is not caused by the user's behavior.

[0029] In step S6, the CPU 100 instructs the image processing circuit 204 to process the moving image data in accordance with the determined emotion.

[0030] Steps S5 and S6 will now be described with a concrete example.

[0031] The CPU 200 of the server device 20 functions as an "emotion determination unit" described later. The emotion determination unit predicts an event experienced by the user using various data received from the smart glasses 10 and an event prediction model M constructed in advance by machine learning, for example, and further determines the emotion felt by the user as a result of experiencing the event. The CPU 200 decides to add text or an image indicating the impression and / or feelings of the user who experienced the event, or to add an image of the user's face. The image processing circuit 204 adds an image effect according to the determined emotion to the video data.

[0032] For example, when a user is experiencing an event called "making soba noodles," the CPU 200 estimates that the event is making soba noodles from the acquired gaze data and movement data and the event prediction model M. Then, based on the gaze and movement of spreading the dough, folding it, and slicing it into strips with a knife, the CPU 200 determines the user's impression and / or feelings, or more directly, the "cry of the heart," that is, "this is fun."

[0033] The image processing circuit 204 of the server device 20 processes the video data according to the determined emotion. In the above-mentioned example of making soba noodles, the image processing circuit 204 adds characters expressing the user's emotion, such as "It's fun," to the video data for a certain playback time, thereby expressing the user's inner cry as characters.

[0034] 1-2-2. Services that can be provided using the imaging device system 1 FIG. 6 shows an example of a service that can be provided to a user using the imaging device system 1. In FIG.

[0035] The menu of services provided is shown in the leftmost column. In this specification, (1) "a. video processing service" and "b. video distribution service" may be provided as video-related services. (2) services to improve quality of life (QoL), (3) marketing services, and (4) monitoring services for the elderly, etc. may also be provided.

[0036] In FIG. 6, data required when providing each of the above services (1) to (4) to a user using the imaging device system 1 is shown as "analysis data". The analysis data is data sent from the user's smart glasses 10 to the server device 20. The main data is as exemplified as "analysis source data", and data used incidentally is shown as "other data". And, in the rightmost column, data provided to the user as each of the above services (1) to (4) is shown as "delivery data".

[0037] (1) Video-related services: A service for processing videos or still images is provided. For example, the estimated emotion, viewpoint data, and audio data are used to process video to reproduce the emotion of the subject in response to the sound. The "original data for analysis" in this service is the data for video, audio, biometric data, viewpoint data, movement data, and the user's own video. The video is the first video data acquired by the out-camera, and the first video data is essential for processing in this service. The user may also wish to combine the user's own still image with the video in this service, and the still image is extracted using the second video data acquired by the in-camera. Note that audio, biometric data, viewpoint data, movement data, and the user's own video are not all essential, and at least one data may be used. Among the "other data," the analysis section refers to a specific section of the video data to be processed. By specifying a certain time section, an image effect showing the user's "cry from the heart" at an event occurring in that section is added. The "delivery data" to the user is a processed video to which, for example, text expressing the user's emotion, such as "It's fun~" is added. It may also be a still image. When a user is experiencing an event that exposes the user to cityscapes, mountain scenery, etc., the CPU 200 of the server device 20 can process the video data to recreate the excitement of natural sounds, particularly by using viewpoints, emotions, and sounds. The processed video data is sent to an email address or the like specified as a "delivery address."

[0038] (2) QoL Improvement Services: The QoL improvement service provides event analysis data. Event analysis data includes emotion classification and its causes, events and objects of interest, and statistical data. If a specific period of the video is specified, analyzed data for that period will be provided. In addition, QOL improvement advice is provided using the analysis results of subjects with high QOL who are similar in age, gender, hobbies, and thoughts.

[0039] For example, event analysis data can determine factors that lead to improved QoL. Factors that improve QoL include being moved by an experience or event. When a user is moved by nature, such as a landscape, sunset, starry sky, or babbling brook, the user's gaze and / or image can be detected. In addition, by acquiring the user's face data and voice data using the in-camera and microphone, it is possible to determine from the user's facial muscles and voice that the user was happy or laughed while watching a video, eating, watching a sport, etc. By counting and displaying these factors that improve QoL, the user's QoL and its changes can be quantified. Figure 8 shows an example of a graph showing the frequency of positive events that improved QoL by month.

[0040] On the other hand, by using event analysis data, it is also possible to determine factors that reduce QoL. From video data, audio data, and biometric data such as heart rate, it is possible to detect when a user was angry, sad, or crying. These are negative factors that lead to sadness and anger and depression. By counting and displaying these factors that reduce QoL, it is also possible to quantify the user's QoL and changes in it.

[0041] Following the example of Figure 8, the frequency of positive events that lowered QoL can be graphed by month. Alternatively, the positive factors, negative factors, and the total of positive and negative factors per month can be calculated. By showing the average value of the user's age, for example, with a dashed line, the relative level of the user's QoL can be known.

[0042] Since it is possible to analyze trends regarding what events improved or decreased a user's QoL, it is also possible to provide advice on how to use the analysis results to improve QoL.

[0043] As with the above (1) video-related service, the "original data for analysis" in this service is data on video, audio, biometrics, viewpoint, movement, and the user's own video. However, not all of these are required. For example, at least one of audio and biometrics may be required for video and / or the user's own video data.

[0044] (3) Marketing services: Event analysis data is provided by marketing services such as sales promotion data and personal ads. Event analysis data includes emotion classifications and their causes, as well as subjects of interest. It may also include statistical data. When a specific period of a video is specified, data analyzed for that period is provided. For example, clients who receive the services (1) and / or (2) above can receive benefits by consenting to secondary use of their data.

[0045] Figure 9 shows an example of a display of items that a user is interested in, extracted from the user's video data, audio data, movement data, biometric data, and gaze data. Using this information, it is possible to display advertisements targeted at the user or to direct them to a purchasing site.

[0046] (4) Monitoring service (action history): Event analysis data is provided, including emotion classifications and their drivers, and objects of interest, and may also include statistical data. If a specific period of the video is specified, analysis data for that period is provided.

[0047] Suppose a request is received from a family member of a certain user to keep an eye on the user. In this case, event analysis data including the number of steps the user took, the number of times the user was moved, the number of times the user was depressed, etc., calculated using the acceleration sensor 122, is presented to the family member of the user. Fig. 10 shows an example of the display of event analysis data provided by the watching service. By displaying such case analysis data, the family member of the user can grasp the state of the user even from a remote location, for example.

[0048] Users can choose from any of the four types of services mentioned above. FIG. 11 shows a procedure in which a selected service is provided by selecting a service menu from a user terminal. The user terminal is assumed to be smart glasses 10, but may be a terminal device owned by the user separately from the smart glasses 10. Each user terminal has an input interface device (not shown) that accepts a user's operation for selecting a service menu. For example, the smart glasses 10 are provided with a function for projecting a service menu. Furthermore, the smart glasses 10 may be provided with an input interface device that selects one of a plurality of service menus by touching and sliding back and forth on the temples of the glasses, and determines the selected service menu by tapping the temples.

[0049] In step S10, the CPU 100 of the smart glasses 10 accepts a selection from the user of one of the four service menus described above in the smart glasses 10. In step S11, the CPU 100 extracts analysis raw data according to the analysis raw data designated in advance by the user in the criteria shown in Fig. 6 and transmits the extracted analysis raw data to the server device 20. In other words, the user does not need to transmit all of the analysis raw data to the server device 20.

[0050] In the next step S12, the CPU 200 of the server device 20 uses the received analysis source data in accordance with the selected service menu to perform image processing or analysis according to the selected service menu to generate cause analysis data or event analysis data. Then, in step S13, the CPU 200 transmits the image-processed video data or cause analysis data as delivery data. The destination is designated in advance by the user as shown in FIG. 6. In the example of FIG. 11, the data can be confirmed by the smartphone 40a or PC 40b.

[0051] 1-3. Details of Operation of Imaging Device System 1 Next, the operation of the imaging device system 1 will be described in detail.

[0052] In the following, the details of the (1) video-related service will be explained, but for (2) to (4), in addition to the configuration of (1), it is sufficient to use a program on the smart glasses 10 side and a program on the server device 20 that perform the processing described in (2) to (4).

[0053] Figure 12 shows various examples of possible attention- and emotion-evoking objects, as well as the events, emotions, and video processing that are detected in relation to each object. Note that these are just examples.

[0054] First, when a user is photographing a landscape or the like, the landscape that is in front of the user and the experiences or events that take place in the environment in which the photograph is being taken can be factors that arouse the user's attention or emotions.

[0055] To determine such an object, the CPU 200 of the server device 20 first determines what kind of environment the user is currently in from the video data acquired by the out-camera 114a of the smart glasses 10 and / or the audio data acquired from the microphone 118. That is, scene determination can be performed from the video data and / or audio data and the event prediction model M. Then, the CPU 200, which also operates as an emotion determination unit, determines the emotion of the user at the time of shooting mainly using the face data acquired by the in-camera 114b. The "user's emotion" is the emotion the user had regarding the event that occurred at the time of shooting the video. The server device 20 also holds a table that associates each of multiple types of emotions (positive emotions and negative emotions) with each of multiple processing processes for the video data for each type of event. The CPU 200 refers to the table based on the predicted type of event and emotion and determines the processing process. Note that when one processing process, for example, superimposing a message, is associated with a certain emotion, there may be multiple messages. That is, there may be multiple processing modes for one processing process. In such a case, it is desirable to randomly select one processing mode from a plurality of processing modes.

[0056] Here, the CPU 200 has a pre-constructed event prediction model M, which is generated by machine learning based on teacher data including video and / or audio including an event as an explanatory variable in association with the type of event as a target variable.

[0057] The event prediction model M is constructed by performing machine learning in advance so that when a video in which the type of event is unknown is input as an explanatory variable, the type of event is output as a predicted result. The CPU 200 can predict an event using the event prediction model M. In particular, by adopting a video containing a wide variety of events as an explanatory variable, it is sufficiently possible to distinguish the events exemplified in, for example, FIG. 7A to FIG. 7E.

[0058] In addition, since the event prediction model M is configured to output a certain predicted value when a certain input is given, it can be implemented as a part of a computer program. Such a computer program may be stored in advance in the storage unit 202, for example. When performing event prediction, the CPU 200 reads such a computer program into the RAM, receives a video or still image as an input, and performs event prediction. Alternatively, the CPU 200 and the event prediction model M can be implemented as hardware using an ASIC (Application Specific IC), an FPGA (Field Programmable Gate Array), or a CPLD (Complex Programmable Logic Device).

[0059] Hereinafter, some of the various "subjects evoking attention or emotions" shown in Fig. 12 will be described as examples. In the following, it will be described that the CPU 200 of the server device 20 executes the processing, but this also includes the CPU 200 instructing the image processing circuit 204 to execute the processing.

[0060] In FIG. 12, when the object that evokes attention or emotion is food, the CPU 200 detects an event (which can also be called a shooting scene) that is food using video data captured by the user with the outer camera 114a and the event prediction model M. If gaze data is further used, the CPU 200 can also determine that the user's gaze is fixed on the food, thereby improving the accuracy of event detection. Furthermore, if olfactory data is used, the accuracy of event detection can be further improved. Next, the CPU 200 detects the user's facial expression captured with the inner camera 114b to detect the user's emotion. In addition, if biological information such as heart rate is used, the strength of the user's emotion can be more accurately grasped.

[0061] When the user has a smiling expression, i.e., when the user has positive emotions, the CPU 200 performs a processing process to add text ("Looks delicious") to the video data, for example, as shown in FIG. 7A. On the other hand, when the user has a distorted expression, i.e., when the user has negative emotions, the CPU 200 performs a processing process to add text ("Looks disgusting") to the video data, for example. In this way, the user's inner cry at the time of shooting can be expressed through image processing.

[0062] Next, referring to FIG. 12, a case will be described in which the object evoking attention or emotion is a landscape. The CPU 200 detects an event (which can also be called a photographed scene) that is a landscape using video data captured by the user with the outer camera 114a and the event prediction model M. If the gaze data is further used, the CPU 200 can also determine that the user's viewpoint is staying at a location with a landscape, thereby improving the accuracy of event detection. Furthermore, if audio data is used, the accuracy of event detection can be further improved. Next, the CPU 200 detects the user's facial expression captured with the inner camera 114b to detect the user's emotion. In addition, if biometric information is used, the strength of the user's emotion can be grasped more accurately.

[0063] If the user's facial expression is smiling and indicates positive emotion, the CPU 200 performs processing to add text ("So relaxing", "Buzzing") to the video data, for example, as shown in FIG. 7B. In this way, the user's inner cry at the time of shooting can be expressed through image processing. As another example, if the user's facial expression is smiling and indicates positive emotion, processing is performed to add text ("It's summer! Hot!", "Buzzing") to the video data, as shown in FIG. 7E. Even if the sound of cicadas cannot be obtained in the audio data, the imitation sound "buzzing" is added to the image in the text data to increase the sense of realism at the time of shooting.

[0064] In this way, the sound of the clear stream and the chirping of the cicadas included in the scene illustrated in Fig. 12 may be factors for executing a video processing process as an event experienced by the user in the mountains, and the processing method may be determined based on them. As an example of the latter, the onomatopoeia "saasa" representing the sound of the clear stream and the onomatopoeia "jiji" representing the chirping of the cicadas may be added to the video data as emphasis.

[0065] Next, in FIG. 12, a case will be described in which the object of attention or emotion is watching a sporting event. The CPU 200 detects an event (which can also be called a shooting scene) that is watching a sporting event using video data captured by the user with the outer camera 114a and the event prediction model M. If gaze data is further used, it is possible to know that the user's viewpoint is staying at a certain point in the sporting event, thereby improving the accuracy of event detection. Furthermore, if audio data is used, the accuracy of event detection can be further improved. Next, the CPU 200 detects the user's facial expression captured by the inner camera 114b to detect the user's emotion. In addition, if biometric information is used, the strength of the user's emotion can be grasped more accurately.

[0066] When the user's facial expression is a smile and indicates a positive emotion, the CPU 200 performs processing to add text ("Go!") to the video data, for example, as shown in FIG. 7C.

[0067] Next, in FIG. 12, when the subject that evokes attention or emotion is travel, the CPU 200 detects a travel event (which can also be called a shooting scene) using video data captured by the user with the outer camera 114a and the event prediction model M. If gaze data is further used, it is possible to know that the user's viewpoint is staying at a travel spot, and the accuracy of event detection can be improved. Furthermore, if voice data is used, the accuracy of event detection can be further improved. Next, the CPU 200 detects the user's emotion by detecting the user's facial expression captured with the inner camera 114b. In addition, if biometric information is used, the strength of the user's emotion can be grasped more accurately.

[0068] If the user's facial expression is one of relief, that is, if the user has positive feelings, CPU 200 performs processing to add text ("Finally arrived") to the video data, for example, as shown in FIG. 7D. In this way, the user's state of mind at the time of shooting is added to the image, thereby increasing the sense of realism at the time of shooting. At the same time, the user's face captured by in-camera 114b, or a previously prepared still image of the user with a relieved facial expression, may be superimposed on the video.

[0069] Above, we have shown examples in which events such as food, scenery, watching sports, and travel are detected mainly from video data, the emotions felt at the time are determined mainly from the facial expression of the photographer, and image effects corresponding to the determined emotions are added to the video data.

[0070] The smart glasses 10 that perform such processing to generate video data may be called an "event log camera" or a "life log camera." The use of a glasses-type imaging device as shown in FIG. 3 allows constant shooting in accordance with the user's activities, making it suitable for obtaining an event log or a life log. However, constant shooting increases the power consumption of the smart glasses 10 and increases the memory capacity required for recording video data, etc. Therefore, if an automatic shooting start function and / or automatic shooting end function is installed, it is effective in reducing power consumption and memory capacity. The "automatic shooting start function" is a function that starts shooting when, for example, biometric data increases to a predetermined threshold or more. The "automatic shooting end function" is a function that ends shooting when, for example, biometric data falls below a predetermined threshold during shooting. Voice data and / or movement data may be used in place of or together with biometric data.

[0071] In this embodiment, when judging emotions, a numerical value representing the user's feelings is determined using gaze data, movement data, olfactory data, face data, image data, voice data, blood pressure data, and electroencephalogram data. In this specification, the above-mentioned "numerical value of feelings" is also called an "emotion quotient." The emotion quotient is a value calculated from each of video data, voice data, movement data, biological data, and gaze data, and can be defined as a numerical value representing emotions that can be grasped from the temporal fluctuation of each data.

[0072] Next, a method for the CPU 200 to determine an emotion will be described. In this embodiment, the CPU 200 calculates an emotion quotient in response to an event that evokes attention or emotion. The emotion quotient is defined as the sum of individual emotion quotients defined for video data, audio data, movement data, biological data, and gaze data.

[0073] FIG. 13 shows a method of determining each individual emotion quotient and an example of the numerical value. In this embodiment, if the emotion quotient calculated as the sum of the individual emotion quotients is ±20 points or more, the CPU 200 determines to process the image and sound according to the determined scene. The processing content is determined in advance according to the determined scene content. It is desirable to change the processing content according to the size of the emotion quotient point number. For example, when the emotion quotient becomes higher than a predetermined value, it is possible to adopt a processing method that overly expresses the content of the text added to the image during processing. More specifically, assume a scene in which a user is moved by watching a natural sunset. The user's gaze stays in one direction (the direction of the sunset) and the sound data shows silence less than a predetermined value, which can be said to be because the user is moved. Therefore, the CPU 200 processes the image by adding the text "I'm so moved I can't put it into words" or adding an image of a character representing the user crying. In this way, by changing the processing content according to the emotion quotient, the cry of the heart can be expressed more accurately.

[0074] Fig. 13 shows an example of a standard for determining individual emotion quotients for each of gaze data, movement data, voice data, and various biological data. Therefore, the numerical values ​​shown in the "Numerical Value Example" of Fig. 13 can be called voice reference data, movement reference data, biological reference data, and gaze reference data.

[0075] The CPU 200 derives the relationship between the voice data and the voice reference data. Similarly, the CPU 200 derives the relationship between the motion data and the motion reference data, the relationship between the biometric data and the biometric reference data, and the relationship between the gaze data and the gaze reference data. If the "relationship" here means the difference, the CPU 200 calculates the sum of these differences as the emotion quotient value.

[0076] According to the example shown in FIG. 13, it can be seen that various data are collected. As in the above calculation example, each individual emotion index may be determined to be positive or negative. Alternatively, some data may be treated as having a "neutral" nature that cannot be determined to be positive or negative by itself. In FIG. 13, an * is marked to the left of the data column that corresponds to neutral data.

[0077] For example, gaze data, movement data, image data, blood pressure data, and pulse data are neutral data. The reason why blood pressure data is included in the neutral data is because blood pressure has a large effect on emotions. The reason why pulse data is also included in the neutral data is to prevent the calculation from becoming complicated.

[0078] For neutral data, the CPU 200 determines whether to assign the individual emotion index as positive or negative based on other biometric information. If the CPU 200 is still unable to determine the emotion, it may determine the emotion of the photographer as positive or negative based on the scene judgment based on machine learning, or may decide that the emotion is undeterminable.

[0079] For example, even if +10 points are given to both gaze data and image data, if the facial data indicates that the person is sleepy or unconscious, no points will be added. Also, the image data and audio data of a video will be reflected in the emotion quotient of the person who took the video.

[0080] In addition, even if the user utters words such as "it was hard" that indicate pain, a comprehensive judgment can be made based on the tone of the voice and other biometric information (for example, if a happy expression is detected from face data), and as a result, it can be judged as positive or negative.

[0081] NEC has also announced an emotion analysis solution that uses pulse data. NEC uses heart rate variability analysis to analyze time series fluctuations from pulse rate and pulse cycle (= 60 seconds / pulse rate), and analyzes emotions such as "excitement / happiness," "calmness / relaxation," "depression / fatigue," and "tension / stress." However, no further details are available.

[0082] Next, with reference to FIGS. 14 and 15, an example of emotion-based processing of moving image data by the CPU 200 and the image processing circuit 204 and an emotion index will be described.

[0083] FIG. 14 is a diagram for explaining an example of a process for enhancing a moving image. In this example, an olfactory sensor (biosensor) is used in addition to the sensors in FIG. 1 and FIG. 2, or by replacing some of the sensors. The olfactory sensor detects the presence of an object that emits a scent or a smell, such as food or flowers, and positions it as a factor that evokes emotions. As a result, the presence of such an object is considered to be an event for which processing is performed. Note that the olfactory sensor and the method for determining the smell can use, for example, the technology announced by the National Institute for Materials Science (NIMS) on June 21, 2021.

[0084] In FIG. 14, first, the CPU 200 performs a scene determination process (= event detection process) using video data and the event prediction model M. Through the scene determination process, the CPU 200 determines that the scene is a close-up of a flower subject based on the video data of the out-camera 114a, and that the user is smelling the flower based on the video data of the in-camera 114b. In this case, if the gaze data is used, if there is a point A where the gaze stays, it is considered that there is a subject that interests the user at point A, so that the accuracy of the above determination can be improved. Similarly, if the motion data is used, the accuracy of the above determination can be improved if there is a point C where the motion stays. Note that gaze data is not essential. This is because it is sufficient to identify the shooting scene from the video data. The gaze data can be used as an auxiliary to improve the accuracy of the determination.

[0085] Next, the CPU 200 mainly detects the user's emotions from the facial expression of the photographer from the image captured by the in-camera 114b. In addition, the CPU 200 uses the olfactory data acquired by the olfactory sensor to determine whether the odor evokes positive emotions. In this example, it is assumed that the odor is a positive odor.

[0086] Furthermore, according to the gaze data, the gaze moves slowly back and forth around a certain point. The same is true for the movement data and blood pressure data. These are neutral data. According to the olfactory data, the smell that evokes positive emotions gradually becomes stronger toward point B. From these, the individual emotion index for smells is +20.

[0087] In summary, CPU 200 estimates that the emotion quotient is 20 and that the user is feeling a pleasant aroma. As a result, in response to an instruction from CPU 200, image processing circuit 204 superimposes an image of the user's profile taken by in-camera 114b and text corresponding to the emotion, "Smells nice...", on the video at the time indicated by the dashed line corresponding to time B. In this image processing, the image of the flower taken by out-camera 114a is reduced, the reduced image is placed on the left side of the composite image, and the image of the user's profile taken by in-camera 114b is placed on the right side of the composite image. This allows image processing circuit 204 to process the video data according to the user's emotion.

[0088] FIG. 15 is a diagram for explaining how an emotion of enjoying dancing is determined and an avatar showing an appearance of enjoying dancing is displayed in video data.

[0089] The video shows the user's hands, parts of his / her arms, and the face and the vicinity of the other person's face. Assume that music is playing as audio data. First, the CPU 200 performs a scene determination process using the video data, the motion data, and the event prediction model M. Through the scene determination process, the CPU 200 determines that the scene is one in which two people are dancing, based on the video data and the motion data.

[0090] Next, the CPU 200 mainly detects the user's emotions from the facial expression of the photographer from the image captured by the in-camera 114b. In addition, by using biometric data related to blood pressure, the intensity of the user's emotions can be grasped. Similarly, by using voice data uttered by the user, positive or negative emotions can be grasped.

[0091] The motion data indicates that the user is moving relatively vigorously, and the music audio data indicates that rhythmic music is playing. If the user's voice in the audio data is a positive voice, it is determined that a positive event has occurred, and the individual emotion indices are added together to calculate the emotion quotient.

[0092] In summary, the CPU 200 can determine that the user has a positive emotional quotient.

[0093] Here, as an example of processing, "face + avatar" is used to further emphasize that it is dancing. When a face photo of the user or the other person is obtained, a composite image is generated using the face photo and the body part using the avatar. The body movement from the neck down is reproduced by the movement obtained from the motion sensor 122. The face image of the user may be an image prepared by the user in advance, or a face image acquired from the in-camera 114b. The face image of the other person may be an image prepared by the user in advance, or a face image acquired from the out-camera 114a. Using these face images, the CPU 200 creates an avatar image with the part from the neck down as an avatar. At the time when the dashed line of the section B of the motion data ends, the CPU 200 superimposes the avatar image on the video. Alternatively, a predetermined frame of the video may be replaced with the avatar image. Also, an animation of the arms and legs of the avatar image may be created. In this way, the image processing circuit 204 performs processing on the video data according to the user's emotions. This makes it possible to express the user enjoying dancing.

[0094] 1-4. Effects etc. As described above, the imaging device system 1 according to the present embodiment includes the imaging sensor 114 that generates video data by shooting, the interface unit 106 that acquires data, the recording unit 202, the emotion determination unit 200, and the image processing unit 204. The interface unit acquires at least one of voice data acquired during shooting and biometric data that is biometric information of the user acquired during shooting. The recording unit records a data set in which at least one of the voice data and the biometric data is associated with the video data. The emotion determination unit uses at least one of the video data, the voice data, and the biometric data to determine the emotion felt by the user regarding the event that occurred at the time of shooting. The image processing unit executes processing on the video data according to the determined emotion.

[0095] According to the above configuration, the video can be processed in accordance with the emotions felt by the user after experiencing the event, thereby making it possible to express the emotions of the user.

[0096] The emotion determination unit grasps the event using the video data, and determines the emotion felt by the user at the event using at least one of the audio data and the biometric data.

[0097] The interface unit acquires gaze data indicating the direction of the user's gaze at the time of shooting. The emotion determination unit grasps an event using the video data and the gaze data, and determines the emotion felt by the user at the event using at least one of the audio data and the biometric data.

[0098] The image sensor includes a first image sensor that captures an image of a predetermined subject to generate first video data, and a second image sensor that captures an image of the user's face to generate second video data. The emotion determination unit grasps an event using the first video data, and determines the emotion felt by the user at the event using the second video data and at least one of the audio data and the biometric data.

[0099] The emotion determination unit calculates a first value from the voice data, calculates a second value from the biometric data, and calculates a third value from the gaze data, and determines the emotion based on a total value of the first value to the third value.

[0100] The emotion determination unit includes an event prediction model that is constructed in advance by machine learning based on teacher data that includes video and / or audio including an event as an explanatory variable in association with the type of event as a target variable. The emotion determination unit predicts the type of event from at least one data and the event prediction model, and determines processing based on the predicted event type and emotion. The image processing unit executes the determined processing on the video data.

[0101] The emotion determination unit holds a table in which, for each type of event, each of a plurality of types of emotions is associated with each of a plurality of processing operations for video data, and refers to the table based on the predicted type of event and emotion to determine the processing operation.

[0102] Each of the multiple processing processes includes at least one of adding text or an image that indicates the photographer's impression and / or feelings, and adding an image of the user.

[0103] Each of the multiple processing steps includes adding text or an image that represents the photographer's impression and / or feelings, which are generated from the voice data or biometric data.

[0104] The server device 20 according to this embodiment is a server device used in an imaging device system 1 having an imaging device 10. The imaging device has an imaging sensor 114, a microphone 118, an interface unit 106, and a transmission circuit 104. The imaging sensor generates video data by capturing images. The microphone generates audio data. The interface unit acquires biometric data. The transmission circuit transmits the video data and at least one of the audio data and the biometric data.

[0105] The server device has a communication circuit, a recording unit, an emotion determination unit, and a data generation unit. The communication circuit communicates with the imaging device. The recording unit records a data set in which at least one of audio data and biometric data is associated with the video data received by the communication circuit. The emotion determination unit uses at least one of the audio data and biometric data to determine the emotion felt by the user regarding an event occurring at the time of shooting. The data generation unit processes the video data according to the determined emotion, or generates factor analysis data by analyzing events indicating factors of the user's emotion.

[0106] According to the above configuration, the user's emotions can be expressed by processing the video, or the factors that have led to the improvement or deterioration of the user's QoL can be determined.

[0107] The server device receives a request for creating processed video data or factor analysis data via a communication circuit. The data generation unit generates the processed video data or creates the factor analysis data according to the content of the request.

[0108] The factor analysis data generated by the data generating unit includes factor analysis data that is preferred by the user and / or factor analysis data that is not preferred by the user.

[0109] The server device receives, via the communication circuit, a designation of a particular section of video data to be subjected to processing.

[0110] 2. Variations "Live streaming" is a configuration example for selecting a video to be used in video processing when live streaming. For example, consider a situation where live streaming is being performed using multiple cameras. Assume that multiple cameras are present and multiple users are being filmed using the multiple cameras. Depending on the magnitude of the estimated emotion value for each user in the live event, the camera can be switched and displayed. The camera switching can be achieved using a switcher.

[0111] 16 shows a schematic configuration of a server device 21 according to a modified example. In addition to the configuration of the server device 20, or instead of the configuration of the server device 20, the server device 21 is configured and operates at least as described below. The CPU 200 has an emotion determination unit 200a and a switcher 200b. The CPU 200 receives the video data of each of the cameras A and B via the ring buffer 202a, performs emotion determination using each video data in the emotion determination unit 200a, and instructs the image processing circuit 204 to perform necessary image processing on the camera A video data or the camera B video data. The emotion determination unit 200a outputs a control signal to the switcher 200b so as to replace the camera A video data or the camera B video data, which is the source of image correction, with an image that has been subjected to image processing in the image processing circuit 204. In this way, the video replaced with the image that has been subjected to the desired image processing is distributed as a live video. The processing of the emotion determination unit 200a is the same as that of the CPU 200 described above. The switcher 200b may be realized as a software switch or a hardware switch. With these, a live distribution system is realized.

[0112] FIG. 17 is a diagram for explaining how, when eating a meal in a live broadcast, the ingredients look delicious and the emotion that the food tastes good is determined and then processed into a video.

[0113] It is assumed that each of Mr. X and Mr. Y wears smart glasses 10 and broadcasts live images captured by the smart glasses 10. The smart glasses 10 are regarded as cameras, and hereinafter, Mr. X's smart glasses 10 are described as "camera X" and Mr. Y's smart glasses 10 are described as "camera Y". In the imaging device system 1 of FIG. 1, 5G communication is used for communication between the smart glasses 10 and the server device 20. At the times shown in A and B, the CPU 200 in the server device 20 uses the first video data acquired from the out-camera 114a of Mr. X's camera X and the above-mentioned event prediction model M to determine that the scene contains shrimp as an ingredient of the dish, and further determines that the scene is a meal scene based on the second video data acquired from the in-camera 114b of Mr. X's camera X. From the times A, B and the captured scene, the CPU 200 switches the switcher (FIG. 16) to an image obtained by performing image processing on a frame capturing Mr. X with Mr. X's camera X from the video of Mr. X's camera X or the video of Mr. Y's camera Y. After this switching, the image is switched back to the image before the switching.

[0114] Here, the smart glasses 10 are provided with a ring buffer 202a capable of temporarily storing first and second video data from about 5 seconds before the current shooting time. The ring buffer 202a may be included in the storage unit 202. The CPU 200 uses the buffered first and second video data to perform event detection, emotion detection by calculating an emotion quotient, and video processing in parallel with video shooting, and outputs the processed video to the outside in place of the shot video. By using this configuration, when Mr. X makes a sound saying "It's delicious" at time C, the CPU 200 can process the sound into an image with Mr. X's face as the face part and an avatar as the body part, and output the image.

[0115] In FIGS. 1 and 2, the smart glasses 10 and the server device 20 are described as separate devices that constitute the imaging device system 1. However, these do not necessarily have to be separate devices. For example, with the improvement of the processing performance of the CPU 100 and / or the CPU 200, the increase in capacity of the recording unit 202 that stores video data, and the reduction in cost, the smart glasses 10 and the server device 20 may be implemented in one device having a common housing. For example, if a smart glass that has the same appearance as the smart glasses 10, generates video data, biometric data, etc., and can determine emotions is developed, all processing can be completed with that one smart glass. Therefore, it is not essential that it is a server-client system. The smart glasses themselves that house the functions and configurations of the smart glasses 10 and the server device 20 in a single housing also fall within the category of an imaging device system.

[0116] Also, an example has been described in which an image sensor, a microphone, a motion sensor, biometric data, and gaze data are provided on the side of the smart glasses 10. However, as described above, they may be positioned as external devices of the smart glasses 10. A system that includes these is the imaging device system 1.

[0117] In the present disclosure, a smart glass-type imaging device has been exemplified, but an ordinary video camera that is held in the hand to capture video, a smartphone, or a digital camera with a video capture function can also be used as the imaging device. When any of these cameras is used, and when a camera configuration including the functions and configuration of the server device 20 is used, it falls within the scope of the imaging device system.

[0118] 6.Examples The following provides examples of aspects of the present disclosure.

[0119] <Aspect 1> an imaging sensor for generating video data by capturing images; An interface unit for acquiring data, Audio data captured during the recording, and Biometric data, which is biometric information of the user acquired during the photographing An interface unit for acquiring at least one of the following: a recording unit that records a data set in which at least one of the audio data and the biometric data is associated with the video data; an emotion determination unit that determines an emotion felt by the user regarding an event that occurred at the time of capturing the image, using at least one of the video data, the audio data, and the biometric data; an image processing unit that processes the video data in accordance with the determined emotion; An imaging device system comprising:

[0120] <Aspect 2> The imaging device system of aspect 1, wherein the emotion determination unit grasps the event using the video data, and determines the emotion felt by the user at the event using at least one of the audio data and the biometric data.

[0121] <Aspect 3> The interface unit acquires gaze data indicating a direction of the user's gaze at the time of photographing, The imaging device system of aspect 1 or 2, wherein the emotion determination unit grasps the event using the video data and the gaze data, and determines the emotion felt by the user at the event using at least one of the audio data and the biometric data.

[0122] <Aspect 4> the imaging sensor includes a first imaging sensor that captures an image of a predetermined subject and generates first video data, and a second imaging sensor that captures an image of a face of the user and generates second video data; The imaging device system of any one of aspects 1 to 3, wherein the emotion determination unit grasps the event using the first video data, and determines the emotion felt by the user at the event using the second video data and at least one of the audio data and the biometric data.

[0123] <Aspect 5> The emotion determination unit is Calculating a first value from the voice data, calculating a second value from the biometric data, and calculating a third value from the gaze data; The imaging device system according to any one of aspects 1 to 4, wherein the emotion is determined based on a total value of the first value to the third value.

[0124] <Aspect 6> The emotion determination unit includes: a pre-constructed event prediction model, the event prediction model being constructed by machine learning based on teacher data including, as explanatory variables, video and / or audio including the event, and, as a response variable, a type of the event, in association with each other; The emotion determination unit is predicting a type of the event from the at least one data and the event prediction model; determining a processing operation based on the predicted type of the event and the emotion; 6. The imaging device system according to any one of aspects 1 to 5, wherein the image processing unit executes the determined processing on the video data.

[0125] <Aspect 7> The emotion determination unit is a table is stored in which each of a plurality of types of emotions is associated with each of a plurality of processing operations of the video data for each type of event; The imaging device system according to any one of aspects 1 to 6, wherein the processing is determined by referring to the table based on the predicted type of the event and the emotion.

[0126] <Aspect 8> The imaging device system according to aspect 7, wherein each of the plurality of processing processes includes at least one of adding text or an image indicating the photographer's impression and / or feelings, and adding an image of the user.

[0127] <Aspect 9> An imaging device system according to aspect 7, wherein each of the multiple processing processes includes adding text or images indicating the photographer's impressions and / or feelings generated from the audio data or the biometric data.

[0128] <Aspect 10> An imaging device system having an imaging device and a server device capable of communicating with each other, the imaging device includes an imaging sensor that generates the video data, a microphone that generates the audio data, and a biometric sensor that generates the biometric data; The server device has a storage device that operates as the recording unit and a signal processing circuit that operates as the emotion determination unit. 10. The imaging device system according to any one of embodiments 1 to 9.

[0129] <Aspect 11> A server device used in an imaging device system having an imaging device, The imaging device includes: an image sensor that captures images to generate video data; A microphone for generating audio data; An interface unit for acquiring the biometric data; and a transmission circuit for transmitting the video data and at least one of the audio data and the biometric data; The server device includes: a communication circuit for communicating with the imaging device; a recording unit that records a data set in which at least one of the voice data and the biometric data is associated with the video data received by the communication circuit; an emotion determination unit that determines an emotion felt by the user regarding an event that occurred at the time of capturing the image, using at least one of the voice data and the biometric data; a data generating unit that processes the video data in accordance with the determined emotion, or analyzes events that indicate factors of the user's emotion to generate factor analysis data; A server device comprising:

[0130] <Aspect 12> the server device receives, via the communication circuit, a request for creating the video data on which the processing has been performed or the factor analysis data; 12. The server device according to claim 11, wherein the data generation unit generates the moving image data on which the processing has been performed, or creates the factor analysis data, in accordance with the content of the request.

[0131] <Aspect 13> 13. The server device according to aspect 11 or 12, wherein the factor analysis data generated by the data generation unit includes factor analysis data preferred by the user and / or factor analysis data not preferred by the user.

[0132] <Aspect 14> 14. The server device according to aspect 12 or 13, wherein the server device receives, via the communication circuit, a designation of a specific section of the video data to be subjected to the processing. [Industrial Applicability]

[0133] The present disclosure is applicable to an imaging device system and a server device. [Explanation of symbols]

[0134] 10. Smart Glasses 20 Server device 100 CPU 102 Memory 104 Communication Circuit 106 Interface Device 110 Eye gaze sensor 114 Image Sensor 118 Mike 122 Motion Sensor 126 Biometric Sensors 200 CPU (emotion determination section, data generation section) 202 Storage section 204 Image Processing Circuit 206 Communication Circuit

Claims

1. an imaging sensor for generating video data by capturing images; An interface unit for acquiring data, Audio data captured during the recording, and Biometric data, which is biometric information of the user acquired during the photographing An interface unit for acquiring at least one of the following: a recording unit that records a data set in which at least one of the audio data and the biometric data is associated with the video data; an emotion determination unit that determines an emotion felt by the user regarding an event that occurred at the time of capturing the image, using at least one of the video data, the audio data, and the biometric data; an image processing unit that processes the video data in accordance with the determined emotion; An imaging device system comprising:

2. The imaging device system according to claim 1 , wherein the emotion determination unit grasps the event using the video data, and determines the emotion felt by the user at the event using at least one of the audio data and the biometric data.

3. The interface unit acquires gaze data indicating a direction of the user's gaze at the time of photographing, The imaging device system according to claim 2 , wherein the emotion determination unit grasps the event using the video data and the gaze data, and determines the emotion felt by the user at the event using at least one of the audio data and the biometric data.

4. the image sensor includes a first image sensor that captures an image of a predetermined subject and generates first video data, and a second image sensor that captures an image of a face of the user and generates second video data; 2. The imaging device system of claim 1, wherein the emotion determination unit grasps the event using the first video data, and determines the emotion felt by the user at the event using the second video data and at least one of the audio data and the biometric data.

5. The emotion determination unit is Calculating a first value from the voice data, calculating a second value from the biometric data, and calculating a third value from the gaze data; The imaging device system according to claim 3 , wherein the emotion is determined based on a total value of the first value to the third value.

6. The emotion determination unit includes: a pre-constructed event prediction model, the event prediction model being constructed by machine learning based on teacher data including, as explanatory variables, video and / or audio including the event, and, as a response variable, a type of the event, in association with each other; The emotion determination unit is predicting a type of the event from the at least one data and the event prediction model; determining a processing operation based on the predicted type of the event and the emotion; The imaging device system according to claim 1 , wherein the image processing unit executes the determined processing on the video data.

7. The emotion determination unit is a table is stored in which each of a plurality of types of emotions is associated with each of a plurality of processing operations of the video data for each type of event; The imaging device system according to claim 1 , wherein the processing is determined by referring to the table based on the predicted type of the event and the predicted emotion.

8. The imaging device system according to claim 7 , wherein each of the plurality of processing steps includes at least one of adding a character or an image indicating a photographer's impression and / or feelings, and adding an image of the user.

9. 8. The imaging device system according to claim 7, wherein each of the plurality of processing steps includes adding a character or an image representing a mental image and / or a feeling of a photographer generated from the voice data or the biometric data.

10. An imaging device system having an imaging device and a server device capable of communicating with each other, the imaging device includes an imaging sensor that generates the video data, a microphone that generates the audio data, and a biometric sensor that generates the biometric data; The server device has a storage device that operates as the recording unit and a signal processing circuit that operates as the emotion determination unit. The imaging device system according to claim 1 .

11. A server device used in an imaging device system having an imaging device, The imaging device includes: an image sensor that captures images to generate video data; A microphone for generating audio data; An interface unit for acquiring biometric data; and a transmission circuit for transmitting the video data and at least one of the audio data and the biometric data; The server device includes: a communication circuit for communicating with the imaging device; a recording unit that records a data set in which at least one of the audio data and the biometric data is associated with the video data received by the communication circuit; an emotion determination unit that determines an emotion felt by the user regarding an event that occurred at the time of capturing the image, using at least one of the voice data and the biometric data; a data generating unit that processes the video data in accordance with the determined emotion, or analyzes events that indicate factors of the user's emotion to generate factor analysis data; A server device comprising:

12. the server device receives a request for creating the video data on which the processing has been performed or the factor analysis data via the communication circuit; The server device according to claim 11 , wherein the data generating section generates the moving image data on which the processing has been performed, or creates the factor analysis data, in accordance with the content of the request.

13. The server device according to claim 11 , wherein the factor analysis data generated by the data generating unit includes factor analysis data preferred by the user and / or factor analysis data not preferred by the user.

14. The server device according to claim 12 , wherein the server device receives, via the communication circuit, a designation of a specific section of the video data to be subjected to the processing.

Citation Information

Patent Citations

  • Information processing device, information processing method, and program

    JP2019047234A