Learning system and data collection device

Abstract

Provided is a learning system that trains a machine learning model, which estimates a camera operation for generating noteworthy content or photographing noteworthy content.　This learning system comprises a data collection device for collecting data and a learning device for training a machine learning model using the data collected by the data collection device. The learning device retrains the machine learning model using: the learning data collected on the basis of the result of having analyzed learning data that affects the training of the machine learning model, said learning data exerting more than a prescribed level of influence upon the training of the machine learning model; insufficient learning data; or data similar to these.

Description

[Technical field]

[0001] The technology disclosed in this specification (hereinafter referred to as "the present disclosure") relates to a learning system that trains a machine learning model, a data collection device that performs collection processing of learning data for training the machine learning model, and an information processing device that performs analysis processing of data. [Background technology]

[0002] Digital cameras are now widely used. For example, in every household, digital video cameras and digital still cameras are used to record daily life and various events such as children's entrance and graduation ceremonies and weddings. It is desirable to shoot more moving scenes, but it is difficult for general users to judge whether the scene they are observing through the viewfinder is a moving scene. Of course, it is also difficult for professional photographers to judge whether the scene they are shooting is moving.

[0003] For example, one possible method is to use artificial intelligence technology equipped with a trained neural network model to handle emotionally charged content. In particular, a deep learning neural network model (hereinafter referred to as "DNN") equipped with multiple convolutional neural network layers can extract features that developers cannot imagine from a large amount of data through training, i.e., deep learning, to develop artificial intelligence functions that can solve complex problems that developers cannot imagine algorithms for.

[0004] To utilize artificial intelligence technology, learning data for learning artificial intelligence is necessary. For example, a learning device has been proposed that includes a learning data collection unit that collects learning data for machine learning of a specified ability, and a learning processing unit that performs machine learning of a learning device so as to acquire the specified ability using the collected learning data (see Patent Document 1). To handle moving content using artificial intelligence technology, a large amount of learning data is first required to be used for learning a machine learning model that generates moving content for humans or estimates camera operations for shooting moving content.

[0005] However, due to the subjective nature of emotions, collecting learning data is extremely difficult. One possible method of collecting data that reflects subjectivity is to utilize preference information that appears on social networking services (SNS), but this is not easy to implement due to the risk of violating privacy. In addition, not only the images and background music of individual scenes in a piece of content, but also the context of the scenes before and after are thought to be elements that create emotions, but it is difficult to collect data that reflects context using a method that relies solely on images. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] JP 2018-149669 A [Non-patent literature]

[0007] [Non-Patent Document 1] Using Brain Data for Sentiment Analysis.Gu et al.JLCL 2014. [Non-Patent Document 2] Music Emotion Classification: A Fuzzy Approach.Yang et al.ACM MM 2006. [Non-Patent Document 3] Visual Social Relationship Recognition.Li et al.arxiv 2018. [Non-Patent Document 4] Understanding Black-box Predictions via Influence Functions,Pang Wei Kho and Percy Liang<https: / / arxiv.org / abs / 1703.04730> [Non-Patent Document 5] Alex Kendall, Yarin Gal, "What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision", NIPS 2017 Summary of the Invention [Problem to be solved by the invention]

[0008] An object of the present disclosure is to provide a learning system that trains a machine learning model that estimates camera operations for generating moving content or shooting moving content, a data collection device that performs a process of collecting learning data for a machine learning model that estimates camera operations for generating moving content or shooting moving content, and an information processing device that analyzes the emotions imparted by the collected data. [Means for solving the problem]

[0009] The present disclosure has been made in consideration of the above problems, and a first aspect thereof is: The present invention includes a data collection device that collects data, and a learning device that uses the data collected by the data collection device to learn a machine learning model, The learning device re-learns the machine learning model using learning data that has a predetermined or greater influence on the learning of the machine learning model, missing learning data, or data similar thereto, which is collected based on a result of analyzing the learning data that influences the learning of the machine learning model. It is a learning system.

[0010] However, the term "system" used here refers to a logical collection of multiple devices (or functional modules that realize a specific function), and it does not matter whether each device or functional module is contained within a single housing.

[0011] The learning device trains a machine learning model that estimates camera operations for generating or capturing moving content, and the analysis can be performed by XAI, confidence score calculation, influence function, or Bayesian DNN.

[0012] The learning device analyzes learning data that affects the learning of the machine learning model, and based on the analysis result, transmits a request signal to the data collection device to request transmission of learning data that has a predetermined or greater effect on the learning of the machine learning model, missing learning data, or data similar thereto. Then, the data collection device transmits data collected based on the received request signal to the learning device, so that the learning device can re-learn the machine learning model based on the data transmitted from the data collection device in response to the request signal.

[0013] Alternatively, when the data collection device receives the request signal from the learning device, the data collection device transmits to the learning device learning data that has a predetermined or greater effect on the learning of the machine learning model, missing learning data, or data similar thereto, among the collected data, based on a result of analyzing the effect of the collected data on the learning of the machine learning model.The learning device can then re-learn the machine learning model based on the data transmitted from the data collection device in response to the request signal.The learning device may transmit information required for the analysis to the data collection device when transmitting the request signal.

[0014] The second aspect of the present disclosure is A receiving unit that receives a request signal from a learning device that performs learning of a machine learning model, the request signal requesting transmission of learning data of the machine learning model; A data collection unit that collects learning data that has a predetermined or greater influence on learning of the machine learning model, missing learning data, or data similar thereto in response to receiving the request signal; a transmission unit that transmits the data collected by the data collection unit to the learning device; The data collection device includes:

[0015] The receiving unit receives the request signal from the learning device, requesting learning data that has a predetermined or greater effect on learning of the machine learning model, missing learning data, or data similar thereto. The data collecting unit then collects data based on the received request signal, and the transmitting unit transmits the data collected by the data collecting unit to the learning device.

[0016] Alternatively, the data collection device according to a second aspect further includes an analysis unit that analyzes the influence of the data collected by the data collection unit on learning of the machine learning model. Then, the transmission unit transmits to the learning device, among the data collected by the data collection unit, learning data that has a predetermined or greater influence on the learning of the machine learning model, missing learning data, or data similar thereto, based on the analysis result by the analysis unit.

[0017] Moreover, a third aspect of the present disclosure is a segment extraction unit that extracts a segment from the content based on content evaluation information of the content and biometric information of a person viewing the content; an expression identification unit that detects the face of a person appearing in the segment and identifies the facial expression; a relationship estimation unit that estimates a relationship between people appearing in the segment; an emotion identification unit that identifies emotion labels of the segments based on facial expressions of people appearing in the segments and relationships between the people; The information processing device includes:

[0018] The segment extraction unit extracts a segment in which a high viewing rate and biometric information indicating positive emotion match from the content, and further extracts a segment in which a high viewing rate and biometric information indicating positive emotion do not match.

[0019] The relationship estimation unit estimates the relationship between people appearing in the current segment using face detection of people appearing in past segments and facial expression identification results of the detected faces as the context.

[0020] The information processing device according to the third aspect further includes a first emotion analysis unit that analyzes emotions based on text information obtained by recognizing a voice included in the segment, and a second emotion analysis unit that detects music included in the segment and analyzes the emotions imparted by the music. The emotion identification unit further considers the emotions identified by the first emotion analysis unit from the text information and the emotions identified by the second emotion analysis unit from the music to identify an emotion label for the segment.

[0021] The data collection device performs the analysis by calculating a reliability score. In this case, the learning device transmits information on the neural network model that has been trained up to the current time to the data collection device as information necessary for the analysis. Effect of the Invention

[0022] According to the present disclosure, it is possible to provide a learning system that trains a machine learning model that estimates camera operations for generating moving content or shooting moving content, a data collection device that performs a process of collecting learning data for a machine learning model that estimates camera operations for generating moving content or shooting moving content, and an information processing device that identifies the emotions provided by content.

[0023] Note that the effects described in this specification are merely examples, and the effects brought about by the present disclosure are not limited thereto. Furthermore, the present disclosure may have additional effects in addition to the effects described above.

[0024] Further objects, features and advantages of the present disclosure will become apparent from the following detailed description based on the embodiments and the accompanying drawings. [Brief description of the drawings]

[0025] [Figure 1] FIG. 1 is a diagram showing the functional configuration of a data collection system 100. As shown in FIG. [Diagram 2] FIG. 2 is a flowchart showing the operation of the data collection system 100. [Diagram 3] FIG. 3 shows a mechanism for estimating relationships between people based on the context of a segment and the results of face detection and facial expression recognition. [Figure 4] FIG. 4 is a diagram showing the overall flow of the learning process of the emotion classifier using the collected learning data. [Diagram 5] FIG. 5 is a diagram showing an example of the configuration of a digital camera 500. As shown in FIG. [Figure 6] FIG. 6 is a diagram showing a functional configuration for performing emotion identification labeling of content captured by digital camera 500. As shown in FIG. [Figure 7] FIG. 7 is a diagram showing the functional configuration of a digital camera 500 for automatically controlling camerawork based on the emotion identification result of the captured content. [Figure 8] FIG. 8 is a diagram showing the functional configuration of a digital camera 500 for automatically generating captions based on emotion identification results for captured content. [Figure 9] FIG. 9 is a diagram showing the functional configuration of a digital camera 500 for automatically adding background music based on the emotion identification result of the captured content. [Figure 10]FIG. 10 is a diagram showing the configuration of a learning system 1000 for efficiently learning a neural network model for automatic camera operation. [Figure 11] FIG. 11 is a diagram showing the configuration of an observation prediction model 1100. [Figure 12] FIG. 12 is a diagram showing the configuration of an operation model 1200. [Figure 13] FIG. 13 is a diagram showing the configuration of an operation estimation model 1300. [Figure 14] FIG. 14 is a diagram showing an impression score estimation model 1400. As shown in FIG. [Figure 15] FIG. 15 is a diagram showing the relationship between the control object and the average impression score. [Figure 16] FIG. 16 shows a neural network model trained to estimate the error of the output. [Figure 17] FIG. 17 is a diagram showing an example of the internal configuration of the data collecting device 1010. [Figure 18] FIG. 18 is a diagram showing another example of the internal configuration of the data collecting device 1010. As shown in FIG. [Figure 19] FIG. 19 is a diagram showing an example of the internal configuration of the learning device 1030. [Figure 20] FIG. 20 is a diagram showing another example of the internal configuration of the learning device 1030. [Figure 21] FIG. 21 is a diagram showing yet another example of the internal configuration of the learning device 1030. [Figure 22] FIG. 22 is a diagram showing an example of the internal configuration of the model utilization device 1020. [Figure 23] FIG. 23 is a diagram showing an example of the internal configuration of a model utilization device 1020 serving as an edge AI. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0026] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings in the following order.

[0027] A. Configuration of a system for collecting and processing data related to machine learning models that generate inspiring content B. System operation to collect and process data for machine learning models that generate inspiring content C. Estimation of relationships between people and other entities included in content D. Emotional Identification E. Learning process using collected data F. Reasoning G. Applications H. Efficient training of machine learning models

[0028] A. Configuration of a system for collecting and processing data related to a learning model that generates inspiring content FIG. 1 shows a schematic functional configuration of a data collection system 100 that performs data collection processing according to the present disclosure.

[0029] The illustrated data collection system 100 performs a process of collecting data used for training a machine learning model that generates moving content from a vast amount of video content 101 or estimates camera operations for shooting moving content. The data collected by the data collection system 100 is basically used as training data for training a machine learning model (e.g., a neural network) for handling moving content (e.g., for inferring whether a scene or content moves people), but of course it may be used for other purposes.

[0030] In this embodiment, the data collection system 100 handles video content 101 that is broadcast on television or an Internet broadcasting station and is provided with content evaluation information such as audience rating information, and the content is mainly composed of video data and audio data. Furthermore, the video content 101 may include various contents such as video sharing sites, so long as the content is provided with content evaluation information that evaluates the content and is similar to audience rating information (or can be replaced with audience rating information), such as "Like!" (like, enjoyable, supportable). Furthermore, the data collection system 100 processes content extracted from the video content 101 in fixed-length or variable-length segment units.

[0031] The data collection system 100 includes a content evaluation information acquisition unit 102, a biometric information acquisition unit 103, a comparison unit 104, a segment extraction unit 105, a context extraction unit 106, a voice recognition unit 107, a music detection unit 108, a face detection unit 109, a first emotion analysis unit 110, a second emotion analysis unit 111, a facial expression identification unit 112, a relationship estimation unit 113, and an emotion identification unit 114. The data collection system 100 is assumed to be configured as a service in which computer resources are provided on a wide-area computer network such as the cloud, i.e., the Internet. The components of the data collection system 100 may be integrated into one computer or distributed across multiple computers. Each unit will be described below.

[0032] The content evaluation information acquisition unit 102 acquires content evaluation information such as a viewer rating that is assigned to the content extracted from the video content 101. The content evaluation information acquisition unit 102 may acquire viewer rating information of the target content from a viewer rating survey company. In this embodiment, the data collection system 100 uses the viewer rating information to determine whether each scene of the content moves people. Advantages of using the viewer rating information include that it does not violate privacy issues and that it is information that already reflects the reactions of a large number of people. In addition, since the viewer rating information also reflects the context of the story of the content, such as the beginning, development, twist, and conclusion, context information before and after a scene with a high viewer rating can also be effectively used. Furthermore, in the case of content to which viewer rating information is assigned, additional information (for example, lines, background music, etc.) that accompanies the scene is frequently attached. As will be described later, the lines and background music of a moving scene play a major role in moving people and should be utilized to expand the scope of applications. In addition, the content evaluation information acquisition unit 102 may acquire content evaluation information such as "Like!" (like, enjoy, support) given on SNS and the like, in addition to (or instead of) the viewing rating information.

[0033] The biometric information acquiring unit 103 acquires biometric information of a person viewing content extracted from the video content 101. For example, a biometric information sensor that detects biometric information such as brain waves, sweating, gaze, and myoelectric potential may be installed in a home, and the biometric information acquiring unit 103 may collect the biometric information detected when viewing content such as television or Internet broadcasts. When a huge amount of biometric information is collected from many homes, the biometric information acquiring unit 103 may use the information after performing statistical processing such as averaging.

[0034] When a person watches a reproduced content, bio-information such as electroencephalograms, sweating, gaze, and myoelectric potentials react to a scene that makes him / her feel tension, sadness, anger, or other emotions. On the other hand, while the viewership rating information is an index showing how many households or people are watching the content, scenes that move people are not necessarily reflected in the viewership rating information. In view of the risk that content evaluation information such as viewership ratings alone cannot accurately determine whether each scene in the content moves people, the data collection system 100 uses bio-information such as electroencephalograms acquired by the bio-information acquisition unit 103 to interpolate the content evaluation information. Research into identifying emotions from electroencephalograms has already been conducted (see, for example, Non-Patent Document 1). Therefore, it is possible to estimate which emotions are responsible for high viewership ratings based on the electroencephalograms of people watching each scene of the content. Note that electroencephalograms are electrical potentials obtained by measuring electrical activity generated from the brain from the scalp. Electroencephalographs are generally configured to measure brain potentials from electrodes placed on the scalp. The International 10-20 system is known as the arrangement position of electrodes, but the present disclosure is not particularly limited to this system.

[0035] The comparison unit 104 compares content evaluation information such as viewing ratings acquired by the content evaluation information acquisition unit 102 with human biometric information acquired by the biometric information acquisition unit 103. Then, the segment extraction unit 105 extracts segments of the content extracted from the video content 101 in which high viewing ratings and biometric information indicating positive emotions (such as "moved") match, as data to be used for emotion identification learning. The segment extraction unit 105 also extracts segments in which high viewing ratings and biometric information indicating positive emotions do not match, as negative samples (such as "not moved").

[0036] The context extraction unit 106 extracts the context of the content extracted from the video content 101. More specifically, the context extraction unit 106 extracts the context before and after a scene with a high viewing rate, or the context before and after a segment extracted by the segment extraction unit 105. The context extraction unit 106 may extract the context using a trained neural network model trained to estimate the context from content consisting of video and audio data.

[0037] The speech recognition unit 107 applies speech recognition processing (ASR: Automatic Speech Recognition) to the speech components of the audio data contained in the segments extracted by the segment extraction unit 105, recognizes speech such as lines, and converts it into text (speech transcription).

[0038] The first emotion analysis unit 110 applies natural language processing to perform emotion analysis on text information such as lines output from the speech recognition unit 107, and analyzes what emotion is composed of the segment (scene). The first emotion analysis unit 110 may perform emotion analysis in a broader context by referring to the lines of the previous and following segments (scenes). Based on the emotion analysis result of the lines, the emotion of the segment (scene) can be grasped more accurately.

[0039] The voice recognition unit 107 and the first emotion analysis unit 110 can each be configured using a trained machine learning model such as a Convolutional Neural Network (CNN). In addition, the voice recognition unit 107 and the first emotion analysis unit 110 can be collectively referred to as a "voice identifier" and configured using one machine learning model such as a CNN.

[0040] The music detection unit 108 applies music detection processing to the audio data contained in the segments extracted by the segment extraction unit 105, and detects background music added to the video.

[0041] The second emotion analysis unit 111 identifies the mood of the background music detected by the music detection unit 108 and whether it is moving or not. Several techniques have been proposed to identify emotions imparted by music (for example, see Non-Patent Document 2). Based on the emotion analysis result of the background music, the emotion of the segment (scene) can be grasped more accurately.

[0042] The music detection unit 108 and the second emotion analysis unit 111 can each be configured using a trained machine learning model such as CNN. Also, the music detection unit 108 and the second emotion analysis unit 111 can be collectively configured as a "music classifier" using one machine learning model such as CNN.

[0043] The face detection unit 109 executes face detection of people appearing in the video data included in the segments extracted by the segment extraction unit 105 .

[0044] The facial expression identification unit 112 identifies the facial expression of the face detected by the face detection unit 109. For example, the facial expression identification unit 112 analyzes a facial image pattern from the detected face and estimates which emotion the facial image pattern corresponds to. The facial emotion can be estimated using artificial intelligence that has previously learned the correlation between the facial image pattern and human emotion. The facial image pattern can be configured as a combination of facial parts such as eyebrows, eyes, nose, mouth, and cheeks, but can also be configured as an image of the entire facial image without dividing it into facial parts.

[0045] The face detection unit 109 and the facial expression classification unit 112 can each be configured using a trained machine learning model such as CNN. In addition, the face detection unit 109 and the facial expression classification unit 112 can be combined as an "facial expression classifier" and configured using one machine learning model such as CNN.

[0046] When two or more people appear in a segment, the relationship estimation unit 113 estimates the relationship and intimacy between the people based on the facial expressions of each person identified by the facial expression identification unit 112 and the context of the previous and next segments extracted by the context extraction unit 106. This is because even if a person has the same facial expression, the likelihood of it inspiring emotions in people who see it varies depending on the relationship and intimacy with other people who appear with them (for example, whether they are photographed with family or close friends), and this affects the emotion identification process at a later stage.

[0047] The emotion identification unit 114 identifies the emotion of the segment based on the emotion identification results of the voice (such as lines) in the segment by the voice recognition unit 107 and the first emotion analysis unit 110, the identification results of the emotion and mood of the background music in the segment by the music detection unit 108 and the second emotion analysis unit 111, the identification results of the facial expressions of the people appearing in the segment by the face detection unit 109 and the facial expression identification unit 112, and the estimation results of the relationships and intimacy between the people appearing in the segment by the relationship estimation unit 113.

[0048] The emotion identification unit 114 estimates the emotion of the segment from the above input data using, for example, a trained neural network model, and outputs an emotion identification label indicating the emotion level of the segment. The emotion identification label may be a simple binary label of two types, positive (e.g., "moved") and negative (e.g., "not moved"), or may be a label indicating emotions differentiated into various types, such as "anger," "disgust," "fear," "happiness," "sadness," "surprise," and so on. The emotion identification unit 114 may output an emotion identification label defined based on an emotion model, such as the Wundt model or the Plutchik model.

[0049] B. System operation to collect and process data for machine learning models that generate inspiring content FIG. 2 illustrates the operation of data collection system 100 in flow chart form.

[0050] First, content that has been broadcast on television or an Internet broadcasting station and has content evaluation information such as audience ratings attached thereto is imported into the data collection system 100 (step SS201). The content evaluation information acquisition unit 102 acquires the content evaluation information attached to this content. It is assumed that the content is composed of multiple segments, and content evaluation information is attached to each segment.

[0051] The biometric information acquiring unit 103 acquires biometric information such as brain waves that indicate the reaction of a person watching the content captured in step S201 (step S202). If the content is a television broadcast, it is assumed that biometric information including at least brain waves is collected from many homes that watch the program.

[0052] The comparison unit 104 compares, for each segment, the content evaluation information acquired by the content evaluation information acquisition unit 102 with the human biometric information acquired by the biometric information acquisition unit 103, and checks whether the biometric information indicating high viewing ratings and positive emotions matches (step S203).

[0053] Here, the segment extraction unit 105 extracts segments in which high viewing ratings and biometric information indicating positive emotions match (Yes in step S203) from the content extracted from the video content 101 as data to be used for emotion identification learning (step S204). Also, the segment extraction unit 105 extracts segments in which high viewing ratings and biometric information indicating positive emotions do not match (No in step S203) as negative samples (step S215).

[0054] If the segment extracted in step S204 contains voice data (Yes in step S205), the voice recognition unit 107 recognizes the voice and converts it into text, and the first emotion analysis unit 110 identifies emotion from the text information (step S206).

[0055] Furthermore, if the segment extracted in step S204 contains music data (Yes in step S207), the music detection unit 108 detects the music, and the second emotion analysis unit 111 identifies the emotion imparted by the music (step S208).

[0056] Furthermore, if a person is photographed in the segment extracted in step S204 (Yes in step S209), the face detection unit 109 detects the person's face, and the facial expression identification unit 112 identifies the facial expression of the detected face (step S210).

[0057] Furthermore, if two or more people are captured in the segment extracted in step S204 (Yes in step S211), the relationship estimation unit 113 estimates the relationship between the people based on the facial expressions of each person identified by the facial expression identification unit 112 and the context of the previous and next segments extracted by the context extraction unit 106 (step S212).

[0058] The emotion identification unit 114 identifies the emotion of the segment based on the emotion identification result of the voice (such as lines) in the segment by the voice recognition unit 107 and the first emotion analysis unit 110, the identification result of the emotion and mood of the background music in the segment by the music detection unit 108 and the second emotion analysis unit 111, the identification result of the facial expressions of the people appearing in the segment by the face detection unit 109 and the facial expression identification unit 112, and the estimation result of the relationship between the people appearing in the segment by the relationship estimation unit 113 (step S213).

[0059] The emotion identification unit 114 estimates the emotion of the segment and outputs an emotion identification label indicating the emotion level of the segment. The emotion identification label may be a simple binary label of two types, positive and negative, or may be a label expressing various types of emotions such as "anger", "disgust", "fear", "happiness", "sadness", "surprise", etc. As a result, samples having labels expressing positive emotions or various types of emotions such as "anger", "disgust", "fear", "happiness", "sadness", "surprise", etc. can be acquired (step S214). In addition, negative samples that do not evoke positive emotions in humans are also acquired (step S215).

[0060] The data collection system 100 extracts emotionally charged segments (scenes) from content such as television and Internet broadcasts according to the processing procedure shown in Fig. 2, and identifies the emotion label of each segment based on the emotion analysis results of the voices and background music contained in the segments, the facial expression identification results of the people appearing in the segments, and the estimation results of the relationships between the people when two or more people appear in the segments. The emotion-labeled segments collected by the data collection system 100 can be used as learning data for training an emotion classifier using artificial intelligence (neural network model).

[0061] C. Estimation of relationships between people and other entities included in content In this section, the process of estimating the relationships between people in a video, which is performed by the relationship estimation unit 113, will be described in detail.

[0062] When a video contains only one person, the emotion can be identified from the facial expression of the detected person. On the other hand, when a video contains two people, the relationship between the people arises, so the relationship between the people must also be taken into consideration when identifying the emotion from the person.

[0063] For example, even if the same person A appears in a video, if person B who appears with person A is a family member or friend with whom person A has a deep relationship and a high level of intimacy, the video is likely to be emotional. On the other hand, if the level of intimacy between person A and person B is low, the video may not evoke any emotion. In this way, understanding the relationship between subjects is very important in estimating whether a video will evoke emotion.

[0064] A technology for machine learning information about a subject based on information posted on SNS and the like is already known. However, when trying to understand human relationships such as the relationship between subjects from information posted on SNS, it is not easy to do so because there is a risk of threatening the privacy of the subject. In contrast, in the present disclosure, the relationship estimation unit 113 estimates the relationship between people based on the facial expressions of each person identified by the facial expression identification unit 112 and the context of the previous and next segments extracted by the context extraction unit 106, so there is no privacy issue.

[0065] Although there are already known technologies that estimate the relationship between people based on the facial expression of each person, most of these technologies are limited to a one-dimensional level of simply connecting positive facial expressions to positive emotions. However, the emotion label superficially identified from the facial expression does not necessarily match the emotion of the person. For example, in a photo taken with a friend, a person may intentionally make an angry expression. Also, in a photo taken at an official event with a heavy atmosphere, even if the person is smiling, it may be clear that the relationship is not clearly intimate. In other words, it is not possible to simply estimate the relationship between people by identifying the facial expressions of people. It is not enough to simply identify whether the facial expressions of the people in the photo are inspiring or not, but it is also necessary to grasp the intimacy between two or more people in the photo. If a video contains people with high intimacy, the possibility of it being inspiring is inevitably high.

[0066] Research into recognizing relationships between people based on visual data has already been actively conducted (see, for example, Non-Patent Document 3). In the present disclosure, the relationship estimation unit 113 estimates relationships between people using not only the information of the segment extracted by the segment extraction unit 105 (for example, facial expressions of people identified from the video in the segment) but also the context of the previous and following segments, so that it is expected that the accuracy of estimation will be improved. Furthermore, it is also conceivable to use meta-information of the content itself (for example, information on broadcast programs) that can be easily obtained from information sources such as the Internet to estimate relationships between people.

[0067] Fig. 3 illustrates how the relationship estimation unit 113 estimates the relationship between people based on the context of a segment and the face detection and facial expression identification results. In the example shown in Fig. 3, the relationship estimation unit 113 uses the face detection and facial expression identification results of a person appearing in a past segment, as well as the face detection and facial expression identification results of the detected face of a person appearing in a current segment extracted by the segment extraction unit 105, as context.

[0068] First, a past segment 301 is input to a CNN 311 constituting the face detection unit 109 and the facial expression classification unit 112, and this CNN 311 detects the face of each person appearing in the past segment 301 and classifies the facial expression of the detected face. Then, the relationship estimation unit 113 registers the people whose faces are detected from the past segment 301 and their facial expression classification results 303 in a person registration unit 313.

[0069] Next, the current segment 302 extracted by the segment extraction unit 105 is input to the CNN 312 constituting the face detection unit 109 and the facial expression classification unit 112, and this CNN performs face detection of each person appearing in the current segment 302 and facial expression classification of the detected face, and outputs the face detection and facial expression classification results 304. Then, the relationship estimation unit 113 queries the person registration unit 313 to check whether the person detected from the current segment 302 is a person who has been pre-registered from the past segment 301.

[0070] Here, if the person detected from the current segment 302 is a person who has been registered in advance from the past segment 301, the relationship estimation unit 113 estimates the relationship between the people using the facial expression identification result estimated from the past segment 301 in addition to the facial expression identification result estimated from the current segment 302 for the same person. If the relationship estimation unit 113 uses the context of the segment to more accurately estimate the relationship between the people, the emotion identification unit 114 at the subsequent stage can more accurately identify the emotion label of the current segment.

[0071] Furthermore, if the person detected from the current segment 302 is not a person previously registered from the past segment 301, the relationship estimation unit 113 estimates the relationship between the people based only on the facial expression classification result estimated from the current segment 302. Alternatively, the emotion classification unit 113 at the latter stage identifies the emotion label of the current segment based on the facial expression classification result estimated from the current segment without using the relationship between the people.

[0072] D. Emotional Identification In this section, the process of identifying the emotion imparted by a segment, which is performed by the emotion identifying unit 114, will be described in detail.

[0073] The emotion identification unit 114 identifies the emotion label of the current segment by combining the following multi-dimensional information:

[0074] (1) The result of emotion recognition of the voice (such as lines) in the segment by the voice recognition unit 107 and the first emotion analysis unit 110 (2) The result of identifying the emotion or mood of the background music in the segment by the music detection 108 and the second emotion analysis unit 111. (3) Recognition results of facial expressions of people in the segment by the face detection unit 109 and the facial expression recognition unit 112 (4) Estimation results of the relationship and intimacy between people appearing in the segment by the relationship estimation unit 113

[0075] If all of the above (1) to (4) indicate positive emotions, the emotion identifying unit 114 labels the current segment as an emotional scene.

[0076] Furthermore, if some of the above information (1) to (4) is missing (for example, if the current segment does not have background music, or if no people are shown and there is no facial expression recognition result), and all of the existing information is positive, the emotion recognition unit 114 may label the current segment as an emotional scene. If some information is missing, the reliability of the emotion recognition may decrease, but since the segment itself has already passed the filter of matching content evaluation information and electroencephalogram information in the segment extraction unit 105, it is believed that the reliability will not decrease suddenly.

[0077] Furthermore, if the current segment cannot be labeled as emotional based on any of the information (1) to (4) above, the emotion classifier 114 may label the segment as negative. Segments labeled as negative can be used as negative samples in the learning data for training the emotion classifier.

[0078] As described above, each classifier that classifies the voice, music, and facial expressions of a person contained in a segment is composed of a trained machine learning model such as CNN. Depending on how each classifier is pre-trained, the emotion classifier 114 can also assign labels that represent emotions differentiated into various types, such as "anger," "disgust," "fear," "happiness," "sadness," "surprise," etc., rather than just two simple binary labels of positive and negative. In other words, the data collection system 100 can collect learning data with various emotion identification labels, and can be used to train an emotion classifier that can distinguish various types of emotions.

[0079] E. Learning process using collected data In this section, a learning process for training an emotion classifier using learning data collected using the data collection system 100 according to the present disclosure will be described in detail. The emotion classifier is a device that identifies what emotion a piece of content evokes. The device referred to here means both a device configured with dedicated hardware and a device that executes software to achieve a given function.

[0080] Figure 4 shows a schematic diagram of the overall learning process of the emotion recognition machine using the collected learning data.

[0081] The learning data collected using the data collection system 100 is stored in the data accumulation unit 410. Each piece of learning data corresponds to what is called a "segment" in sections A to D above. A segment as learning data is an element of content such as television or Internet broadcasting, and is composed of video data consisting of a frame sequence and audio data synchronized with the video. Furthermore, an emotion identification label is finally assigned as teacher data to the segment as learning data during the processing in the data collection system 100. The emotion identification label may be differentiated into various types, but in this section, for the sake of simplicity of explanation, it is assumed that two simple binary labels, positive and negative, are assigned.

[0082] The emotion classifier 420 is composed of a network unit 421 consisting of multiple network models to which video and audio data are input as learning data, and a classifier 422 that classifies emotion labels of content based on features acquired by the network unit 421. Each network in the network unit 421 has parameters (such as connection weight coefficients between nodes).

[0083] In the case of video, each feature is acquired for each frame through a trained neural network model (CNN) in the network unit 421, and the features are compiled to obtain the feature (Video Feature) of the entire video. There are various methods for compiling the frame features, including a simple concatenation method or a method of sequentially inputting the features into a time-series network such as LSTM (Long Short Term Memory). In this embodiment, when two or more people appear in a frame, the relationship between the people estimated based on the context of multiple consecutive frames is acquired as a video feature.

[0084] On the other hand, in the case of audio data, audio features can be acquired by converting the data into features such as Mel-Frequency Cepstrum Coefficients (MFCC) or mel-spectrograms and inputting them into a time-series network such as LSTM in the network unit 421. Also, speech features can be transcribed into text (transcription) and inputted into a time-series network such as LSTM in the network unit 421 to acquire text features.

[0085] The classification unit 422 projects the overall video feature (Video Feature), audio feature (Audio Feature), and text feature (Text Feature) obtained as described above into a common space, and assigns emotion classification labels to the learning data input to the emotion classifier 420. Here, the classification unit 422 assigns either a positive or negative binary label.

[0086] The evaluation unit 423 calculates a loss function Loss such as softmax based on the error between the emotion identification label assigned to the training data by the classification unit 422 and the emotion identification label as teacher data assigned to the training data by the data collection system 100. Alternatively, the loss function Loss may be used to individually classify the emotion identification label based on the video feature amount, the emotion identification label based on the audio feature amount, and the emotion identification label based on the audio feature amount, and calculate the error L video , L text , L Audio The loss function Loss may be calculated based on the sum of the errors. The loss function Loss calculated based on the errors is backpropagated to each network having parameters in the network unit 421, and the parameters of each network (CNN, LSTM) in the network unit 421 are updated so that the loss function Loss is minimized. As a result, the emotion classifier 420 progresses with learning so as to output an emotion classification label for the input learning data that is equal to the teacher data.

[0087] F. Reasoning Section A above has described a method in which the data collection system 100 collects data by using content evaluation information and biological information such as electroencephalograms from content such as television and Internet broadcasts. Section E above has described a process in which the emotion classifier 420 learns using the data collected by the data collection system 100.

[0088] The emotion identifier 420, having been trained through the learning process, can identify emotion-inducing content or what type of emotion content inspires. The content referred to here includes a variety of content, such as television or Internet broadcast content, video content shared on video sharing sites, and video and still image content taken by users with digital cameras, and does not need to be provided with content evaluation information or biological information such as brain waves.

[0089] G. Applications As described above, the learning data collected by the data collection system according to the present disclosure can be used to train a machine learning model for an emotion classifier that identifies the emotions that content evokes in humans. In addition, the emotion classifier developed in this way is expected to have various applications.

[0090] G-1. Digital Camera Configuration The emotion identifier developed based on the present disclosure can be installed in various content processing devices, such as digital cameras, that process content, such as recording, playing, and editing content.

[0091] 5 shows an example of the configuration of a digital camera 500. The illustrated digital camera 500 includes an optical system 501, an imaging unit 502, an AFE (Analog Front End: analog pre-processing) unit 503, a camera signal processing unit 504, a codec 505, a main processing unit 506, a microphone 514, an A / D conversion unit 515, a display unit 516, an audio playback unit 517, and a recording unit 518. It is assumed that this digital camera 500 is equipped with an emotion classifier developed based on the present disclosure.

[0092] The optical system 501 includes a lens for collecting light from a subject on the imaging surface of the imaging unit 502, a drive mechanism for moving the lens to perform focusing and zooming, a shutter mechanism for allowing light from the subject to enter the imaging surface for a predetermined period of time by opening and closing the shutter, and an iris (aperture) mechanism for limiting the direction and range of the light beam from the subject (all not shown). A driver (not shown) controls camerawork such as driving each mechanism in the optical system 501 (for example, subject focusing, iris, pan and tilt, shutter or self-timer setting timing) based on control signals from a main processing unit 506 described later.

[0093] The imaging unit 502 is composed of imaging elements such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxyde Semiconductor), has an imaging surface on which pixels with photoelectric conversion effect are arranged two-dimensionally, and converts incident light from a subject into an electrical signal. For example, a G checkered RB color coding single plate is arranged on the light receiving side. Signal charges corresponding to the amount of incident light that has passed through each color filter are accumulated in each pixel, and the color of the incident light at the pixel position can be reproduced from the amount of each of the three color signal charges read out from each pixel. Note that the analog image signals output from the imaging unit 502 are primary color signals of each color of RGB, but may also be complementary color signals.

[0094] The AFE unit 503 performs sample-and-hold after highly accurately suppressing low noise in the imaging signal (Correlated Double Sampling: CDS), and then performs appropriate gain control using an AGC (Automatic Gain Control), performs AD conversion, and outputs a digital image signal. The AFE unit 503 also outputs to the imaging unit 502 a timing pulse signal for driving the imaging element, and a drive signal for outputting the charge of each pixel of the imaging element in the vertical direction line by line in accordance with this timing pulse signal.

[0095] The camera signal processing unit 504 performs pre-processing such as defective pixel correction, digital clamping, and digital gain control on the image signal sent from the AFE unit 503, applies a white balance gain using AWB, performs image quality correction processing such as sharpness and contrast adjustment to reproduce the appropriate color state, and creates an RGB screen signal using demosaic processing. The camera signal processing unit 504 also performs resolution conversion depending on whether the captured image is to be displayed and output as a through image on the display unit 516 or stored in the recording unit 518, and performs codec processing such as MPEG (Moving Picture Experts Group).

[0096] The main processing unit 506 is composed of a processor, a RAM (Random Access Memory), and a ROM (Read Only Memory), and controls the overall operation of the digital camera 500. The processor is a CPU (Central Processing Unit) or a GPU (Graphic Processing Unit) having multiple cores. During recording, the main processing unit 506 stores video data captured by the imaging unit 502 and audio data collected by the microphone 514 in the recording unit 518. During playback, the main processing unit 506 reads out the video and audio data from the recording unit 518 and outputs them on the display unit 516 and the sound playback unit 517. In this embodiment, it is assumed that the main processing unit 506 is equipped with an emotion classifier developed based on the present disclosure.

[0097] Display unit 516 is a device that displays images being captured or recorded images, such as a liquid crystal display panel mounted on digital camera 500, or an external television or projector.

[0098] The sound playback unit 517 is a device that plays back recorded sound, such as a speaker built into the digital camera 5100 or an external speaker.

[0099] The recording unit 518 is a large-capacity recording device such as a hard disk drive (HDD) or a solid state drive (SSD). The recording unit 518 records content consisting of video captured by the imaging unit 502 and audio data collected by the microphone 514 in synchronization with the video. In addition, parameters of a machine learning model for an emotion classifier (for example, a coupling weight coefficient between neurons in a neural network model) are recorded in the recording unit 518.

[0100] G-2. Labeling the captured content Next, functions realized by mounting the emotion classifier developed based on the present disclosure on the digital camera 500 will be described.

[0101] By subjecting content captured by digital camera 500 to an emotion identifier, it is possible to assign an emotion identification label to the content and to extract emotional scenes (segments) from the content.

[0102] Fig. 6 shows a functional configuration for performing emotion identification labeling of content captured by a digital camera 500. In the figure, the configuration of emotion identifier 420 is assumed to be the same as that shown in Fig. 4. It is assumed that emotion identifier 420 operates in main processing unit 506 within digital camera 500.

[0103] The video of the content captured by the digital camera 500 is passed through a trained neural network model (CNN) in the network unit 421 for each frame to acquire each feature, and the features of the entire video are obtained by combining these. In addition, audio data recorded in synchronization with the video is converted into features such as MFCC and input into a time-series network such as LSTM in the network unit 421 to obtain audio features. In addition, the voice features are rewritten into text and input into a time-series network such as LSTM in the network unit 421 to obtain text features.

[0104] The classification unit 422 projects the overall video feature amount, audio feature amount, and text feature amount obtained as described above into a common space, and assigns an emotion classification label to the learning data input to the emotion classifier 420. The classification unit 422 assigns a binary label, either positive or negative. Alternatively, the classification unit 422 may assign labels representing various differentiated types of emotions.

[0105] In either case, the identification unit 422 assigns an emotion identification label to each scene (or segment) of the shot content. The assigned emotion identification label is recorded, for example, as meta information of the content. When playing back the recorded content, the emotion identification label can be used as a search key to select and view emotional scenes. Furthermore, when editing the recorded content, the emotion identification label can be used as a search key to select only scenes that evoke specific emotions, such as "happy" scenes and "sad" scenes.

[0106] In this way, the data collection system 100 according to the present disclosure can assign an emotion identification label that identifies the type of emotion that can be given to humans to each input content or each segment in the content. Therefore, by using the data collection system 100, content or segments to which a specific emotion identification label (e.g., "happy", "sad", etc.) is assigned can be collected from a vast amount of video content 101, and used to train a machine learning model that generates content that gives the specific emotion or estimates camera operations for shooting moving content.

[0107] G-3. Camerawork support or control Camera work can be assisted or automatically controlled based on the results obtained by subjecting content captured by digital camera 500 to an emotion identification device. For example, the line of sight direction or zoom for photographing a subject may be automatically controlled to increase the level of emotion or to acquire a specific type of emotion identification label, or the brightness, color, angle of view, composition, focus, etc. of a captured image may be automatically controlled, or a guidance display on display unit 516 or audio guidance from sound playback unit 517 may be used to suggest recommended angles.

[0108] Fig. 7 shows the functional configuration of a digital camera 500 for automatically controlling camerawork based on the emotion identification result of shot content. In the figure, the emotion identification device 420 has the same configuration as shown in Fig. 4. It is assumed that the emotion identification device 420 operates in the main processing unit 506 in the digital camera 500.

[0109] The video of the content captured by the digital camera 500 is passed through a trained neural network model (CNN) in the network unit 421 for each frame to acquire each feature, and the features of the entire video are obtained by combining these. In addition, audio data recorded in synchronization with the video is converted into features such as MFCC and input into a time-series network such as LSTM in the network unit 421 to obtain audio features. In addition, the voice features are rewritten into text and input into a time-series network such as LSTM in the network unit 421 to obtain text features.

[0110] The classification unit 422 projects the overall video feature amount, audio feature amount, and text feature amount obtained as described above onto a common space, and assigns an emotion identification label to the learning data input to the emotion identification device 420. Then, the evaluation unit 423 outputs a camerawork control signal to the optical system 501 so as to increase the degree of emotion or to obtain a specific type of emotion identification label.

[0111] G-4. Adding captions Based on the results obtained by subjecting content captured by digital camera 500 to an emotion identifier, captions that are appropriate for the video scenes and that increase the emotion can be automatically added.

[0112] Fig. 8 shows the functional configuration of a digital camera 500 for automatically generating captions based on emotion identification results for captured content. In the figure, the configuration of emotion identifier 420 is assumed to be the same as that shown in Fig. 4. It is assumed that emotion identifier 420 operates in main processing unit 506 within digital camera 500.

[0113] The video of the content captured by the digital camera 500 is passed through a trained neural network model (CNN) in the network unit 421 for each frame to acquire each feature, and the features of the entire video are obtained by combining them. Audio data recorded in synchronization with the video is converted into features such as MFCC and input into a time-series network such as LSTM in the network unit 421 to obtain audio features. Voice features are rewritten into text and input into a time-series network such as LSTM also in the network unit 421 to obtain text features. The caption generation unit 801 generates a caption for each scene using, for example, a trained machine learning model. The text information of the caption generated by the caption generation unit 801 is superimposed on the text information obtained from the voice recognition of the content, and is input into a time-series network such as LSTM in the network unit 421.

[0114] The classification unit 422 projects the overall video feature amount, audio feature amount, and text feature amount obtained as described above into a common space, and assigns an emotion identification label to the learning data input to the emotion identification device 420. The evaluation unit 423 calculates a loss function for the emotion identification label output from the classification unit 422. Then, the caption generation unit 801 re-learns so as to increase the degree of emotion or to acquire a specific type of emotion identification label. The caption generated by the caption generation unit 801 is recorded by being superimposed on the audio data of the content.

[0115] G-5. Adding background music Based on the results obtained by subjecting content captured by digital camera 500 to an emotion identifier, background music that is appropriate for the video scene and that enhances the emotion can be automatically added.

[0116] Fig. 9 shows the functional configuration of a digital camera 500 for automatically adding background music based on the emotion identification result of photographed content. In the figure, the emotion identification device 420 has the same configuration as shown in Fig. 4. It is assumed that the emotion identification device 420 operates in the main processing unit 506 in the digital camera 500.

[0117] The video of the content captured by the digital camera 500 is passed through a trained neural network model (CNN) in the network unit 421 for each frame to acquire each feature, and the feature of the entire video is obtained by combining the features. Audio data recorded in synchronization with the video is converted into features such as MFCC and input into a time-series network such as LSTM in the network unit 421 to obtain audio features. Voice features are rewritten into text and input into a time-series network such as LSTM also in the network unit 421 to obtain text features. The music search unit 901 searches for music that will be background music for the scene, for example, using a trained machine learning model. The music search unit 901 may search for music that will be background music for the scene, for example, in a music database (not shown) that accumulates a huge amount of music data. The music data of the music found by the music search unit 901 is superimposed on the audio data of the content and input into a time-series network such as LSTM in the network unit 421.

[0118] The classification unit 422 projects the overall video feature amount, audio feature amount, and text feature amount obtained as described above into a common space, and assigns an emotion identification label to the learning data input to the emotion identification device 420. The evaluation unit 423 calculates a loss function for the emotion identification label output from the classification unit 422. The music search unit 901 then re-learns to find background music that increases the degree of emotion or that can acquire a specific type of emotion identification label. Music The music data is recorded while being superimposed on the audio data of the content.

[0119] H. Efficient Training of Models As described in Section G above, according to the present disclosure, by equipping the digital camera 500 with a neural network model, it is possible to assign an emotion identification label that indicates the emotion level to the content (or each segment that constitutes the content), provide camerawork support or control (automatic camera operation) for shooting content that provides a high level of emotion, and assign captions and background music to improve the emotion score of the content. In this Section H, Equipped A method for efficiently training a neural network model is described.

[0120] For convenience, the following description will be limited to the training method of a neural network model for automatic camera operation. However, it should be understood that a similar method can also be used to efficiently train a neural network model for adding captions or background music.

[0121] H-1. Learning system configuration 10 is a schematic diagram showing the configuration of a learning system 1000 for efficiently learning a neural network model for camera operation assistance or automatic operation. In this embodiment, the neural network model to be learned is mainly assumed to be a neural network model that estimates camera operations for generating moving content or shooting moving content. Of course, the learning system 1000 can also be used to learn other types of neural network models.

[0122] The learning system 1000 shown in FIG. 10 is composed of a data collection device 1010 that collects learning data, a learning device 1030 that learns a neural network model based on the learning data collected by the data collection device 1010, a data analysis device 1040 that analyzes the learning data, and a model utilization device 1020 that uses the neural network model 1050 learned by the learning device 1030.

[0123] The data collection device 1010 collects data consisting of image data captured by a camera and operation information at the time of camera capture, for example. The data collection device 1010 includes a huge variety of devices, such as still cameras, video cameras, image sensors used in cameras, multi-function information terminals such as smartphones, TVs, headphones or earphones, game consoles, IoT devices such as refrigerators and washing machines, and mobile devices such as drones and robots, and can collect a huge amount of data from these huge numbers of data collection devices 1010. For example, if the data collection device 1010 is a camera used by an expert such as a professional photographer, it can collect impressive captured images and camera operation information for capturing such images. In addition, the data collection device 1010 may include not only devices that collect data in real time as exemplified above, but also devices that have already accumulated a large amount of data, such as a content database.

[0124] The learning device 1030 transmits a request signal to each data collecting device 1010 to request transmission of collected data. Of course, the data collecting device 1010 may transmit data spontaneously, not in response to a request signal. The learning device 1030 then uses the vast amount of data collected by the multiple data collecting devices 1010 to learn and re-learn various neural network models, such as an "observation prediction model," an "operation model," an "operation estimation model," and an "impression score estimation model." Details of the neural network models used in the learning system 1000 will be described later.

[0125] Although the learning accuracy of the neural network model is improved by using a huge amount of data, it is inefficient to perform learning or re-learning using data that does not contribute much to learning. Therefore, in the learning system 1000 according to the present disclosure, the data analysis device 1040 analyzes the learning data that affects the learning of the neural network model to be learned, and based on the analysis results, extracts meaningful learning data such as learning data that has a predetermined or greater effect on the learning of the neural network model, missing learning data, and data similar to these, and the learning device 1030 uses the meaningful learning data extracted by the data analysis unit 1040 to efficiently perform learning and re-learning of the neural network model.

[0126] The data analysis device 1040 can analyze the data collected by the data collection device 1010 based on techniques such as XAI (eXplainable AI), confidence score calculation of learning data, influence function calculation, and data deficiency estimation using Bayesian DNN (Deep Neural Network).

[0127] The model utilization device 1020 is a device that shoots moving content using the neural network model 1050 trained by the learning device 1030. The model utilization device 1020 is, for example, a camera used by a general user who is not familiar with camera operation. The model utilization device 1020 can use the neural network model trained by the learning device 1030 to automatically operate a camera comparable to a photography expert such as a professional cameraman, and can automatically shoot video with a high impression score.

[0128] Here, a modified example of the learning system 1000 will be described.

[0129] The learning device 1030 requires huge computational resources to learn various neural network models. Therefore, in FIG. 10, it is assumed that the learning device 1030 is constructed on a cloud, for example (i.e., cloud AI (Artificial Intelligence)). The learning device 1030 may perform distributed learning using a plurality of computation nodes. However, it is also assumed that the learning device 1030 is configured integrally with a model utilization device that uses a trained neural network model (i.e., edge AI). Alternatively, the learning device 1030 may be configured integrally with a data collection device 1010 that provides training data.

[0130] The data analysis device 1040 may be constructed in either a cloud or an edge. For example, the data analysis device 1040 may be configured as an integrated device with the learning device 1030. In this case, the learning device 1030 may internally analyze learning data that affects the learning of the neural network model, and request the data collection device 1010 to transmit learning data that has a predetermined or greater influence on the learning of the neural network model, missing learning data, or data similar thereto.

[0131] Alternatively, the data analysis device 1040 may be configured as an integrated device with the data collection device 1010. In this case, the learning device 1030 provides the data collection device 1010 with information required for data analysis (for example, information on the neural network model that has already been trained at that point in time) when making a request to transmit training data. The data collection device 1010 may then analyze the influence of the collected data on the neural network model, and transmit to the learning device 1030 training data that has a predetermined or greater influence on the training of the neural network model, missing training data, or data similar thereto.

[0132] 10, the data collection device 1010 that collects learning data and the model utilization device 1020 that uses a neural network model are depicted as separate devices, but it is also possible that one device operates as both the data collection device 1010 and the model utilization device 1020. For example, one camera may operate as the data collection device 1010 in a manual operation mode, collect data such as shooting data and camera operation information, and transmit the data to the learning device 1030, while operating as the model utilization device 1020 when switched to an automatic operation mode, and automatically capture images using a neural network model that has been trained by the learning device 1030.

[0133] H-2. Construction of neural network model The neural network models used in the learning system 1000 shown in Fig. 10 will be described. In the learning system 1000, the learning device 1030 uses data collected by the data collection device 1040 to learn and re-learn neural networks such as an "observation prediction model," an "operation model," an "operation estimation model," and an "impression score estimation model." In addition, the model utilization device 1020 utilizes at least a part of these neural network models trained by the learning device 1030.

[0134] Observation-forecast model: 11 shows a schematic configuration of an observation prediction model 1100. The observation prediction model 1100 is a neural network model that predicts an image to be captured by the camera at the next time (i.e., the "next time image") 1111 from video information 1101 captured by the camera up to the current time and operation information 1102 on the camera up to the current time.

[0135] The operation information 1102 referred to here is, for example, information on operations performed on the camera to determine imaging conditions such as frame rate, aperture, exposure value, magnification, focus, etc. (The same applies below.) In addition, when performing operations on a mobile device such as a robot or drone, remote control operations (camera work indicated by roll, pitch, yaw, etc.) performed on the mobile device may also be included in the operation information (The same applies below.)

[0136] The observation prediction model 1100 also outputs a reliability score 1112 for the image 1111 at the next time predicted. The reliability score 1112 is a value indicating how accurately the image 1111 at the next time is predicted. In the present disclosure, the reliability score is used to identify data that is insufficient for learning or data that has a high influence on learning. Alternatively, the data analysis device 1040 extracts meaningful learning data, such as learning data that has a predetermined or greater influence on the learning of the observation prediction model 1100, insufficient learning data, and data similar to these, based on a method such as an explanation of the grounds for the prediction of the observation prediction model 1100 by XAI, reliability score calculation, influence function calculation, and data insufficiency estimation by Bayesian DNN. The calculation function of the reliability score 1112 by the observation prediction model 1100 may be implemented as a part of the data analysis device 1040.

[0137] , In the learning system 1000, the learning device 1030 can learn the observation prediction model 1100 based on a data set consisting of video information and operation information transmitted from the data collection device 1010. The learning device 1030 can learn the observation prediction model 1100 so as to predict more moving images through reinforcement learning. In addition, the data analysis device 1040 extracts meaningful learning data from the data set transmitted from the data collection device 1010, and the learning device 1030 can efficiently re-learn the observation prediction model 1100 using this meaningful learning data.

[0138] Operation Model: FIG. 12 shows a schematic configuration of the operation model 1200. The operation model 1200 is a neural network model that predicts an operation 1211 to be performed on the camera at the next time from video information 1201 captured by the camera up to the current time and operation information 1202 to the camera up to the current time. The operation model 1200 also outputs a reliability score 1212 for the predicted operation 1211 at the next time. The reliability score 1212 is a value indicating how accurately the image 1111 at the next time is predicted. Alternatively, the data analysis device 1040 extracts meaningful learning data such as learning data that has a predetermined or greater influence on the learning of the operation model 1200, missing learning data, and data similar to these, based on a method such as an explanation of the grounds for the prediction of the operation model 1200 by XAI, a reliability score calculation, an influence function calculation, and a data shortage estimation by Bayesian DNN. The computation of the confidence score 1212 according to the operational model 1200 may be implemented as part of the data analysis device 1040 .

[0139] In the learning system 1000, the learning device 1030 can learn the operation model 1200 based on a data set consisting of video information and operation information transmitted from the data collection device 1010. The learning device 1030 can learn the operation model 1200 so as to predict camera operation information that can capture a more impressive image by reinforcement learning. In addition, the data analysis device 1040 extracts meaningful learning data from the data set transmitted from the data collection device 1010, and the learning device 1030 can efficiently re-learn the operation model 1200 using this meaningful learning data.

[0140] Operation estimation model: 13 is a schematic diagram showing the configuration of an operation estimation model 1300. The operation estimation model 1300 is a neural network model that estimates time-series operation information 1311 for capturing video time-series information 1301 up to the current time with a camera. For example, by using the operation estimation model 1300, it is possible to estimate time-series operation information of a camera performed by an expert who is well versed in camera operation, such as a professional cameraman, from high-quality video time-series information captured by the expert.

[0141] The manipulation estimation model 1300 may also output a reliability score for the estimated time-series manipulation information 1311. Alternatively, the data analysis device 1040 extracts meaningful learning data, such as learning data that has a predetermined or greater influence on the learning of the manipulation estimation model 1300, missing learning data, and data similar thereto, based on a method such as explanation of the grounds for prediction of the manipulation estimation model 1300 by XAI, reliability score calculation, influence function calculation, and data shortage estimation by Bayesian DNN. The function of calculating the reliability score by the manipulation estimation model 1300 may be implemented as a part of the data analysis device 1040.

[0142] In the learning system 1000, the learning device 1030 can learn the operation estimation model 1300 based on a data set consisting of video information and operation information transmitted from the data collection device 1010. The learning device 1030 can learn the operation estimation model 1300 so as to predict time-series operation information that can capture a more moving image by reinforcement learning. In addition, the data analysis device 1040 extracts meaningful learning data from the data set transmitted from the data collection device 1010, and the learning device 1030 can efficiently re-learn the operation estimation model 1300 using this meaningful learning data.

[0143] Emotion score estimation model: 14 is a schematic diagram showing the configuration of an emotion score estimation model 1400. The emotion score estimation model 1400 is a neural network model that estimates an emotion score 1411 of video information 1401, and corresponds to the emotion classifier described above. For example, the emotion score estimation model 1400 can be trained according to the learning process shown in FIG.

[0144] FIG. 15 shows the relationship between the control target (frame rate, resolution, etc.) and the average impression score. Note that the predicted value output from the neural network model is shown by a solid line, and the variance is shown by dots. FIG. 15 shows the average impression score obtained when an operation model is learned by reinforcement learning so as to increase the impression score based on video data acquired at a certain frame rate, and the learned operation model is used to operate the camera. In the figure, data points indicated by black circles are points that already have data. Points with data have small variance. Points without data have large variance. Points with large variance and where a high score can be expected are more worth observing. For example, an implementation can be considered in which data is obtained from a point where the highest score can be expected after optimistically taking variance into account.

[0145] The emotional score estimation model 1400 may also output a reliability score for the estimated emotional score 1411. Alternatively, the data analysis device 1040 extracts meaningful learning data, such as learning data that has a predetermined or greater influence on the learning of the emotional score estimation model 1400, missing learning data, and data similar thereto, based on methods such as explanation of the basis for the prediction of the emotional score estimation model 1400 by XAI, reliability score calculation, influence function calculation, and data shortage estimation by Bayesian DNN. The function of calculating the reliability score by the emotional score estimation model 1400 may be implemented as a part of the data analysis device 1040.

[0146] In the learning system 1000, the learning device 1030 can learn the emotion score estimation model 1400 based on a data set consisting of video information and operation information transmitted from the data collection device 1010. The learning device 1030 can learn the emotion score estimation model 1400 by reinforcement learning so as to estimate a higher emotion score for video information that can inspire. In addition, the data analysis device 1040 extracts meaningful learning data from the data set transmitted from the data collection device 1010, and the learning device 1030 can efficiently re-learn the emotion score estimation model 1400 using this meaningful learning data.

[0147] H-3. Analysis of learning data In the learning system 1000, the data analysis device 1040 analyzes the learning data of the neural network model to be learned, and identifies meaningful learning data, such as learning data that has a certain or greater influence on learning, missing learning data, and data similar to these, and the learning device 1020 uses such meaningful learning data to efficiently learn or re-learn the neural network model.

[0148] The data analysis device 1040 may be constructed in either a cloud or an edge. For example, the data analysis device 1040 may be configured as an integrated device with the learning device 1030. In this case, the learning device 1030 internally analyzes the learning data that affects the learning of the neural network model, and requests the data collection device 1010 to transmit the learning data that has a predetermined or greater influence on the learning of the neural network model, the missing learning data, or data similar thereto.

[0149] Alternatively, the data analysis device 1040 may be configured as an integrated device with the data collection device 1010. In this case, the learning device 1030 provides the data collection device 1010 with information required for data analysis (for example, information on the neural network model that has already been trained at that point in time) when making a request to transmit training data, etc. Then, the data collection device 1010 analyzes the influence of the collected data on the neural network model, and transmits to the learning device 1030 training data that has a predetermined or greater influence on the training of the neural network model, missing training data, or data similar thereto.

[0150] Methods by which the data analysis device 1040 analyzes data include XAI, reliability score calculation of learning data, influence function calculation, and data deficiency estimation using Bayesian DNN.

[0151] The reliability score is a numerical representation of the accuracy of a predicted value by a neural network model. Among the neural network models described above, the observation prediction model 1100 and the operation model 1200 are configured to output a reliability score together with a predicted value.

[0152] In the learning system 1000, the data collection device 1010 uses the confidence scores to filter the learning data to be sent to the learning device 1030, but details of this will be given later.

[0153] There are several methods for calculating the reliability score of the output value of the neural network model, and the present disclosure is not limited to a specific calculation method. Here, three types of reliability score calculation methods (1) to (3) will be described.

[0154] (1) A neural network model trained to estimate the error of the output As shown in FIG. 16, a neural network model 1500 is trained to output an error of an original output as a confidence score.

[0155] (2) Bayesian estimation method Using random numbers such as image augmentation and dropout, perturbations are added to the input data and the configuration of intermediate neurons, and learning is performed to obtain correct predictions even under perturbations. During inference, estimations are made multiple times while adding perturbations. The larger the variance of the results, the lower the confidence score.

[0156] (3) Using predicted probabilities (for classification problems) For classification problems where predictions can be made with a probability between 0.0 and 1.0, results such as 0.0 or 1.0 indicate a high confidence score, for binary classification a confidence score of 0.5 (close to 50%) is considered high, and for multi-class classification a confidence score of low is considered low if the probability of the most probable class is low.

[0157] The influence function is a formulation of the influence that the presence or absence of individual learning data and perturbations have on the prediction results of the neural network model (for example, see Non-Patent Document 4). In addition, the Bayesian DNN is configured by combining Bayesian estimation and deep learning, and by using Bayesian estimation, it is possible to evaluate the uncertainty caused by a lack of data when the neural network model outputs the prediction results.

[0158] H-4. Data collection device configuration The data collection device 1010 is a variety of devices, including still cameras, video cameras, image sensors used in cameras, multi-function information terminals such as smartphones, TVs, headphones or earphones, game consoles, IoT devices such as refrigerators and washing machines, and mobile devices such as drones and robots.

[0159] Fig. 17 shows an example of the internal configuration of a data collection device 1010. The data collection device 1010 shown in Fig. 17 includes a sensor unit 1011, an operation input unit 1012, a control unit 1013, a log transmission unit 1014, and a data analysis unit 1015. However, Fig. 17 illustrates an abstraction of a representative functional configuration related to realizing the present disclosure among many types of data collection devices 1010, and each data collection device 1010 is assumed to include various components not shown.

[0160] The sensor unit 1011 is composed of an image sensor such as a CMOS sensor and other sensors equipped in the data collection device 1010, and performs observations such as taking images and videos. In addition, when the data collection device 1010 is mounted on a mobile device such as a robot or a drone, the sensor unit 1011 also includes various sensors equipped in the mobile device such as an IMU (Inertial Measurement Unit).

[0161] The operation input unit 1012 is a functional module that performs an input operation for adjusting operation information such as the shooting conditions in the data collection device 1010. The operation input unit 1012 is composed of operators such as buttons and knobs, a touch panel screen, etc. In addition, when the data collection device 1010 is mounted on a mobile device such as a robot or a drone, the operation input unit 1012 also includes a remote control used to remotely operate the mobile device.

[0162] The control unit 1013 performs overall control of the operation of the data collection device 1010. In addition, the control unit 1013 controls observation such as capturing images and videos in the sensor unit 1011 based on operation information input via the operation input unit 1012.

[0163] The log transmission unit 1014 transmits a data set consisting of an observation log observed by the sensor unit 1011 and an operation log input to the operation input unit 1012 to the learning device 1030. Basically, the log transmission unit 1014 transmits a data set to the learning device 103 in response to receiving a request signal requesting data transmission from the learning device 1030. The data collection device 1010 collects new data in response to receiving a request signal from the learning device 1030 and transmits it to the learning device 1030, but may also transmit data extracted from already collected data based on the request signal to the learning device 1030. Of course, the data collection device 1010 may also transmit data spontaneously, not in response to a request signal.

[0164] The data analysis unit 1015 analyzes the influence of a data set consisting of the observation log observed by the sensor unit 1011 and the operation log input to the operation input unit 1012 on the learning of each neural network model to be trained, and identifies whether the data is meaningful for learning, such as learning data that has a predetermined or greater influence on learning, missing learning data, or data similar to these.

[0165] The learning device 1030 may transmit a request signal specifying data that is useful for learning, or may transmit a request signal instructing analysis of collected data. In the former case, the data analysis unit 1015 checks whether a data set consisting of an observation log observed by the sensor unit 1011 and an operation log input to the operation input unit 1012 corresponds to the data specified in the request signal, and the log transmission unit 1014 transmits to the learning device 1030 only the data that matches the request.

[0166] In the latter case, the data analysis unit 1015 corresponds to the data analysis device 1040 in FIG. 10. In this case, the data analysis unit 1015 analyzes the learning data that affects the learning of the neural network model to be learned in the learning device 1030, and checks whether a data set consisting of the observation log observed by the sensor unit 1011 and the operation log input to the operation input unit 1012 corresponds to meaningful learning data such as learning data that has a predetermined or greater influence on the learning of the neural network model, missing learning data, or data similar thereto. Then, the log transmission unit 1014 transmits only the data set that is meaningful learning data to the learning device 1030. The data analysis unit 1015 analyzes the neural network model using, for example, at least one of XAI, confidence score, influence function, and Bayesian DNN, or a combination of two or more of them. For example, when the data analysis unit 1015 analyzes data using the reliability score, it receives information on the neural network model that has been trained up to the current time from the learning device 1030, and calculates the reliability score of the inference by the neural network model when the data set is input. For a method of calculating the reliability score, please refer to Section H-3 above. It can be said that observation logs and operation logs with a high reliability score are learning data that are not insufficient and have low value, while observation logs and operation logs with a low reliability score are learning data that are insufficient and have high value. By collecting learning data with a low reliability score and insufficient learning data and providing it to the learning device 1030, the learning device 103 side can efficiently learn and re-learn the neural network model.

[0167] Furthermore, the data analysis unit 1015 may feed back the analysis results to the sensor unit 1011 and the operation input unit 1012. The sensor unit 1011 and the operation input unit 1012 may capture images while changing the resolution, frame rate, brightness, color, angle of view, viewpoint position, or line of sight based on the degree of influence on learning of the neural network model to be learned.

[0168] Fig. 18 shows another example of the internal configuration of the data collection device 1010. The data collection device 1010 shown in Fig. 18 includes a sensor unit 1011, an operation input unit 1012, a control unit 1013, and a log transmission unit 1014. The main difference from the example of the configuration shown in Fig. 17 is that it does not include a data analysis unit 1015. The data collection device 1010 shown in Fig. 18 does not analyze the collected data; in other words, it transmits all the collected data to the learning device 1030 regardless of whether the collected data is learning data that is meaningful for learning or relearning a neural network model.

[0169] H-5. Configuration of the learning device Fig. 19 shows an example of the internal configuration of the learning device 1030. The learning device 1030 shown in Fig. 19 includes a model learning unit 1031, an observation / operation log accumulation unit 1032, an observation prediction model / operation model accumulation unit 1033, an operation estimation model accumulation unit 1034, an operation estimation unit 1035, and a data analysis unit 1036.

[0170] The model learning unit 1031 uses the learning data to learn various neural network models. Specifically, the model learning unit 1031 learns an observation prediction model (see FIG. 11), an operation model (see FIG. 12), an operation estimation model (see FIG. 13), an impression score estimation model (see FIG. 14), and the like. In the case of a cloud AI in which the learning device 1030 is installed on a cloud, the model learning unit 1031 may be configured using, for example, a plurality of computation nodes to perform distributed learning of the neural network model.

[0171] The observation / operation log accumulation unit 1032 accumulates the observation logs and operation logs transmitted from the data collection device 1010. It is assumed that the observation logs and operation logs transmitted from the data collection device 1010 are meaningful learning data.

[0172] The model learning unit 1031 learns an observation prediction model based on a data set consisting of observation logs and operation logs stored in the observation and operation log storage unit 1032. Then, the learned observation prediction model is stored in the observation prediction model and operation model storage unit 1033.

[0173] In addition, the model learning unit 1031 learns an operation model based on a data set consisting of observation logs and operation logs stored in the observation and operation log storage unit 1032. Then, the learned operation model is stored in the observation prediction model and operation model storage unit 1033.

[0174] Furthermore, the model learning unit 1031 learns an operation estimation model based on a data set made up of the observation logs and operation logs stored in the observation / operation log storage unit 1032. Then, the learned operation estimation model is stored in the operation estimation model storage unit 1034.

[0175] The data analysis unit 1036 corresponds to the data analysis device 1040 in FIG. 10. The data analysis unit 1036 analyzes the learning data that influences the learning of each trained neural network stored in the observation prediction model / operation model storage unit 1033 and the operation estimation model storage unit 1034. Then, based on the analysis result, a request signal is transmitted from the learning device 1030 to the data collection device 1010, requesting transmission of learning data that has a predetermined or greater influence, missing learning data, or data similar thereto. Therefore, the data collection device 1010 transmits an observation log and an operation log that are meaningful learning data. The data analysis unit 1036 analyzes the learning data based on a method such as XAI, confidence score calculation, influence function, or Bayesian DNN.

[0176] The operation estimation unit 1035 estimates time-series operation information for shooting a video shot by an expert who is well versed in camera operations, such as a professional cameraman, from the video (hereinafter also referred to as "professional video"). Then, a data set consisting of the professional video (observation log) input to the operation estimation unit 1035 and the operation information (estimated operation log) estimated by the operation estimation unit 1035 is accumulated in the observation and operation log accumulation unit 1032 as high-quality learning data for learning a professional-level operation model. Therefore, the model learning unit 1031 can learn an operation model capable of predicting professional-level camera operations, using the high-quality learning data accumulated in the observation and operation log accumulation unit 1032.

[0177] The professional video content is optional and is assumed to be provided to the cloud via the network. By collecting a large amount of high-quality learning data using professional video content, it will be possible to learn an operation model to achieve professional-level automatic camera operation.

[0178] Fig. 20 shows another example of the internal configuration of the learning device 1030. The learning device 1030 shown in Fig. 20 includes a model learning unit 1031, an observation / operation log accumulation unit 1032, an observation prediction model / operation model accumulation unit 1033, an operation estimation model accumulation unit 1034, and an operation estimation unit 1035. The main difference from the example of the configuration shown in Fig. 19 is that the data analysis unit 1036 is not included.

[0179] 20 does not analyze the learning data itself, but instead transmits information on the neural network model to be learned to the data collecting device 1010. In this case, the data analyzing unit 1015 on the data collecting device 1010 side analyzes the learning data that affects the learning of the neural network model to be learned in the learning device 1030, and checks whether a data set consisting of an observation log observed by the sensor unit 1011 and an operation log input to the operation input unit 1012 corresponds to meaningful learning data such as learning data that has a predetermined or greater influence on the learning of the neural network model, missing learning data, or data similar thereto (as described above). Then, only the data set that is meaningful learning data is transmitted from the data collecting device 1010 to the learning device 1030.

[0180] Fig. 21 shows yet another example of the internal configuration of the learning device 1030. The learning device 1030 shown in Fig. 21 includes a model learning unit 1031, an observation and operation log accumulation unit 1032, an observation prediction model and operation model accumulation unit 1033, an operation estimation model accumulation unit 1034, an operation estimation unit 1035, and a data analysis unit 1037. The main difference from the configuration examples shown in Fig. 19 and Fig. 20 is that the learning device 1030 receives all collected data from the data collection device 1010 having the configuration example shown in Fig. 18, and checks in the data analysis unit 1037 whether each received data is meaningful learning data or not.

[0181] The data analysis unit 1037 corresponds to the data analysis device 1040 in FIG. 10. The data analysis unit 1037 analyzes the influence of each received data stored in the observation and operation log storage unit 1032 on the learning of each trained neural network stored in the observation prediction model and operation model storage unit 1033 and the operation estimation model storage unit 1034. Then, the data analysis unit 1037 extracts only meaningful learning data from the observation and operation log storage unit 1032, such as learning data that has a predetermined or greater influence on the learning of each neural network model, missing learning data, and data similar thereto, and outputs the data to the model learning unit 1031. Therefore, the model learning unit 1031 can efficiently learn and re-learn various neural network models using the meaningful learning data.

[0182] H-6. Configuration of the model utilization device 22 shows an example of the internal configuration of the model utilization device 1020. The model utilization device 1020 includes a sensor unit 1021, an automatic operation unit 1022, a control unit 1023, and a presentation unit 1024.

[0183] The sensor unit 1021 is made up of an image sensor such as a CMOS sensor, or other sensors equipped in the model utilization device 1020, and performs observations including taking images and videos. In addition, when the model utilization device 1020 is mounted on a mobile device such as a robot or drone, the sensor unit 1021 also includes various sensors equipped in the mobile device, such as an IMU.

[0184] The automatic operation unit 1022 uses the operation model provided by the learning device 1030 to predict operation information for the next time from the observation information of the sensor unit 1021 (video information up to the current time and operation information up to the current time), and outputs data analysis results such as a reliability score of the predicted operation information.

[0185] The control unit 1023 controls observation, such as capturing images or videos, in the sensor unit 1011 based on operation information for the next time predicted by the automatic operation unit 1022.

[0186] When an operation model that has been sufficiently learned in the learning device 1030 is provided, even if the user of the model utilization device 1020 himself / herself is not familiar with camera operation, he / she can take pictures equivalent to those of an expert who is familiar with camera operation, such as a professional cameraman, based on the operation information predicted by the automatic operation unit 1022.

[0187] The presentation unit 1024 presents the data analysis results, such as a reliability score of the operation information predicted by the automatic operation unit 1022. The user of the model utilization device 1020 can judge from the presented reliability score whether the video is shot with professional skill by the automatic operation.

[0188] H-7. Edge AI Configuration It is also assumed that the model utilization device 1020 that utilizes a trained neural network model is configured integrally with the functions of a learning device that trains the neural network model (i.e., an edge AI). Fig. 23 shows an example of the internal configuration of the model utilization device 1020 as an edge AI. The illustrated model utilization device 1020 includes a sensor unit 1021, an automatic operation unit 1022, a control unit 1023, and a presentation unit 1024, as well as a model learning unit 1031, an observation / operation log accumulation unit 1032, an observation / prediction model / operation model accumulation unit 1033, an operation estimation model accumulation unit 1034, and an operation estimation unit 1035.

[0189] The observation and operation log storage unit 1032 stores the observation logs and operation logs transmitted from the data collecting device 1010. The model learning unit 1031 learns an observation prediction model based on a data set made up of the observation logs and operation logs stored in the observation and operation log storage unit 1032. The model learning unit 1031 also learns an operation model based on a data set made up of the observation logs and operation logs stored in the observation and operation log storage unit 1032. Then, the learned observation prediction model and operation model are stored in the observation prediction model and operation model storage unit 1033.

[0190] Moreover, the model learning unit 1031 learns an operation estimation model based on a data set consisting of the observation log and the operation log stored in the observation and operation log storage unit 1032. Then, the learned operation estimation model is stored in the operation estimation model storage unit 1034. The operation estimation unit 1035 uses the learned operation estimation model to estimate time-series operation information for shooting a video shot by an expert who is well versed in camera operations, such as a professional cameraman, and stores the information in the observation and operation log storage unit 1032 as high-quality learning data for learning a professional-level operation model. Therefore, the model learning unit 1031 can learn an operation model capable of predicting professional-level camera operations, using the high-quality learning data stored in the observation and operation log storage unit 1032.

[0191] The sensor unit 1021 is made up of an image sensor such as a CMOS sensor, or other sensors equipped in the model utilization device 1020, and performs observations including capturing images and videos.

[0192] The automatic operation unit 1022 uses the operation model read out from the observation prediction model / operation model storage unit 1033 to predict operation information at the next time from the observation information of the sensor unit 1021 (video information up to the current time and operation information up to the current time), and outputs data analysis results such as a reliability score of the predicted operation information.

[0193] The control unit 1023 controls the observation of capturing images and videos in the sensor unit 1011 based on the operation information for the next time predicted by the automatic operation unit 1022. When an operation model that has been sufficiently learned in the learning device 1030 is provided, even if the user of the model utilization device 1020 himself / herself is not familiar with camera operation, he / she can take pictures based on the operation information predicted by the automatic operation unit 1022 at the same level as an expert who is familiar with camera operation.

[0194] The presentation unit 1024 presents the results of the data analysis, such as a reliability score of the operation information predicted by the automatic operation unit 1022. The model utilization device 1020 can determine, from the presented reliability score, whether or not the video will be shot with professional skill by the automatic operation. [Industrial Applicability]

[0195] Although the present disclosure has been described in detail above with reference to specific embodiments, it is obvious that those skilled in the art can modify or substitute the embodiments without departing from the gist of the present disclosure.

[0196] The present disclosure can be applied to the collection process of data used to train a machine learning model that generates moving content for humans. Data collected based on the present disclosure is used as training data for training a machine learning model (e.g., a neural network model) for handling moving content for humans (specifically, for generating moving content or estimating camera operations for shooting moving content), but of course it can also be used for training machine learning models for other purposes.

[0197] In addition, data collected based on the present disclosure can be used to train a machine learning model that identifies the emotions that content evokes in humans, and an emotion identifier equipped with the machine learning model thus developed can be installed in content processing devices such as digital cameras.

[0198] In short, the present disclosure has been described in the form of examples, and the contents of the specification should not be interpreted as being limited. In order to judge the gist of the present disclosure, the claims should be taken into consideration.

[0199] The present disclosure can also be configured as follows.

[0200] (1) A data collection device that collects data; and a learning device that uses the data collected by the data collection device to learn a machine learning model, The learning device re-learns the machine learning model using learning data that has a predetermined or greater influence on the learning of the machine learning model, missing learning data, or data similar thereto, which is collected based on a result of analyzing the learning data that influences the learning of the machine learning model. Learning system.

[0201] (2) The learning device learns a machine learning model that estimates a camera operation for generating moving content or for shooting moving content. A learning system as described in (1) above.

[0202] (3) Perform the analysis using XAI, confidence score calculation, influence function, or Bayesian DNN; A learning system according to either (1) or (2) above.

[0203] (4) The learning device analyzes learning data that affects the learning of the machine learning model, and based on the analysis result, transmits a request signal to the data collection device to request the transmission of learning data that has a predetermined or greater effect on the learning of the machine learning model, missing learning data, or data similar thereto; the data collection device transmits data collected based on the received request signal to the learning device; The learning device re-learns the machine learning model based on the data transmitted from the data collection device in response to the request signal. A learning system according to any one of (1) to (3) above.

[0204] (5) The data collection device is a camera or imager that captures images, and transmits image data captured by changing the resolution, frame rate, brightness, color, angle of view, viewpoint position, or line of sight direction based on the degree of influence on the learning of the machine learning model to the learning device. A learning system according to any one of (1) to (4) above.

[0205] (6) the learning device transmits a request signal to the data collection device to request transmission of the learning data of the machine learning model; The data collection device transmits to the learning device, based on a result of analyzing the influence of the collected data on the learning of the machine learning model, learning data that has a predetermined or greater influence on the learning of the machine learning model, missing learning data, or data similar thereto, from the collected data; The learning device re-learns the machine learning model based on the data transmitted from the data collection device in response to the request signal. A learning system according to any one of (1) to (3) above.

[0206] (7) The learning device transmits information required for the analysis to the data collection device when the request signal is transmitted. A learning system as described in (6) above.

[0207] (8) a receiving unit that receives a request signal requesting transmission of learning data of the machine learning model from a learning device that learns the machine learning model; A data collection unit that collects learning data that has a predetermined or greater influence on learning of the machine learning model, missing learning data, or data similar thereto in response to receiving the request signal; a transmission unit that transmits the data collected by the data collection unit to the learning device; A data collection device comprising:

[0208] (9) The receiving unit receives, from the learning device, the request signal requesting learning data that has a predetermined or greater effect on learning of the machine learning model, missing learning data, or data similar thereto; The data collection unit collects data based on the received request signal, The transmission unit transmits the data collected by the data collection unit to the learning device. The data collection device according to (8) above.

[0209] (10) The data collection unit collects new data based on the received request signal, or the transmission unit transmits data extracted from the data already collected by the data collection unit based on the request signal. The data collection device according to (9) above.

[0210] (11) Further comprising an analysis unit that analyzes the effect of the data collected by the data collection unit on learning of the machine learning model, The transmission unit transmits to the learning device, based on the analysis result by the analysis unit, learning data that has a predetermined or greater effect on learning of the machine learning model, missing learning data, or data similar thereto, among the data collected by the data collection unit. The data collection device according to (8) above.

[0211] (12) The data collection unit collects image data captured by changing the resolution, frame rate, brightness, color, angle of view, viewpoint position, or line of sight of a camera or imager that captures an image based on the degree of influence on learning of the machine learning model. A data collection device according to any one of (8) to (11) above.

[0212] (13) a segment extraction unit that extracts segments from the content based on content evaluation information of the content and biometric information of a person viewing the content; an expression identification unit that detects the face of a person appearing in the segment and identifies the facial expression; a relationship estimation unit that estimates a relationship between people appearing in the segment; an emotion identification unit that identifies emotion labels of the segments based on facial expressions of people appearing in the segments and relationships between the people; An information processing device comprising:

[0213] (14) The biological information includes at least electroencephalogram information. The information processing device according to (13) above.

[0214] (15) The segment extraction unit extracts a segment of the content in which a high rating and biometric information having a positive emotion match. 13. The information processing device according to claim 12, wherein the first and second information processing devices are disposed in a first and second spaced apart position.

[0215] (16) The segment extraction unit further extracts a segment in which high ratings and biometric information having positive emotions do not match. The information processing device according to (15) above.

[0216] (17) The relationship estimation unit estimates a relationship between people appearing in a segment based on a context of the previous and next segments and facial expressions of the people. An information processing device according to any one of (13) to (16) above.

[0217] (18) The relationship estimation unit estimates the relationship between people appearing in the current segment by using, as the context, face detection of people appearing in past segments and facial expression identification results of the detected faces. The information processing device according to (17) above.

[0218] (19) A first emotion analysis unit that analyzes emotions based on text information obtained by recognizing a voice included in the segment, The emotion identification unit identifies an emotion label for the segment by further considering the emotion identified by the first emotion analysis unit from the text information. 19. An information processing device according to any one of (13) to (18) above.

[0219] (20) A second emotion analysis unit is further provided for detecting music included in the segment and analyzing an emotion imparted by the music, The emotion identification unit identifies an emotion label for the segment by further considering the emotion identified from the music by the second emotion analysis unit. 19. An information processing device according to any one of claims 13 to 19.

[0220] (21) a segment extraction step of extracting segments from the content based on content evaluation information of the content and biometric information of a person viewing the content; an expression identification step of detecting a face of a person appearing in the segment and identifying a facial expression; A relationship estimation step of estimating a relationship between people appearing in the segment; an emotion identification step of identifying emotion labels for the segments based on facial expressions of people appearing in the segments and relationships between the people; An information processing method comprising the steps of:

[0221] (22) a segment extraction unit that extracts segments from the content based on content evaluation information of the content and biometric information of a person viewing the content; an expression recognition unit that detects the faces of people in the segments and recognizes their facial expressions; a relationship estimation unit that estimates relationships between people appearing in the segments; an emotion identification unit that identifies emotion labels of the segments based on facial expressions of people appearing in the segments and relationships between people; A computer program written in a computer-readable form to cause a computer to function as a

[0222] (23) A method for generating a trained machine learning model for identifying an emotion imparted by content, comprising: inputting content into a machine learning model; acquiring video features estimated from the content by the machine learning model; obtaining audio features estimated from the content by the machine learning model; acquiring text features of speech estimated from the content by the machine learning model; identifying an emotion label for the content based on the video features, audio features, and text features estimated by the machine learning model; Calculating a loss function based on an error between the identified emotion label and an emotion label assigned to the content; updating parameters of the machine learning model based on the loss function; A method for generating a trained machine learning model having the following structure:

[0223] (24) The video feature includes, when two or more people appear in a video frame, a relationship between the people estimated based on a context of multiple consecutive frames. A method for generating a trained machine learning model as described in (23) above. [Explanation of symbols]

[0224] 100...Data collection system, 101...Video content 102: content evaluation information acquisition unit, 103: biometric information acquisition unit 104: comparison unit; 105: segment extraction unit 106 ... context extraction unit, 107 ... speech recognition unit 108...music detection unit, 109...face detection unit 110: first emotion analysis unit; 111: second emotion analysis unit 112... facial expression recognition unit, 113... relationship estimation unit 114…Emotional Identification Department 410...data storage unit, 420...emotion identification device 421: network unit, 422: identification unit, 423: evaluation unit 500: digital camera, 501: optical system, 502: imaging unit 503: AFE unit, 504: camera signal processing unit, 506: main processing unit 514: microphone, 515: A / D converter, 516: display 517: sound playback unit, 518: recording unit 1000...Learning system, 1010...Data collection device 1011: sensor unit, 1012: operation input unit, 1013: control unit 1014: Log transmission unit, 1015: Data analysis unit 1020: model utilization device, 1021: sensor section 1022: automatic operation unit, 1023: control unit, 1024: display unit 1030... learning device, 1031... model learning unit 1032: Observation and operation log storage unit 1033…Observation and prediction model / operation model storage unit 1034: Operation estimation model storage unit, 1035: Operation estimation unit 1036…Data Analysis Department, 1037…Data Analysis Department 1040...Data analysis equipment 1100: Observation and prediction model, 1200: Operational model 1300: Operation estimation model, 1400: Emotion score estimation model

Claims

1. A learning system for learning a machine learning model that estimates camera operations for generating moving content or shooting moving content, comprising: The present invention includes a data collection device that collects data, and a learning device that uses the data collected by the data collection device to learn a machine learning model, The data collection device collects learning data including image data captured by a camera and operation information of the camera at the time of capturing an image, The learning device re-learns the machine learning model using learning data that has a predetermined or greater influence on the learning of the machine learning model, missing learning data, or data similar thereto, which is collected based on a result of analyzing the learning data that influences the learning of the machine learning model. Learning system.

2. The analysis is performed by XAI (Explainable AI), confidence score calculation, influence function, or Bayesian DNN. The learning system according to claim 1 .

3. The learning device analyzes learning data that affects the learning of the machine learning model, and based on the analysis result, transmits a request signal to the data collection device to request transmission of learning data that has a predetermined or greater effect on the learning of the machine learning model, missing learning data, or data similar thereto; the data collection device transmits data collected based on the received request signal to the learning device; The learning device re-learns the machine learning model based on the data transmitted from the data collection device in response to the request signal. The learning system according to claim 1 .

4. The data collection device is a camera or imager that captures images, and transmits image data captured by changing resolution, frame rate, brightness, color, angle of view, viewpoint position, or line of sight direction based on the degree of influence on learning of the machine learning model to the learning device. The learning system according to claim 1 .

5. The learning device transmits a request signal to the data collection device to request transmission of learning data of the machine learning model; The data collection device transmits to the learning device, based on a result of analyzing the influence of the collected data on the learning of the machine learning model, learning data that has a predetermined or greater influence on the learning of the machine learning model, missing learning data, or data similar thereto, from the collected data; The learning device re-learns the machine learning model based on the data transmitted from the data collection device in response to the request signal. The learning system according to claim 1 .

6. The learning device transmits information necessary for the analysis to the data collection device when the request signal is transmitted. The learning system according to claim 5 .

7. A receiving unit that receives a request signal requesting transmission of training data of the machine learning model from the learning device in the learning system according to claim 1; A data collection unit that collects learning data that has a predetermined or greater influence on learning of the machine learning model, missing learning data, or data similar thereto in response to receiving the request signal; a transmission unit that transmits the data collected by the data collection unit to the learning device; A data collection device comprising:

8. The receiving unit receives the request signal from the learning device, requesting learning data that has a predetermined or greater effect on learning of the machine learning model, missing learning data, or data similar thereto; The data collection unit collects data based on the received request signal, The transmission unit transmits the data collected by the data collection unit to the learning device. The data collection device of claim 7.

9. The data collection unit collects new data based on the received request signal, or the transmission unit transmits data extracted from the data already collected by the data collection unit based on the request signal. The data collection device of claim 8.

10. Further comprising an analysis unit that analyzes the effect of the data collected by the data collection unit on learning of the machine learning model, The transmission unit transmits to the learning device, based on the analysis result by the analysis unit, learning data that has a predetermined or greater effect on learning of the machine learning model, missing learning data, or data similar thereto, among the data collected by the data collection unit. The data collection device of claim 7.

11. The data collection unit collects image data captured by changing a resolution, frame rate, brightness, color, angle of view, viewpoint position, or line of sight of a camera or imager that captures an image based on a degree of influence on learning of the machine learning model. The data collection device of claim 7.

12. A segment extraction unit that extracts segments from content based on content evaluation information of the content and biometric information of a person viewing the content; an expression identification unit that detects the face of a person appearing in the segment and identifies the facial expression; a relationship estimation unit that estimates a relationship between people appearing in the segment; an emotion identification unit that identifies emotion labels of the segments based on facial expressions of people appearing in the segments and relationships between the people; Equipped with The emotion identification unit identifies an emotion label using an emotion score estimation model that has been trained by the learning device in the learning system according to claim 1. Information processing device.

13. The biological information includes at least electroencephalogram information. The information processing device according to claim 12.

14. The segment extraction unit extracts segments of the content in which high ratings and biometric information indicating positive emotions match. The information processing device according to claim 12.

15. The segment extraction unit further extracts a segment in which high ratings and biometric information having positive emotions do not match. The information processing device according to claim 14.

16. The relationship estimation unit estimates a relationship between people appearing in a segment based on a context of previous and next segments and facial expressions of the people. The information processing device according to claim 12.

17. The relationship estimation unit estimates the relationship between people appearing in a current segment by using, as the context, face detection of people appearing in a past segment and facial expression identification results of the detected faces. The information processing device according to claim 16.

18. Further comprising a first emotion analysis unit configured to analyze emotion based on text information obtained by recognizing a voice included in the segment; The emotion identification unit identifies an emotion label for the segment by further considering the emotion identified by the first emotion analysis unit from the text information. The information processing device according to claim 12.

19. The present invention further comprises a second emotion analysis unit that detects music included in the segment and analyzes an emotion imparted by the music; The emotion identification unit identifies an emotion label for the segment by further considering the emotion identified from the music by the second emotion analysis unit. The information processing device according to claim 12.

Images