Information processing device and information processing method

Abstract

To improve the accuracy of extracting moving scenes that move a user.SOLUTION: The present invention is an information processing device 100 that includes an analysis unit 3 that analyzes a response output from a generation AI 9 that receives an inquiry regarding image data extracted from video data for each piece of image data and outputs the analysis results as a vector, an inference unit 4 that inputs the image data into an inference model 8 and assigns an emotional label output from the inference model 8 to the image data, and an extraction unit 5 that extracts emotional scenes from the video data using the vector and the emotional label.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an information processing device and an information processing method. [Background technology]

[0002] In recent years, technological developments aimed at helping users, such as foreign tourists, create memories have been progressing, and related inventions have been published. For example, Patent Document 1 discloses an emotion information providing device that records the level of emotion experienced by a user in real time. According to the invention of Patent Document 1, emotion information is recorded that associates a user ID with peripheral information indicating the date and time of the emotion and the device's location at that date and time, and the user's emotion level at that date and time. Furthermore, additional information is added to the emotion information based on a request from the device. Furthermore, emotion information for users that matches search criteria using peripheral information is extracted. Furthermore, the user's emotion level is determined by the number of times the device's hardware buttons are operated per unit time, the number of times the device is shaken per unit time, the number of times the device is pressed per unit time, the number of images taken per unit time, etc. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2016-71394 Summary of the Invention [Problem to be solved by the invention]

[0004] The invention of Patent Document 1 can be said to be an invention in which the level of emotion is determined mainly by the user's terminal operation. Therefore, according to the invention of Patent Document 1, if the user operates the terminal excessively, there is a risk of erroneously determining that the user is emotional (high level of emotion) even when the user is not actually emotional. Also, there is a risk of erroneously determining that the user is not emotional (low level of emotion) if the user is so emotional that they forget to operate the terminal. In other words, conventional technologies including the invention of Patent Document 1 cannot improve the accuracy of extracting emotional scenes that move the user. In view of the above circumstances, an object of the present invention is to improve the accuracy of extracting moving scenes that move users. [Means for solving the problem]

[0005] The present invention, which solves the above problems, an analysis unit that analyzes a response output from the generation AI that has received an inquiry regarding image data extracted from video data for each of the image data and outputs the analysis result as a vector; an inference unit that inputs the image data into an inference model and assigns an emotion label output from the inference model to the image data; and an extraction unit that extracts emotional scenes from the video data using the vectors and the emotional labels.

[0006] The present invention also provides The information processing device A step of analyzing a response output from a generation AI that has received an inquiry regarding image data extracted from video data for each of the image data, and outputting the analysis result as a vector; inputting the image data into an inference model and assigning an emotion label output from the inference model to the image data; and extracting emotional scenes from the video data using the vectors and the emotional labels. [Effects of the Invention]

[0007] According to the present invention, it is possible to improve the accuracy of extracting moving scenes that move a user. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a diagram illustrating an example of a functional configuration of an information processing device according to an embodiment of the present invention. [Figure 2] 1 is an example of a flowchart illustrating an information processing method according to the present embodiment. [Figure 3] 10 is an example of a flowchart illustrating an analysis process. [Figure 4] 10 is an example of a flowchart illustrating an extraction process. [Figure 5] FIG. 10 is an explanatory diagram of a specific example of emotional scene extraction. DETAILED DESCRIPTION OF THE INVENTION

[0009] [composition] FIG. 1 is an example of a functional configuration diagram of an information processing device according to this embodiment. The information processing device 100 is a computer that edits video data. The information processing device 100 includes hardware such as an input unit, an output unit, a control unit, and a storage unit. For example, if the control unit is configured from a CPU (Central Processing Unit), information processing by the computer including the control unit is realized by program execution processing by the CPU. Furthermore, the storage unit included in the computer stores various programs for realizing the functions of the computer in response to instructions from the CPU. This realizes collaboration between software and hardware. The programs can be provided by recording them on a recording medium or via a network. The output unit may include the function of a display unit that displays information on a screen.

[0010] As shown in FIG. 1, the information processing device 100 includes a cutout unit 1, a query unit 2, an analysis unit 3, an inference unit 4, and an extraction unit 5. The information processing device 100 also stores a video DB 6, an emotion DB 7, and multiple types of inference models 8. "DB" stands for database. The information processing device 100 can also use a generation AI 9 (AI: Artificial Intelligence) via a network (e.g., the Internet). The information processing device 100 has an access key and password for accessing the generation AI 9. There may be multiple types of generation AI 9. The information processing device 100 has an access key and password for each generation AI 9.

[0011] The cropping unit 1 crops image data from video data. The cropping interval can be set as appropriate. The video data may be, for example, but is not limited to, data taken by a user (e.g., a foreign tourist) using a fixed camera or a smartphone with a camera function, or data uploaded from a separate information terminal. The query unit 2 transmits the image data cut out by the cutout unit 1 to the generation AI 9 and makes an inquiry (e.g., a prompt) about the image data. The generation AI 9 outputs a response to the inquiry and transmits it to the information processing device 100. The analysis unit 3 analyzes the answer output from the generation AI 9 for each image data.

[0012] The inference unit 4 inputs the image data cut out by the cutout unit 1 into the inference model 8 and assigns the emotion label output from the inference model 8 to the image data. The emotion label is a marker that indicates whether the target image data will move the user. For example, emotion labels can be prepared as "0" (not moved), "S" (start of emotion), "R" (currently moved), or "E" (end of emotion), but are not limited to these. For example, the state in which a subject is moved can be captured by quantifying characteristics such as the subject's heart rate, facial expression, voice, and movement, and a rule can be established that image data in which the numerical value of each characteristic exceeds a threshold will be assigned an emotion label of "S," "R," or "E." The extraction unit 5 uses the vectors from the analysis unit 3 and the emotion labels from the inference unit 4 to extract emotional scenes from the video data.

[0013] The video DB 6 stores video data. The emotion DB 7 stores samples of emotion parameters, which are parameters that express emotions. The output format of the analysis results of the analysis unit 3 matches the output format of the emotion parameters. The inference model 8 is a learning device that receives image data as input and outputs emotion labels.

[0014] In this embodiment, the inference model 8 has learned the correspondence between image data and emotion labels, for example, using LSTM (Long Short Term Memory). More specifically, "correct image data" is prepared, which is image data to which an emotion label of "0," "S," "R," or "E" has already been assigned. The inference model 8 performs arithmetic processing on the input correct image data using a CNN (Convolutional Neural Network) layer, a Pooling layer, a vector conversion, an LSTM layer, an Affine layer, and a Softmax layer. The inference model 8 also associates the output results of the arithmetic processing with the emotion label for each correct image data. Note that the neural network used in the inference model 8 is not limited to LSTM.

[0015] In this embodiment, the query made by the query unit 2 is text data. The response output by the generation AI 9 is text data. In this embodiment, the analysis unit 3 can be implemented as, for example, BERT (Bidirectional Encoder Representations from Transformers). However, this is not limiting, and the analysis unit 3 may be implemented as other word embedding language modeling. When BERT is used, the analysis result of the analysis unit 3 is output as a vector. The output format of the emotion parameter samples stored in the emotion DB 7 is a vector.

[0016] The query unit 2 can make inquiries to the generation AI 9 regarding the correct image data in the form of text data. The generation AI 9 can output responses to the inquiries in the form of text data. The analysis unit 3 can analyze the text data responses output from the generation AI 9 and output the analysis results as vectors. The vectors output by the analysis unit 3 for the correct image data are called "emotion vectors." The emotion DB 7 can store emotion vectors as samples of emotion parameters.

[0017] The video DB 6 can store management data in association with the video data. The management data can include, for example, an "experience category" that classifies the video content of the video data (e.g., pottery, calligraphy, or washi paper making), a "user ID" that identifies the owner of the video data, a "video start time" that indicates when filming of the video data started, a "video end time" that indicates when filming of the video data ended, and a "file path" that indicates where the video data is stored, but is not limited to these. The image data cut out from the video data by the cutout unit 1 and the correct image data can be classified by experience category. The emotion DB 7 can store emotion vectors in association with emotion labels. That is, the emotion DB 7 can store, for each emotion vector, the image ID of the correct image data and the emotion label associated with the correct image data by the inference model 8 in association with each other. For example, multiple types of inference models 8 can be prepared for each experience category.

[0018] [process] The following describes the process performed by the information processing device 100 during prediction. Fig. 2 is a flowchart showing an example of an information processing method according to this embodiment. The information processing device 100 reads video data from the video DB 6 and starts processing.

[0019] First, the cutout unit 1 of the information processing device 100 performs cutout processing on the video data (step A1). Specifically, the cutout unit 1 acquires (cuts out) image data at predetermined intervals from the start of the video data, that is, image data at predetermined number of frames from the start. The cutout unit 1 can divide one video data into multiple chapters and set the cutout image data as the cue position of the chapter.

[0020] Next, the query unit 2 of the information processing device 100 performs query processing on the cut-out image data (step A2). Specifically, the query unit 2 accesses the generation AI 9 and queries about the image data. The query is text data. The generation AI 9 outputs a response to the query for each image data. The query unit 2 can obtain the response from the generation AI 9. The response is text data. Next, the analysis unit 3 of the information processing device 100 performs an analysis process on the acquired answer (step A3). Details of the analysis process will be described later.

[0021] Furthermore, in parallel with steps A2 and A3, the inference unit 4 of the information processing device 100 performs inference processing on the cut-out image data (step A4). Specifically, the inference unit 4 inputs the image data to the inference model 8. The inference model 8 outputs an emotion label for each image data. The inference unit 4 assigns the output emotion label to the corresponding image data. Note that it is preferable that the inference model 8 selects data whose experience category matches that of the loaded video data. Thereafter, the extraction unit 5 of the information processing device 100 performs extraction processing (step A5). Details of the extraction processing will be described later. This completes the processing in FIG.

[0022] (Details of the analysis process) FIG. 3 is an example of a flowchart showing the analysis process. First, the analysis unit 3 performs loop processing on the image data for which an answer has been obtained from the generation AI 9 (step B1). Next, the analysis unit 3 analyzes the answer text data for the target image data using language processing by BERT, and outputs the analysis result as a vector (step B2). Next, the analysis unit 3 sets the judgment flag for the target image data to "F" (step B3). The judgment flag is a flag that indicates whether the vector resulting from the analysis corresponds to an emotional vector ("T") or not ("F"). A judgment flag can be prepared for each image data.

[0023] Next, the analysis unit 3 performs a loop process on the emotion vectors stored in the emotion DB 7 (step B4). Next, the analysis unit 3 calculates the cosine similarity between the vector output in step B2 and the emotion vector targeted in step B4 (step B5). If the calculated cosine similarity is equal to or greater than the threshold (Yes in step B6), the analysis unit 3 sets the judgment flag to "T" (step B7). On the other hand, if the calculated cosine similarity is less than the threshold (No in step B6), the judgment flag remains "F". A vector with the judgment flag set to "T" is treated as a vector expressing emotion. A vector with the judgment flag set to "F" is treated as a vector not expressing emotion.

[0024] As a result, the judgment flag "F" or "T" is set for the image data for which a response is obtained from the generation AI 9. Note that, as a result of the loop processing (step B4), there may be image data that shows a cosine similarity above the threshold for two or more impression vectors, that is, there may be image data for which the judgment flag "T" may be set multiple times. In this case, for example, it is preferable to assign the impression vector with the maximum cosine similarity as the vector output as the analysis result. However, the assignment method is not limited to this. This completes the processing in FIG.

[0025] (Details of extraction process) FIG. 4 is an example of a flowchart showing the extraction process. First, the extraction unit 5 performs a loop process on the image data for which a response is obtained from the generation AI 9 (step C1). Next, the extraction unit 5 sets the emotional scene flag for the target image data to "0" (step C2). The emotional scene flag indicates the intensity of the emotion imparted by the target image data. The emotional scene flag can take values ​​of 0, 1, 2, etc., and the larger the value, the greater the intensity of the emotion.

[0026] Next, the extraction unit 5 determines whether the emotional label assigned to the target image data is one of "S," "R," or "E" (step C3). If it is one of "S," "R," or "E" (Yes in step C3), the extraction unit 5 adds "1" to the emotional scene flag (step C4). Then, the process proceeds to step C5. On the other hand, if it is not one of "S," "R," or "E" (No in step C3), that is, if the emotional label is "0," the value of the emotional scene flag is maintained. Then, the process proceeds to step C5.

[0027] Next, the extraction unit 5 determines whether the determination flag set for the target image data is "T" (step C5). If the determination flag is "T" (Yes in step C5), the extraction unit 5 adds "1" to the moving scene flag (step C6). On the other hand, if the determination flag is not "T" (No in step C5), that is, if the determination flag is "F", the value of the moving scene flag is maintained.

[0028] As a result, the value of the touching scene flag is set for the image data for which a response is obtained from the generation AI 9. The extraction unit 5 then determines whether or not there is a time period for the entire video data of interest in which the total value indicated by the touching scene flag is equal to or greater than a threshold value (step C7). For example, if the image data for one video data is arranged in chronological order according to the time at which the extracted image data was obtained, a distribution of the values ​​of the touching scene flag can be obtained. Therefore, the extraction unit 5 can calculate the total value of the touching scene flag values ​​included in a predetermined time range by applying the distribution to the predetermined time range. In step C7, the extraction unit 5 can determine whether or not the calculated total value is equal to or greater than a predetermined threshold value while shifting the time range as appropriate. The time range of the time period can be determined as appropriate.

[0029] If the time period exists (Yes in step C7), the extraction unit 5 determines the portion of the target video data that is included in the time period, i.e., the clipping of the time period, as a touching scene (step C8). Then, the processing of FIG. 4 ends. As a result, the extraction unit 5 is able to extract a touching scene from the video data. On the other hand, if the time period does not exist (No in step C7), the processing of FIG. 4 ends as is. As a result, no touching scene is extracted.

[0030] [Specific example] A specific example of this embodiment will be described. FIG. 5 is an explanatory diagram of a specific example of emotional scene extraction. The information processing device 100 reads video data 10 from the video DB 6. The clipping unit 1 clips image data 11-20 from the video data 10. The query unit accesses the generation AI 9 and makes a query 30 regarding the image data 11-20. The query 30 is text data such as "What is happening at this moment in the image?" The generation AI 9 makes answers 41-50 to the query 30. The answers 41-50 are text data and correspond to the image data 11-20, respectively. Note that the arrows in FIG. 5 indicate the input and output of information.

[0031] The analysis unit 3 analyzes the answers 41 to 50 and outputs vectors 61 to 70. The vectors 61 to 70 correspond to the image data 11 to 20, respectively. The analysis unit 3 also references the impression DB 7 and sets a determination flag for each of the vectors 61 to 70. As a result, the determination flag "F" is set for the vectors 61, 62, 67 to 70, and the determination flag "T" is set for the vectors 63 to 66 (shown in bold).

[0032] The inference unit 4 inputs image data 11 to 20 into the inference model 8. The inference model 8 outputs emotion labels 81 to 90. The emotion labels 81 to 90 correspond to the image data 11 to 20, respectively. Furthermore, the values ​​of the emotion labels 81, 82, 87 to 90 are "0", the value of the emotion label 83 is "S", the value of the emotion labels 84 and 85 is "R", and the value of the emotion label 86 is "E".

[0033] The extraction unit 5 extracts clippings of time periods including image data 13 to 16 as emotional scenes using vectors 61 to 70 and emotion labels 81 to 90. Note that the start and end times of the time periods and the number of time periods in step C7 of Fig. 4 can be set as appropriate. The analysis process (step A3 in FIG. 2) and the inference process (step A4 in FIG. 2) are performed separately. For this reason, there may be image data in which the judgment flag is set to "T" but the emotion label is assigned "0." There may also be image data in which the judgment flag is set to "F" but the emotion label is assigned one of "S," "R," or "E." The extraction unit 5 considers such image data to express emotion and does not prevent it from being incorporated into an emotional scene.

[0034] According to this embodiment, it is possible to improve the accuracy of extracting moving scenes that move the user. More specifically, the accuracy of extracting emotional scenes can be improved by introducing an ensemble of the inference results (emotional labels) when using the inference model 8 and the analysis results (vectors) when using the generation AI 9. Furthermore, by using the generation AI 9, the work time required to extract emotional scenes can be shortened even with high extraction accuracy.

[0035] [Other embodiments] A plurality of types of generation AIs 9 may be prepared for use by the information processing device 100. The inquiry unit 2 transmits the image data cut out by the cutout unit 1 to each of the generation AIs 9 and makes inquiries about the image data. Each of the generation AIs 9 outputs a response to the inquiry for each piece of image data.

[0036] Here, weighting can be set for each generation AI9. The analysis unit 3 analyzes the answers output from each generation AI9 for each image data. When BERT is used, the analysis unit 3 can output the analysis results as a weighted vector for each generation AI9. The analysis unit 3 can also calculate an average vector, which is the average of the weighted vectors for each generation AI9. The extraction unit 5 can extract emotional scenes from the video data using the mean vector and the emotional label.

[0037] The accuracy of generation AI is improving every day, and there are limits to stabilizing the accuracy of extracting moving scenes by simply using a specific generation AI. According to another embodiment, the influence of multiple types of generation AI can be adjusted, thereby stabilizing the accuracy of extracting moving scenes.

[0038] Furthermore, the analysis unit 3 may identify the vector closest to the average vector from among the weighted vectors for each generated AI 9. The extraction unit 5 can extract emotional scenes from the video data using the identified vectors and emotion labels.

[0039] This allows the extraction of moving scenes using vectors derived from the actual cut-out image data, thereby reliably avoiding hallucination in the generation AI caused by using average vectors.

[0040] [Variations] (a) The query made by the query unit 2 is not limited to text data. For example, it may be image data or voice data. (b): The answer from the generation AI 9 is not limited to text data. For example, it may be image data or audio data. In particular, images and sound effects that emphasize the touching scenes extracted by the extraction unit 5 may be answered as additional data. The extracted touching scenes may not be limited to a portion of the original video data, but may also be accompanied by creative image data (and / or creative audio data) that forms the answer from the generation AI 9. (c) In response to the text data, subtitles of the text data may be added to the emotional scenes. The subtitles to be displayed are not limited to the native language, but may be translated into other languages. (d): In the information processing method of this embodiment, among the multiple types of inference models 8, one that has a different experience category from the loaded video data may be selected. As a result, it is possible to increase the possibility of extracting moving scenes that include emotions of different flavors.

[0041] (e) It is also possible to realize a technology that appropriately combines the various technologies described in this embodiment. (f) The software described in this embodiment can be realized as hardware, and vice versa. (g) Other modifications may be made to the hardware, software, flowcharts, etc. as appropriate without departing from the spirit of the present invention. [Explanation of symbols]

[0042] 100 Information processing device 1 Cutout 2. Inquiry Department 3 Analysis section 4 Reasoning part 5 Extraction part 6 Video DB 7 Kando DB 8 Inference Model 9 Generation AI

Claims

1. an analysis unit that analyzes a response output from the generation AI that has received an inquiry regarding image data extracted from video data for each of the image data and outputs the analysis result as a vector; an inference unit that inputs the image data into an inference model and assigns an emotion label output from the inference model to the image data; an extraction unit that extracts emotional scenes from the video data using the vectors and the emotional labels.

2. A plurality of types of the generation AI are prepared, and weighting is set for each of the generation AI, The analysis unit analyzes the answers output from each of the generation AIs for each of the image data, outputs the analysis results as a weighted vector for each of the generation AIs, and calculates an average vector of the vectors. The information processing device according to claim 1 , wherein the extraction unit extracts emotional scenes from the video data using the mean vector and the emotion label.

3. The analysis unit identifies a vector closest to the average vector among the weighted vectors for each of the generated AIs, The information processing device according to claim 2 , wherein the extraction unit extracts emotional scenes from the video data using the specified vectors and the emotional labels.

4. The information processing device A step of analyzing a response output from a generation AI that has received an inquiry regarding image data extracted from video data for each of the image data, and outputting the analysis result as a vector; inputting the image data into an inference model and assigning an emotion label output from the inference model to the image data; and extracting emotional scenes from the video data using the vectors and the emotional labels.

Images