Information processing device, information processing method, and program

Abstract

Provided is an information processing device including a processor, the processor acquiring moving image data including sound data, acquiring range-specifying information specifying a reproduction time range satisfying a predetermined condition related to a speaker or a scene by using the sound data included in the moving image data, and using the acquired range-specifying information to execute predetermined guide processing for guiding the reproduction of a portion satisfying the predetermined condition for the acquired moving image data.

Description

Information processing device, information processing method, and program

[0001] The present invention relates to an information processing device, an information processing method, and a program.

[0002] 2. Description of the Related Art A process known as "cueing" is known in which a desired scene is selected from video data to be played back and played back.

[0003] In recent years, image data analysis processing has become more sophisticated, and processing methods have been developed to identify people captured in a scene, output text explaining the content of the scene, etc. By utilizing such image data analysis processing, it becomes possible to cue and play back scenes that satisfy conditions such as "the person you want to see appears."

[0004] However, when dealing with relatively long moving image data such as movies and TV dramas, there is a problem in that the processing load increases if the entire image data is subjected to analysis processing each time the moving image data is played back.

[0005] The present invention has been made in consideration of the above-mentioned circumstances, and one of its objects is to provide an information processing device, an information processing method, and a program that can guide a playback start point that satisfies a condition using a method with a relatively low load.

[0006] One aspect of the present invention that solves the problems of the above-mentioned conventional examples is an information processing device having a processor, which acquires video data including audio data, uses the audio data included in the video data to acquire range identification information that identifies a playback time range that satisfies predetermined conditions related to a speaker or scene, and uses the acquired range identification information to execute a predetermined guidance process that guides playback of portions of the acquired video data that satisfy the predetermined conditions.

[0007] According to the present invention, since the processing of audio data has a relatively low processing load, even when playing back moving image data, for example, it is possible to guide the playback of a portion that satisfies a predetermined condition.

[0008] Fig. 1 is a block diagram showing a configuration and connection example of an information processing device according to an embodiment of the present invention; Fig. 2 is a functional block diagram showing an example of an information processing device according to an embodiment of the present invention; Fig. 3 is an explanatory diagram showing information on a playback range generated by an information processing device according to an embodiment of the present invention; Fig. 4 is an explanatory diagram showing an example of guidance information provided by an information processing device according to an embodiment of the present invention; and Fig. 5 is a flowchart showing an operation example of an information processing device according to an embodiment of the present invention.

[0009] An information processing device 1 according to an embodiment of the present invention is a program-controlled device such as a personal computer or a home game console, and includes a control unit 11, a storage unit 12, an operation unit 13, a display unit 14, and a communication unit 15, as shown in FIG.

[0010] The control unit 11 is a processor such as a CPU, and operates according to a program stored in the storage unit 12. In one example of the present embodiment, the control unit 11 acquires video data including audio data and processes it. The control unit 11 subjects the audio data included in the video data to be processed to processing such as speaker identification processing, and acquires range identification information that identifies a playback time range that satisfies predetermined conditions related to the speaker or scene. The control unit 11 also uses the acquired range identification information to execute a predetermined guidance process that guides playback of portions of the video data to be processed that satisfy the predetermined conditions. The operation of the control unit 11 will be described in detail later.

[0011] The storage unit 12 is a memory device, a disk device, or the like, and stores a program executed by the control unit 11. This program may be provided by being stored in a computer-readable, non-transitory recording medium and may be copied to the storage unit 12. The storage unit 12 also operates as a work memory for the control unit 11.

[0012] The operation unit 13 receives information indicating the content of user operations input from a mouse, keyboard, game controller, or the like, and outputs the received information to the control unit 11. The display unit 14 displays images on a display device or the like in accordance with instructions input from the control unit 11.

[0013] The communication unit 15 is a network interface or the like, and sends information to an external server device connected via a network in accordance with instructions input from the control unit 11. The communication unit 15 also outputs information received via the network from an external server device or the like to the control unit 11.

[0014] Next, an example of the operation of the control unit 11 of this embodiment will be described. In this example of this embodiment, the control unit 11 operates according to a program stored in the storage unit 12, and functionally includes a video data acquisition unit 21, an area identification information acquisition unit 22, and a guidance processing unit 23, as exemplified in FIG.

[0015] The video data acquisition unit 21 acquires video data to be processed and stores it in the storage unit 12. In the example of this embodiment, the video data to be processed includes audio data.

[0016] The range identification information acquisition unit 22 uses audio data contained in the video data acquired by the video data acquisition unit 21 to acquire range identification information that identifies a playback time range that satisfies specified conditions related to the speaker or scene, and specifically, in one example of this embodiment, it operates as follows.

[0017] Specifically, the range identification information acquisition unit 22 extracts audio data from the video data, and then subjects the extracted audio data to a process of identifying portions of the audio data that contain audio that satisfies predetermined conditions.

[0018] An example of this processing may be processing to identify portions of speech spoken by different speakers (speaker recognition processing). This speaker recognition processing (processing to assign unique identification information to different speakers and identify portions of speech spoken by each speaker identified by each identification information, without necessarily specifying proper names that identify the speakers) can be performed using a widely known method, and therefore a detailed description of the processing content will be omitted.

[0019] The range identification information acquisition unit 22 that performs the speaker recognition process generates playback time range information that associates, for each speaker identified by the speaker recognition process, identification information ID that identifies the speaker with information on the speaker's speech start time Ts and speech end time Te, and acquires the information as range identification information, as shown in Fig. 3. Here, the speech start time Ts and the speech end time Te represent the start and end times of the speech when the playback start time of the audio data used for processing is set to 0 seconds and the audio data is played back at a normal playback speed (an assumed playback speed without fast or slow playback).

[0020] If the same speaker (for example, a speaker identified by identification information = "1") speaks twice in the audio data subjected to processing, the range identification information acquisition unit 22 acquires the speech start time Ts and speech end time Te for the first utterance, and the speech start time Ts and speech end time Te for the second utterance, and records them as part of the range identification information.

[0021] It should be noted that since the processing of this range identification information acquisition unit 22 is processing of audio data, which has a relatively low processing load, the range identification information acquisition unit 22 does not need to permanently store the range identification information in a disk device of the storage unit 12 or the like (so that it is not deleted due to a power outage), but may store the range identification information in a memory device of the storage unit 12 and generate and acquire the range identification information in real time each time processing is performed.

[0022] The guidance processing unit 23 uses the range identification information acquired by the range identification information acquisition unit 22 to perform a process (guidance process) to guide the playback of parts of the video data that satisfy specified conditions for the video data that is the processing target acquired by the video data acquisition unit 21.

[0023] For example, the guidance processing unit 23 generates a list of playback start times for each speaker's identification information ID from the range identification information exemplified in Fig. 3. Specifically, the guidance processing unit 23 generates a list (a list with no overlaps) of the speaker's identification information IDs included in the range identification information, and generates a list of the speech start times Ts associated in the range identification information for each identification information ID.

[0024] The guidance processing unit 23 extracts image data from the video data being processed that corresponds to the speech start time Ts included in the generated list (which may be still image data from the video data at the playback time corresponding to the speech start time Ts), and generates a thumbnail image by performing a reduction process, etc.

[0025] The guidance processing unit 23 then displays, for each identification information ID, a list of entry information (E) including a playback time corresponding to the utterance start time Ts and a thumbnail image corresponding to the playback time (FIG. 4). When the user selects one of the listed entry information, the guidance processing unit 23 plays the video data to be processed from the playback time associated with the selected entry information.

[0026] In the explanation so far, it has been assumed that the information processing device 1 does not need to permanently store the generated / acquired range identification information, but this is just one example. In one example of this embodiment, the information processing device 1 may permanently store the generated / acquired range identification information and / or a list of the speech start times Ts generated by the guidance processing unit 23 in the storage unit 12 (i.e., in a disk device of the storage unit 12 so that it will not be deleted due to a power outage) in association with information identifying the video data that was the subject of processing. In this example, when the video data is played again later, if there is range identification information, etc. associated with the information identifying the video data, this information may be read and used. According to this example, there is no need to generate this information again when the video data is played again.

[0027] [Operation] The information processing device 1 according to the embodiment of the present invention basically has the above-described configuration and operates as follows: In the following, an example will be described in which moving image data (moving image data including audio data) with a playback time of 30 minutes is processed.

[0028] When the user inputs the video data to be processed into the information processing device 1 (for example, by instructing the information processing device 1 to specify the URL of the video data distributed on an external web server so that it can be downloaded), the information processing device 1 starts the process illustrated in Figure 5 and extracts audio data from the video data (S11).

[0029] The information processing device 1 then subjects the extracted voice data to speaker recognition processing. The information processing device 1 may perform the speaker recognition processing itself, or, if a server connected via a network provides a speaker recognition processing service, may send the voice data to the server and obtain the results of the speaker recognition processing.

[0030] Here, we will explain a case where the information processing device 1 sends voice data to a server connected via a network and obtains the results of speaker recognition processing. In the following example, it is assumed that the server can process a maximum of 10 minutes of voice data.

[0031] The information processing device 1 divides the audio data extracted in step S11 into 10-minute pieces of audio data (a data size that can be processed by the server) (S12). Here, the information processing device 1 stores the playback time offset for each piece of audio data obtained by division (hereinafter referred to as divided audio data for the sake of distinction). Specifically, of the extracted audio data, the offset for the divided audio data corresponding to the first 10 minutes is set to "0 minutes." The offset for the divided audio data corresponding to the next 10 minutes is set to "10 minutes." Similarly, for each divided audio data, an offset that associates the divided audio data with the playback time (here, the playback start time) in the original audio data is recorded.

[0032] The information processing device 1 sequentially transmits each piece of divided audio data obtained by dividing to the server (S13), and acquires range identification information from the server that is included in the divided audio data and indicates the speech period (speech start time and speech end time) for each speaker (S14). Since the acquired range acquisition information is the time from the playback start time of the divided audio data, the information processing device 1 adds the corresponding offset to each speech start time and speech end time (S15).

[0033] For example, in the second divided audio data, if one of the speech periods for speaker A starts at 1 minute 20 seconds and ends at 1 minute 30 seconds, in this example the original audio data was divided into 10-minute intervals to obtain the divided audio data, so the offset of 10 minutes for the second divided audio data is added to these to make the speech start time of the speech period for speaker A 11 minutes 20 seconds, and similarly the speech end time 11 minutes 30 seconds.

[0034] When the information processing device 1 obtains the range specifying information for each piece of divided audio data, it synthesizes (concatenates) the pieces of divided audio data to generate range specifying information for the original audio data (S16).

[0035] Next, the information processing device 1 generates a list of speech start times for each speaker (each speaker's identification information) from the range specification information generated in step S16 (S17). This results in a list such as the following, for example: Speaker A: speech start time = [1 minute 0 seconds, 5 minutes 20 seconds, 11 minutes 20 seconds, ...] Speaker B: speech start time = [1 minute 15 seconds, 5 minutes 0 seconds, 13 minutes 30 seconds, ...]

[0036] The information processing device 1 further extracts image data from the video data to be processed that corresponds to the speech start time included in the list generated here, and generates a thumbnail image by performing a reduction process, etc. (S18).

[0037] Then, the information processing device 1 displays, for each speaker (their identification information ID), a list (guidance information) of entry information (E) representing the playback time corresponding to the speech start time included in the list corresponding to that speaker and the thumbnail image corresponding to that playback time (S19; Figure 4).

[0038] Hereinafter, when the user selects any of the entry information included in the guidance information and instructs playback, the information processing device 1 plays the video data to be processed from the playback time associated with the selected entry information.

[0039] According to this embodiment, by processing audio data with a relatively low processing load, it is possible to guide playback of portions that satisfy a predetermined condition, such as portions in a movie where a specific character is speaking.

[0040] [Cast / Scene Identification] In the description so far, in the processing as range identification information acquisition unit 22, control unit 11 performs speaker recognition processing to identify portions of the audio data extracted from the video data being processed that contain audio that satisfies predetermined conditions, without necessarily identifying proper names that identify the speakers, by assigning unique identification information to different speakers and identifying the portions of the audio that are spoken by each speaker identified by the identification information. However, in this embodiment, the processing of audio data performed in the processing as range identification information acquisition unit 22 is not limited to the speaker recognition processing.

[0041] For example, the information processing device 1 may be used for speaker identification processing that identifies the proper name of the person speaking (such as the name of a cast member). In this example, the information processing device 1 generates, for each speaker, playback time range information that correlates the proper name as identification information that identifies the speaker with information on the utterance start time Ts and the utterance end time Te of the speaker identified by the proper name. Also in this example, if the same speaker (a speaker identified by a common proper name) speaks twice in the audio data used for processing, the range identification information acquisition unit 22 acquires the utterance start time Ts and the utterance end time Te for the first utterance, and the utterance start time Ts and the utterance end time Te for the second utterance, respectively.

[0042] In this example, the information processing device 1 generates a list of speech start times for each speaker (each proper name of the speaker). Then, the information processing device 1 associates the information on the obtained proper names with entry information included in the list shown in Fig. 4 as a guidance process, the entry information including a playback time corresponding to the speech start time of the attention range identification information, and stores the information.

[0043] The information processing device 1 may refer to information on a proper name associated with the entry information and display the information on the referenced proper name (such as the name of a cast member) instead of the speaker's identification information shown in association with the entry information, as exemplified in Figure 4.

[0044] There are various well-known methods for speaker identification, including the proper name of the person speaking, such as machine learning and methods using known classification methods, so a detailed explanation will be omitted here.

[0045] If this speaker identification process is a relatively heavy process, range identification information that identifies the playback time of each different speaker obtained by speaker recognition processing that does not identify the proper names of the speakers may be referenced, and audio data included in the playback time range identified by the referenced range identification information may be extracted from the video data being processed, and the speaker identification process may then be performed on the extracted audio data. In this way, the speaker identification process is performed partially, thereby reducing the load. Furthermore, for example, by first performing speaker recognition processing when the video data is played back, and then performing speaker identification processing later, such as after the video data playback has finished, it is possible to provide convenient guidance information including proper names while reducing the processing load at each point in time.

[0046] Furthermore, the information processing device 1 may extract image data included in the playback time range specified by each range specification information acquired based on the audio data from the video data to be processed, and perform image analysis processing using the extracted image data. The results of this image analysis processing may also be used in the guidance processing.

[0047] Here, the image analysis process may be, for example, a process of distinguishing people or objects captured in the image data, a process of determining a scene, etc. Such image analysis process can be performed using a neural network (machine learning model) that has machine-learned the relationship between the image data and character strings representing the captured people, objects, scene content, etc. This type of process is widely known, so a detailed description of the process will be omitted.

[0048] Specifically, this process makes it possible to obtain information such as proper names (such as the names of cast members) based on image data that falls within a playback time range specified by certain range specifying information (hereinafter referred to as attention range specifying information) from among the video data that is the subject of the process. When a proper name is obtained, the information processing device 1 associates and stores the obtained information such as the proper name of the cast member with entry information included in the list shown in Figure 4 as a guidance process, which entry information includes a playback time that corresponds to the speech start time of the attention range specifying information.

[0049] In this example, the information processing device 1 may also refer to information on the proper name of the cast member associated with the entry information, and display the information on the proper name of the cast member referred to instead of the identification information of the speaker shown in association with the entry information, as illustrated in Figure 4.

[0050] Furthermore, in the guidance process, the information processing device 1 may further acquire a list of character names corresponding to the proper names of the cast members (in the case of movie video data, the names of the characters in the movie). Such a list can be acquired by referring to a database of movies, etc. Note that if the metadata of the video data being processed contains information on the proper names of the cast members and the corresponding character names, the list may be acquired by referring to the metadata. In this example, the information processing device 1 displays the cast names and the corresponding character names (if acquired) as information on each speaker in the list illustrated in FIG. 4.

[0051] In addition, when recognizing and obtaining the names of the cast members captured in the image data included in the playback time determined by the range identification information in this manner, the names of multiple cast members may be obtained (for example, in the case of a scene in which multiple people appear).

[0052] Therefore, the information processing device 1 may compile a list of cast names related to the entry information associated with each speaker, check the frequency of appearance of the listed cast names, and display the cast name with the highest frequency as the cast name corresponding to the speaker. Furthermore, if information on character names can be obtained in association with the cast name, the information on the character names may be displayed instead of or together with the cast name.

[0053] In this example of the present embodiment, although analysis processing of the video data is performed, the analysis processing is performed only on a portion of the video data indicated by the range identification information determined based on the audio data, so that it is possible to guide playback of portions that satisfy specified conditions while suppressing an increase in processing load.

[0054] [Scene Analysis] As described above, the information processing device 1 of the present embodiment acquires range identification information that uses audio data to identify a playback time range that satisfies a predetermined condition related to a speaker or a scene. Here, the predetermined condition related to a scene may be, for example, a condition such as: - a predetermined type of sound effect, such as an explosion, is included; - a specific instrument is being played; or - a piece of music is being played.

[0055] This process can be performed using a machine learning model that has learned the relationship between the sound of a sound effect or instrument and the corresponding string information (such as a string representing the type of sound effect or the name of an instrument). Furthermore, the name of the song being played may be obtained using a well-known process for recognizing music.

[0056] In this example, the information processing device 1 acquires range specification information that indicates the range of playback times (ringing start time and ringing end time) during which audio corresponding to each predetermined condition related to a scene (such as the type of sound effect, the name of an instrument, or the name of a song) is played back.The information processing device 1 then stores the predetermined condition related to the scene in association with the corresponding range specification information, and uses the information for a predetermined guidance process.

[0057] For example, the information processing device 1 extracts the ringing start time from the range identification information for each sound effect for which range identification information has been obtained, generates entry information for displaying information representing the playback time of the video data corresponding to the extracted ringing start time, a thumbnail image obtained from the video data at the playback time, and a character string representing the sound of the corresponding sound effect, and displays them in a list.

[0058] In this way, a list of playback start times corresponding to sound effects is displayed, such as Gunshot sound: 12 minutes 24 seconds Gunshot sound: 13 minutes 12 seconds Explosion sound: 13 minutes 20 seconds, etc. The user can preview the desired scene by playing back the video data from the playback start time selected from this list.

[0059] In this example of the present embodiment, too, by processing audio data with a relatively low processing load, it is possible to guide the playback of parts of a movie that satisfy predetermined conditions, such as parts where specific sound effects are used.

[0060] REFERENCE SIGNS LIST 1 information processing device, 11 control unit, 12 storage unit, 13 operation unit, 14 display unit, 15 communication unit, 21 video data acquisition unit, 22 range specification information acquisition unit, 23 guidance processing unit

Claims

1. An information processing device having a processor, which acquires video data including audio data, acquires range identification information using the audio data included in the video data to identify a playback time range that satisfies specified conditions related to a speaker or scene, and uses the acquired range identification information to execute a specified guidance process that provides guidance on playback of portions of the acquired video data that satisfy the specified conditions.

2. An information processing device as described in claim 1, wherein the predetermined condition is that different speakers are speaking, and the guidance process includes a process of presenting a guidance screen for playing back the speech portions of each speaker in the acquired video data.

3. An information processing device as described in claim 1, wherein the guidance process includes extracting image data from the acquired video image data that is included in a playback time range specified by the acquired range specification information, and processing using the results of image analysis processing using the extracted image data.

4. An information processing device as described in claim 3, wherein the image analysis process is a process of recognizing people included in the extracted image data, and the guidance process includes a process of using the results of the recognition to obtain recognition results for each speaker who is speaking within the playback time range corresponding to the extracted image data, and presenting a guidance screen for playing back the speech portion for each recognized speaker.

5. An information processing method using an information processing device having a processor, which uses the processor to acquire video data including audio data, acquire range identification information using the audio data included in the video data to identify a playback time range that satisfies specified conditions related to a speaker or scene, and use the acquired range identification information to execute a specified guidance process that provides guidance on playback of portions of the acquired video data that satisfy the specified conditions.

6. A program for controlling a processor to: acquire video data including audio data; acquire, using the audio data included in the video data, range identification information that identifies a playback time range that satisfies specified conditions related to a speaker or scene; and use the acquired range identification information to execute a specified guidance process that provides guidance for playback of a portion of the acquired video data that satisfies the specified conditions.

Images