Moving image information search device, search method, search program, and use method of search result

Abstract

To provide a moving image information search device, search method, search program, and use method of a search result capable of easily searching for an important scene in a congress or a lecture.SOLUTION: A moving image information search device 1 comprises: input means 2 to which moving image data is input; query generation means 3 generating a query; extraction means 4 for extracting a scene from the moving image data; display means 5 displaying the extracted scene; and search query storage means 6. An operation screen 10 comprises: an input area 11 receiving input of the moving image data; a query generation area 13 capable of generating a search query 30; and a search result area 14 capable of displaying a search result scene 20S. A selection area 12 display a usable query and an operator, the query generation area 13 is an area where the search query 30 can be created by combining a query made into an object and the operator and the search query 30 can be visually generated. The search query 30 is generated by a combination of a language query 31, a voice query 32, and an image query 34.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to an apparatus, method, program, and method for searching for important scenes from video data of a video conference, a lecture, or the like, and a method for using the search results. [Background technology]

[0002] In recent years, with the spread of teleworking, the number of opportunities to use video conferencing has increased. In addition, in sales activities, the number of opportunities to use video conferencing is increasing, in addition to sales activities where sales representatives visit customers directly. Furthermore, online lectures and other events are being held frequently.

[0003] Patent Document 1 discloses a system for presenting relevant information related to a conference to conference participants, which generates a query based on a common word that is common to multiple text data contained in at least two of the following data: audio data, image data, and keyword data indicating registered keywords, obtains search results based on the query from a specified database, and outputs the search results or a portion of the search results to a specified output device.

[0004] The device in Patent Document 1 is said to be able to reduce erroneous recognition in the voice recognition process performed when converting voice into text and the image recognition process performed when converting images into text when presenting participants with information related to voice uttered during a meeting or images displayed. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2019-102060 A Summary of the Invention [Problem to be solved by the invention]

[0006] In the technology described in Patent Document 1, in voice recognition processing and image recognition processing performed when converting images into text, a query is created using a common word common to multiple text data contained in at least two of the voice data, image data, and registered keywords, and search results for the query are obtained from a database. By using the common word common to these multiple data, misrecognition in image processing is reduced.

[0007] However, although the technology of Patent Document 1 can reduce erroneous recognition in image processing, it is difficult to grasp the main points of a meeting, such as which conversations were important in the meeting.

[0008] Furthermore, when searching for important scenes from long video data, the query used to perform the search becomes important; however, in Patent Document 1, the query only uses common words that are common to the audio data, image data, and registered keywords, and no consideration is given to whether the audio data, etc. is an important scene in the conference or not.

[0009] In view of the above problems, the present invention aims to provide a video information retrieval device, a retrieval method, and a retrieval program that can easily retrieve important scenes from a conference or lecture. Another object of the present invention is to provide a method for effectively utilizing the search results obtained by the above search method. [Means for solving the problem]

[0010] In order to achieve the above-mentioned object, the video information search device of the present invention is a device for searching for desired scenes from video data, and is characterized in that it comprises an input means for inputting video data having image data and audio data, a query creation means for creating a search query using a language query for searching for linguistic information about the audio data and a voice query for searching for audio information about the audio data, an extraction means for extracting search result scenes from the video data that match the conditions of the search query, and a display means for displaying the search result scenes extracted by the extraction means.

[0011] When searching for a desired scene from video data, the video information retrieval device of the present invention generates a search query using a language query and a voice query, and extracts search result scenes from the video data using the search query. In this way, the present invention can create queries for multiple types of information in the video data, making multimodal searches possible. Here, linguistic information refers to information about the content and meaning of words in the audio of the video data. Audio information refers to information about the voice quality, volume, tone, tempo, etc. of the audio of the video data.

[0012] In the video information search device of the present invention, the query creation means may create an image query for searching image information for the image data, and the search query may be created using at least two types of queries, the language query, the voice query, or the image query. This configuration allows queries to be created for even more types of information in the video data, enabling multimodal search. Here, the image information is information in the video data, mainly about a person's appearance, gestures, facial expressions, gaze, movement, etc.

[0013] In the video information search device of the present invention, the query creation means may display the language query, the voice query, and an operator used to create a search expression as objects on the display means, and may combine these queries with the operators to create the search query.Furthermore, the image query may also be displayed as an object on the display means, and may combine these queries with the operators to create the search query.

[0014] According to this configuration, each query and operator can be treated as an object and can be combined to create a search query. Therefore, the search query can be created visually, which makes it easy to create a search query. Note that an object refers to a unit that performs independent information processing, and in the present invention, it is a visually represented query and operator.

[0015] In the video information search device of the present invention, the query creation means may create a combined query capable of searching both the audio data and the image data. According to this configuration, even a scene that is difficult to detect using only the audio data or the image data can be searched for using the combined query. An example of a combined query is mainly a person's emotion.

[0016] In this configuration, the composite query may be capable of searching for at least one of the utterance and behavior of a speaker in the video data and the listener's reaction thereto by using at least a part or all of the linguistic information, the audio information, and the image information, i.e., it may be possible to search for what the speaker in the video data conveyed and how he / she conveyed it, and what the listener received and how he / she received it.

[0017] According to this configuration, since a search can be performed based on the speaker's speech and behavior, and the receiver's reaction to these, it is possible to accurately search for changes in the state of mind of the parties involved that occur during the exchange of information. For example, a search can be performed for the receiver's facial expression when the speaker greets in a loud voice, or for a scene in which the receiver becomes frightened by the speaker's remark. In this case, the compound query may also be displayed as an object and used or edited as a query.

[0018] In the video information search device of the present invention, a search query storage means is further provided for storing the created search query, and the query creation means may use the search query stored in the search query storage means as is, or may edit the stored search query to make it usable.

[0019] According to this configuration, when searching for video data, if the search query stored in the search query storage means has the same search purpose, it can be used as is. Also, if a search query similar to the search purpose is stored, it is possible to easily create a search query for the search purpose by editing the search query.

[0020] The present invention can also be a video information retrieval program that causes a computer to operate as each of the video information retrieval devices described above. The video information retrieval program can be installed in a computer, stored in a CD-ROM or DVD-ROM, or uploaded or downloaded via a network.

[0021] In addition, the video information search method of the present invention is a method for searching for a desired scene from video data, and is characterized by comprising an input step for inputting video data having at least audio data, a query creation step for creating a search query using a language query for searching for language information about the audio data and a voice query for searching for voice information about the audio data, an extraction step for extracting search result scenes from the video data that match the conditions of the search query, and a display step for displaying the search result scenes extracted in the extraction step.

[0022] In the video information search method, image data is obtained in an input step, and an image query for searching image information for the image data can be created in the query creation step, and the search query can be created using at least two types of queries, namely, the language query, the voice query, or the image query.

[0023] In addition, in the query creation step, the language query, the voice query, and operators used to create a search formula may be displayed as objects, and the search query may be created by combining these queries with the operators. In addition, in the query creation step, the language query, the voice query, the image query, and operators used to create a search formula may be displayed as objects, and the search query may be created by combining these queries with the operators.

[0024] In the video information search method, the query creating step may create a combined query capable of searching both the audio data and the image data. The combined query may be capable of searching for emotions of people in the video data by using at least a part or all of the linguistic information, the audio information, and the image information.

[0025] Furthermore, the video information search method may further include a search query storing step of storing the created search query, and in the query creating step, the search query stored in the search query storing step may be used as is, or the stored search query may be edited and made usable.

[0026] In addition, the method of utilizing search result scenes using the video information search method of the present invention is characterized by comprising a command creation step of creating a command using the search result scenes, and an answer acquisition step of inputting the command and the search result scenes into an AI system that provides an answer in response to the input to obtain the answer.

[0027] According to this method of use, the command created in the command creation step is created using a search result scene that is an important scene, and the command and the search result scene are input into an AI system, and an answer is obtained in the answer obtaining step, so that an answer of high importance can be obtained by using the AI ​​system. Effect of the Invention

[0028] According to the present invention, it is possible to provide a video information retrieval device, a retrieval method, a retrieval program, and a method for using the retrieval results, which are capable of easily retrieving important scenes in a conference or lecture. [Brief description of the drawings]

[0029] [Figure 1] FIG. 1 is an explanatory diagram showing a functional configuration of a video information search device according to an embodiment of the present invention; [Diagram 2] FIG. 2 is an explanatory diagram showing an operation screen of the video information search device of the embodiment. [Diagram 3] FIG. 2 is an explanatory diagram showing a configuration of video data and a configuration of a search query. [Figure 4] FIG. 2 is an explanatory diagram showing an example of a scene for which a search is desired. [Diagram 5] FIG. 13 is an explanatory diagram showing the state of a saved search query. [Figure 6]FIG. 11 is an explanatory diagram showing an example of a compound search query. [Figure 7] FIG. 4 is an explanatory diagram showing a method for using search results obtained by the video information search device of the present embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0030] Next, a video information retrieval device, a retrieval method, a retrieval program, and a method for using retrieval results according to an embodiment of the present invention will be described with reference to Figures 1 to 7. As shown in Figure 1, the video information retrieval device 1 according to this embodiment includes an input means 2 for inputting video data, a query creation means 3 for creating a query for extracting a desired scene from the video data, an extraction means 4 for extracting a scene from the video data using the query, a display means 5 for displaying the extracted scene, and a search query storage means 6 for storing the created query.

[0031] The video information search device 1 is composed of a computer as hardware and a video information search program 7, which is software stored in the computer. The computer includes a processor such as a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit), a hard disk, a storage means such as a memory, a connection means for various networks, a keyboard, a mouse, a display, etc. (not shown).

[0032] 2 shows an operation screen 10 displayed by the display means 5 in the video information retrieval device 1 of this embodiment. The operation screen 10 is provided with an input area 11, which is a part of the input means 2 into which video data is input. Video data 20 from which a scene is to be extracted can be input into this input area 11 by dragging and dropping or the like.

[0033] The operation screen 10 is provided with a selection area 12 in which a query displayed by the query creation means 3 and an operator to be used in creating a search expression can be selected, a query creation area 13 in which objectified queries and operators can be arranged and linked to create a search query 30, and a search result area 14 in which a search result scene 20S that is displayed as a result of a search performed using the search query 30 can be displayed.

[0034] The selection area 12 displays queries and operators that can be used when creating a search query 30. These queries and operators can be selected with a mouse or the like and moved to the query creation area 13.

[0035] The query creation area 13 is an area where a search query 30 can be created by combining queries and operators that have been turned into objects. Each query and operator that has been turned into an object is provided with a connection port P, and it is possible to connect these connection ports P with each other using a mouse or the like. In this way, each query and operator is turned into an object, making it possible to visually create the search query 30.

[0036] The query creation area 13 is provided with a + button 30a, a - button 30b, a delete button 30c, a code generation button 30d, and an execute button 30e, which are used to create a search query 30.

[0037] Next, the structure of the video data 20 and the relationship between each query constituting the search query 30 will be described with reference to Fig. 3. As shown in Fig. 3, the video data 20 has audio data 21 and image data 22. The audio data 21 includes linguistic information 21a, which is linguistic information such as human conversation or lecture, and audio information 21b capable of detecting the volume, pitch, or intonation of a voice. The image data 22 includes image information 22a including a person's facial expression, etc.

[0038] The search query 30 includes a language query 31, a voice query 32, a compound query 33, and an image query 34, and an operator 35. The language query 31 is a query for searching the language information 21a, and is a query for searching linguistic features. The voice query 32 is a query that can be used to search the voice information 21b, and is a query for searching quasi-linguistic features. The compound query 33 is a query that can be used to search both the language information 21a and the voice information 21b, and is also a query that can be used to search the image information 22a. The image query 34 is a query for searching the image information 22a, and is a query for searching non-linguistic features.

[0039] Specific examples of language queries 31 include whether the utterance is a question, whether there is a specific backchannel response, whether there is a specific word such as a delivery date, whether the utterance contains fillers (meaningless phrases such as "um" or "well"), etc.

[0040] Specific examples of voice queries 32 include the volume of the voice during an utterance, the pitch of the voice, the variation in pitch of the voice during an utterance, the length between utterances, whether adjacent utterances are spoken by the same person, the speed of speech, etc.

[0041] Specific examples of the compound query 33 include at least one of the speaker's utterance and behavior, and the listener's reaction to them. When a conversation takes place, one of the parties becomes the speaker and the other becomes the listener. At this time, if the speaker's utterance and behavior (facial expression, gestures, head movement, volume of voice, etc.) can search for the listener's reaction (joy, surprise, confusion, bewilderment, anger, sadness, neutral, etc.), it becomes easy to extract important scenes.

[0042] For example, if a speaker greets with a smile and a loud voice, and the receiver's facial expression softens, it can be said that the speaker and receiver have built a good relationship. On the other hand, if the speaker's speech and behavior contain anger and the receiver's facial expression appears frightened, it is assumed that the scene is not favorable for conversation. In this way, by making it possible to search by combining the speaker's speech and the receiver's reaction, it is possible to easily extract important scenes.

[0043] Furthermore, a specific example of the compound query 33 may be a person's emotion. As with the above-mentioned reaction of the receiver, the emotion may be joy, surprise, confusion, bewilderment, anger, sadness, neutrality, etc. For example, if the compound query 33 is "joy," it can be identified from information such as positive language in the linguistic information 21a, a relatively high-pitched voice in the audio information 21b, and eyes wide open and corners of the mouth turned up in the image information 22a.

[0044] Next, a method for searching for a desired scene from the video data 20 using the video information retrieval device 1 and the video information retrieval program 7 of this embodiment will be described. An example of an important scene to be detected is a case in which, as shown in Fig. 4, a speaker in a business negotiation speaks in the form of a question in the language information 21a, has a strong tone in the voice information 21b, and leans forward in the image information 22a. Thus, a scene in a business negotiation where a speaker asks a question with emphasis is often an important scene in the business negotiation.

[0045] When searching for such a scene, first, the video data 20 to be searched is input (input step) into the input area 11. Specifically, for example, the video data 20 stored in a computer constituting the video information search device 1, or stored in a computer or server connected to a network, etc., is dragged and dropped into the input area 11 using a device such as a mouse.

[0046] Next, a search query 30 is created using any one of the language query 31, the voice query 32, and the image query 34 (query creation step). Specifically, each query displayed in the selection area 12 shown in Fig. 2 is selected with a mouse or the like, and placed in the query creation area 13 by dragging and dropping. In Fig. 2, queries such as "voice strength" 32a, "is the utterance a question?" 31a as the language query 31, and "vertical head direction" 34a as the image query 34, as well as operators 35 such as "greater than" 35a, 35b, "And" 35c, and "detect" 35d are placed.

[0047] Next, each object is combined to create a search query 30. As shown in FIG. 2, the connection ports P of the "word strength" 32a and "greater than" 35a objects are connected to each other. For the "greater than" 35a object, a parameter with a maximum value of 1 is input. In FIG. 2, a parameter of "0.6" is input as an example. The "greater than" 35a and the "And" 35c and "Detect" 35d objects are also connected by connection ports P.

[0048] Connections are also made between the objects "Is the utterance a question?" 31a and "And" 35c. Furthermore, "Vertical head direction" 34a is connected to "Greater than" 35b, whose parameter is set to "0.6", and this "Greater than" 35b is connected to "And" 35c. When the search query 30 is complete, a search process can be executed using the created search query 30 by clicking the code generation button 30d in the query creation area 13.

[0049] The search query 30 created in this way is a query for searching for scenes in which a person is filmed with a tone strength of greater than 0.6, speaking as a question, and with a vertical head orientation of greater than 0.6. When the execute button 30e in the query creation area 13 is clicked, scenes matching the conditions formed by the search query 30 are searched and extracted from within the video data 20 (extraction step), and if there is a scene matching the conditions, one or more search result scenes 20S are displayed in the search result area 14 (display step).

[0050] A user who has searched video data 20 can determine the validity of the search query 30 that he or she created by checking the search result scene 20S. If the search result scene 20S is different from the scene that the user imagined as the search result, the user can modify the search query 30. For example, if the search result scene 20S is a scene in which a speaker asks a question, but the tone is weak and the question is not emphasized, the parameter of "greater than" 35a, which is the operator 35 in the query creation area 13, can be set to greater than 0.6.

[0051] In this way, in the video information retrieval device 1 of the present embodiment, the search query 30 can be visually created and edited in the query creation area 13, so that the user performing the search can create the search query 30 by an intuitive operation. In addition, since the validity of the created search query 30 can be confirmed by viewing the search result scenes 20S, the search query 30 can be easily fine-tuned.

[0052] In addition, in the video information search device 1 of this embodiment, an emotion label 33a, which is a composite query 33, can be used as the search query 30. The emotion label 33a is a query for searching for emotions of a person, and can select emotions such as joy, surprise, confusion, bewilderment, anger, sadness, and neutral. Alternatively, the user can set a query related to an emotion such as "bad mood" separately from these.

[0053] The emotion label 33a is a query created using at least a part or all of the language information 21a, the voice information 21b, and the image information 22a. For example, if the language information 21a contains information such as "I'm in trouble," the emotion of confusion can be searched for using only the language information 21a. Alternatively, if the voice volume of the voice information 21b increases suddenly, the emotion of anger can be searched for using only the voice information 21b. Alternatively, since the facial expression of a person can be determined using the image information 22a, the emotion can be searched for using only the image information 22a. Furthermore, by combining the information of the language information 21a, the voice information 21b, and the image information 22a, it is possible to search for an emotion with high accuracy.

[0054] The search query 30 created in this way can be used in the next and subsequent query creation steps by storing it in the search query storage means 6. The search queries 30 stored in the search query storage means 6 can be displayed in a list on a library screen 36 as shown in Fig. 5. On the library screen 36, queries and operators that have been converted into objects can be viewed, and the name of each query and an explanation of its contents are displayed.

[0055] Using this library screen 36, the user can select and use a desired search query 30 from among the stored search queries 30. Furthermore, if the user cannot find the desired search query 30, the user can extract a query close to the desired search query 30 and edit it in the query creation area 13 to create the desired search query 30.

[0056] Here, as an example of the composite query 33, a composite query 33b for searching for "a listener's reaction to a speaker's utterance and behavior in a conversation" shown in Fig. 6 will be described. This composite query 33b includes a speaker input section 33c into which the speaker's utterance and behavior can be input, and a listener input section 33d into which the listener's reaction can be input.

[0057] In the speaker input section 33c, it is possible to input what kind of utterance the speaker made, and what kind of behavior (facial expression, etc.) accompanied the utterance. In the receiver input section 33d, two input areas are provided, and it is possible to input the reaction of the receiver, such as what he or she received and how. Input may be made in only one of these input fields, or in both.

[0058] Next, a method of using search result scenes by the video information search method will be described with reference to Fig. 7. The search result scenes 20S extracted by the video information search device 1 and search method of this embodiment are important scenes desired by the user. For example, if the video data 20 is video data related to online business negotiations and the search query 30 is for extracting points that a salesperson should be careful of in business negotiations, the search result scenes 20S will be a collection of scenes that the salesperson should be careful of.

[0059] It is expected that useful information can be obtained by creating commands 9 for the AI ​​system 8 using the search result scenes 20S, which are a collection of such important scenes, and inputting the search result scenes 20S and commands 9 into the AI ​​system 8.

[0060] First, a command 9 is created using the search result scenes 20S (command creation step). Specifically, the command 9 is created using keywords related to the search query 30. For example, if the search query 30 is to extract points that a sales representative should be careful about during a business negotiation, a command 9 such as "These video data (search result scenes 20S) are scenes in which a sales representative was asked a question and was unable to answer it properly. Please extract from these scenes topics necessary for the future training of this sales representative." can be created.

[0061] In this command creation step, the command 9 may be created using an AI system or a data table associated with the search query 30. Furthermore, the command 9 may be editable by the user performing the operation.

[0062] Here, a model known as a large-scale model can be used as the AI ​​system 8. For example, OpenAI's GPT can be used. The search result scene 20S and the created command 9 are input to such an AI system 8, and an answer 8a is obtained (answer acquisition step).

[0063] The answer 8a obtained by this answer acquisition step is an answer obtained by inputting the search result scene 20S obtained by the video information search device 1 and the search method, and a command 9 related to the search query 30, so that the user can obtain the answer he or she desires with a high probability.

[0064] In the above embodiment, in the query creation step, the search query 30 is created using the language query 31, the voice query 32, and the image query 34, but it is sufficient to create the search query 30 using at least two types of queries. Furthermore, the use of three types of queries is not limited to this, and the search query 30 may be created using only the language query 31 and the voice query 32. Even in this case, a search can be performed on the voice data 21 using the language information 21a and the voice information 21b, thereby enabling a multimodal search.

[0065] In the above embodiment, the video information search program 7 is installed in the video information search device 1 and operated, but the present invention is not limited to this. The program may be provided by cloud computing, or may be provided in a state stored in a storage medium such as a CD-ROM or DVD-ROM. The video information search program 7 may be installed in a server (not shown), or may be a program using API (Application Programming Interface) linkage. [Explanation of symbols]

[0066] 1...Video information search device 2. Input method 3. How to create a query 4...Extraction means 5...Display means 6…Search query storage means 7...Video information search program 8. AI system 10…Operation screen 11...Input area 12...Selection area 13…Query creation area 14…Search results area 20...Video data 20S…Search result scene 21...Audio data 21a…Language information 21b…Audio information 22...Image data 22a…Image information 30…Search queries 31…Language query 32…Voice query 33…Combined queries 33a…Emotion label 34…Image Query 35…Operator 36...Library screen

Claims

1. An apparatus for searching for a desired scene from video data, comprising: an input means for inputting video data having image data and audio data; a query creation means for creating a search query using a language query for searching for language information about the voice data and a voice query for searching for voice information about the voice data; an extraction means for extracting search result scenes, which are scenes that match the conditions of the search query, from the video data; A video information retrieval device comprising display means for displaying the search result scenes extracted by the extraction means.

2. 2. A video information retrieval device according to claim 1, The query creation means is capable of creating an image query for searching image information for the image data, and the search query is created using at least two types of queries: the language query, the voice query, or the image query.

3. 2. A video information retrieval device according to claim 1, The query creation means displays the language query, the voice query, and operators used to create a search expression as objects on the display means, and is capable of creating the search query by combining these queries and the operators.

4. 3. A video information retrieval device according to claim 2, The query creation means displays the language query, the voice query, the image query, and operators used to create a search expression as objects on the display means, and is capable of creating the search query by combining these queries and the operators.

5. 3. A video information retrieval device according to claim 2, The video information search device, wherein the query creation means is capable of creating a composite query capable of searching both the audio data and the image data.

6. 6. A video information retrieval device according to claim 5, A video information search device characterized in that the composite query is capable of searching for at least one of the speaker's speech and behavior in the video data, and the listener's reaction to these, using at least a part or all of the language information, the audio information, and the image information.

7. 3. A video information retrieval device according to claim 1, The system further includes a search query storage unit that stores the created search query, The video information search device, wherein the query creation means is capable of using the search query stored in the search query storage means as is, or editing the stored search query for use.

8. A video information retrieval program for causing a computer to operate as the video information retrieval device according to any one of claims 1 to 6.

9. A video information retrieval program for causing a computer to operate as the video information retrieval device according to claim 7.

10. A method for searching for a desired scene from video data, comprising the steps of: an input step of inputting video data having at least audio data; a query generation step of generating a search query using a language query for searching language information for the voice data and a voice query for searching voice information for the voice data; an extraction step of extracting search result scenes from the video data that match the search query; a display step of displaying the search result scenes extracted in the extraction step.

11. A method for utilizing the search result scenes by the video information search device according to claim 1, comprising: A command creation step of creating a command that can be input to an AI system that responds to an input using the search result scene; A method for utilizing search results, comprising an answer obtaining step of inputting the command and the search result scene into the AI ​​system to obtain the answer.

Images