Information processing method, information processing device, and computer program

Abstract

The information processing method of the present embodiment converts voice data into character string data, extracts a first word from the converted character string data and generates question data, extracts a second word from the character string data using the question data including the first word, and outputs the extracted second word, and storing the sound data, the first word, and the second word in association.

Description

Technical Field

[0001] The present disclosure relates to an information processing method, an information processing device, and a computer program. Background Art

[0002] Patent document 1 discloses a technology having a shooting unit, a recording unit, and a conversion unit that converts sound included in the recording data into a character string, extracts nouns from the character string, obtains related words corresponding to the extracted nouns from a dictionary unit, and stores the shooting data, nouns and related words in an associated manner. Prior art literature Patent Literature

[0003] Patent Document 1: Japanese Patent Application Publication No. 2016-170654 Summary of the invention Technical problem to be solved by the invention

[0004] Patent Document 1 has a technical problem that nouns and related words simply extracted from a character string of audio do not necessarily accurately represent the content of the audio recording data.

[0005] The present disclosure proposes an information processing method, an information processing device, and a computer program that can associate video or sound data obtained by shooting or recording with index information that accurately represents the content of the data. Technical solutions adopted to solve technical problems

[0006] An information processing method according to a first aspect of the present disclosure converts voice data into character string data, extracts a second word from the character string data using question data including a first word, and stores the voice data, the first word, and the second word in association with each other.

[0007] The information processing method of the second viewpoint of the present invention is based on the information processing method of the first viewpoint, and is preferably constructed by extracting a second word from the string data by inputting the string data and the question data into a language learning model that has completed learning, and the language learning model that has completed learning outputs a word equivalent to an answer to the question data from the string data when the string data and the question data are input.

[0008] The information processing method according to the third aspect of the present disclosure is preferably configured such that, in addition to the information processing method according to the first aspect or the second aspect, the first word is extracted from the character string data to generate the question data.

[0009] An information processing method according to a fourth aspect of the present disclosure is the information processing method according to the third aspect, and is preferably configured such that the first word is a verb or an adjective, and the second word is a noun.

[0010] The information processing method of the fifth viewpoint of the present disclosure is based on the information processing method of the third viewpoint or the fourth viewpoint, and is preferably constructed to extract a word existing in the dictionary data storing specified words from among the multiple verb or adjective words included in the character string data as a first word, and generate the question data.

[0011] An information processing method according to a sixth aspect of the present disclosure is the information processing method according to any one of the first to fifth aspects, and is preferably configured such that there are plural numbers of each of the first word and the second word.

[0012] The information processing method of the seventh viewpoint of the present invention is based on the information processing method of any one of the first viewpoints to the sixth viewpoints, and is preferably constructed such that the sound data is divided into multiple scenes, the second word is extracted from the character string data of each segment, and the first word and the second word are stored in association with each segment.

[0013] The information processing method of the eighth aspect of the present disclosure is based on the information processing method of the seventh aspect, and is preferably configured to extract the second word from the entire character string data of the sound data, and store the first word and the second word in association with the file of the sound data.

[0014] The information processing method of the ninth viewpoint of the present invention is based on the information processing method of any one of the first viewpoints to the eighth viewpoints, and is preferably constructed to extract a first word from a template of a report containing characters and generate the question data, input a second word extracted from the character string data into the template, and store the report data obtained by inputting the second word into the template in association with the sound data.

[0015] The information processing method of the tenth aspect of the present invention is based on the information processing method of any one of the first to ninth aspects, and is preferably constructed to obtain video data obtained by on-site filming and recording during maintenance and inspection of equipment, convert the sound data contained in the acquired video data into character string data, use question data containing a first word to extract a second word from the character string data, and store the video data, the first word, and the second word in association with each other.

[0016] The information processing method according to an eleventh aspect of the present disclosure is the information processing method according to the tenth aspect, and is preferably configured such that a first word and a second word related to the video of the video data are overlapped.

[0017] The information processing method of the twelfth aspect of the present invention is based on the information processing method of any one of the first to tenth aspects, and is preferably constructed to accept a search request containing characters, and detect the sound data associated with the first word and the second word related to the characters of the search request from a plurality of the sound data stored in a database.

[0018] The information processing method of the thirteenth aspect of the present disclosure converts sound data included in video data into character string data, uses question data containing a first word, extracts a second word from the character string data, and outputs the question data containing the first word and the second word together with the video data.

[0019] The information processing device of the fourteenth aspect of the present disclosure includes: a processing unit, which converts sound data into character string data, uses question data including a first word, and extracts a second word from the character string data; and a storage unit, which stores the sound data, the first word, and the second word in association with each other.

[0020] The information processing device of the fifteenth aspect of the present disclosure includes: a processing unit, which converts sound data included in video data into character string data, uses question data containing a first word, and extracts a second word from the character string data; and an output unit, which outputs the question data containing the first word and the second word together with the video data.

[0021] The computer program of the sixteenth aspect of the present disclosure causes a computer to execute the following processing: converting voice data into character string data, extracting a second word from the character string data using question data including a first word, and storing the voice data, the first word, and the second word in association with each other.

[0022] The computer program of the seventeenth aspect of the present disclosure causes a computer to perform the following processing: converting sound data included in video data into character string data, using question data containing a first word, extracting a second word from the character string data, and outputting the question data containing the first word and the second word together with the video data. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 This is a schematic diagram showing an overview of the information processing system according to the first embodiment. Figure 2 This is a block diagram showing the structure of a server device according to the first embodiment. Figure 3 This is a conceptual diagram showing an example of a video DB according to the first embodiment. Figure 4 This is a block diagram showing the structure of the language learning model according to the first embodiment. Figure 5 This is a block diagram showing the structure of BERT which is an example of a language learning model according to the first embodiment. Figure 6 This is a block diagram showing the structure of a terminal device according to the first embodiment. Figure 7 This is a flowchart showing the index information generation processing procedure of the first embodiment. Figure 8 This is a conceptual diagram showing the index information generation processing method according to the first embodiment. Fig. 9 This is a flowchart showing the video search processing steps of the first embodiment. Fig.10 This is a schematic diagram showing an example of a video playback screen according to the first embodiment. Fig.11 This is a flowchart showing the information processing steps of the second embodiment. Fig.12 This is a flowchart showing the steps of generating scene index information. Fig.13 This is a conceptual diagram showing a method for matching video scenes with utterance sentence data. Fig.14 This is a flowchart showing the steps of generating the file index information. Fig.15 This is a flowchart showing the report preparation procedure of the second embodiment. Fig.16 This is a schematic diagram showing an example of a report template. Fig.17 This is a conceptual diagram showing an example of a video DB according to the second embodiment. Fig.18 This is a flowchart showing the video search processing steps of the second embodiment. Fig.19 This is a schematic diagram showing an example of a video playback screen according to the second embodiment. Fig. 20 This is a block diagram showing the structure of a server device according to a third embodiment. Fig.21 This is a flowchart showing the index information generation processing steps of the fourth embodiment. DETAILED DESCRIPTION

[0024] Hereinafter, the information processing method, the information processing device, and the computer program of the present disclosure will be described in detail based on the drawings showing embodiments thereof.

[0025] (Implementation Method 1) Maintenance, inspection, repair, or construction of various equipment such as air conditioners and chemical plants require technical proficiency, and the efficiency of the work varies greatly depending on the proficiency of the operator. As a method of assisting unskilled workers in their work, it is considered to collect and accumulate video data obtained by shooting the work of skilled workers and provide the accumulated video data to the unskilled workers. In order to retrieve the video data required by the unskilled workers from the accumulated video data, it is necessary to give appropriate index information to the video data.

[0026] The present disclosure proposes an information processing method, an information processing device, and a computer program (program product) capable of associating video or sound data obtained by shooting or recording with index information that accurately represents the content of the data.

[0027] <System Structure> Figure 1 1 is a schematic diagram showing an overview of an information processing system of Embodiment 1. The information processing system of Embodiment 1 includes a server device (information processing device, computer) 1, a head mounted device 2, and a terminal device 3. The server device 1 is connected to the head mounted device 2 and the terminal device 3 via a wired or wireless communication network such as a mobile phone communication network, a wireless LAN (local area network), and the Internet.

[0028] The head mounted device 2 is a device mounted on the head of a worker who performs maintenance, inspection, repair, or construction of the air conditioner A, particularly a skilled worker B. The head mounted device 2 has a camera 2a, a microphone 2b, and headphones, and captures images of the working conditions of the skilled worker B and collects sounds. The video data includes sound data collected by the microphone 2b. The head mounted device 2 is an example of a device for capturing images and collecting sounds of the working conditions of the skilled worker B, and may also be other wearable devices or mobile terminals having the functions of capturing images and collecting sounds. The head mounted device 2 may also be replaced by a camera 2a and a microphone 2b provided around the air conditioner A and the skilled worker B.

[0029] The video data obtained by shooting and sound collection is provided to the server device 1. For example, when the head mounted device 2 has a communication circuit, the head mounted device 2 sends the video data to the server device 1 through wired or wireless communication. The head mounted device 2 may also be configured to send the video data to the server device 1 via a communication terminal such as a PC (personal computer) or a smartphone. When the head mounted device 2 does not have a communication circuit, the head mounted device 2 records the video data in a recording device such as a memory card or an optical disk. The video data is provided from the head mounted device 2 to the server device 1 via the recording device. The above-described method of providing video data from the head mounted device 2 to the server device 1 is merely an example, and any known method may be employed.

[0030] The server device 1 acquires the video data provided from the head mounted device 2, and accumulates the acquired video data in the video DB 12b. The terminal device 3 is a general communication terminal such as a smartphone or a PC used by an unskilled worker C who studies and performs maintenance, inspection, repair, or construction of the air conditioning equipment A. The terminal device 3 accesses the server device 1, and requests the server device 1 to retrieve the video data desired by the unskilled worker C. The server device 1 retrieves the video data according to the request from the terminal device 3, and sends the required video data to the terminal device 3. The terminal device 3 receives the video data sent according to the request. The terminal device 3 displays a video recording the status of the operation performed by the skilled worker B by playing the received video data. The unskilled worker C can learn the skills of the skilled worker B through the video displayed on the terminal device 3.

[0031] <Device Structure> Figure 2 1 is a block diagram showing the configuration of the server device 1 according to Embodiment 1. The server device 1 according to Embodiment 1 is configured to include a control unit 11 , a storage unit (memory) 12 , and a communication unit (transceiver) 13 .

[0032] The control unit 11 has a CPU (central processing unit), an MPU (microprocessor), a GPU (graphics processing unit) or a quantum processor, a ROM (read-only memory), a RAM (random access memory), etc. The control unit 11 reads out and executes a server program (computer program) 12a stored in the storage unit 12, thereby performing a process of assigning index information to the accumulated video data. The index information is information indicating the content of the video data by a plurality of words. The control unit 11 performs a process of retrieving the required video data by referring to the index information and sending it to the terminal device 3. The control unit 11 functions as a voice recognition unit 11a, a natural language processing unit 11b, an AI processing unit 11c, a marker 11d, and a video processing unit 11e. Each functional unit may be implemented by software by the control unit 11 reading and executing the server program 12a, or may be implemented by hardware by partially or entirely implementing the program through a circuit. The overview of each functional unit is as follows.

[0033] The voice recognition unit 11a is a component that converts the voice data included in the video data into speech data (character string data). The speech data is character string data obtained by converting the speech content of the skilled worker B into text.

[0034] The natural language processing unit 11b is a component that divides the character string represented by the utterance sentence data into morphemes by morpheme analysis, extracts the first word (verb or adjective), and generates question sentence data using the extracted first word. The natural language processing unit 11b is a component that performs processing based on a rule base that does not use the language learning model 12c obtained by machine learning. The question sentence data is data for extracting meaningful nouns from the utterance sentence data.

[0035] The AI ​​processing unit 11c is a component that performs processing to output answer data corresponding to an answer to the question sentence from the utterance sentence data by inputting the question sentence data and the utterance sentence data into the learned language learning model 12c. The answer data includes the second word which is a noun.

[0036] The tagger 11d is a word parser, and has a function as an encoder that encodes the question data and speech data into data that can be processed by the language learning model 12c. When BERT is used as the language learning model 12c, the tagger 11d encodes the question data and speech data into tensor data that has been embedded. Specifically, the tagger 11d divides the question data and speech data into tags (words) that are the smallest units of language, and converts them into tensor data of tag columns in which tag IDs are arranged. The tagger 11d inserts a special tag [CLS] at the beginning of the sentence, and embeds a special tag [SEP] between the tag column of the question data and the tag column of the speech data. The tagger 11d adds segment information to the tensor data of the tag column, and the segment information is used to identify whether each tag is equivalent to a question tag or a speech tag. The tagger 11d adds position information to the tensor data of the tag column, and the position information indicates the arrangement order of multiple tags equivalent to question sentences and speech sentences. The tokenizer 11d also has a function as a decoder that decodes the tensor data output from the language learning model 12c into data of a character string.

[0037] The video processing unit 11e is a component that performs processing such as parsing video data and dividing the video data as a file into a plurality of scenes. In the following, in the first embodiment, an example of adding index information to the video data as a file is described. In the second embodiment, a method of adding index information to each of the plurality of scenes obtained by division is described.

[0038] The storage unit 12 is, for example, a large-capacity storage device such as a hard disk. The storage unit 12 stores the server program 12a executed by the control unit 11 and various data required for the processing of the control unit 11. The storage unit 12 constitutes a video DB (database) 12b that accumulates video data obtained by shooting and sound collection using the camera 2a and the microphone 2b. The storage unit 12 stores a language learning model 12c for generating index information assigned to video data. The storage unit 12 may also be an external storage device connected to the server device 1.

[0039] The server program 12a may be recorded in the recording medium 10 in a computer-readable manner. The storage unit 12 stores the server program 12a read from the recording medium 10 by a reading device. The recording medium 10 is a semiconductor memory, an optical disk, a magnetic disk, a magneto-optical disk, etc. The server device 1 may also download the server program 12a of the first embodiment from an external server connected to the network N and store it in the storage unit 12.

[0040] Figure 3 1 is a conceptual diagram showing an example of a video DB 12b. The video DB 12b is a database that stores video data obtained by shooting and sound collection using the camera 2a and the microphone 2b, the shooting date, and index information generated by the information processing method of the first embodiment in association with each other. The index information is information including the first word and the second word described later.

[0041] Figure 4 This is a block diagram showing the structure of the language learning model 12c of the first embodiment. The language learning model 12c is a machine learning model that has completed learning, and when question sentence data and speech sentence data are input, the model outputs answer data corresponding to the answer to the question expressed in the question sentence data from the speech sentence data. The language learning model 12c is constructed using, for example, a deep neural network. The structure of the language learning model 12c is not particularly limited, but is preferably BERT. The following description assumes that the language learning model 12c is constructed using BERT.

[0042] Figure 5 This is a block diagram showing the structure of BERT as an example of a language learning model 12c in Embodiment 1. The language learning model 12c formed by BERT has a plurality of connected transformer encoders (Transformer encoders) (Trm) 12d. The transformer encoder 12d of the first layer corresponding to the input layer has a plurality of nodes for inputting element values ​​of tensor data of question sentence data and speech sentence data. Figure 5In the figure, "Tok1" to "TokN" on the lower side represent the tag ID of the question sentence data, "Tok1" to "TokM" represent the tag ID of the speech sentence data, and "CLS" and "SEP" represent special tags. The multiple transformer encoders 12d equivalent to the middle layer perform calculations corresponding to the required tasks on the values ​​output from the nodes of the transformer encoder 12d of the previous layer, and output them to the transformer encoder 12d of the next layer. The BERT of this first embodiment performs calculations to extract tags corresponding to the responses to the question sentences. The transformer encoder 12d of the final layer, which is equivalent to the output layer, has the same number of nodes as the transformer encoder 12d of the first layer, and outputs tensor data of the response sentence. Figure 5 In the figure, "Tok1", "Tok2", ... on the upper side indicate the tag IDs of the reply data.

[0043] BERT, i.e., language learning model 12c, can be learned through pre-learning and fine-tuning. Pre-learning is performed using unlabeled learning data. Specifically, word prediction learning (MLM: Masked LM) and next sentence prediction (NSP: Next Sentence Prediction) learning are used to learn the neural network. In word prediction learning, a portion of the tag column of the input sentence serving as learning data is masked, and the weighting coefficient of the transformer encoder 12d is optimized to be able to predict the masked tags. In next sentence prediction learning, the weighting coefficient of the transformer encoder 12d is optimized to be able to correctly determine whether the first character string and the second character string are subsequent character strings. In fine tuning, when tensor data of question sentence data and utterance sentence data are input, the weighting coefficient of the transformer encoder 12d is finely corrected so as to output tensor data of desired answer data. The language learning model 12c may fine-tune BERT using question sentence data and utterance sentence data actually used, or may use BERT fine-tuned using general character string data.

[0044] The communication unit 13 communicates with the head mounted device 2 and the terminal device 3 via a network N including a mobile phone communication network, a wireless LAN, and the Internet. The communication unit 13 sends data provided from the control unit 11 to the head mounted device 2 or the terminal device 3, and provides data received from the head mounted device 2 or the terminal device 3 to the control unit 11.

[0045] Although the example in which the server device 1 is composed of a single computer device has been described, the server device 1 may be a multi-computer including a plurality of computers and performing distributed processing. The server device 1 may also be a virtual machine virtually constructed by software.

[0046] Figure 6This is a block diagram showing the configuration of the terminal device 3 according to Embodiment 1. The terminal device 3 includes a control unit 31 , a storage unit (memory) 32 , a communication unit (transceiver) 33 , a display unit (display) 34 , and an operation unit 35 .

[0047] The control unit 31 includes, for example, a CPU or MPU, a processing unit, a ROM, etc. The control unit 31 reads and executes a terminal program 32a stored in the storage unit 32 to perform a search request process for video data accumulated in the video DB 12b of the server device 1 and a playback process (display process) of video data provided from the server device 1. The terminal program 32a may be a dedicated program in the information processing method of the first embodiment, or a general-purpose program such as an Internet browser or a web browser.

[0048] The storage unit 32 is, for example, a non-volatile storage element such as a flash memory or a storage device such as a hard disk. The storage unit 32 stores the terminal program 32a executed by the control unit 31 and various data required for the processing of the control unit 31. It can also be recorded in the recording medium 30 in a computer-readable manner. The storage unit 32 stores the terminal program 32a read from the recording medium 30 by the reading device. The recording medium 30 is a semiconductor memory, an optical disk, a magnetic disk, a magneto-optical disk, etc. The terminal device 3 can also download the terminal program 32a of the first embodiment from an external server connected to the network N and store it in the storage unit 12.

[0049] The communication unit 33 communicates with the server device 1 via the network N. The communication unit 33 transmits data supplied from the control unit 31 to the server device 1 , and supplies data received from the server device 1 to the control unit 31 .

[0050] The display unit 34 is a liquid crystal panel, an organic EL display, etc. The display unit 34 displays a video, a still image, characters, etc. according to the data supplied from the control unit 31 .

[0051] The operation unit 35 is an input device such as a touch panel, a software key, a hardware key, a keyboard, a mouse, etc. The operation unit 35 receives an operation by, for example, an unskilled person C, and notifies the control unit 31 of the received operation.

[0052] <Information Processing Method (Generation and Assignment of Index Information)> The server device 1 can generate index information that accurately indicates the content of video data obtained by capturing a situation of work such as maintenance, inspection, repair, or construction of the air-conditioning equipment A performed by the skilled worker B. Figure 7 is a flowchart showing the index information generation processing steps of the first implementation mode, Figure 81 is a conceptual diagram showing an index information generation processing method of the first embodiment. The control unit 11 of the server device 1 obtains video data (step S111). For example, the server device 1 obtains the video data by receiving the video data sent from the head mounted device 2 using the communication unit 13. The video data is obtained by photographing and collecting sound of the working conditions of the skilled worker B, and includes sound data. The server device 1 can also obtain the video data by reading the video data stored in the storage unit 12 or an external storage device.

[0053] The control unit 11 extracts sound data from the acquired video data (step S112). The control unit 11 or the sound recognition unit 11a converts the extracted sound data into text speech sentence data through sound recognition processing (step S113). The control unit 11 or the natural language processing unit 11b divides the speech sentence data into morphemes through morpheme analysis processing, and extracts one or more first words that are verbs or adjectives (step S114). For example, the first word is a verb such as "repair" or "replace", or an adjective such as "hot" or "slow". The control unit 11 may also extract all verbs and adjectives included in the speech sentence data as the first word, or may extract a specified number of verbs and adjectives as the first word. The control unit 11 may also randomly extract a specified number of verbs and adjectives as the first word. The control unit 11 may also extract a specified number of verbs and adjectives as the first word in a manner that the variance of the similarity becomes larger. The control unit 11 may also extract the first word in a manner that the playback time is deviated. The control unit 11 may extract, as the first word, verbs and adjectives whose application frequencies are within a predetermined range, for example, a 1σ range.

[0054] The control unit 11 or the natural language processing unit 11b generates one or more question sentence data based on the one or more first words (step S115). For example, the control unit 11 uses the first word "repair" to generate question sentence data such as "What was repaired?" For example, the control unit 11 uses the first word "replace" to generate question sentence data such as "What was replaced?" It is also possible to generate a plurality of question sentence data based on one first word. For example, the control unit 11 may generate question sentence data such as “What was repaired?”, “What was used to repair it?”, and “How was it repaired?”. The storage unit 12 may also be configured to store a dictionary of related words. When the storage unit 12 stores the dictionary of related words, the control unit 11 generates question sentence data using the related words of "repair". For example, when the related words of "repair" are "problem", "component", "error code", etc., question sentence data such as "What is the problem?", "What is the component?", and "What is the error code?" are generated. The storage unit 12 may be configured to store fixed question data. The control unit 11 may add the fixed question data read from the storage unit 12 to the generated question data. For example, question data such as "What is the model of the device?" may be added as a fixed question.

[0055] The control unit 11 outputs the reply data by inputting the question sentence data and the speech sentence data into the language learning model 12c (step S116). In the case where there are multiple question sentence data, multiple corresponding reply data are obtained. The reply data includes a second word as a noun. Specifically, the marker 11d encodes the question sentence data and the speech sentence data into tensor data. The control unit 11 outputs tensor data related to the reply sentence by inputting the encoded tensor data into the language learning data 12c. The marker 11d decodes the tensor data output from the language learning model 12c into reply data.

[0056] The control unit 11 generates index information based on the first word and the second word (step S117). For example, the index information is data obtained by arranging the first word and the second word.

[0057] The control unit 11 stores the generated index information in association with the video data in the storage unit 12 (step S118). Specifically, the control unit 11 stores the video data and the index information in the video DB 12b.

[0058] <Video retrieval processing> The unskilled worker C can use the terminal device 3 to search and view the video data stored in the video DB 12 b of the server device 1 . Fig. 9 1 is a flowchart showing the video search processing steps of the first embodiment. The control unit 31 of the terminal device 3 displays a search screen for searching the video data stored in the video DB 12b of the server device 1 on the display unit 34 (step S171). The control unit 31 receives the search word through the operation unit 35 (step S172). The control unit 31 transmits the search request data for requesting the search of the video data including the received search word to the server device 1 through the communication unit 33 (step S173).

[0059] The server device 1 receives the search request data sent from the terminal device 3 through the communication unit 13 (step S174). The control unit 11 of the server device 1 that receives the search request data uses the search word included in the search request data as a keyword and refers to the index information stored in the video DB 12b to search for video data that matches the search word (step S175). The control unit 11 sends the search result of step S175 to the terminal device 3 of the search request source through the communication unit 13 (step S176). The search result includes the file name, thumbnail image, shooting date, playback time, index information, etc. of the video data.

[0060] The control unit 31 of the terminal device 3 receives the search result transmitted from the server device 1 via the communication unit 33 (step S177). The control unit 31 displays the information of the search result on the display unit 34, and accepts the selection of the video to be played via the operation unit 35 (step S178).

[0061] The control unit 31 transmits video request data requesting video data including information indicating the selected video, such as the file name of the video data, to the server device 1 via the communication unit 33 (step S179 ).

[0062] The control unit 11 of the server device 1 receives the video request data sent from the terminal device 3 through the communication unit 13 (step S180). The control unit 11 obtains the video data and index information represented by the video request data from the video DB 12b (step S181). The control unit 11 sends the read video data and index information to the terminal device 3 of the video request source through the communication unit 13 (step S182).

[0063] The control unit 31 of the terminal device 3 receives the video data and index information transmitted from the server device 1 via the communication unit 33 (step S183). The control unit 31 plays the received video data and displays it on the display unit 34 (step S184). The control unit 31 displays the index information superimposed on the video image (step S185).

[0064] Fig.10 1 is a schematic diagram showing an example of a video playback screen 34a of the first embodiment. The terminal device 3 displays the video playback screen 34a on the display unit 34, for example. The terminal device 3 displays a video based on the video data received from the server device 1 in the center of the video playback screen 34a. The terminal device 3 displays index information in an overlapping manner on the upper or lower part of the video. The terminal device 3 displays operation buttons such as a play button, a pause button, a stop button, a fast forward button, and a fast rewind button at the lower part of the video playback screen 34a, and the video displayed in the center of the screen of the display unit 34 is displayed. When various buttons are operated, the control unit 31 controls the playback of the video according to the operated button.

[0065] According to the information processing system of the first embodiment, the video data and the index information accurately representing the content of the video can be stored in the video DB 12b in association. Since the second word is extracted from the utterance data using the question data including the first word, the second word includes information meaningful in terms of the content corresponding to the question data. The first word and the second word are information accurately representing the content of the video data, and the first word and the second word can be associated with the video data as index information.

[0066] By using the language learning model 12c as a machine learning model, it is possible to extract a second word that more accurately represents the content of the utterance sentence data. In particular, by using BERT, it is possible to extract a second word that is more meaningful in content from the utterance sentence data.

[0067] Since the question data is generated using the first word extracted from the utterance, the second word that more accurately represents the content of the utterance data can be extracted. The first word is information included in the utterance data of the video data, so the question data based on the content of the video data can be obtained.

[0068] Since the first word constituting the question sentence data is a verb or an adjective, it is possible to generate question sentence data suitable for extracting a second word related to the verb or adjective, that is, extracting a noun.

[0069] Since there are a plurality of first words and second words associated with the video data, it is possible to generate index information that more specifically indicates the content of the video data.

[0070] The first word and the second word of the index information associated with the video data obtained by shooting and recording at the site of equipment maintenance and inspection represent the content of the video data. The content of the video data can be confirmed by referring to the first word and the second word of the index information.

[0071] In the video of the video data, index information including the first word and the second word can be displayed on the video.

[0072] By referring to the index information, desired video data can be retrieved.

[0073] In addition, in the first embodiment, the video data obtained by photographing and collecting sound of the operation status of the air conditioning equipment A is used as an example for explanation, but the operation object is not limited to maintenance, repair, or construction. The information processing method of the first embodiment can also be applied to the video data obtained by photographing and collecting sound of the maintenance and inspection status of chemical plants and other various equipment. The information processing method and the like of the first embodiment can also be applied to video data or audio data captured or recorded for call center support, sales support, or employee training.

[0074] In the first embodiment, the index information is associated with the video data as an example, but the information processing method of the first embodiment can also be applied to the sound data. That is, the index information generated by the information processing method of the first embodiment can be associated with the sound data and stored.

[0075] (Implementation Method 2) The information processing device of the second embodiment is different from the first embodiment in that it divides the video data into a plurality of scenes and also adds index information to each scene. The information processing device of the second embodiment is different from the first embodiment in that it automatically creates a work report for the video data capturing the status of the work such as the maintenance and inspection of the air conditioner A. The video data playback method of the information processing device of the second embodiment is different from that of the first embodiment. The other structures and processes of the information processing system are the same as those of the information processing system of the first embodiment, and therefore, the same reference numerals are given to the same parts and detailed descriptions are omitted.

[0076] <Information Processing Method (Generation and Assignment of Index Information)> Fig.11 1 is a flowchart showing the information processing steps of the second embodiment. The control unit 11 of the server device 1 obtains video data (step S211). The control unit 11 or the video processing unit 11e analyzes the video data and divides the video data as a file into multiple scenes (step S212). For example, the video processing unit 11e divides the video content into multiple scenes based on the change in brightness of each frame image constituting the video, the change in the feature quantity of the object, etc. The control unit 11 associates the scene data with the video data and stores it in the video DB 12b (refer to Fig.17 ), wherein the scene data includes information such as a scene number for identifying each scene, an end frame number of each scene, a playback time indicating a start position and an end position of each scene, etc., as information representing multiple scenes.

[0077] The control unit 11 extracts the sound data from the acquired video data (step S213). The control unit 11 or the sound recognition unit 11a converts the extracted sound data into text speech data through sound recognition processing (step S214). Specifically, the control unit 11 or the sound recognition unit 11a converts the sound data into text speech data for each interval of speech. The control unit 11 or the sound recognition unit 11a temporarily stores the speech data group including the numbers identifying the plurality of speech data, the playback time indicating the playback start position and end position of each speech data, and the speech data in the storage unit 12.

[0078] The control unit 11 executes a process of generating index information based on a plurality of utterance sentence data of each scene (step S215). Hereinafter, the index information generated based on the utterance sentence data of each scene is referred to as scene index information.

[0079] Fig.12 2 is a flowchart showing the procedure for generating scene index information. The control unit 11 matches each scene of the video data with the utterance sentence data (step S231).

[0080] Fig.13 is a conceptual diagram showing the matching method between video scenes and speech sentence data. Fig.13 As shown, the control unit 11 refers to the scene data and compares the start position and the end position of each scene with the start position and the end position of each of the plurality of speech sentence data converted in step S214. The control unit 11 identifies the speech sentence data having a start position close to the start position of the scene. The control unit 11 identifies the speech sentence data having an end position close to the end position of the scene. The control unit 11 integrates the identified speech sentence data at the start position of the scene, the speech sentence data between the start position and the end position, and the speech sentence data at the end position of the scene. For example, the start position of the scene of scene number 1 is 00:00, and the end position is 00:12. The speech data corresponding to the start position and the end position of the scene are speech data No. 1 to No. 3, and the control unit 11 integrates the speech data No. 1 to No. 3. Similarly, the start position of the scene of scene number 2 is 00:12, and the end position is 00:23. The speech data corresponding to the start position and the end position of the scene are speech data No. 4 to No. 7, and the control unit 11 integrates the speech data No. 4 to No. 7.

[0081] The control unit 11 or the natural language processing unit 11b divides the speech sentence data of a scene into morphemes through morpheme analysis processing, and extracts one or more first words that are verbs or adjectives (step S232). The control unit 11 or the natural language processing unit 11b generates one or more question sentence data based on the one or more first words (step S233). The control unit 11 outputs the reply data by inputting the question sentence data and the speech sentence data into the language learning model 12c (step S234). In the case where there are multiple question sentence data, the corresponding multiple reply data are obtained. The reply data includes a second word that is a noun. The control unit 11 generates scene index information based on the first word and the second word (step S235).

[0082] The control unit 11 determines whether the process of generating scene index information for all scenes is completed (step S236). If it is determined that there is a scene for which scene index information has not been generated (step S236: No), the control unit 11 returns the process to step S232. If it is determined that scene index information for all scenes has been generated (step S236: Yes), the process of generating scene index information is completed.

[0083] Back to Fig.11 The control unit 11 executes a process of generating index information based on the video data as a file (step S216). Hereinafter, the index information generated based on the video data as a file is referred to as file index information.

[0084] Fig.14 25 is a flowchart showing the steps of generating the file index information. The control unit 11 or the natural language processing unit 11b divides the speech sentence data (the entire character string data) of the entire video data into morphemes through morpheme analysis processing, and extracts one or more first words that are verbs or adjectives (step S251). The control unit 11 or the natural language processing unit 11b generates one or more question sentence data based on the one or more first words (step S252). The control unit 11 outputs the reply data by inputting the question sentence data and the speech sentence data into the language learning model 12c (step S253). The reply data includes the second word that is a noun. The control unit 11 generates the file index information based on the first word and the second word (step S254), and ends the file index information generation processing.

[0085] Back to Fig.11 The control unit 11 creates a report based on the utterance sentence data (step S217). The report includes information related to the maintenance and inspection of the air conditioner A.

[0086] Fig.15This is a flowchart showing the report creation procedure of Embodiment 2. The storage unit 12 of the server device 1 stores the report template, and the control unit 11 of the server device 1 obtains the report template from the storage unit 12 (step S271).

[0087] Fig.16 This is a schematic diagram showing an example of a report template. The report template includes a plurality of input item characters representing items of information to be input. The input item characters are, for example, "item", "repair location", "inquiry number", "customer name", "customer address", "telephone number", "model name", "repair date", etc.

[0088] The control unit 11 extracts a plurality of first words, i.e., a plurality of input item characters, from the acquired report template (step S272). The control unit 11 or the natural language processing unit 11b generates a plurality of question sentence data based on the plurality of first words (step S273). The control unit 11 outputs the reply data by inputting the question sentence data and the speech sentence data into the language learning model 12c (step S274). The reply data includes a second word as a noun. The second word is information that should be input into the item represented by the input item character. The control unit 11 generates the report data obtained by inputting the reply data into the report template (step S275), and ends the report making process. The form of the report data is not particularly limited, and the report data is, for example, data that establishes a corresponding arrangement of the input item characters of the report template and the reply data corresponding to the item. The report data may also be image data that displays the reply data at each item of the report template.

[0089] Back to Fig.11 The control unit 11 stores the generated scene index information, file index information, and report data in the storage unit 12 in association with the video data (step S218).

[0090] Fig.17 1 is a conceptual diagram showing an example of the video DB 12b according to the second embodiment. Fig.17 As shown, the control unit 11 associates the video data as one file with the file index information. The control unit 11 associates the scene index information with each of the multiple scenes. Specifically, the video data is associated with the scene number, the end frame number, and the information indicating the playback time of the start position and the end position of each of the multiple scenes, and the control unit 11 stores the scene index information corresponding to the scene in the video DB 12b in association with each scene number. The control unit 11 associates the report data with the video data.

[0091] <Video retrieval processing> Fig.181 is a flowchart showing the video search processing steps of the second embodiment. The control unit 31 of the terminal device 3 and the control unit 11 of the server device 1 perform the same processing as steps S171 to S180 described in the first embodiment, and the server device 1 receives the video request data through the communication unit 13 (steps S271 to S280). In addition, in step S275, the control unit 11 refers to the file index information associated with the video data to search for the video data. The substance of the processing is the same as that of the first embodiment.

[0092] The control unit 11 of the server device 1 obtains the video data, file index information and report data indicated by the video request data (step S281). The control unit 11 uses the search word included in the search request data as a keyword and refers to the scene search information to determine the scene matching the search word (step S282).

[0093] The control unit 11 transmits the acquired video data, file index information, scene data, and scene specifying information specifying the scene determined in step S282 to the terminal device 3 that is the video request source via the communication unit 13 (step S283).

[0094] The control unit 31 of the terminal device 3 receives the video data, file index information, scene data, scene index information, and scene designation information sent from the server device 1 through the communication unit 33 (step S284). The control unit 31 starts playing the received video data from the scene indicated by the scene designation information and displays it on the display unit 34 (step S285). The control unit 31 displays the file index information and the scene index information corresponding to the scene being played on the image of the video in an overlapping manner (step S286). Specifically, the control unit 31 determines the scene being played and the scene index information corresponding to the scene by referring to the scene data. The control unit 31 displays the file index information and the determined scene index information in an overlapping manner on the video.

[0095] The control unit 31 displays the received report data on the display unit 34 (step S287 ). The control unit 31 may be configured to display the report data in accordance with the operation of the operation unit 35 .

[0096] Fig.19This is a schematic diagram showing an example of a video playback screen 34a of the second embodiment. The terminal device 3 displays the video playback screen 34a on the display unit 34, for example. The terminal device 3 displays a video based on the video data received from the server device 1 in the center of the video playback screen 34a. The control unit 31 of the terminal device 3 displays the file index information and the scene index information in an overlapping manner on the upper and lower parts of the video, respectively. The control unit 31 displays the scene number in an overlapping manner on the lower right side of the video. The control unit 31 can also be configured to display a character string obtained by summarizing the speech sentence data of the video data by a known technique in an overlapping manner on the video. The display position of the file index information, the scene index information, the scene number, and the summary is an example.

[0097] The control unit 31 displays the report on the video playback screen 34a based on the report data. For example, the control unit 31 displays the report data and the video in parallel.

[0098] According to the information processing system and the like of the second embodiment, a plurality of scenes obtained by dividing the video data can be stored in the video DB 12 b in association with each other and with scene index information accurately indicating the contents thereof. The files of the undivided video data are stored in the video data DB 12 b in association with the scene index information accurately indicating the contents thereof.

[0099] The video data can be automatically played starting from the scene related to the search word.

[0100] A report on maintenance and inspection of air conditioner A can be automatically prepared based on video data. A first word is extracted from a report template and question data is generated. The first word represents an item to be input into the report. A second word extracted from speech data using question data is information corresponding to the item. Report data representing the content of video data can be prepared by inputting the second word into the template. The terminal device 3 can display reports and play video data.

[0101] (Implementation Method 3) The information processing device of the third embodiment differs from the first and second embodiments in that the first word is extracted from the utterance data using the dictionary data 312d and the question sentence data is generated. The other structures and processes of the information processing system are the same as those of the information processing system of the first and second embodiments, and therefore, the same reference numerals are given to the same parts and detailed descriptions are omitted.

[0102] Fig. 20This is a block diagram showing the structure of the server device 1 according to Embodiment 3. The storage unit 12 of the server device 1 according to Embodiment 3 stores dictionary data 312d. The dictionary data 312d stores verbs and adjectives (prescribed words) suitable for generating question data and verbs and adjectives unsuitable for generating question data.

[0103] When extracting the first word from the speech sentence data, the control unit 11 selects the dictionary data 312d and performs selection. For example, the control unit 11 determines whether the verb or adjective extracted from the speech sentence data is consistent with the verb or adjective stored in the dictionary data 312d as the verb or adjective suitable for the generation of the question sentence data, and extracts it as the first word if it is determined to be consistent. The control unit 11 determines whether the verb or adjective extracted from the speech sentence data is consistent with the verb or adjective stored in the dictionary data 312d as the verb or adjective not suitable for the generation of the question sentence data, and does not extract it as the first word if it is determined to be consistent. If the verb or adjective extracted from the speech sentence data does not exist in the dictionary data 312d, the control unit 11 can extract it as the first word.

[0104] The processing after the first word extraction is the same as that in the first and second embodiments, and question sentence data is generated, and answer data is acquired from the utterance sentence data to generate index information.

[0105] According to the third embodiment, the server device 1 can generate more accurate question sentence data. By inputting appropriate question sentence data and speech sentence data into the language learning model 12c, more accurate answer data (second data) can be output. Therefore, index information that more accurately represents the content of the video data can be generated and associated with the video data.

[0106] (Implementation Method 4) The information processing device of the fourth embodiment differs from the first to third embodiments in that it outputs the generated index information to the outside. The other configurations and processes of the information processing system are the same as those of the first to third embodiments, and therefore the same reference numerals are used for the same parts and detailed descriptions are omitted.

[0107] Fig.21It is a flowchart showing the index information generation processing steps of Implementation Method 4. The control unit 11 of the server device 1 performs the same processing as steps S111 to S116 described in Implementation Method 1, and the server device 1 obtains the first word and the answer data (second word) representing the content of the video data (steps S411 to S416). The control unit 11 outputs the question sentence data including the first word, the answer data (second word) together with the video data to the outside (step S417). The control unit 11, for example, plays the video data and displays the question sentence data and the answer data on an external display device. The control unit 11 can also output or send the video data, the question sentence data, and the answer data to an external computer. The control unit 11 that executes the process of step S417 functions as an output unit that outputs the question data including the first word and the second word together with the video data.

[0108] According to the fourth embodiment, video data and index information accurately indicating the content of the video can be output to the outside.

[0109] The embodiments are described above, but the present invention is not limited to these examples, and it should be understood that various changes in form and details can be made without departing from the spirit and scope of the claims. In addition, at least a part of the above embodiments can be arbitrarily combined. Explanation of symbols

[0110] 1. Server device (information processing device, computer); 2Head mounted device; 2a Camera; 2b Microphone; 3. Terminal device; 11. Control Department; 11a voice recognition unit; 11b Natural Language Processing Department; 11c AI processing unit; 11d marker; 11e video processing unit; 12 storage unit; 12a Server programs (computer programs); 12b videodb; 12c language learning model; 12d transformer encoder; 312d dictionary data; 13 Ministry of Communications; 31 Control Department; 32 storage unit; 32a terminal program; 33 Ministry of Communications; 34 display unit; 34a video playback screen; 35 operation unit; 10, 30 recording medium; A. Air conditioning equipment; B. Skilled personnel; C unskilled personnel; N network.

Claims

1. An information processing method, characterized in that: Convert the sound data into string data. Using the question data including the first word, extracting the second word from the character string data, The voice data, the first word, and the second word are stored in association with each other.

2. The information processing method according to claim 1, characterized in that: The second word is extracted from the string data by inputting the string data and the question data into a language learning model that has completed learning. The language learning model that has completed learning outputs a word from the string data that is equivalent to an answer to the question data when the string data and the question data are input.

3. The information processing method according to claim 1 or 2, characterized in that: The question data is generated by extracting a first word from the character string data.

4. The information processing method according to claim 3, characterized in that: The first word is a verb or an adjective, The second word is a noun.

5. The information processing method according to claim 3, characterized in that: The question data is generated by extracting, as a first word, a word present in dictionary data storing predetermined words from among a plurality of verb or adjective words included in the character string data.

6. The information processing method according to claim 1 or 2, characterized in that: There are plural numbers of the first word and the second word respectively.

7. The information processing method according to claim 1 or 2, characterized in that: The sound data is divided into a plurality of scenes, Extract the second word from each segment of the string data. The first word and the second word are stored in association with each segment.

8. The information processing method according to claim 7, characterized in that: extracting a second word from the entire character string data of the sound data, The first word and the second word are stored in association with the file of the voice data.

9. The information processing method according to claim 1 or 2, characterized in that: Extracting a first word from a report template containing characters and generating the question data, inputting a second word extracted from the character string data into the template, The report data obtained by inputting the second word into the template is stored in association with the voice data.

10. The information processing method according to claim 1 or 2, characterized in that: Obtain video data obtained by on-site shooting and recording during equipment maintenance and inspection. Convert the sound data included in the acquired video data into character string data, Using the question data including the first word, extracting the second word from the character string data, The video data, the first word, and the second word are stored in association with each other.

11. The information processing method according to claim 10, characterized in that: A first word and a second word associated with the video of the video data are overlapped.

12. The information processing method according to claim 1 or 2, characterized in that: Accepts search requests containing characters, The voice data associated with the first word and the second word related to the characters of the search request are detected from the plurality of voice data stored in the database.

13. An information processing method, characterized in that: Convert the sound data included in the video data into character string data, Using the question data including the first word, extracting the second word from the character string data, Question data including the first word and the second word are output together with the video data.

14. An information processing device, characterized in that: include: a processing unit that converts the voice data into character string data and extracts a second word from the character string data using question data including the first word; as well as A storage unit stores the voice data, the first word, and the second word in association with each other.

15. An information processing device, characterized in that: include: a processing unit that converts the sound data included in the video data into character string data and extracts a second word from the character string data using the question data including the first word; as well as An output unit outputs question data including a first word and a second word together with the video data.

16. A computer program, characterized in that The computer program is used to make the computer execute the following processing: Convert the sound data into string data. Using the question data including the first word, extracting the second word from the character string data, The voice data, the first word, and the second word are stored in association with each other.

17. A computer program, characterized in that The computer program is used to make the computer execute the following processing: Convert the sound data included in the video data into character string data, Using the question data including the first word, extracting the second word from the character string data, Question data including the first word and the second word are output together with the video data.

Images