Information processing method, program, and storage medium

Abstract

To provide an information processing method, a program, and a storage medium that can more accurately obtain the contents of statements made by participants in a specified Web conference and reduce the burden of obtaining the contents.SOLUTION: In an information processing method according to an embodiment of the present invention, when a subtitle display screen 205 of a Web conference is displayed, an HTML document related to the screen display of the Web conference is analyzed, HTML elements related to the display of comments acquired by a subtitle generation function are referenced to acquire the contents of the comments displayed on the subtitle display screen (steps S31, S33), the acquired contents of the comments are stored in memory 104 as a comment list (step S33), and the stored comment list is output in a predetermined file format (step S35). Every time an HTML element related to the display of a comment is updated, updated information of the HTML element is acquired (steps S32, S33), and the updated information is added to the comment content information (step S33).SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to an information processing method, a program, and a storage medium. [Background technology]

[0002] Due to the recent COVID-19 pandemic, an increasing number of companies are introducing so-called teleworking, where employees work from home or other locations instead of coming to the office, and the demand for online communication via communication networks such as the Internet is increasing rapidly. Web conferencing systems are being used as one of these online communication tools.

[0003] As the demand for such Web conference systems increases, various functions are being implemented. For example, Patent Document 1 discloses that a text chat screen for performing a text chat with users participating in a certain Web conference as participants is generated as an independent screen separate from the Web conference screen, and access information for accessing the text chat screen is output to each of the users. By utilizing the text chat screen, each user as a participant can communicate with the other participants even after the Web conference has ended. Therefore, if participants of the certain Web conference want to share some materials, there is no need to build a separate environment for communication between the participants. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 7162953 specification Summary of the Invention [Problem to be solved by the invention]

[0005] According to the technology disclosed in Patent Document 1, even after a Web conference has ended, participants can easily communicate with each other, so when sharing meeting minutes, meeting notes, etc. with other participants, these files can be sent without constructing a communication environment, which is very useful. However, the creation of the meeting contents such as minutes or a concise summary of them is done by a human being, for example, by creating them based on notes describing the meeting contents, or by recording the meeting contents and creating them from the recorded contents.

[0006] As such, if minutes are created by humans, there is a risk of incorrect note-taking or missing information, the time required to transcribe the recording, and an increased burden on resources required for recording (increased capacity, etc.).

[0007] The present invention has been made in consideration of such problems, and its purpose is to provide an information processing method, program, and storage medium that can more accurately obtain the content spoken by participants in a given Web conference and reduce the burden of obtaining such content. [Means for solving the problem]

[0008] In order to achieve this object, a first aspect of the present invention is an information processing method for converting remarks into text using a subtitle generation function of a Web conference application, and when a subtitle display screen for displaying subtitles for remarks entered by attendees of the Web conference is displayed, the method comprises the steps of: analyzing an HTML document related to the screen display of the Web conference, and acquiring the content of the remarks displayed on the subtitle display screen by referring to HTML elements related to the display of the remarks acquired by the subtitle generation function; saving the acquired content of the remarks in a storage means as remark content information including the content of the remarks; and outputting the saved remark content information in a predetermined file format, wherein the acquiring step acquires updated information of the HTML element each time the HTML element related to the display of the remarks is updated, and the saving step adds the updated information to the remark content information.

[0009] A second aspect of the present invention is a program for causing a computer to execute an information processing method for converting remarks into text using a subtitle generation function of a Web conferencing application, the information processing method comprising the steps of: analyzing an HTML document related to the screen display of the Web conference when a subtitle display screen is displayed that displays subtitles for remarks entered by attendees of the Web conference, and acquiring the content of the remarks displayed on the subtitle display screen by referring to HTML elements related to the display of the remarks acquired by the subtitle generation function; saving the acquired content of the remarks in a storage means as remark content information including the content of the remarks; and outputting the saved remark content information in a predetermined file format, wherein the acquiring step acquires updated information of the HTML element each time the HTML element related to the display of the remarks is updated, and the saving step adds the updated information to the remark content information.

[0010] A third aspect of the present invention is a computer-readable storage medium storing the program according to the second aspect of the present invention. Effect of the Invention

[0011] According to the present invention, it is possible to obtain the contents of statements made by participants in a given Web conference more accurately and reduce the burden of obtaining such contents. [Brief description of the drawings]

[0012] [Figure 1] 1 is a diagram showing a schematic configuration of an information processing apparatus according to an embodiment of the present invention; [Diagram 2] FIG. 13 is a diagram showing a display screen of a Web conference according to an embodiment of the present invention, in which a speech recording function is being executed. [Diagram 3] 10 is a flowchart showing a processing procedure for executing a speech recording function in a Web conference according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, an embodiment of the present invention will be described with reference to the drawings, but the present invention is not limited to this embodiment. In the drawings described below, parts having the same functions are given the same reference numerals, and repeated explanations of those parts may be omitted.

[0014] One embodiment of the present invention involves holding a specific Web conference using a Web conferencing system (an application for conducting Web conferences), which is a method of online communication using the Internet, and using the subtitle generation function of the Web conferencing system (a function that performs voice recognition on the content of a specific speaker's speech and transcribes it based on the recognized speech) to convert the speech of a specific person in the Web conference into text (speech recording function), analyzing the HTML document related to the screen display of the Web conference, obtaining the content of the speech based on the HTML elements related to the content of the speech of the specific speaker obtained by the subtitle generation function, and outputting the obtained content of the speech as text data.

[0015] In the present invention, a "Web conference" is a mechanism that connects devices equipped with communication functions, such as PCs (Personal Computers), smartphones, and tablets, via the Internet, and enables communication with remote parties through the screens of these devices. In other words, the "Web conference" according to the present invention is a mechanism that enables users of the devices communicating via the Internet to communicate with each other, such as through meetings and conversations, and to distribute / view content, such as webinars, through screens displayed by HTML documents.

[0016] FIG. 1 is a diagram showing a schematic configuration of an information processing device 100 according to this embodiment. The information processing device 100 includes a CPU (Central Processing Unit) 101, a GPU (Graphics Processing Unit) 102, a storage 103, a memory 104, an output interface (I / F) 105, a reader 106, a reader interface (I / F) 107, an input interface (I / F) 108, and a communication module 109. A display unit 110 is connected to the information processing device 100 via the output I / F 105, and an input unit 111 is connected to the information processing device 100 via the input I / F 108.

[0017] The CPU 101 executes various processing operations such as various calculations, controls, and determinations, and functions to execute various processes, for example, by loading a predetermined program from the storage 103 into the memory 104 and executing a series of instructions included in the program. The GPU 102 is a processor for performing image processing, and draws an image in response to a predetermined instruction from the CPU 101 and causes the display unit 110 to display the image.

[0018] The storage 103 stores programs (e.g., the program shown in FIG. 3, a Web browser 103a, a program for executing a Web conference application (app), etc.) and various data to be read and executed by the CPU 101. The memory 104 temporarily holds data and programs. The memory 104 is a volatile memory that temporarily stores data being processed by the CPU 101 and GPU 102, input data, and further data (e.g., images) read by the reader 106, and is, for example, a RAM (Random Access Memory).

[0019] The output I / F 105 displays an image on a display unit 110 such as a display according to instructions from the CPU 101 and the GPU 102. The reader 106 is provided in the information processing device 100 and is configured to be able to mount an external storage medium such as a memory card or a DVD (Digital Versatile Disc) in which various data, programs, etc. are stored, and reads predetermined data from the mounted external storage medium. The reader I / F 107 reads data from the external storage medium mounted in the reader 106 according to instructions from the CPU 101. The input I / F 108 accepts an input signal related to an operation input by a user through an input unit 111 such as a keyboard or a microphone as a means for inputting voice. The communication module 109 is connected to a communication network and communicates with other computers connected to the communication network. This communication may be wired or wireless. Predetermined data such as the original image may be acquired via the communication module 109.

[0020] In one embodiment of the present invention, the speech recording function can be executed as a "bookmarklet" that can be registered in the bookmarks of the Web browser 103a. A "bookmarklet" is a script that resides in the form of an icon on the bookmarks toolbar of the Web browser, or exists temporarily, and performs simple processing in the Web browser. Note that the means for executing the speech recording function according to one embodiment of the present invention is not limited to a bookmarklet, but may be anything that can add a function to the Web browser, such as a plug-in.

[0021] Hereinafter, an example of recording comments using a bookmarklet will be described. FIG. 2 is a diagram showing a display screen (a screen displayed when a Web conference application is used) of a Web conference according to this embodiment, and a display screen when a speech recording function is executed. A bookmark toolbar 201 of a Web conference display screen 200 displays a bookmarklet 202 for speech recording. In addition, the attendees of the Web conference are displayed individually on their individual display screens (for example, their own images, etc.), but the speaker display screen 203a displays the current speaker, and the attendee display screens 203b to 203d display images of attendees who are not speaking. Furthermore, the attendee list screen 204 displays a list of names of attendees (users currently accessing this Web conference) of the Web conference. Therefore, when participating in the Web conference, images of each participant (attendee) are appropriately displayed on the screens 203a to 203d, and the attendee list screen 204 displays the name.

[0022] The subtitle display screen 205 is a screen that displays the contents of the voice uttered by the speaker in the Web conference, and is a screen that displays the contents of what a specific person has said in real time. For example, when "Haruko Yamada" displayed on the speaker display screen 203a utters "Hello everyone," the voice of the utterance is input via the input unit (microphone) 111 of the information processing device 100 used by "Haruko Yamada." The Web conference application implemented in the information processing device 100 used by "Haruko Yamada" transcribes the voice input via the input unit 111 to generate subtitle data (text data) related to the contents uttered by "Haruko Yamada" of "Hello everyone," and displays it on the subtitle display screen 205. The process of generating subtitle data from the voice by the Web conference application (subtitle generation function) may be performed by the information processing device 100 of the speaker (Haruko Yamada in this case), or may be performed by a server (not shown) connected to the information processing device 100 via a network, or may be performed by both the server and the information processing device 100. The essence of the present invention is not to display the contents (voice) of a speaker as subtitles. Therefore, a typical method may be used for the function of recognizing the voice input through a microphone and displaying the recognized contents as characters (subtitle generation function, etc.).

[0023] The exit button 206 is selected by each user when he or she wants to leave the Web conference, and the information processing device 100 on which the exit button 206 is selected is blocked from accessing the Web conference. For example, in Fig. 2, a display screen 200 is displayed on the display unit 100 of the information processing device 100 used by Suzuki Ichiro, and when Suzuki Ichiro selects the exit button 100 via the input unit (keyboard and mouse) 111, access to the Web conference of the information processing device 100 used by Suzuki Ichiro is blocked, and Suzuki Ichiro has exited the Web conference.

[0024] In this embodiment, when the statement recording bookmarklet 202 is selected or the subtitle generation function of the Web conference application is selected, the process described below is executed and the subtitle display screen 205 is displayed.

[0025] FIG. 3 is a flowchart showing a processing procedure for executing a speech recording function in a Web conference according to this embodiment. In this embodiment, the Web conference application has a subtitle generation function. Therefore, when a user wishes to display subtitles, the user operates a user interface such as a subtitle generation button of the Web conference application via the input unit (keyboard) 111, and the remarks made by the user are displayed as subtitles. In this embodiment, the main processing (such as voice recognition) of the Web conference application is performed by a server connected to the information processing device 100 of each participant via a network. In this case, when the subtitle generation function is started by operating the subtitle generation button, the server acquires the voice inputted via the input unit (microphone) 111 of a specific speaker from the information processing device 100 to which the voice was inputted, executes voice recognition, and transmits data related to the remarks acquired by the voice recognition to each information processing device 100 to display them on the subtitle display screen 205. In this way, the remarks made by each user are displayed as subtitles (transcribed).

[0026] As described above, in this embodiment, the speech recording function is executed by a bookmarklet, and therefore a speech recording bookmarklet is prepared and registered in the bookmarks of the Web browser 103a before participating in the Web conference. In this embodiment, the speech recording function is executed when the speech recording bookmarklet 202 provided in the bookmark toolbar 201 is selected. When this selection is input by a user (here, Suzuki Ichiro) via the input unit 111, the information processing device 100 operated by Suzuki Ichiro starts this process. When the speech recording bookmarklet 202 is selected, the information processing device 100 creates a speech list in which the speech contents displayed on the subtitle display screen 205 are written, and stores the list in the memory 104.

[0027] In this embodiment, a "statement list" is a list including the contents of statements made by each speaker (user) in the corresponding Web conference (a list that links the speaker with the statement). In this embodiment, the statement list is a list of each statement, with the speaker's name, statement content, and statement time (the time the statement was made (entered)) grouped together as a single entity.

[0028] When a user who wants to execute the speech recording function selects the speech recording function bookmarklet 202 through the input unit 111, the processing of the speech recording function is started. Note that, what is important in this embodiment is that the speech recording function is executed as a program separate from the Web conference application, and the contents of the speech are automatically recorded by utilizing the subtitle generation function of the Web conference application, and for this purpose, the HTML document generated by the Web conference application is analyzed. Therefore, it is important to display the subtitle display screen 205 and to be able to analyze the HTML document for displaying the subtitle display screen 205. That is, in order to execute the processing shown in FIG. 3, it is a prerequisite that the subtitle display screen 205 is displayed. Therefore, before selecting the speech recording bookmarklet 202, the user needs to start the subtitle generation function of the Web conference application through the input unit 111. Alternatively, when the speech recording bookmarklet 202 is selected, the CPU 101 may display a message on the display unit 110 to prompt the user to start the subtitle generation function (prompt the display of the subtitle display screen 205), or may forcibly start the subtitle generation function and display the subtitle display screen 205.

[0029] In step S31, the CPU 101 starts monitoring an update event of an HTML element related to the display of a comment in the corresponding Web conference. In this embodiment, an "update event" is a notification generated each time an update occurs, such as a change, addition, or deletion of a specific HTML element, for an HTML document for displaying the subtitle display screen 205. The content of the HTML document for displaying the subtitle display screen 205 is rewritten each time an event related to a comment occurs, such as when a user makes a comment. In this embodiment, the CPU 101 functions to generate an update event when the HTML document for displaying the subtitle display screen 205 is rewritten (updated). In this embodiment, an update event occurs when an HTML element related to the display of a comment is updated. Therefore, by checking the update event one by one, when some comment is made, the information processing device 100 can recognize the timing and content of the comment by analyzing the HTML document.

[0030] In step S32, the CPU 101 judges whether an update event has occurred. In this embodiment, when any of the users participating in the Web conference inputs a voice via the input unit 111, the HTML element related to the display of the remark is updated according to the input voice, and an update event occurs. However, since the information processing device 100 monitors the update event, the CPU 101 can recognize that the HTML element related to the display of the remark has been updated due to the occurrence of the update event. That is, in this step, the CPU 101 can be said to judge whether any of the users participating in the Web conference has made a remark. If it is judged that an update event has not occurred, the CPU 101 repeats this step until it is judged that an update event has occurred. If it is judged that an update event has occurred, the CPU 101 proceeds to step S33.

[0031] In step S33, the CPU 101 acquires the content of the comment displayed on the subtitle display screen 205 by referring to the HTML element related to the display of the comment that has been updated by any of the comments (input corresponding to the comment) made by the users participating in the Web conference, and adds the acquired content of the comment to the comment list. At this time, if it is determined in step S32 that an update event has occurred, the HTML element related to the display of the comment will be updated, and the CPU 101 adds data related to the updated value to the comment list. In this embodiment, the data added to the comment list is the speaker's name, comment content, and comment time, so the CPU 101 extracts the speaker's name and comment content by referring to the HTML element related to the display of the comment, and acquires the time when the update occurred by referring to a clock function (not shown) of the information processing device 100, thereby adding each item to the comment list. 2, for example, when Yamada Haruko says "Hello everyone," the voice "Hello everyone" is input via input unit (microphone) 111 of information processing device 100 used by Yamada Haruko, but among the HTML elements related to the display of the comment, the HTML element related to the speaker is updated to information indicating "Yamada Haruko," and the HTML element related to the comment content is updated to information indicating "Hello everyone." Therefore, CPU 101 adds the data of the speaker "Yamada Haruko" and the comment content "Hello everyone" to the comment list, together with the time obtained by referring to the clock function.

[0032] In step S34, the CPU 101 judges whether the speech recording function has ended. In this embodiment, the speech recording function ends when a user leaves the Web conference or when the user does not leave the Web conference but stops the speech recording function. In the former case, this occurs when the exit button 206 is selected, and in the latter case, this occurs when the speech recording function stop button (not shown) is selected. Therefore, when the exit button 206 is selected or the speech recording function stop button is selected, the CPU 101 judges that the speech recording function has ended and proceeds to step S35. On the other hand, when it is not judged that the exit button 206 or the speech recording function stop button has been selected, the process returns to step S32 and steps S32 to S34 are repeated.

[0033] In step S35, CPU 101 outputs the comment list as a text file. Note that in this embodiment, the comment list is automatically output as a text file when the comment recording function ends, but a file output button may be provided as a user interface for the comment recording bookmarklet, and the comment list at that time (when the file output button is selected) may be output as a specified file at the user's discretion (even when the comment recording function is running).

[0034] In this embodiment, the target of utterance recording is the voice (utterance) uttered by the user, but a text message may be recorded in addition to the voice. For example, if a Web conference application has a message transmission function such as chat, in addition to the utterances of each user during a Web conference, a text message may also be input. Such input text messages may also be acquired by the utterance recording function and added to the utterance list. In such a case, for example, an element related to the message input by text may be added to the HTML element related to the display of the utterance.

[0035] According to this embodiment, the HTML document for displaying the subtitle display screen 205 is analyzed, and the updated contents are added to the comment list every time the HTML elements related to the display of comments are updated, so that the comments of users who participated in the Web conference can be acquired more accurately. Also, each user can acquire the contents of each user's comments without having to take notes or refer to the recorded contents of the conference, so the burden of recording comments in the Web conference can be reduced.

[0036] Thus, the present embodiment focuses on the subtitle display screen 205 that is actually displayed in the target Web conference, and in a sense, generates text data according to the user's remarks based on that screen. More specifically, the Web conference application itself is not modified or a new function is added, and the operation of the Web conference application remains unchanged. Text data (remarks record) showing the contents of the remarks (transcribed subtitle contents) is generated using information for constructing the screen displayed by the application (HTML elements related to the display of remarks). Therefore, it can be implemented as a program that runs on a Web browser like a bookmarklet or plug-in, and the program can be created and run separately from the corresponding Web conference application. Therefore, the remarks record function can be customized by the user, and can also be applied to other Web conference applications.

[0037] Also, in this embodiment, the speech recording function is not built into the program of the Web conference app, but is separate software from the Web conference app, so it can be operated with the same UI even if the Web conference app changes.

[0038] (Other embodiments) The above-mentioned embodiment also includes a processing method in which a program (for example, a program that performs the processing shown in FIG. 3) that operates the configuration of the above-mentioned embodiment to realize the functions of the above-mentioned embodiment is stored in a storage medium, the program stored in the storage medium is read as code, and the program is executed by a computer. That is, a computer-readable storage medium is also included in the scope of the embodiment. In addition, the above-mentioned embodiment includes not only the storage medium in which the above-mentioned computer program is stored, but also the computer program itself. Examples of such a storage medium include a floppy disk, a hard disk, an optical disk, a magneto-optical disk, a CD-ROM, a magnetic tape, a non-volatile memory card, a ROM, and the like. In addition, the above-mentioned embodiment also includes not only a program stored in the above-mentioned storage medium that executes processing by itself, but also a program that operates on an OS in cooperation with other software and functions of an extension board to execute the operations of the above-mentioned embodiment. [Explanation of symbols]

[0039] 100 Information processing device 101 CPU 103 Storage 103a Web Browser 104 Memory 200 Web conference display screens 202 Bookmarklet for recording comments 205 Subtitle display screen

Claims

1. An information processing method for converting a speech into text using a subtitle generation function of a Web conference application, comprising the steps of: When a subtitle display screen for displaying subtitles of comments input by attendees of the Web conference is displayed, an HTML document related to the screen display of the Web conference is analyzed, and the content of the comments displayed on the subtitle display screen is acquired by referring to HTML elements related to the display of comments acquired by the subtitle generation function; storing the acquired content of the comment in a storage means as comment content information including the content of the comment; outputting the stored comment content information in a predetermined file format; the acquiring step includes acquiring updated information of an HTML element each time the HTML element related to the display of the message is updated; The step of saving includes adding the updated information to the comment content information.

23. An information processing method comprising:

2. A program for causing a computer to execute an information processing method for converting a speech into text using a subtitle generation function of a Web conference application, comprising: The information processing method includes: When a subtitle display screen for displaying subtitles of comments input by attendees of the Web conference is displayed, an HTML document related to the screen display of the Web conference is analyzed, and the content of the comments displayed on the subtitle display screen is acquired by referring to HTML elements related to the display of comments acquired by the subtitle generation function; storing the acquired content of the comment in a storage means as comment content information including the content of the comment; outputting the stored comment content information in a predetermined file format; the acquiring step includes acquiring updated information of an HTML element each time the HTML element related to the display of the message is updated; The step of saving includes adding the updated information to the comment content information. A program characterized by:

3. A computer-readable storage medium storing the program according to claim 2.

Images