Information processing system, text still image display system, text still image display method, and program

Abstract

To associate a still picture included in moving image data with utterance data.SOLUTION: An information processing system acquires a still picture from moving image data acquired from a first terminal device, acquires first time information acquired from the moving image data by the still picture, and acquires a plurality of utterance data corresponding to voice data included in the moving image data and second time information obtained by uttering the utterance data in the moving image data. The information processing system transmits screen information obtained by associating the still picture with the utterance data on the basis of the first time information and the second time information.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an information processing system, a text still image display system, a text still image display method, and a program. [Background technology]

[0002] There is a technology known that transcribes video data sent between locations, such as during remote conferences, to create records such as transcripts (documented versions of what was said). Such technology can be used to support the creation of meeting minutes and to store technical information.

[0003] A technique has been devised in which voice data is divided into a plurality of segments and recorded in association with speaker identification information (see, for example, Patent Document 1). Patent Document 1 discloses a technique for identifying the speaker of a corresponding segment of voice data based on image data including the face of the participant. Summary of the Invention [Problem to be solved by the invention]

[0004] However, conventional technologies have a problem in that they do not associate still images contained in video data with speech data. For example, in online conferences, the conference organizer often displays materials on a shared screen and explains them along with audio data. Even if a user who wants to know the content of the conference after the conference has ended views the video data on the shared screen during the conference or the text data generated by speech recognition, it is difficult to understand the content of the conference because it is unclear which scenes in the video data correspond to which text data.

[0005] In view of the above-mentioned problems, the present invention provides a technique for associating still images included in video data with speech data. [Means for solving the problem]

[0006] In view of the above problems, the present invention provides an information processing system having a still image acquisition unit that acquires a still image and first time information at which the still image was acquired from video data acquired from a first terminal device, an utterance data acquisition unit that acquires a plurality of utterance data corresponding to audio data contained in the video data and second time information at which the utterance data was uttered in the video data, and a communication control unit that transmits screen information in which the still image and the utterance data are associated to a second terminal device based on the first time information and the second time information. [Effects of the Invention]

[0007] The present invention can provide a technique for associating still images included in video data with speech data. [Brief explanation of the drawings]

[0008] [Figure 1] 10A and 10B are diagrams illustrating a process in which a text still image display system associates a still image acquired from a video with text data. [Figure 2] FIG. 1 is a diagram illustrating an example of a system configuration of a text still image display system. [Figure 3] FIG. 2 is a diagram illustrating a hardware configuration of an example of a computer. [Figure 4] FIG. 2 is a diagram illustrating an example of functional configurations of an information processing system, a first terminal device, a second terminal device, and a third terminal device. [Figure 5] FIG. 10 is a conceptual diagram illustrating an example of a text information management table. [Figure 6] FIG. 10 is a conceptual diagram illustrating an example of a still image information management table. [Figure 7] FIG. 10 is a conceptual diagram illustrating an example of a record bibliographic information management table. [Figure 8] FIG. 10 is a diagram illustrating a screen displayed when uploading a video file. [Figure 9] 10 is a sequence diagram illustrating an example of a process flow in which a first terminal device uploads a video file to an information processing system and the information processing system acquires a still image. [Figure 10A] FIG. 10 is a diagram showing an example of a video file list screen displayed by the first terminal device. [Figure 10B] FIG. 10 is a diagram showing an example of a video file list screen when new record bibliographic information is saved. [Figure 11] 10 is an example of a sequence diagram illustrating a process in which a third terminal device displays a text still image display screen. [Figure 12] FIG. 10 is a diagram showing an example of a text still image display screen displayed by a third terminal device. [Figure 13] FIG. 10 is a diagram illustrating a screen displayed by the first terminal device when uploading a video file. [Figure 14] FIG. 10 is a diagram showing an example of an area designation screen displayed by the first terminal device. [Figure 15] FIG. 10 is a diagram illustrating an example of still image acquisition area information. [Figure 16] FIG. 10 is a diagram illustrating an area designation screen displayed by the first terminal device. [Figure 17] 10A and 10B are diagrams illustrating an example of still image acquisition area information when the layout of a moving image is changed. [Figure 18] 10 is an example of a sequence diagram illustrating a processing flow in which a first terminal device uploads a video file to an information processing system and the information processing system acquires a still image in pattern 1. FIG. [Figure 19] 10 is an example of a sequence diagram illustrating a process flow in which a first terminal device uploads a video file to an information processing system and the information processing system acquires a still image in pattern 2. FIG. [Figure 20] FIG. 10 is a diagram illustrating an example of data contained in a transcript file. [Figure 21] FIG. 10 is a diagram showing an example of a video file list screen displayed by the first terminal device. [Figure 22] FIG. 10 is a diagram showing an example of an upload screen (part 1). [Figure 23] 10 is a sequence diagram illustrating an example of a process flow in which a first terminal device uploads a video file to an information processing system and the information processing system acquires a still image. [Figure 24] 10 is a diagram illustrating the correspondence between speech data obtained by a voice recognition service and speech data acquired from a transcript file, which is stored in a text information management table storage unit. FIG. [Figure 25] 10 is an example of a sequence diagram illustrating a process in which a third terminal device displays a text still image display screen. [Figure 26] FIG. 10 is a diagram showing an example of a text still image display screen displayed by a third terminal device. DETAILED DESCRIPTION OF THE INVENTION

[0009] A text still image display system and a text still image display method performed by the text still image display system will be described below as an example of an embodiment of the present invention.

[0010] [First embodiment] <Summary> 1 is a diagram illustrating the process of associating text data with still images acquired from a video by the text still image display system 100. In FIG. 1, it is assumed that the online conference has already ended and the terminal device 10 has saved a video file including audio.

[0011] (1) A participant of a conference uploads a video file 301 from a terminal device 10 to the information processing system 30 so that the contents of the conference can be easily viewed later.

[0012] (2) The information processing system 30 performs speech recognition on the speech included in the video file 301 and extracts text data 302. The terminal device 10 may transmit a transcript of the speech that has been recognized together with the video file. The text data 302 is appropriately divided into segments, such as segments of silence, corresponding to the divisions of each utterance. The divided text data 302 is called speech data. The information processing system 30 stores the speech data in association with the start time and end time (both of which are the elapsed time from the start of the conference).

[0013] (3) Next, the information processing system 30 extracts and stores a still image at regular intervals (for example, every 30 seconds). Instead of extracting a still image at regular intervals, the still image may be extracted at each start time of the speech data.

[0014] (4) When a user views a video, the information processing system 30 acquires still images in order from the start time, and acquires speech data recorded between the still images and the time when the next still image is acquired. The information processing system 30 provides the terminal device 10 with a screen in which the still images and speech data are associated, so that the user can check the scene (still image) of the video when the speech data was spoken.

[0015] In this way, the speech data and the still image of the scene corresponding to the speech data are displayed in association with each other, so that the user can see the association between the content of the speech and the scene in the video, and can easily grasp the content of the video.

[0016] <Terminology> Video data is data that includes moving images. Video is often also called video. A still image, in contrast to a video, is a single, discontinuous image. Video data may also include audio data. In this embodiment, video data will be described using the term video file.

[0017] The speech data corresponding to the voice data is data obtained by speech recognition of the voice data containing the speech. The speech data is expressed in character codes. The conversion from the voice data to the speech data may be performed by the information processing system or may be requested from an external party by the information processing system. The information processing system may receive the speech data from a terminal device.

[0018] A screen in which a still image and speech data are associated means that both are displayed in a manner that allows the user to recognize the speech data corresponding to the still image or the still image corresponding to the speech data. The still image and speech data may be displayed synchronously.

[0019] <System configuration example> FIG. 2 is a diagram showing an example of a system configuration of a text still image display system 100. As shown in FIG. 2, in the text still image display system 100, a first terminal device 10A to a third terminal device 10C can communicate with an information processing system 30 via a communication network N. Any of the first terminal device 10A to the third terminal device 10C will be referred to as a "terminal device 10." It is also assumed that the first terminal device 10A (an example of a first terminal device) is used by a user A, the second terminal device 10B is used by a user B, and the third terminal device 10C (an example of a second terminal device) is used by a user C. The communication network N may be, for example, a LAN (including a wireless LAN) within a facility where each of the first terminal device 10A to the third terminal device 10C is located, or may further include the Internet established by a provider network or the like. The communication network N may also include a telephone line network such as 4G, 5G, or 6G.

[0020] A speech recognition server 9 is connected to the communication network N. A speech recognition service 9a (application) runs on the speech recognition server 9. The speech recognition service 9a performs speech recognition on the speech data and converts it into text data. The speech recognition server 9 also divides the text data into speech data based on rules such as silent sections of the speech data or fixed time intervals. The speech recognition server 9 can convert speech data into text data in real time while a conference is in progress, and can also convert speech data extracted from a video file into text data after the conference has ended.

[0021] A teleconference service system 8 is connected to the communication network N. A teleconference service 8a (application) runs on the teleconference service system 8. The teleconference service 8a enables remote communication between users at multiple locations. The teleconference service 8a transmits and receives image data and audio data between a location where a first terminal device 10A is located and a location where a second terminal device 10B is located via the communication network N. That is, the first terminal device 10A collects facial images and audio data such as speech of user A, converts them into digital signals, and transmits them to the teleconference service system 8. The teleconference service system 8 transmits digital signals of image data and audio data to the second terminal device 10B, which is communicating in the same session. The second terminal device 10B displays images on a display and outputs audio from a speaker. Similar digital signals are also transmitted from the second terminal device 10B, enabling bidirectional communication between the first terminal device 10A and the second terminal device 10B.

[0022] The information processing system 30 is one or more information processing devices. An information recording service 30a (application) runs in the information processing system 30. The information recording service 30a has the following functions. The information recording service 30a acquires text data extracted from the voice data by transmitting the voice data included in the video file received from the first terminal device 10A to the voice recognition server 9. The text data is divided into speech data. The information recording service 30a extracts still images from the video file and associates the speech data with the still images. The information recording service 30a provides a text still image display screen 250 (see FIG. 12) in which still images and speech data are associated with each other in response to a request from the terminal device 10.

[0023] The information processing system 30 may be realized by cloud computing or by a single information processing device. Cloud computing refers to a form in which resources on a network are used without being aware of specific hardware resources. The information processing system 30 may exist on the Internet or on-premise. Cloud computing includes SaaS (Software as a Service), PaaS (Platform as a Service), IaaS (Infrastructure as a Service), etc., but the service form is not important in this embodiment.

[0024] The functions of the information processing system 30 may be distributed among multiple information processing devices, or multiple information processing systems 30 with the same functions may exist, and the number of information processing systems 30 that deliver videos may be increased or decreased depending on the processing load.

[0025] 2, the text still image display system 100, the voice recognition server 9, and the teleconferencing service system 8 are shown as separate systems, but these three may be integrated. Also, any two of the voice recognition server 9 and the teleconferencing service system 8 may be integrated. If a transcript file for the video file is available, the voice recognition server 9 may not be required.

[0026] The first terminal device 10A to the third terminal device 10C are, for example, PCs (Personal Computers) or smart devices. There is no clear definition of a smart device, but it refers to a smartphone, tablet terminal, etc. that does not include an existing computer such as a PC or a server. Both the PC and the smart device can connect to the information processing system 30 using at least one of a web browser or a client application dedicated to the information processing system 30.

[0027] When a web browser is executed on the first terminal device 10A to the third terminal device 10C, the first terminal device 10A to the third terminal device 10C execute a web application provided by the information processing system 30. A web application is software or a mechanism thereof that operates by cooperation between a program written in a programming language (for example, JavaScript (registered trademark)) that runs on the web browser and a program on the web server side. The first terminal device 10A to the third terminal device 10C can dynamically change web pages using the web application.

[0028] The first terminal apparatus 10A to the third terminal apparatus 10C are not limited to PCs or smart devices, but may be any apparatuses on which a web browser or a dedicated client application runs in the information processing system 30. The first terminal apparatus 10A to the third terminal apparatus 10C may be any apparatuses that have a communication function and can connect to the communication network N. The first terminal apparatus 10A to the third terminal apparatus 10C may be game consoles, PDAs (Personal Digital Assistants), digital cameras, wearable PCs, MFPs (Multi-Function Peripherals), PJs (Projectors), electronic whiteboards (whiteboards with electronic whiteboard functions), etc. Although the illustrated terminal apparatuses 10 are the first terminal apparatus 10A to the third terminal apparatus 10C, the number of terminal apparatuses 10 may be three, namely, the first terminal apparatus 10A to the third terminal apparatus 10C, but the number of terminal apparatuses 10 may be two, four, or more.

[0029] In this embodiment, for convenience, it is assumed that user A is the organizer, user B is the participant, and user C is the viewer. The organizer is the person who holds the conference and sends an invitation email to the participants via the remote conference service system 8. The invitation email contains the URL and password of the conference. In this embodiment, the participants are the people who lead the conference and explain the contents of the proceedings.

[0030] The first terminal device 10A displays the voices, images, or shared screens (such as materials) of the participants on a display. An information recording application is running on the first terminal device 10A, which acquires voice data from the host (microphone + sound card) and other participants (sound cards), and also captures (records) the screens displayed by the first terminal device 10A and the shared screens. This becomes a video file. However, the second terminal device 10B may also create the video file.

[0031] A participant is someone who takes part in a conference. It is known that participants can participate in a conference by receiving an invitation email from the remote conference service system 8 and connecting to the URL contained in the invitation email. Participants may be employees of the same company as the organizer or of a business partner. Participants may also be potential customers who the organizer expects to purchase products or services from. Participants are only required to be people who take part in a conference, and the type of person a participant is likely to depend on the content of the conference.

[0032] A viewer may be anyone who wants to view the video of the conference or the text still image display screen. A viewer can log in to the information processing system 30 and view the video file for which they have viewing authority. Alternatively, user A or user B can send a URL for viewing the video to a viewer they want to recommend to watch the video of the conference by email or the like. This allows the viewer to connect third terminal device 10C to the URL and view the video of the conference. Third terminal device 10C plays the video and displays a text still image display screen in which speech data and still images are associated, making it easy for user C to understand the content of each scene of the video.

[0033] <Hardware configuration example> The information processing system 30 and the first terminal device 10A to the third terminal device 10C in Fig. 2 are realized, for example, by a computer having the hardware configuration shown in Fig. 3. Fig. 3 is a hardware configuration diagram of an example of a computer. As shown in Fig. 3, the computer 500 includes a CPU 501, a ROM 502, a RAM 503, an HD 504, an HDD controller 505 (Hard Disk Drive), a display 506, an external device connection I / F 508 (Interface), a network I / F 509, a bus line 510, a keyboard 511, a pointing device 512, an optical drive 514, and a media I / F 516.

[0034] Of these, the CPU 501 controls the overall operation of the computer. The ROM 502 stores programs, such as the IPL, used to drive the CPU 501. The RAM 503 is used as a work area for the CPU 501. The HD 504 stores various data, such as programs. The HDD controller 505 controls the reading and writing of various data from and to the HD 504 under the control of the CPU 501. The display 506 displays various information, such as a cursor, menus, windows, characters, or images. The external device connection I / F 508 is an interface for connecting various external devices. In this case, external devices include, for example, USB (Universal Serial Bus) memories and printers. The network I / F 509 is an interface for data communication using a communication network. The bus line 510 is an address bus, a data bus, or the like, for electrically connecting the components, such as the CPU 501, shown in FIG. 3.

[0035] The keyboard 511 is a type of input means having multiple keys for inputting characters, numbers, various instructions, etc. The pointing device 512 is a type of input means for selecting and executing various instructions, selecting a processing target, moving a cursor, etc. The optical drive 514 controls reading and writing of various data from an optical storage medium 513, which is an example of a removable storage medium. The optical drive 514 is a CD, DVD, Blu-Ray (registered trademark), etc. The media I / F 516 controls reading and writing (storing) of data from a storage medium 515, such as a flash memory.

[0036] <Function block diagram> Next, functions of the text still image display system 100 will be described with reference to Fig. 4. Fig. 4 is a diagram showing an example of the functional configuration of the information processing system 30 and the first to third terminal devices 10A to 10C.

[0037] <<First Terminal Device 10A>> The first terminal device 10A has a communication unit 11, a display control unit 12, and an operation reception unit 13. Each of these functional units is a function or means realized by the CPU 501 shown in FIG. 3 executing instructions contained in one or more programs installed in the first terminal device 10A. Note that this program may be a web browser or a dedicated client application. For example, when a web browser is used in the first terminal device 10A, the program is the web browser. When a client application of the information processing system 30 is used in the first terminal device 10A, the program is the client application of the information processing system 30.

[0038] Also, the first terminal device 10A has already acquired a video file that records the screen and audio displayed by the first terminal device 10A during the conference from the teleconferencing service system 8. The teleconferencing service system 8 creates a video file by capturing (recording) the audio data of the participants, the images displayed by the first terminal device 10A, and the shared screen (such as materials). The teleconferencing service system 8 also creates a transcript file from the audio data and provides it to the first terminal device 10A. The first terminal device 10A may transmit the video file to the information processing system 30 in real time during the conference, or may transmit it after the conference ends.

[0039] The communication unit 11 transmits and receives various types of information between the remote conference service system 8 and the information processing system 30. The communication unit 11 receives video files and transcript files from the remote conference service system 8. The communication unit 11 also receives various types of screen information displayed by Web applications and client applications from the information processing system 30. The communication unit 11 transmits video files (and transcript files, if necessary) to the information processing system 30.

[0040] The display control unit 12 interprets screen information of various screens and displays it on the display 506. The operation receiving unit 13 receives various operations by the user A on the various screens displayed on the display 506.

[0041] <<Second Terminal Apparatus 10B>> The second terminal device 10B has a communication unit 14, a display control unit 15, and an operation reception unit 16. Each of these functional units is a function or means realized by the CPU 501 shown in FIG. 3 executing instructions contained in one or more programs installed in the second terminal device 10B. Note that this program may be a web browser or a dedicated client application. For example, when a web browser is used on the second terminal device 10B, the program is the web browser. When a client application of the information processing system 30 is used on the second terminal device 10B, the program is the client application of the information processing system 30.

[0042] The communication unit 14 transmits and receives image data and audio data to and from the remote conference service system 8. The communication unit 14 may also communicate with the information processing system 30.

[0043] The display control unit 15 interprets screen information of various screens and displays it on the display 506. The operation accepting unit 16 accepts various operations by the user B on the various screens displayed on the display 506.

[0044] <<Third Terminal Device 10C>> The third terminal device 10C has a communication unit 17, a display control unit 18, and an operation reception unit 19. Each of these functional units is a function or means realized by the CPU 501 shown in FIG. 3 executing instructions contained in one or more programs installed in the third terminal device 10C. Note that this program may be a web browser or a dedicated client application. For example, when a web browser is used in the third terminal device 10C, the program is the web browser. When a client application of the information processing system 30 is used in the third terminal device 10C, the program is the client application of the information processing system 30.

[0045] The communication unit 17 transmits and receives various types of information to and from the information processing system 30. In this embodiment, the communication unit 17 receives various types of screen information, videos, etc. displayed by the Web application or client application from the information processing system 30. The communication unit 17 receives screen information of the text still image display screen 250 from the information processing system 30. In addition, the communication unit 17 transmits operation information of the user C to the information processing system 30.

[0046] The display control unit 18 interprets screen information of various screens and displays it on the display 506. The operation accepting unit 19 accepts various operations by the user C on the various screens displayed on the display 506.

[0047] <<Information Processing System 30>> The information processing system 30 has a first communication control unit 31, a screen control unit 32, a second communication control unit 33, a voice recognition control unit 34, an utterance data acquisition unit 35, a still image acquisition unit 36, and a table management unit 37. Each of these functional units is a function or means realized by the CPU 501 shown in FIG. 3 executing instructions included in a program installed in the information processing system 30. The information processing system 30 also has a text information management table storage unit 3001, a still image information management table storage unit 3002, and a record bibliographic information management table storage unit 3003. Each of these storage units is formed in the HD 504, RAM 503, etc. shown in FIG. 3.

[0048] The first communication control unit 31 transmits and receives various types of information to and from the first terminal device 10A and the third terminal device 10C. The first communication control unit 31 receives a video file (and a transcript file, if necessary) from the first terminal device 10A, and transmits screen information of the text still image display screen 250 to the third terminal device 10C.

[0049] The screen control unit 32 generates screens to be displayed by the first terminal device 10A and the third terminal device 10C. When the first terminal device 10A and the third terminal device 10C execute a Web application, the screen information is created using HTML, XML, CSS (Cascade Style Sheet), JavaScript (registered trademark), etc. When the first terminal device 10A and the third terminal device 10C execute a client application, the screen information is held by the first terminal device 10A and the third terminal device 10C, and the information to be displayed is transmitted in XML, etc.

[0050] The second communication control unit 33 communicates with the voice recognition server 9. The second communication control unit 33 transmits the entire video file or only the voice data to the voice recognition server 9, and receives text data divided into speech data.

[0051] The voice recognition control unit 34 requests the voice recognition server 9 to perform voice recognition (conversion into text data) on the voice data via the second communication control unit 33. The voice recognition control unit 34 passes the text data divided into speech data, which has been acquired from the voice recognition server 9, to the speech data acquisition unit 35.

[0052] The utterance data acquisition unit 35 acquires the utterance data and the time of utterance from the text data, and stores the utterance data and the time in the text information management table storage unit 3001 via the table management unit 37. Furthermore, when a transcript file is transmitted from the first terminal device 10A, the utterance data acquisition unit 35 acquires the utterance data and the time of utterance from the transcript file, and stores the utterance data and the time in the text information management table storage unit 3001 via the table management unit 37.

[0053] The still image acquisition unit 36 extracts still images from the video file at regular time intervals and stores them in the still image information management table storage unit 3002 via the table management unit 37. The still image acquisition unit 36 may extract still images at each time associated with the utterance data.

[0054] The table management unit 37 manages each table stored in the text information management table storage unit 3001, the still image information management table storage unit 3002, and the recorded bibliographic information management table storage unit 3003. The table management unit 37 stores information in each table and retrieves information from each table.

[0055] <<Text information management table>> Fig. 5 is a conceptual diagram showing an example of a text information management table. The text information management table is a table for managing text data converted by the speech recognition server 9 or text data included in a transcript file. The text information management table storage unit 3001 contains a text information management table such as that shown in Fig. 5. A recording ID is identification information assigned to a video file. A recording ID may also be assigned to an online conference. -Text ID is unique identification information assigned to speech data. The start time (an example of second time information) is the elapsed time from the start of the conference to when the speech data segmented from the text data was spoken. The start of the conference is assumed to coincide with the start of the video. The end time is the time elapsed from the start of the conference when the speech data segmented from the text data ended. The speech data is segmented, for example, when the silent section of the voice data exceeds a threshold, but how the data is segmented depends on the rules of the speech recognition service 9a. The text item indicates the speech data or its path. A transcript provided by the remote conference service system 8 may be used for the text item.

[0056] <<Still Image Information Management Table>> Fig. 6 is a conceptual diagram showing an example of a still image information management table. The still image information management table is a table for managing still images acquired (captured) from a moving image file. The still image information management table storage unit 3002 includes a still image information management table storage unit 3002 configured with a still image information management table such as that shown in Fig. 6. The record ID is the same as the record ID in the text information management table. The still image ID is unique identification information assigned to a still image. The still image item stores the still image or its path. The item "acquisition time" (an example of first time information) stores the time when the still image was acquired (the time elapsed since the start of the conference).

[0057] <<Bibliographic Record Information Management Table>> Fig. 7 is a conceptual diagram showing an example of a recorded bibliographic information management table. The recorded bibliographic information management table is a table for managing video files. The recorded bibliographic information management table storage unit includes a recorded bibliographic information management table storage unit 3003 that is configured using the recorded bibliographic information management table shown in Fig. 7. In the record name field, any information such as the conference name can be set. The record name can be transcribed from the conference information managed by the remote conference service system, or can be written by the participants. The participants of the conference are set in the participant item. The participants may be copied from the conference information managed by the remote conference service system, or may be entered by the participants themselves. The meeting notes section allows participants to enter any notes they wish. The start date and time of the conference is set in the Start Date and Time field. The start date and time may be transcribed from the conference information managed by the remote conference service system, or may be entered by the participant or from the timestamp of the video file. The end date and time of the conference is set in the end date and time field. The end date and time may be transcribed from the conference information managed by the remote conference service system, or may be entered by the participant or from the timestamp of the video file. The audio data path item is set with the URL of the destination where the audio data file is saved, etc. The URL of the destination where the information processing system 30 saves the audio data is set. The user ID field contains the identification information of the user who registered the record bibliographic information. The conference URL field contains the URL to which the first terminal device 10A to the third terminal device 10C connect in order to hold an online conference. The information processing system 30 sets the URL.

[0058] <Screen example when uploading> Next, a screen displayed by the first terminal apparatus 10A when uploading a video file will be described with reference to Fig. 8. Fig. 8(a) shows an upload screen (part 1) 210 displayed by the first terminal apparatus 10A when a video file upload button (described later) is pressed.

[0059] The upload screen (1) 210 has a drag and drop field 211 and an upload button 212. User A drags and drops the video file acquired from the remote conference service system 8 into the drag and drop field 211. When User A presses the upload button 212, the screen transitions to the upload screen (2) 220 shown in FIG. 8(b).

[0060] FIG. 8(b) shows upload screen (2) 220. Upload screen (2) 220 is a confirmation screen displayed immediately before uploading. Upload screen (2) 220 has a file name 221 of the video file selected by user A, capacity 222, upload button 223, and cancel button 224. The user checks capacity 222, for example, to determine whether or not to upload. Upload button 223 is a button for executing the upload. Cancel button 224 is a button for returning to upload screen (1) 210. When user A presses upload button 223, the screen transitions to upload screen (3) 230 shown in FIG. 8(c).

[0061] The upload screen (3) 230 has a message 231 saying, "File is being uploaded. Please wait for a while." When the first terminal apparatus 10A receives a notification of upload completion from the information processing system 30, the upload screen (3) 230 transitions to a video file list screen 240, which will be described later.

[0062] <Upload process flow> Next, a process flow in which the first terminal device 10A uploads a moving image file to the information processing system 30 will be described with reference to Fig. 9. Fig. 9 is a sequence diagram illustrating a process flow in which the first terminal device 10A uploads a moving image file to the information processing system 30 and the information processing system 30 acquires a still image. Note that in the description of Fig. 9, it is assumed that the conference has already ended and the first terminal device 10A is holding the moving image file.

[0063] S1: User A performs an operation on the first terminal device 10A to display the upload screen (type 1) 210. The operation to display the upload screen (type 1) 210 is to press the video file upload button 247 on the video file list screen 240, which will be described later.

[0064] S2: The operation receiving unit 13 of the first terminal apparatus 10A receives an operation, and the communication unit 11 transmits a request for the upload screen (first) 210 to the information processing system 30.

[0065] S3: The first communication control unit 31 of the information processing system 30 receives the request for the upload screen, and the screen control unit 32 transmits screen information of the upload screen (first) 210 to the first terminal apparatus 10A.

[0066] S4: The communication unit 11 of the first terminal apparatus 10A receives the screen information of the upload screen (first) 210, and the display control unit 12 displays the upload screen (first) 210.

[0067] S5: User A drops the video file to be uploaded into the drag and drop field 211 and presses the upload button 212. The operation acceptance unit 13 of the first terminal device 10A accepts the operation, and in response to User A's operation, an upload screen (2) 220 or an upload screen (3) 230 is displayed.

[0068] S6: The communication unit 11 transmits a request for recognizing the audio data to the information processing system 30 together with the video file.

[0069] S7: The first communication control unit 31 of the information processing system 30 receives the request to recognize the moving image file and the audio data, and the audio recognition control unit 34 extracts the audio data from the moving image file.

[0070] S8: The second communication control unit 33 sends a voice recognition request to the voice recognition server 9 together with the voice data.

[0071] S9: The voice recognition service 9a of the voice recognition server 9 performs voice recognition processing on the voice data, and returns the text data divided into utterance data to the information processing system 30.

[0072] S10: The second communication control unit 33 of the information processing system 30 receives the text data, and the utterance data acquisition unit 35 acquires the utterance data, start time, and end time from the text data. The utterance data acquisition unit 35 stores the utterance data, etc. in the text information management table storage unit 3001 via the table management unit 37.

[0073] S11: Meanwhile, the still image acquisition unit 36 acquires still images from the video file at regular intervals (e.g., 30 seconds). The still image acquisition unit 36 may acquire still images at the start time, end time, or between the start time and end time of utterance data divided in the text data. The reason for acquiring still images at regular intervals is that if a still image is acquired at the start time of utterance data, there will be too many still images. Also, there is no guarantee that the still images will change significantly at the end of the utterance data. However, there is no problem if the still image acquisition unit 36 acquires a still image at the start time of utterance data, etc. In this case, one still image is associated with one utterance data.

[0074] Furthermore, the still image acquisition unit 36 compares the time-series frames that make up the video file, detects the timing at which the still images change significantly, and acquires the still images. Since this alone may result in a small number of still images, the still image acquisition unit 36 may further acquire still images at regular intervals. In this case, the interval at which still images are acquired may be one section (described later). Note that the processing of step S10 may be performed in parallel with steps S8 and S9.

[0075] S12: The still image acquisition unit 36 causes the table management unit 37 to store the still image in the still image information management table storage unit 3002.

[0076] S13: Furthermore, for a video file uploaded by user A, at the time user A uploads the video file, the table management unit 37 stores the initial value of the recorded bibliographic information in the recorded bibliographic information management table storage unit 3003. Furthermore, user A can edit the recorded bibliographic information at any time.

[0077] <Video file list screen based on bibliographic information> 10A shows a video file list screen 240 displayed by the first terminal device 10A. The video file list screen 240 is a screen that displays the recorded bibliographic information of video files. The video file list screen 240 displays the recorded bibliographic information of video files uploaded by logged-in user A, or the recorded bibliographic information of video files that can be viewed by logged-in user A (included in the shared range).

[0078] For example, when user A logs in, the screen control unit 32 obtains the recorded bibliographic information of the user A from the recorded bibliographic information management table storage unit 3003, and creates a screen such as that shown in Fig. 10A. Alternatively, when user A logs in, the screen control unit 32 obtains the recorded bibliographic information that user A includes in the shared range from the recorded bibliographic information management table storage unit 3003, and creates a screen such as that shown in Fig. 10A.

[0079] The video file list screen 240 has the following items: date 241 , conference name 242 , sharing range 243 , recorder 244 , capacity 245 , and participant / memo 246 . The date 241 is preferably the start date and end date of the conference. The table management unit 37 stores the date 241 from the timestamp or recording time of the video file. The conference name 242 is the name of the conference held in the remote conference service system 8, but the table management unit 37 sets an initial value (untitled conference) and the user can edit it later. The sharing range 243 specifies the users with whom the video file is shared. In other words, the sharing range indicates the range of users who can view the video file. The sharing range can be, for example, within the same team as the recorder, the same department, only the recorder, or no restrictions. The table management unit 37 sets the initial value (within the team), and the user can edit it later. The recorder 244 is the name of the user who uploaded the video file of the conference. The recorder 244 is User A who logged in to the information processing system 30. The capacity 245 is the size of the video file. Participants / Memo 246 is an item that displays participants and any notes. User A can set it as desired.

[0080] Furthermore, the upload video file button 247 is a button for the first terminal device 10A to upload a video file. Pressing the upload video file button 247 displays an upload screen (first) 210. As described in FIGS. 8 and 9, when still images and speech data are acquired from a video file and the recorded bibliographic information is saved, the recorded bibliographic information of the video file is displayed on the video file list screen 240.

[0081] 10B shows the video file list screen 240 when new record bibliographic information has been saved. Comparing FIG. 10A with FIG. 10B, the fourth conference is registered in the last row.

[0082] When user A or user C selects a video file on the video file list screen 240, the first terminal device 10A or the third terminal device 10C can download the video file. Similarly, when user A or user C selects a video file, the first terminal device 10A or the third terminal device 10C may receive screen information for the text still image display screen and display the text still image display screen.

[0083] <Viewing speech data and still images> 11 and 12, a description will be given of a screen when the third terminal device 10C displays the text still image display screen 250. Fig. 11 is a sequence diagram illustrating a process in which the third terminal device 10C displays the text still image display screen 250.

[0084] S21: For example, assume that user C selects a video file from the video file list screen 240.

[0085] S22: The operation acceptance unit 19 of the third terminal device 10C accepts the selection, and the communication unit 17 specifies the recording ID of the moving image file and transmits a request for a text still image display screen to the information processing system 30.

[0086] S23: The first communication control unit 31 of the information processing system 30 receives a request for a text still image display screen, and the table management unit 37 acquires the still image from the still image information management table storage unit 3002. The table management unit 37 also acquires utterance data from the acquisition time associated with the still image to the acquisition time associated with the next acquired still image from the text information management table storage unit 3001. The interval between the acquisition time and the next acquisition time corresponds to the fixed time in step S11 of Fig. 9. The table management unit 37 acquires utterance data for all still images.

[0087] S24: The screen control unit 32 generates a text still image display screen 250 in which the speech data and the still image are associated with each other.

[0088] S25: The first communication control unit 31 transmits the screen information of the text still image display screen 250 to the third terminal apparatus 10C.

[0089] S26: The communication unit 17 of the third terminal apparatus 10C receives the screen information of the text still image display screen 250, and the display control unit 18 displays the text still image display screen 250.

[0090] <Example of a text still image display screen> 12 shows a text still image display screen 250 displayed by the third terminal device 10C. The text still image display screen 250 may be displayed on any terminal device. The text still image display screen 250 has a recorded bibliographic information display field 251 and a text still image display field 252. The recorded bibliographic information display field 251 has a conference name field 253, a participant field 254, and a memo field 255. The recorded bibliographic information display field 251 is a field for displaying and editing the recorded bibliographic information. For example, when users A and C view the text still image display screen 250, they can edit the information in the recorded bibliographic information display field 251. Furthermore, if the information processing system 30 can acquire conference information from the remote conference service system 8, the acquired conference information may be transcribed into the recorded bibliographic information.

[0091] The text still image display field 252 displays the speech data 258, 259 and the still images 256, 257 in chronological order. The speech data 258, 259 are divided into sections. These sections correspond to the fixed time (e.g., 30 seconds) from which the still images 256, 257 are extracted. Therefore, one still image 256 or 257 is displayed for each section. In this way, the speech data 258, 259 and the still images 256, 257 are displayed in association with each other, so that user C can see the association between the utterance content and the scene, making it easier to understand the content of the video.

[0092] The number of still images 256, 257 per section does not need to be limited to one, and one or more sub-still images may be acquired with the still images 256, 257 as the main images. The still image acquisition unit 36 acquires the still images 256, 257 at regular intervals, and also acquires sub-still images every 1 / N seconds of the regular intervals. The sub-still images are displayed as thumbnails or the like, and are enlarged by the operation of users A and C.

[0093] Below the text still image display field 252, a play button 262, a forward button 263, a back button 261, a playback speed button 264, a current playback time 265, a seek bar pointer 266, and a total video time 267 are displayed. Therefore, user C can press the play button 262 to automatically play (scroll) the still images 256 and 257 and the speech data 258 and 259 in a synchronized state. User C can also operate the seek bar pointer 266 to scroll the still images and the speech data in a synchronized state. Furthermore, when user C scrolls the speech data 258 and 259 up and down, the still images 256 and 257 also scroll in synchronization with the scrolling, and when user C scrolls the still images 256 and 257 up and down, the speech data 258 and 259 also scroll in synchronization with the scrolling.

[0094] <Major Effects> The text still image display system 100 of this embodiment displays speech data in association with still images of scenes corresponding to the speech data, allowing the user to understand the correspondence between the content of the speech and the scenes, making it easier for the user to grasp the content of the video.

[0095] [Second embodiment] In this embodiment, an information processing system 30 will be described that allows user A to specify an area of a video from which a still image is to be captured. A video may contain presentation materials as well as facial images of participants, but there may be cases where the facial images are not desired to be recorded. Furthermore, because materials are also displayed in the video file, there may be information that should not be recorded by the information processing system 30 for confidentiality or security reasons.

[0096] In this embodiment, the hardware configuration diagram of FIG. 3 and the functional block diagram of FIG. 4 described in the above embodiment can be used.

[0097] <Screen example when uploading> Fig. 13 is a diagram illustrating a screen displayed by the first terminal device 10A when uploading a video file. The explanation of Fig. 13 will mainly focus on the differences from Fig. 8. First, upload screen (part 1) 210 may be the same as Fig. 8(a).

[0098] Fig. 13 shows an upload screen (part 2) 260 of this embodiment. The upload screen (part 2) 260 of Fig. 13 newly includes a check box 275 for specifying an area for acquiring a still image. "Specifying an area for acquiring a still image" means that user A specifies an area in a video from which a still image is to be acquired. When the upload button 223 is pressed while the check box 275 for specifying an area for acquiring a still image is in an on state, the screen transitions to an area specification screen 270 shown in Fig. 14.

[0099] FIG. 14 shows an area designation screen 270 displayed by the first terminal device 10A. The area designation screen 270 displays a still image or video when the video starts. The method for displaying the area designation screen 270 will be described in detail below. The area designation screen 270 displays a rectangular frame 271. While checking the still image or video, user A moves the rectangular frame 271 with a pointing device such as a mouse to designate the area from which the user A wants to acquire a still image. The area designated by user A in this manner becomes still image acquisition area information. When user A presses the OK button 274, the area designation screen 270 transitions to the upload screen (part 3) 230 shown in FIG. 8(c). User A may be able to set multiple rectangular frames 271. For example, if the areas that user A wants to acquire are located far apart on the screen, the still image acquisition unit 36 can save each of them at the minimum size.

[0100] 15 is a diagram showing still image acquisition area information. The still image acquisition area information is coordinate position information relating to the acquisition area where a still image is acquired. The still image acquisition area information is information in which the top left coordinate and bottom right coordinate of the acquisition area are associated. While the upload screen (part 3) 230 is displayed, the video file and still image acquisition area information are transmitted to the information processing system 30.

[0101] <<If the video layout changes midway>> Since the video layout may change during the meeting, it is a good idea to allow User A to change the capture area during the meeting. The video layout is the arrangement of the area in the entire video where materials are displayed, the area where participants are displayed, the area where meeting information is displayed, and the area where operation menus are displayed. When the video layout changes, the capture area captured as still images such as materials may also change.

[0102] 16 is a diagram illustrating an area designation screen 270 displayed by the first terminal device 10A. The area designation screen 270 in FIG. 16(a) displays a seek bar pointer 272 below the still image or video at the start of the video. The video file is also played in conjunction with the playback time indicated by the seek bar pointer 272. User A moves the seek bar pointer 272 to search for the time when the layout changed.

[0103] 16(b) shows an area designation screen 270 in which the layout has changed. For example, the number of participants increases, and the layout of the moving image displayed by the first terminal device 10A changes during the conference. Each time the layout changes, user A moves a rectangular frame 271 with a pointing device such as a mouse to designate the area from which he or she wants to acquire a still image. The area designated by user A is reflected in the capture of still images from the time designated by the seek bar pointer 266 onwards.

[0104] Note that since it is a heavy workload for user A to search for the time when the layout changed while moving the seek bar pointer 272, the first terminal device 10A or the information processing system 30 may detect the time when the layout of the video changed significantly and display it on the seek bar. The first terminal device 10A or the information processing system 30 periodically compares frames of the video, and if the difference is equal to or greater than a threshold, displays a mark 273 on the time on the seek bar. In Figures 16(a) and 16(b), the mark 273 is displayed at the time when the difference between image frames is equal to or greater than the threshold. User A moves the seek bar pointer 266 while referring to the mark 273, thereby reducing the workload.

[0105] Fig. 17 shows an example of still image acquisition area information when the layout of the video changes. In the still image acquisition area information in Fig. 17, the upper left coordinate and lower right coordinate of the acquisition area are associated with the elapsed time from the start of the conference. The still image acquisition area information is transmitted to the information processing system 30. In this way, the information processing system 30 can acquire still images from an appropriate area even if the layout of the video changes.

[0106] <Timing patterns for displaying the area specification screen> The timing at which the first terminal apparatus 10A displays the area designation screen includes the following patterns 1 and 2.

[0107] Pattern 1: Before the first terminal device 10A uploads a video file In this case, the information processing system 30 transmits in advance to the first terminal device 10A a function for acquiring still images from a video file. That is, the function is included in the upload screen (type 1) or the like. If the upload screen (type 1) is a web application, this function is written in JavaScript (registered trademark) or the like. When user A selects a video file in a web browser and presses the upload button 223, the function acquires still images from the video file and displays the area designation screen 270. After user A designates an area, when user A presses the OK button 274 on the area designation screen 270, the first terminal device 10A transmits still image acquisition area information together with the video file to the information processing system 30.

[0108] In the case of pattern 1, responsiveness is high because the first terminal apparatus 10A can display the area designation screen 270 without waiting for completion of transmission of the moving image file to the information processing system 30. However, since the first terminal apparatus 10A displays the moving image file, the load on the first terminal apparatus 10A is high.

[0109] Pattern 2: After the first terminal device 10A uploads video data In this case, the first terminal apparatus 10A first transmits the moving image file to the information processing system 30, which then acquires a still image (which may remain as a moving image) from the moving image file and transmits screen information for the area designation screen 270 displaying the still image to the first terminal apparatus 10A. The first terminal apparatus 10A displays the area designation screen 270. User A designates an area and presses the OK button 274, and the first terminal apparatus 10A transmits only information about the designated still image acquisition area to the information processing system 30.

[0110] In this pattern 2, the processing of the video file is performed by the information processing system 30, so the load on the first terminal device 10A is low. However, if the transmission of the video file to the information processing system 30 is not completed, the first terminal device 10A cannot display the area designation screen 270, so the waiting time for user A will be long.

[0111] <Upload process flow> 18 is a sequence diagram illustrating the flow of processing in which the first terminal device 10A uploads a video file to the information processing system 30 and the information processing system 30 acquires a still image in pattern 1. The explanation of FIG. 18 will mainly focus on the differences from FIG. 9. The processing from steps S1 to S5 may be the same as in FIG. 9.

[0112] S5-2: In FIG. 18, the upload button 223 on the upload screen (part 2) is pressed in step S5, causing the display control unit 12 of the first terminal apparatus 10A to display the area designation screen 270.

[0113] S5-3: User A moves rectangular frame 271 with a pointing device such as a mouse to specify an area from which a still image is to be acquired. Operation accepting unit 13 accepts the specification and creates still image acquisition area information.

[0114] S6: The communication unit 11 transmits a request for recognizing the audio data together with the moving image file and the still image acquisition area information to the information processing system 30. The subsequent processing from steps S7 to S10 may be the same as in FIG.

[0115] S11: The still image acquisition unit 36 acquires, at regular time intervals, still images of the acquisition area specified by the still image acquisition area information from the moving image file. The subsequent processing may be the same as in FIG.

[0116] 19 is a sequence diagram illustrating the flow of processing in pattern 2 in which the first terminal device 10A uploads a video file to the information processing system 30 and the information processing system 30 acquires a still image. The explanation of FIG. 19 will mainly focus on the differences from FIG. 9. The processing from steps S1 to S6 may be the same as in FIG. 9.

[0117] S6-2: In step S6, when the first communication control unit 31 of the information processing system 30 receives a recognition request for audio data together with the moving image file, the screen control unit 32 uses the moving image file to create the area designation screen 270. The first communication control unit 31 transmits screen information of the area designation screen 270 to the first terminal device 10A.

[0118] S6-3: When the communication unit 11 of the first terminal apparatus 10A receives the screen information of the area designation screen 270, the display control unit 12 causes the area designation screen 270 to be displayed.

[0119] S6-4: User A moves rectangular frame 271 with a pointing device such as a mouse to specify an area from which a still image is to be acquired. Operation accepting unit 13 accepts the specification and creates still image acquisition area information.

[0120] S6-5: The communication unit 11 transmits the still image acquisition region information to the information processing system 30. The subsequent processes from steps S7 to S10 may be the same as those in FIG.

[0121] S11: The still image acquisition unit 36 acquires still images of the acquisition area specified by the still image acquisition area information from the moving image file at regular intervals. The subsequent processing may be the same as in FIG.

[0122] <Major Effects> In addition to the effects of the first embodiment, the text still image display system 100 of this embodiment allows the user to specify the capture area of the still image. Even when the video layout changes, the user can specify the capture area at the timing of the layout change.

[0123] [Third embodiment] In this embodiment, an information processing system 30 will be described that uses a transcript to display still images and speech data in association with each other. Some remote conference service systems 8 recognize speech during a conference and provide a transcript file. The transcript file stores the start time, end date and time, and speech data in association with each other. By using the transcript file, the information processing system 30 can eliminate the need for speech recognition by a speech recognition service 9a and selectively display text data from the speech recognition service 9a and text data from the transcript file.

[0124] In this embodiment, the hardware configuration diagram of FIG. 3 and the functional block diagram of FIG. 4 described in the above embodiment can be used.

[0125] Fig. 20 shows an example of data contained in a transcript file. As shown in Fig. 20, in the transcript file, speech data is associated with the start and end times of utterances. Therefore, the transcript file has a similar structure to the text information management table.

[0126] <Example of video file list screen 240> Figure 21 shows a video file list screen 240 displayed by the first terminal device 10A. The explanation of Figure 21 will mainly focus on the differences from Figure 10A. The video file list screen 240 in Figure 21 newly includes a video file & transcript file upload button 248. The video file & transcript file upload button 248 is a button that allows the first terminal device 10A to send a video file and a transcript file to the information processing system 30. When user A presses the video file & transcript file upload button 248, an upload screen (1) 290 is displayed.

[0127] FIG. 22 shows an upload screen (part 1) 290 in this embodiment. The upload screen (part 1) 290 has a video file drag and drop field 291, a transcript file drag and drop field 292, and an upload button 293. User A drags and drops the video file into the video file drag and drop field 291. User A drags and drops the transcript file into the transcript file drag and drop field 292. When User A presses the upload button 293, the screen transitions to an upload screen (part 2) 220. The upload confirmation screen (part 2) may be the same as FIG. 8(b).

[0128] Also in this embodiment, as in the second embodiment, user A may specify the acquisition area.

[0129] <Upload process flow> 23 is a sequence diagram illustrating the flow of processing in which the first terminal device 10A uploads a video file to the information processing system 30 and the information processing system 30 acquires a still image. The explanation of FIG. 23 will mainly focus on the differences from FIG. 9. The processing from steps S1 to S4 may be the same as in FIG. 9.

[0130] S5: User A drops the video file to be uploaded into video file drag and drop field 291, drags and drops the transcript file into transcript file drag and drop field 292, and presses upload button 293. Operation acceptance unit 13 of first terminal device 10A accepts the operation, and upload screen (2) 220 and upload screen (3) 230 are displayed in accordance with User A's operation.

[0131] S6: The communication unit 11 sends a request to recognize the voice data along with the video file and the transcript file to the information processing system 30. Because the transcript file contains text data (speech data), voice recognition is not necessary. However, voice recognition by the voice recognition service 9a allows user C to switch between the speech data acquired from the voice recognition server 9 and the speech data in the transcript file. Therefore, the processing of steps S7 to S9 may be the same as in FIG. 9.

[0132] S10: The second communication control unit 33 of the information processing system 30 receives the text data, and the utterance data acquisition unit 35 acquires the utterance data, start time, and end time from the text data. The utterance data acquisition unit 35 stores the utterance data, etc. in the text information management table storage unit 3001 via the table management unit 37. The utterance data acquisition unit 35 also acquires the utterance data, start time, and end time included in the transcript file. The utterance data acquisition unit 35 stores the utterance data in the text information management table storage unit 3001 via the table management unit 37. The subsequent processing may be similar to that shown in FIG. 9.

[0133] FIG. 24 is a diagram illustrating the correspondence between the speech data generated by the speech recognition service 9a and the speech data acquired from the transcript file, which are stored in the text information management table storage unit 3001. As shown in FIG. 24, when the two are associated, the speech data in the transcript file is added to the text information management table. As shown in FIG. 24, the speech data in the transcript file may be added to the text information management table storage unit 3001, or the text still image display screen 250 may be created without the speech data being added. In FIG. 24, text A indicates the speech data generated by the speech recognition service 9a, and text B indicates the speech data acquired from the transcript file.

[0134] It should be noted that the start time and end time of the speech data recognized by the speech recognition server 9 may not be the same as the speech data in the transcript file. This is because the rules for dividing text data are different. For example, the table management unit 37 associates the speech data in the transcript file whose start time falls between the start time and end time of the speech data divided by the speech recognition server 9 with the start time and end time in the text information management table storage unit 3001.

[0135] <Viewing speech data and still images> 25 is a sequence diagram illustrating the process in which the third terminal device 10C displays the text still image display screen 250. The explanation of FIG. 25 will mainly focus on the differences from FIG. 11. The processes of steps S21 and S22 may be the same as those of FIG. 11.

[0136] S23: The first communication control unit 31 of the information processing system 30 receives a request to acquire a video file, and the table management unit 37 acquires a still image from the still image information management table storage unit 3002. The table management unit 37 also acquires speech data (text A) from the acquisition time associated with the still image to the acquisition time associated with the next acquired still image from the text information management table storage unit 3001. The table management unit 37 also acquires speech data (text B) from the acquisition time associated with the still image to the acquisition time associated with the next acquired still image from the text information management table storage unit 3001.

[0137] S24: The screen control unit 32 generates a text still image display screen 250 in which a still image is associated with one of the two pieces of utterance data acquired from the text information management table storage unit 3001. The utterance data with no associated still image is embedded in the text still image display screen 250 using JavaScript (registered trademark) or the like so that it can be displayed by an operation of the user C, or the first communication control unit 31 transmits it to the third terminal device 10C in response to a request from the third terminal device 10C.

[0138] <Text still image display screen 250> FIG. 26 illustrates a text still image display screen 250 displayed by the third terminal device 10C in this embodiment. The explanation of FIG. 26 will mainly focus on the differences from FIG. 12. The text still image display screen 250 in FIG. 26 newly includes an entire text switch button 268 and individual text switch buttons 269a and 269b. The entire text switch button 268 is a button that allows user C to collectively switch all of the utterance data on the screen from those recognized by the speech recognition service 9a to those in a transcript file. Alternatively, the entire text switch button 268 is a button that allows user C to collectively switch all of the utterance data on the screen from those in a transcript file to those recognized by the speech recognition service 9a. The entire text switch button 268 may display the origin of the currently displayed utterance data so that user C can see which utterance data is currently being displayed by the third terminal device 10C. User C can switch all of the utterance data with a single button.

[0139] The individual text switching buttons 269a and 269b are buttons that allow the user C to switch between speech data recognized by the speech recognition service 9a and that in a transcript file for each piece of speech data. For example, when speech data recognized by the speech recognition service 9a is displayed in the initial state, the individual text switching button 269a displays "Switch to transcript file." When the user C presses this individual text switching button 269a, the individual text switching button 269b displays "Switch to speech recognition result." The user C can switch the text to be displayed for each piece of speech data. For example, the speech data 281 has been switched to speech data in a transcript file, so the individual text switching button 269b is displayed.

[0140] The speech data recognized by the speech recognition service 9a and the speech data included in the transcript file should basically be the same, but the recognition accuracy differs, so user C can display the speech data that has been correctly recognized.

[0141] <Major Effects> In addition to the effects of the first embodiment, the text still image display system 100 of this embodiment allows the user to switch between multiple pieces of utterance data and display them, so that correctly recognized utterance data can be displayed.

[0142] <Other application examples> The best mode for carrying out the present invention has been described above using examples, but the present invention is not limited to these examples in any way, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention.

[0143] For example, in this embodiment, the information processing system 30 creates a text still image display screen from a video of an online conference, but the video content can be anything. For example, the video can be a seminar, product explanation, learning content, or any other content.

[0144] Furthermore, although the information processing system 30 acquires the video file from the terminal device 10, the video file may be provided from anywhere. For example, a server device that handles videos may send the video file to the information processing system 30 via an API. This server device can provide a text still image display screen to users who use its service.

[0145] Furthermore, the configuration examples shown in Figure 4 and other figures are divided according to main functions to make it easier to understand the processing by the text still image display system 100. The method of dividing the processing units or the names of the processing units does not limit the present invention. The processing by the text still image display system 100 can be divided into even more processing units depending on the processing content. Furthermore, it can also be divided so that one processing unit includes even more processes.

[0146] Furthermore, each function of the above-described embodiments can be realized by one or more processing circuits. Here, the term "processing circuit" in this specification includes a processor programmed to perform each function by software, such as a processor implemented by an electronic circuit, as well as devices such as an ASIC (Application Specific Integrated Circuit), a DSP (Digital Signal Processor), an FPGA (Field Programmable Gate Array), and conventional circuit modules designed to perform each of the above-described functions.

[0147] Embodiments of the present invention provide significant improvements in computer power and functionality. These improvements allow users to utilize computers that provide more efficient and robust interaction with tables, which are ways of storing and presenting information in information processing devices. Furthermore, embodiments of the present invention provide a better user experience through the use of more efficient, powerful, and robust user interfaces. Such user interfaces provide better interaction between humans and machines.

[0148] <Additional Notes> [Claim 1] a still image acquisition unit that acquires a still image from video data acquired from a first terminal device and first time information at which the still image was acquired from the video data; an utterance data acquisition unit that acquires a plurality of pieces of utterance data corresponding to audio data included in the video data and second time information when the utterance data was uttered in the video data; a communication control unit that transmits screen information in which the still image and the utterance data are associated with each other to a second terminal device based on the first time information and the second time information; An information processing system having the above. [Claim 2] the communication control unit acquires still image acquisition area information relating to an acquisition area for acquiring the still image from the first terminal device; 2. The information processing system according to claim 1, wherein the still image acquisition unit acquires the still image of the acquisition area from the video data. [Claim 3] the still image acquisition area information includes a plurality of acquisition areas associated with a time from the start of the video data, 3. The information processing system according to claim 2, wherein the still image acquisition unit switches the acquisition area for acquiring the still image from the video data according to the time. [Claim 4] the communication control unit transmits to the first terminal device screen information for accepting designation of the acquisition area using the video data; 4. The information processing system according to claim 2, wherein the still image acquisition area information including the acquisition area accepted for the screen displayed by the first terminal device based on the screen information is received from the first terminal device. [Claim 5] The first terminal device displayed the video data before the communication control unit received the video data. On a screen that accepts designation of the acquisition area using the video data, 4. The information processing system according to claim 2, wherein the still image acquisition area information including the acquisition area accepted by the first terminal device is received from the first terminal device. [Claim 6] the screen has a seek bar that indicates the playback time of the video data; a mark is displayed on the seek bar in association with a time when the layout of the video data is changed; 6. The information processing system according to claim 4, wherein the first terminal device accepts the still image acquisition area information based on the mark. [Claim 7] the still image acquisition unit acquires still images from the video data at regular intervals; the communication control unit transmits to the second terminal device, during the fixed time period that is an interval between the first time information and the next first time information, the screen information in which the utterance data including the second time information and the still image associated with the first time information are associated with each other; 7. The information processing system according to claim 1, wherein the second terminal device displays a still image and the speech data in association with each other at regular intervals. [Claim 8] a voice recognition control unit that requests a voice recognition service to recognize voice data included in the video data and acquires text data from the voice recognition service; 8. The information processing system according to claim 1, wherein the utterance data acquisition unit acquires a plurality of utterance data included in the text data and the second time information. [Claim 9] the communication control unit receives a transcript file together with the video data; 9. The information processing system of claim 8, wherein the speech data acquisition unit acquires a plurality of pieces of speech data included in the transcript file and the second time information. [Claim 10] The communication control unit associates the still image with the speech data based on the first time information and the second time information, and The information processing system according to claim 9, wherein the screen information in which the speech data contained in the text data and the speech data contained in the transcript file are associated with the second time information is transmitted to the second terminal device. [Claim 11] The information processing system of claim 10, wherein the second terminal device switches between and displays all of the speech data contained in the text data and all of the speech data contained in the transcript file in response to a user operation. [Claim 12] the second terminal device displays, for each of the utterance data, a button for switching between the utterance data included in the text data and the utterance data included in the transcript file; 12. The information processing system according to claim 10, wherein the utterance data included in the text data and the utterance data included in the transcript file are switched and displayed for each utterance data in response to a user's operation of the button. [Explanation of symbols]

[0149] 8. Teleconference service system 9. Speech Recognition Server 10A First terminal device 10B Second terminal device 10C Third terminal device 30 Information Processing Systems [Prior art documents] [Patent documents]

[0150] [Patent Document 1] Special Publication No. 2023-501728

Claims

1. a still image acquisition unit that acquires a still image from video data acquired from a first terminal device and first time information at which the still image was acquired from the video data; an utterance data acquisition unit that acquires a plurality of pieces of utterance data corresponding to audio data included in the video data and second time information when the utterance data was uttered in the video data; a communication control unit that transmits screen information in which the still image and the utterance data are associated with each other to a second terminal device based on the first time information and the second time information; An information processing system having the above.

2. the communication control unit acquires still image acquisition area information relating to an acquisition area for acquiring the still image from the first terminal device; The information processing system according to claim 1 , wherein the still image acquisition unit acquires the still image of the acquisition area from the video data.

3. the still image acquisition area information includes a plurality of acquisition areas associated with a time from the start of the video data, The information processing system according to claim 2 , wherein the still image acquisition unit switches the acquisition area for acquiring the still image from the video data according to the time.

4. the communication control unit transmits to the first terminal device screen information for accepting designation of the acquisition area using the video data; The information processing system according to claim 2 or 3, wherein the still image acquisition area information including the acquisition area accepted for the screen displayed by the first terminal device based on the screen information is received from the first terminal device.

5. The first terminal device displayed the video data before the communication control unit received the video data. On a screen that accepts designation of the acquisition area using the video data, The information processing system according to claim 2 or 3, wherein the still image acquisition area information including the acquisition area accepted by the first terminal device is received from the first terminal device.

6. the screen has a seek bar that indicates the playback time of the video data; a mark is displayed on the seek bar in association with a time when the layout of the video data is changed; The information processing system according to claim 4 , wherein the first terminal device accepts the still image acquisition area information based on the mark.

7. the still image acquisition unit acquires still images from the video data at regular intervals; the communication control unit transmits to the second terminal device, during the fixed time period that is an interval between the first time information and the next first time information, the screen information in which the utterance data including the second time information and the still image associated with the first time information are associated with each other; The information processing system according to claim 1 , wherein the second terminal device displays a still image and the speech data in association with each other at regular intervals.

8. a voice recognition control unit that requests a voice recognition service to recognize voice data included in the video data and acquires text data from the voice recognition service; The information processing system according to claim 1 , wherein the speech data acquisition unit acquires a plurality of pieces of speech data included in the text data and the second time information.

9. the communication control unit receives a transcript file together with the video data; The information processing system according to claim 8 , wherein the speech data acquisition unit acquires a plurality of pieces of speech data included in the transcript file and the second time information.

10. The communication control unit associates the still image with the speech data based on the first time information and the second time information, and The information processing system of claim 9, wherein the screen information in which the speech data contained in the text data and the speech data contained in the transcript file are associated with the second time information is transmitted to the second terminal device.

11. The information processing system according to claim 10 , wherein the second terminal device switches between displaying all of the speech data contained in the text data and all of the speech data contained in the transcript file in response to a user operation.

12. the second terminal device displays, for each of the utterance data, a button for switching between the utterance data included in the text data and the utterance data included in the transcript file; 12. The information processing system according to claim 10, wherein the display switches between the speech data included in the text data and the speech data included in the transcript file for each piece of speech data in response to a user's operation of the button.

13. A text still image display system in which a terminal device and an information processing system communicate with each other via a network, The information processing system includes: a still image acquisition unit that acquires a still image from video data acquired from a first terminal device and first time information at which the still image was acquired from the video data; an utterance data acquisition unit that acquires a plurality of pieces of utterance data corresponding to audio data included in the video data and second time information when the utterance data was uttered in the video data; a communication control unit that transmits screen information in which the still image and the utterance data are associated with each other to a second terminal device based on the first time information and the second time information, The second terminal device a communication unit that receives screen information in which the still image and the speech data are associated with each other; a display control unit that displays a screen in which the still image and the speech data are associated with each other based on the screen information; A text still image display system having:

14. A text still image display method performed by an information processing system, comprising: A process of acquiring a still image and first time information at which the still image was acquired from the video data acquired from the first terminal device; a process of acquiring a plurality of pieces of speech data corresponding to audio data included in the video data, and second time information when the speech data was uttered in the video data; a process of transmitting screen information in which the still image and the utterance data are associated with each other to a second terminal device based on the first time information and the second time information; A text still image display method having the above construction.

15. A program for causing a computer to function as the information processing system according to claim 1.

Images