Information processing device, information processing system, and program

Abstract

To reduce the workload on a user when processing specific data of a plurality of pieces of data of static images generated from video image data of a plurality of sheets of documents.SOLUTION: A management server 10 serving as an information processing apparatus includes a control unit 11. The control unit 11 includes: a video voice management unit 102 which manages, on the same time axis, video image data obtained by imaging a plurality of sheets of documents and voice data recorded in parallel with imaging, in association with each other; a voice detection unit 103 which detects a voice command from the recorded voice data; an image generation unit 104 which generates static image data for the plurality of documents from the video image data; and a processing control unit 106 which controls executing predetermined processing on the static image data of the documents, the data being to be captured at the time when the voice command is issued.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing system, and a program. [Background technology]

[0002] Even when a user wants to digitize an image formed on a paper document, there are cases where a device with an image reading function (for example, a scanner or a multifunction printer) cannot be used depending on the situation. In such a situation, the user may capture a moving image of the document using the imaging function of a smartphone or the like, and generate still image data of the document from the moving image data, and related technology also exists (for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2010-098615 Summary of the Invention [Problem to be solved by the invention]

[0004] However, when there are multiple documents, it may be necessary to process a specific document differently from other documents. In such cases, the user must search for the target document among the multiple generated still images and process it manually. However, when there is a large amount of generated still image data, searching for the target document takes time, which places a heavy burden on the user.

[0005] An object of the present invention is to reduce the workload on a user when processing specific data among a plurality of still image data generated from moving image data of a plurality of originals. [Means for solving the problem]

[0006] The invention described in claim 1 is an information processing device comprising a processor, which associates moving image data of multiple sheets of photographed original documents with audio data recorded in parallel with the photographing and manages them on the same time axis, detects predetermined audio from the audio data, and controls the execution of predetermined processing on still image data of the multiple sheets of original documents that was the subject of photographing at the timing when the predetermined audio was emitted, among the still image data of the multiple sheets of original documents generated from the moving image data. The invention described in claim 2 is the information processing device described in claim 1, characterized in that the moving image data is data of a single moving image obtained by continuously capturing images formed on each of the multiple original documents. The invention described in claim 3 is the information processing device described in claim 2, characterized in that the data of the first moving image is captured based on a first action and a second action of a user, and the audio data is a recording of audio uttered by the user. The invention described in claim 4 is the information processing device described in claim 3, characterized in that the first operation is an operation of capturing an image of the multiple documents, and the second operation is an operation for making each of the multiple documents the subject. The invention described in claim 5 is the information processing device described in claim 4, characterized in that the operation for making each of the multiple documents the subject is the operation of turning over the multiple documents one by one. The invention described in claim 6 is the information processing device described in claim 1, characterized in that the processor controls the execution of the predetermined processing by controlling the execution of processing corresponding to the predetermined voice. The invention described in claim 7 is the information processing device described in claim 6, characterized in that there are multiple of the predetermined voices and the predetermined processes, and each of the predetermined voices and the predetermined processes is stored in correspondence with each other in a predetermined database. The invention described in claim 8 is the information processing device described in claim 7, characterized in that the predetermined sound is at least one of a sound indicating that it is the predetermined sound, a sound for specifying the document to be processed, a sound indicating the content of the processing, and a sound for deleting the recorded sound. The invention described in claim 9 is an information processing device described in claim 8, characterized in that the audio for specifying the document to be processed and the audio for deleting the recorded audio include audio for specifying one document to be processed out of multiple double-page spread documents. The invention described in claim 10 is the information processing device described in claim 8, characterized in that the audio indicating the content of the processing is one or more of the audio for specifying each of the output format, attributes, and configuration of the still image data to be processed. The invention described in claim 11 is the information processing device described in claim 10, characterized in that the audio for specifying the output format of the still image data is audio for specifying one or more of the file format, color, and orientation of the still image data. The invention described in claim 12 is an information processing device described in claim 10, characterized in that the audio for specifying the attributes of the still image data is audio for specifying one or more of whether the still image data can be printed, whether it can be edited, whether it can be transcribed, whether it is encrypted, and the language used during character recognition. The invention described in claim 13 is an information processing device described in claim 10, characterized in that the audio for specifying the configuration of the still image data is audio for specifying one or more of inserting a document and deleting a document in the still image data. The invention described in claim 14 is an information processing system characterized by having an imaging means for capturing moving images of multiple documents; a recording means for recording audio in parallel with the imaging; a management means for acquiring data of the captured moving images and the recorded audio data, and managing the moving image data and the audio data on the same time axis in association with each other; a detection means for detecting a predetermined audio from the audio data; a generation means for generating data of still images of the multiple documents from the moving image data; and a processing execution control means for controlling the execution of a predetermined process on the still image data of the document that was the subject of imaging at the timing when the predetermined audio was emitted, among the generated still image data. The invention described in claim 15 is a program for enabling a computer to realize the following functions: a function of associating moving image data of multiple sheets of photographed original documents with audio data recorded in parallel with the photographing and managing them on the same time axis; a function of detecting a predetermined sound from the audio data; and a function of controlling the execution of a predetermined process on the still image data of the multiple sheets of photographed original documents at the timing when the predetermined sound is emitted, among the still image data of the multiple sheets of photographed original documents generated from the moving image data. [Effects of the Invention]

[0007] According to the present invention of claim 1, it is possible to provide an information processing device that can reduce the workload on a user when performing predetermined processing on specific data among multiple still image data generated from moving image data of multiple original documents. According to the present invention of claim 2, the workload on the user when performing predetermined processing on specific data among multiple still image data generated from one moving image data obtained by continuously capturing multiple sheets of original document can be reduced more than before. According to the present invention of claim 3, the workload on the user can be reduced more than ever before when performing predetermined processing on specific data among multiple still image data generated from one moving image data obtained by continuously capturing multiple pages of a document using two types of user actions. According to the present invention of claim 4, the workload on the user when performing predetermined processing on specific data among multiple still image data generated from data of a single moving image captured by the user's action of capturing multiple sheets of original document and action of making each of the multiple sheets of original document the subject can be reduced more than in the past. According to the present invention of claim 5, the workload on the user when performing predetermined processing on specific data among multiple still image data generated from one moving image data captured by the user's actions of capturing multiple sheets of original document and turning over the multiple sheets of original document one by one can be reduced more than in the past. According to the present invention of claim 6, the workload on the user can be reduced more than ever before when processing specific data among multiple still image data generated from moving image data of multiple original documents in accordance with a predetermined sound. According to the present invention of claim 7, the workload on the user can be reduced more than ever before when processing specific data among multiple still image data generated from moving image data of multiple original documents in accordance with a predetermined sound. According to the present invention of claim 8, it is possible to specify processing for still image data of a specific document to be processed using at least one of a voice indicating that the voice is a predetermined voice, a voice for specifying the document to be processed by the predetermined voice, a voice indicating the content of the predetermined processing, and a voice for deleting the recorded voice. According to the present invention of claim 9, even when a specific document to be processed is included in a spread of multiple pages, it is possible to specify processing for that document by voice. According to the present invention, it is possible to specify the output format, attributes, and configuration of data of a still image of a specific document to be processed by voice. According to the present invention of claim 11, it is possible to specify by voice the file format, color, and orientation as the output format of the still image data of a specific document to be processed. According to the present invention of claim 12, it is possible to specify by voice whether or not the still image data of a specific document to be processed can be printed, edited, transcribed, encrypted, and the language used for character recognition. According to the present invention of claim 13, it is possible to specify by voice whether to insert or delete a document as a data configuration of a still image of a specific document to be processed. According to the present invention of claim 14, it is possible to provide an information processing system that can reduce the workload of a user when performing predetermined processing on specific data among multiple still image data generated from moving image data of multiple original documents. According to the present invention of claim 15, it is possible to provide a program that enables a user to reduce the workload when performing predetermined processing on specific data among multiple still image data generated from moving image data of multiple original documents. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a diagram illustrating an example of the overall configuration of an information processing system to which the present embodiment is applied. [Figure 2] FIG. 1 is a diagram illustrating a hardware configuration of a management server as an information processing apparatus to which the present embodiment is applied. [Figure 3] FIG. 1 is a diagram illustrating a hardware configuration of a user terminal as an information processing device to which the present embodiment is applied. [Figure 4] FIG. 2 is a diagram illustrating a functional configuration of a control unit of the management server. [Figure 5] FIG. 2 is a diagram illustrating the functional configuration of a control unit of the user terminal. [Figure 6] 10 is a flowchart showing the flow of processing of a user terminal. [Figure 7] 10 is a flowchart showing the flow of processing by the management server. [Figure 8] FIG. 10 is a diagram showing a specific example of an operation of a user terminal by a user. [Figure 9] 9 is a diagram showing a specific example of still image data generated from the moving image and audio data of FIG. 8. FIG. [Figure 10] 10A and 10B are diagrams showing a specific example of processing for extracting a document from moving image data of moving image and audio data to generate still image data. [Figure 11] FIG. 10 is a diagram showing a specific example of a process for dividing a two-page spread document and generating data for two still images. [Figure 12] FIG. 10 is a diagram showing a specific example of a voice command indicating the content of a predetermined process. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. (Configuration of information processing system) FIG. 1 is a diagram showing an example of the overall configuration of an information processing system 1 to which the present embodiment is applied. The information processing system 1 is configured by connecting a management server 10 and a user terminal 30 via a network 90. ​​The network 90 is, for example, a LAN (Local Area Network), the Internet, or the like.

[0010] Management server 10 is an information processing device that serves as a server that manages information processing system 1. For example, management server 10 associates video image data capturing the entire process of multiple sheets of paper being turned over one by one with user voice data recorded simultaneously with the image capture and manages them on the same time axis. Management server 10 then detects predetermined voice from the recorded voice data and performs predetermined processing on the still image data of the document that was the subject of image capture at the timing when the detected voice was uttered by the user, among the still image data of multiple sheets of paper generated from the video image data.

[0011] Here, "manuscript" refers to a sheet of paper with characters or graphics formed on the front or both the front and back. "Multiple sheets of manuscript" refers to a bundle of multiple manuscripts. Note that "bundled" here may mean, for example, that multiple manuscripts are bound together, or that some of the manuscripts are stapled together. It may also mean that the manuscripts are not stapled together, but each manuscript is independent but is collected as a single document.

[0012] Furthermore, the "predetermined voice" refers to a voice stored in association with information indicating a predetermined process in a database in the storage unit 13 (see FIG. 2), which will be described later. Examples of the "predetermined voice" include a voice indicating that the voice is a predetermined voice, a voice for specifying a document to be subjected to the predetermined process, a voice indicating the content of the process, and a voice for deleting a recorded voice. Hereinafter, the "predetermined voice" will be referred to as a "voice command." Specific examples of voice commands and details of the above-mentioned process by the management server 10 will be described later.

[0013] The user terminal 30 is an information processing device such as a smartphone or tablet terminal operated by a user. For example, the user terminal 30 generates data for one moving image by continuously capturing images of multiple documents based on the user's imaging operation. The user performing the imaging operation holds the user terminal 30 in one hand and captures moving images while displaying multiple documents on the display unit 36 ​​(see FIG. 3) of the user terminal 30. Then, with the other hand, the user turns over the multiple documents one by one from the top, capturing moving images of the front and back sides of all the documents. For example, the user turns over the multiple documents one by one from the top with their right hand, while capturing moving images of the documents using the user terminal 30 held in their left hand.

[0014] Here, when a document to be processed in advance is displayed on the display unit 36 ​​during the image capture process, the user performing the image capture operation issues a voice command at any timing during the time period in which the document is being imaged. The user terminal 30 records the voice command issued by the user and stores the voice data and video data of the captured documents in association with each other on the same time axis. The user terminal 30 transmits the stored combination of the video data and voice data in association with each other to the management server 10. Details of these processes performed by the user terminal 30 will be described later.

[0015] The functions of the management server 10 and the user terminal 30 constituting the information processing system 1 described above are merely examples, and the information processing system 1 as a whole may have the functions to realize the above-described processes. Therefore, some or all of the functions to realize the above-described processes may be shared or cooperated within the information processing system 1. That is, some or all of the functions of the management server 10 may be functions of the user terminal 30, or some or all of the functions of the user terminal 30 may be functions of the management server 10. Furthermore, some or all of the functions of the management server 10 and the user terminal 30 constituting the information processing system 1 may be transferred to another server, imaging device, or the like (not shown). This facilitates processing by the information processing system 1 as a whole and also enables the processes to complement each other.

[0016] (Management server hardware configuration) FIG. 2 is a diagram showing the hardware configuration of a management server 10 as an information processing device to which this embodiment is applied. The management server 10 has a control unit 11, a memory 12, a storage unit 13, a communication unit 14, an operation unit 15, and a display unit 16. These units are connected by a data bus, an address bus, a PCI (Peripheral Component Interconnect) bus, etc.

[0017] The control unit 11 is a processor that controls the functions of the management server 10 through the execution of various software such as an OS (operating system) and application software. The control unit 11 is configured, for example, by a CPU (Central Processing Unit). The memory 12 is a storage area that stores various software and data used for executing the software, and is used as a working area for calculations. The memory 12 is configured, for example, by a RAM (Random Access Memory).

[0018] The storage unit 13 is a storage area for storing input data for various software programs, output data from various software programs, etc. The storage unit 13 is configured with, for example, a hard disk drive (HDD), a solid state drive (SSD), a semiconductor memory, etc. used for storing programs, various setting data, etc. The storage unit 13 stores databases for storing various information, such as a video / audio DB 801 storing data combining video data and audio data transmitted from the user terminal 30, a still image DB 802 storing data of still images of multiple pages of documents generated from the video data, and a voice command DB 803 storing voice commands and information indicating predetermined processes in association with each other.

[0019] The communication unit 14 transmits and receives data to and from the user terminal 30 and the outside via the network 90. ​​The operation unit 15 is composed of, for example, a keyboard, a mouse, mechanical buttons, and switches, and accepts input operations. The operation unit 15 also includes a touch sensor that forms a touch panel integrally with the display unit 16. The display unit 16 is composed of, for example, a liquid crystal display or an organic EL (Electro Luminescence) display used to display information, and displays images, text data, and the like.

[0020] (Hardware configuration of user terminal) FIG. 3 is a diagram showing the hardware configuration of a user terminal 30 as an information processing device to which this embodiment is applied. The user terminal 30 has a control unit 31, memory 32, storage unit 33, communication unit 34, operation unit 35, and display unit 36, which correspond to the control unit 11, memory 12, storage unit 13, communication unit 14, operation unit 15, and display unit 16 of the management server 10 in Fig. 2, respectively, and in addition to these components, has an imaging unit 37 and a sound recording unit 38. These units are connected by a data bus, an address bus, a PCI bus, etc.

[0021] The imaging unit 37 is configured with a camera or the like, and captures images of multiple documents as subjects displayed on the display unit 36, which also functions as a camera viewfinder, and acquires the images as moving image data. The recording unit 38 records the voice uttered by the user and acquires the voice data.

[0022] (Functional configuration of the control unit of the management server) FIG. 4 is a diagram showing the functional configuration of the control unit 11 of the management server 10. As shown in FIG. In the control unit 11 of the management server 10, an information acquisition unit 101, a video and audio management unit 102, an audio detection unit 103, an image generation unit 104, a command management unit 105, a processing control unit 106, and a transmission control unit 107 function.

[0023] The information acquisition unit 101 acquires information via the communication unit 14 (see FIG. 2). For example, the information acquisition unit 101 acquires data that is a combination of video data and audio data transmitted from the user terminal 30. Specifically, the information acquisition unit 101 acquires data that is a combination of video data that captures the entire process of multiple pages of a document being turned over one by one, and audio data uttered by the user during the capturing of the video. Hereinafter, the data that is a combination of video data and audio data will be referred to as "video audio data."

[0024] As a management means, the video / audio management unit 102 stores and manages the video / audio data acquired by the information acquisition unit 101 in a video / audio DB 801 (see FIG. 2) in the storage unit 13. The video / audio data includes the appearance of each of the multiple originals and the image formed on each of the multiple originals. Here, the "formed image" refers to the "image of printed characters, figures, etc." on the printing surface of the paper that serves as the original, and the "printed surface" may be the front surface only or both surfaces (front and back).

[0025] As described above, the video / audio management unit 102 manages video / audio data in which video data and audio data are associated on the same time axis. For example, if a user issues a voice command 10 seconds after starting image capture and recording, information indicating the voice command issued by the user is associated with information indicating the nth sheet of original (n is an integer greater than or equal to 1) that was the image capture target at that time. Specific examples of video / audio data will be described later with reference to FIG. 9.

[0026] The voice detection unit 103, as a detection means, analyzes voice data from the video and audio data stored and managed in a database by the video and audio management unit 102 to detect voice commands. The method for detecting voice commands is not particularly limited, and conventional methods can be used. For example, a technique is used in which feature quantities such as the strength, frequency, and interval of sounds contained in the voice data are extracted, and the matching rate with phoneme or word models stored in advance in a database (for example, the voice command DB 803 of the storage unit 13) is calculated to recognize the command as a word.

[0027] The types of voice commands detected by the voice detection unit 103 include, for example, a voice command indicating that the spoken content is a voice command, a voice command for specifying a document to be subjected to a predetermined process, a voice command indicating the content of a predetermined process, a voice command for deleting a recorded voice command, etc.

[0028] Among the types of voice commands described above, the voice command for specifying a document to be processed in advance and the voice command for deleting a recorded voice command include, for example, a voice command for specifying one document to be processed among a plurality of double-page spreads. Furthermore, the voice command indicating the content of the predetermined processing includes, for example, a voice command for specifying the output format, attributes, and configuration of still image data to be processed in advance. Specific examples of voice commands indicating the content of the predetermined processing will be described later with reference to FIG. 12.

[0029] Image generation unit 104, as a generation means, generates still image data for each of a plurality of pages of a document from moving image data among the moving image and audio data stored in a database and managed by moving image and audio management unit 102. The still image data generated by image generation unit 104 is stored and managed in still image DB 802 (see FIG. 2) of storage unit 13.

[0030] The command management unit 105 associates the voice command with information indicating a predetermined process, and stores and manages the information in the voice command DB 803. Specific examples of the information stored in the voice command DB 803 will be described later with reference to FIG.

[0031] The processing control unit 106, as a processing execution control means, controls the execution of predetermined processing on the still image data of the document that is the subject of imaging at the time the voice command is issued, among the still image data of each of the multiple documents generated by the image generation unit 104.

[0032] The transmission control unit 107 controls the transmission of various information to the user terminal 30 or to the outside via the communication unit 14 (see FIG. 2). For example, the transmission control unit 107 controls the transmission of data of still images of each of the multiple pages of a document, generated by the image generation unit 104, to the user terminal 30.

[0033] (Functional configuration of the control unit of the user terminal) FIG. 5 is a diagram showing the functional configuration of the control unit 31 of the user terminal 30. As shown in FIG. In the control unit 31 of the user terminal 30, a display control unit 301, an imaging control unit 302, a sound recording control unit 303, a transmission control unit 304, and an information acquisition unit 305 function.

[0034] The display control unit 301 controls the display of various information on the display unit 36 ​​(see FIG. 3). For example, the display control unit 301 controls the display of multiple sheets of document images to be captured on the display unit 36. The display control unit 301 also controls the display of still image data acquired by an information acquisition unit 305 (described later) on the display unit 36.

[0035] The imaging control unit 302, as an imaging means, controls the imaging unit 37 (see FIG. 3) to capture moving images of multiple documents as subjects. Specifically, the imaging control unit 302 controls the imaging unit 37 to continuously capture the entire process of multiple documents being turned over one by one. The sound recording control unit 303, as a sound recording means, controls the sound recording unit 38 to record sounds emitted by the user in parallel with the imaging by the imaging unit 37.

[0036] The transmission control unit 304 controls the transmission of various information to the management server 10 or to the outside via the communication unit 34 (see FIG. 3). For example, the transmission control unit 304 controls the transmission of video and audio data, which is a combination of video data generated as a result of imaging by the imaging unit 37 and audio data generated as a result of recording by the audio recording unit 38, to the management server 10.

[0037] The information acquisition unit 305 acquires various types of information via the communication unit 34 (see FIG. 3). For example, the information acquisition unit 305 acquires still image data transmitted from the management server 10. The still image data acquired by the information acquisition unit 305 includes still image data processed in response to a voice command.

[0038] (User terminal processing) FIG. 6 is a flowchart showing the flow of processing performed by the user terminal 30. First, based on the imaging operation by the user, multiple sheets of originals to be imaged are displayed on the display unit 36 ​​(step 601), and imaging of the multiple sheets of originals being turned over one by one and audio recording are started (step 602). As a result, audio commands uttered by the user during imaging and audio recording are associated with the moving image data and stored.

[0039] When the user terminal 30 has completed capturing moving images and recording audio (YES in step 603), it transmits moving image and audio data to the management server 10 (step 604). On the other hand, if the user terminal 30 has not completed capturing moving images and recording audio (NO in step 603), it repeats step 603.

[0040] Thereafter, when still image data is transmitted from management server 10 (YES in step 605), user terminal 30 acquires the transmitted data (step 606) and displays it on display unit 36 ​​(step 607). On the other hand, if still image data has not been transmitted (NO in step 605), step 605 is repeated. The still image data acquired in step 606 includes still image data processed in response to a voice command.

[0041] (Management server processing) FIG. 7 is a flowchart showing the flow of processing by the management server 10. When video and audio data is transmitted from the user terminal 30 (YES in step 701), the management server 10 acquires the transmitted data (step 702), stores it in a database, and manages it (step 703). Specifically, the management server 10 stores and manages the video and audio data in the video and audio DB 801 (see FIG. 2) of the storage unit 13. On the other hand, if video and audio data has not been transmitted (NO in step 701), the processing of step 701 is repeated until video and audio data is transmitted.

[0042] The management server 10 generates still image data for each of the multiple pages of the document from the moving image data of the moving image and audio data (step 704), and stores and manages the generated data in a database (step 705). Specifically, the management server 10 stores and manages the still image data in a still image DB 802 (see FIG. 2) in the storage unit 13.

[0043] Management server 10 analyzes the audio data from the video and audio data stored in the database, and if it detects an audio command (YES in step 706), it executes a predetermined process (step 707) on the still image data of the document that was the subject of imaging at the timing when the detected audio command was issued, among the still image data of each of the multiple documents generated in step 704. On the other hand, if no audio command is detected (NO in step 706), the process ends.

[0044] (Example) FIG. 8 is a diagram showing a specific example of an operation of the user terminal 30 by the user. Using the user terminal 30, the user simultaneously captures moving images of multiple pages of a document and records the user's voice. The combination of the captured moving image data and recorded voice data by the user terminal 30 is transmitted to the management server 10 as moving image and voice data, and is stored and managed in the moving image and voice DB 801. For example, as shown in FIG. 8, assume that the multiple pages of a document are a book. In this case, the user captures moving images while turning pages with the hand opposite to the hand holding the user terminal 30. Note that a "page" here corresponds to one side of a document on which images such as characters and figures are formed on both sides.

[0045] A user capturing moving images of a book issues a voice command at any timing during the time period when the document on which the user wishes to perform a predetermined process is being captured. For example, suppose the user wishes to perform a predetermined process on each of the third page (left page) and the fourth page (right page) of a spread. In this case, the user starts capturing images before turning over the cover, and issues a voice command at any timing from the time when the cover is turned over to the first and second pages of the spread, turning over another page to the time when the pages to be captured become the third and fourth pages of the spread, or turning over another page to the time when the pages to be captured become the fifth and sixth pages of the spread.

[0046] For example, suppose a user wants to output page 3 of a spread as an electronic file in Comma-Separated Values ​​(CSV) format instead of the default Portable Document Format (PDF) format for page 3, and does not want to output page 4. In this case, the user issues a series of voice commands, such as "start command," "left page," "CSV output," "start command," "right page," and "prohibit output." In this case, "start command," "left page," and "CSV output" constitute one set of voice commands for page 3, and "start command," "right page," and "prohibit output" constitute one set of voice commands for page 4.

[0047] Among the voice commands, for example, "command start" is a voice indicating that it is a predetermined voice, that is, a voice command indicating that the voices to be emitted from now on, including the voice currently being emitted, are voice commands. Also, for example, "left page" is a voice command indicating that the document on which the predetermined process is to be executed is the left page (page 3) of a double-page spread, and "right page" is a voice command indicating that the document on which the predetermined process is to be executed is the right page (page 4) of a double-page spread. Also, for example, "CSV output" and "prohibit printing" are both voice commands indicating the content of a predetermined process. When a voice command indicating the content of a predetermined process is issued, it is recognized that one instruction has been completed.

[0048] FIG. 9 is a diagram showing a specific example of still image data generated from the moving image and audio data of FIG. The left diagram in Figure 9 is a conceptual diagram showing the video and audio data generated in the specific example in Figure 8 on a time axis, with the top row representing video data, the bottom row representing audio data, and the horizontal axis representing time. "Page 1,2" indicates pages 1 and 2 of the spread, and "Page 3,4" indicates pages 3 and 4 of the spread. The dividing line between the two indicates the timing when the imaging subject changes.

[0049] As described above, the subjects for which the user issued voice commands were pages 3 and 4 of the spread. Therefore, the voice data when the subjects to be imaged were pages 3 and 4 of the spread contains voice commands, but the voice data when the subjects to be imaged were pages 1 and 2 of the spread does not contain voice commands.

[0050] The management server 10 generates still image data by extracting images page by page from the moving image data of the moving image and audio data. The generated still image data is stored and managed in the still image DB 802. Still image data may be generated uniformly for all pages, or the audio data of the moving image and audio data may be analyzed to generate still image data only for pages on which audio commands are recorded. In the specific example of FIG. 9, still image data (PDF format) is generated for each of the third and fourth pages on which audio commands are recorded (see the right diagram of FIG. 9).

[0051] The management server 10 executes processing according to the voice command for the still image of the page on which the voice command is recorded. In the specific example of Fig. 9, the voice command instructs that the third page be output in CSV format, so the still image data generated in PDF format is converted to SCV format and then output (see the right diagram of Fig. 9). Also, the voice command instructs that the fourth page not be output, so the still image data generated in PDF format is not output (see the right diagram of Fig. 9).

[0052] FIG. 10 is a diagram showing a specific example of processing for extracting a document from moving image data of moving image and audio data to generate still image data. The management server 10 extracts still image data from the moving image data of the video and audio data at predetermined intervals, compares the data of the still images before and after the still images from the extracted data of the multiple still images to extract the difference, and determines the timing when the page is turned and the timing when the page of the imaged object is changed based on whether the difference exceeds a predetermined threshold.

[0053] 10, the management server 10 compares still image data G1 extracted at timing t1 with still image data G2 extracted at timing t2 to extract the difference. Here, since there is no difference between still image data G1 and still image data G2, the management server 10 determines that "no page was turned" in the time period between timing t1 and timing t2.

[0054] Furthermore, management server 10 compares still image data G2 extracted at timing t2 with still image data G3 extracted at timing t3 to extract the difference. Here, still image data G2 is still image data extracted just before the page begins to be turned, and still image data G3 is still image data extracted while the page is being turned, so the difference between the two is large and exceeds a predetermined threshold. In this case, management server 10 determines that the time period between timing t2 and timing t3 is "while the page is being turned."

[0055] Furthermore, management server 10 compares still image data G3 extracted at timing t3 with still image data G4 extracted at timing t4 to extract the difference. Here, still image data G3 is still image data extracted while a page is being turned, and still image data G4 is still image data extracted immediately after the page is turned, so the difference between the two is large and exceeds a predetermined threshold. In this case, management server 10 determines that "the page has finished turning" at some point in the time period between timing t3 and timing t4.

[0056] Based on the result of such a determination, the management server 10 records timestamps indicating the start and end of imaging for each page of the image data included in the video and audio data, thereby enabling the generation of still image data for each page.

[0057] FIG. 11 shows a specific example of a process for dividing a two-page spread document and generating data for two still images. If the document to be imaged, which is included in the moving image data of the moving image and audio data, is a two-page spread document, it is divided into a left page and a right page, and each is stored as still image data in a database (for example, still image DB802 of memory unit 13 in Figure 2).

[0058] For example, as shown in the upper part of Fig. 11, if a page on which the letter "A" is formed (the left page) and a page on which the letter "B" is formed (the right page) are stored as moving image data of a two-page spread manuscript, the manuscript is split to generate still image data of the page on which the letter "A" is formed and still image data of the page on which the letter "B" is formed. Also, for example, in the specific example of Fig. 10 described above, the two-page spread manuscript on which the letters "A" and "B" are formed and the two-page spread manuscript on which the letters "C" and "D" are formed are each split to generate four still image data on which the letters "A" to "D" are formed.

[0059] FIG. 12 shows specific examples of voice commands indicating the contents of predetermined processes. 12, voice commands indicating the content of predetermined processing include, for example, those for specifying the output format of still image data to be processed by the predetermined processing, those for specifying attributes, those for specifying the configuration at the time of output, etc. Of these, voice commands for specifying the output format of still image data include, for example, those for specifying the file format of still image data, those for specifying color, those for specifying orientation, etc.

[0060] Specific examples of voice commands for specifying the file format of still image data include "output in PDF format" and "output in JPEG format." Of these, "output in PDF format" is a voice command for outputting the generated still image data in PDF format. Furthermore, "output in JPEG format" is a voice command for outputting the generated still image data in JPEG (Joint Photographic Experts Group) format.

[0061] Specific examples of voice commands for specifying the color of still image data include "full color" and "black and white." Of these, "full color" is a voice command for outputting the generated still image data in full color. "Black and white" is a voice command for outputting the generated still image data in black and white. Specific examples of voice commands for specifying the orientation of still image data include "rotate 90 degrees right." "Rotate 90 degrees right" is a voice command for outputting the generated still image data rotated 90 degrees to the right.

[0062] Specific examples of voice commands for specifying attributes of still image data include "output prohibited," "edit prohibited," "transcription prohibited," "password setting (yes / no)," and "OCR language setting." Of these, "output prohibited" is a voice command for preventing the generated still image data from being output (for example, printed, converted to a predetermined file format, etc.). "edit prohibited" is a voice command for preventing the generated still image data from being edited (for example, changing the content, etc.).

[0063] "Prohibit transcription" is a voice command for preventing at least a portion of the content of the generated still image data from being copied and used. "Set password (Yes / No)" is a voice command for specifying whether to set a password for the generated still image data. "Set OCR language" is a voice command for setting the language when documents containing documents in a language different from the other documents are mixed.

[0064] Furthermore, examples of voice commands for specifying the configuration of still image data when it is output include "insert blank page" and "delete page." Of these, "insert blank page" is a voice command for inserting a blank page into the data of multiple generated still images. Furthermore, "delete page" is a voice command for preventing still image data from being generated.

[0065] Although not shown, one voice command for dealing with a case where the voice command issued by the user is incorrect is a voice command for deleting the voice recorded by the user. For example, suppose that the voice command issued during the time period when the image capture target is the third and fourth pages of a spread is incorrect. In this case, the user who issued the incorrect voice command issues, in that order, a voice command preset as a voice command for canceling the voice command and a voice command for specifying the pages to be canceled during the time period when the image capture target is the third and fourth pages of a spread.

[0066] For example, if the user wants to cancel a voice command for page 3 (the left page), the user issues a voice command such as "command clear" or "left page." Also, if the user wants to cancel a voice command for page 4 (the right page), the user issues a voice command such as "command clear" or "right page." Also, if the user wants to cancel all voice commands for pages 3 and 4, the user issues a voice command such as "command clear" or "all pages."

[0067] Although the present embodiment has been described above, the present invention is not limited to the above-described embodiment. Furthermore, the effects of the present invention are not limited to those described in the above-described embodiment. For example, the configuration of the information processing system 1 shown in FIG. 1, the hardware configuration of the management server 10 shown in FIG. 2, and the hardware configuration of the user terminal 30 shown in FIG. 3 are merely examples for achieving the object of the present invention, and are not particularly limited.

[0068] Furthermore, the functional configuration of the management server 10 shown in Fig. 4 and the functional configuration of the user terminal 30 shown in Fig. 5 are merely examples and are not particularly limited. It is sufficient that the information processing system 1 in Fig. 1 is provided with a function that can execute the above-mentioned processes as a whole, and the functional configuration used to realize this function is not limited to the examples in Figs. 4 and 5.

[0069] 6 and 7 are merely examples and are not particularly limited. The steps are not necessarily performed in chronological order according to the illustrated order of steps, but may be performed in parallel or individually. Specific examples shown in FIGS. 8 to 12 are also merely examples and are not particularly limited.

[0070] Furthermore, while the voice commands in the above-described embodiments are meaningful words that humans can distinguish by themselves, they may be any voice commands that can be distinguished and recognized by an information processing device. Therefore, they can be replaced with other voices and set. For example, the voice command "Command Start" is long, so the word "Command" or the abbreviated word "Start" may be set. Furthermore, voice commands such as "Prohibit Output," "Prohibit Editing," and "Prohibit Transcription" may be replaced with symbols such as "Prohibit A," "Prohibit B," and "Prohibit C." By replacing voice commands with abbreviated words or symbols, the burden on the user issuing the voice commands can be reduced. [Explanation of symbols]

[0071] 1...information processing system, 10...management server, 11...control unit, 30...user terminal, 90...network, 101...information acquisition unit, 102...video and audio management unit, 103...audio detection unit, 104...image generation unit, 105...command management unit, 106...processing control unit, 107...transmission control unit, 301...display control unit, 302...imaging control unit, 303...sound recording control unit, 304...transmission control unit, 305...information acquisition unit

Claims

1. a processor; The processor: The moving image data of the captured multiple sheets of originals and the audio data recorded in parallel with the capturing of the images are associated with each other and managed on the same time axis, detecting a predetermined voice from the voice data; Among the still image data of the plurality of sheets of original document generated from the moving image data, control is performed to execute a predetermined process on the still image data of the original document that is the image capture target at the timing when the predetermined sound is emitted. Information processing device.

2. The data of the moving image is data of one moving image obtained by successively capturing images formed on each of the plurality of originals. The information processing device according to claim 1 .

3. The first moving image data is captured based on a first action and a second action of a user, and the audio data is a recording of an audio uttered by the user. The information processing device according to claim 2 .

4. the first operation is an operation of capturing an image of the plurality of originals, and the second operation is an operation of capturing an image of each of the plurality of originals as a subject, The information processing device according to claim 3 .

5. The operation for making each of the plurality of originals into a subject is an operation for turning over the plurality of originals one by one. The information processing device according to claim 4 .

6. The processor controls execution of a process corresponding to the predetermined voice as the control for executing the predetermined process. The information processing device according to claim 1 .

7. a plurality of the predetermined voices and a plurality of the predetermined processes exist, and the predetermined voices and the predetermined processes are stored in a predetermined database in association with each other; The information processing device according to claim 6 .

8. The predetermined voice is at least one of a voice indicating that the voice is the predetermined voice, a voice for specifying the document to be processed, a voice indicating the content of the processing, and a voice for deleting the recorded voice. The information processing device according to claim 7 .

9. The voice for specifying the document to be processed and the voice for deleting the recorded voice include a voice for specifying one document to be processed among a plurality of document pages in a double-page spread. The information processing device according to claim 8 .

10. The sound indicating the content of the processing is at least one of sounds for specifying an output format, an attribute, and a configuration of the still image data to be processed. The information processing device according to claim 8 .

11. The sound for specifying the output format of the still image data is a sound for specifying one or more of a file format, a color, and an orientation of the still image data. The information processing device according to claim 10.

12. The voice for specifying the attributes of the still image data is a voice for specifying one or more of whether the still image data can be printed, whether it can be edited, whether it can be transcribed, whether it is encrypted, and the language used for character recognition. The information processing device according to claim 10.

13. The voice for specifying the configuration of the still image data is a voice for specifying one or more of inserting a document and deleting a document in the still image data. The information processing device according to claim 10.

14. an imaging means for capturing moving images of a plurality of originals; a recording means for recording sound in parallel with the imaging; a management means for acquiring the captured moving image data and the recorded audio data, and managing the moving image data and the audio data in association with each other on the same time axis; a detection means for detecting a predetermined voice from the voice data; a generating means for generating still image data of the plurality of originals from the moving image data; a processing control means for controlling the execution of a predetermined process on the still image data of the originals that are the subject of imaging at the timing when the predetermined sound is emitted, among the generated still image data; characterized in that it has Information processing system.

15. On the computer, a function of associating the moving image data of the captured multiple sheets of originals with the audio data recorded in parallel with the image capture and managing them on the same time axis; a function of detecting a predetermined voice from the voice data; a function of controlling the execution of a predetermined process on the still image data of the document that is the image capture target at the timing when the predetermined sound is emitted, among the still image data of the plurality of documents generated from the moving image data; A program to achieve this.

Images