Apparatus system, device, terminal device, image generation method, and program

Abstract

To provide a technique capable of sharing the state of an operator operating a display device with a terminal device in another foothold.SOLUTION: An apparatus system includes: an input display device for displaying an input display area; an imaging apparatus for imaging an operator image including the input display device and a person inputting to the input display device; and an information processing system including an acquisition part for acquiring the operator image and a sharing part for allowing the input display device and a terminal device to share the input display area. The information processing system includes a transmission part for transmitting the operator image to the terminal device, to display it on the input display area.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to an equipment system, a device, a terminal apparatus, an image generating method, and a program. [Background technology]

[0002] Input display devices are known that display data handwritten on a touch panel using an input means such as a pen or a finger. Input display devices equipped with relatively large touch panels are placed indoors or outdoors and are used by multiple users as electronic whiteboards or the like.

[0003] There is known a technology for estimating the operator of an input display device using a camera mounted on an electronic whiteboard (see, for example, Patent Document 1). Patent Document 1 discloses a technology for estimating a user who is included in a first image but not included in a second image as the current touch panel operator. Summary of the Invention [Problem to be solved by the invention]

[0004] However, the conventional technology has a problem in that it is not possible to generate an image of a partial area from the surrounding image based on the range indicated by the plurality of specific images in response to a predetermined action of the user.

[0005] In view of the above-mentioned problems, the present invention aims to provide an equipment system that can generate an image of a partial area from a surrounding image based on the range indicated by multiple specific images in accordance with a specified action of the user. [Means for solving the problem]

[0006] In view of the above problems, the present invention provides a system comprising a photographing unit that photographs a surrounding image, a detection unit that detects a predetermined action of the user, an image generation unit that generates an image of a partial area from the surrounding image based on a specific image contained in the surrounding image in accordance with the predetermined action, and a display unit that displays the image of the partial area, wherein the image generation unit generates the image of the partial area based on the range indicated by a plurality of the specific images. [Effects of the Invention]

[0007] A technology can be provided that can generate an image of a partial area from a surrounding image based on the range indicated by the plurality of specific images in response to a predetermined action of the user. [Brief explanation of the drawings]

[0008] [Figure 1] 10A and 10B are diagrams illustrating an outline of creating record information that saves the screen of an application executed during a remote conference together with a panoramic image of the surrounding area. [Figure 2] 10 is a diagram illustrating an outline of a process in which a terminal device at another location displays an object displayed on an electronic whiteboard together with an operator image. FIG. [Figure 3] FIG. 1 is a diagram illustrating an example of the configuration of a record information creation system. [Figure 4] FIG. 2 is a diagram illustrating an example of the hardware configuration of an information processing system and a terminal device. [Figure 5] FIG. 2 is a diagram illustrating an example of the hardware configuration of a meeting device. [Figure 6] FIG. 10 is a diagram illustrating the imaging range of the meeting device. [Figure 7] 10A and 10B are diagrams illustrating the cutting out of a panoramic image and a speaker image. [Figure 8] FIG. 2 is a diagram illustrating a hardware configuration of an example of an electronic whiteboard. [Figure 9] 1 is an example of a functional block diagram illustrating functions of a terminal device, a meeting device, and an information processing system in a record information creation system, divided into blocks. [Figure 10]4 is a diagram showing an example of moving image recording information stored in an information storage unit. FIG. [Figure 11] FIG. 10 is a diagram illustrating an example of conference information managed by a communication management unit. [Figure 12] 10 is a diagram showing an example of association information stored in an association information storage unit, in which conference IDs are associated with device identification information; FIG. [Figure 13] FIG. 4 is a diagram illustrating an example of account information stored in an account information storage unit. [Figure 14] FIG. 1 is an example of a functional block diagram illustrating functions of an electronic whiteboard divided into blocks. [Figure 15] 4 is a diagram showing an example of device identification information stored in a device information storage unit; FIG. [Figure 16] FIG. 2 is a diagram illustrating object information stored in an object information storage unit. [Figure 17] 10 is an example of a sequence diagram illustrating a process in which a terminal device at another location displays handwritten strokes on an electronic whiteboard by a user together with an image of the operator. [Figure 18] 10A and 10B are diagrams illustrating examples of an object and an operator image displayed by a terminal device at another location. [Figure 19] FIG. 10 is a diagram illustrating an example of a conference registration screen displayed by the terminal device. [Figure 20] 10 is an example of a sequence diagram illustrating a process in which a user registers a meeting device and an electronic whiteboard in an information processing system. [Figure 21] 10 is an example of a sequence diagram illustrating a process in which a user registers a meeting device and an electronic whiteboard in an information processing system when no meeting information has been created. [Figure 22] 10 is an example of a sequence diagram illustrating a communication method between a terminal device and an electronic whiteboard when the terminal device and the electronic whiteboard are connected to the same network. FIG. [Figure 23] 10A and 10B are diagrams illustrating an operation method for setting the direction of the electronic whiteboard by pressing a position registration button. [Figure 24] FIG. 10 is a diagram showing a screen on which the user can confirm the direction set by the user. [Figure 25] 10 is an example of a sequence diagram illustrating a process in which the information recording application determines the direction of the electronic whiteboard based on specific video and audio. [Figure 26] FIG. 10 is a diagram showing an example of a two-dimensional code displayed as a specific image on an electronic whiteboard. [Figure 27] FIG. 10 is a diagram illustrating a method for determining a direction based on a two-dimensional code. [Figure 28] 10A and 10B are diagrams illustrating a method for determining the direction of an electronic whiteboard when the electronic whiteboard outputs a specific sound. [Figure 29] 10A and 10B are diagrams illustrating a direction for determining the orientation of an electronic whiteboard based on the electronic whiteboard detected by image processing such as machine learning. [Figure 30] FIG. 10 is a diagram showing an example of an initial screen displayed by an information recording application running on the terminal device after login. [Figure 31] FIG. 10 is a diagram showing an example of a recording setting screen displayed by an information recording application. [Figure 32] FIG. 10 is a diagram showing an example of a recording screen displayed by the information recording application during recording. [Figure 33] FIG. 10 is a diagram illustrating an example of a conference list screen displayed by the information recording application. [Figure 34] 10 is an example of a sequence diagram illustrating a process in which a meeting device starts and ends cutting out a device image in response to the start and end of an operation on an electronic whiteboard. [Figure 35] 10 is a sequence diagram illustrating an example of a process in which a terminal device at another location displays handwritten strokes on an electronic whiteboard by a user together with an operator image 7 (embodiment 2); [Figure 36] 10 is an example of a sequence diagram illustrating a process in which, when a user writes handwriting on an electronic whiteboard and then a different user writes handwriting, a terminal device at another location displays the strokes together with an operator image. [Figure 37] FIG. 10 is a diagram showing an example of an operator image displayed on a terminal device at another location when two users 1 and 2 are operating an interactive whiteboard. [Figure 38]10 is a flowchart illustrating an example of a process in which a terminal device at another location receives an operator image and displays the operator image. [Figure 39] 10 is a diagram illustrating an example of a mark indicating that an operator image is attached to a stroke displayed by a terminal device at another location. [Figure 40] FIG. 2 is a diagram illustrating an input display area. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, a device system and an information processing method performed by the device system will be described as an example of an embodiment of the present invention.

[0010] <An example of how to create minutes for a remote meeting> Information about the operators who operated the objects displayed on the electronic whiteboard was not shared. For example, if an information processing system shared object data displayed on an electronic whiteboard at one location with other locations in real time, users at other locations would be able to see who was writing what. Also, if the information processing system associated operators with objects, it would be clear who input the objects, improving the quality of meeting minutes.

[0011] First, an outline of a method for creating minutes using a panoramic image and an application screen will be described with reference to Fig. 1. Fig. 1 is a diagram for explaining an outline of creating record information that saves the screen of an application executed during a remote conference together with a panoramic image of the surrounding area. As shown in Fig. 1, a user at a home location 102 is using a remote conference service system 90 to hold a remote conference with another location 101.

[0012] The record information creation system 100 of this embodiment creates record information (such as minutes) using a horizontal panoramic image (hereinafter referred to as a panoramic image) acquired by a meeting device 60 equipped with an imaging means capable of capturing 360° images of the surroundings, a microphone, and a speaker, by processing information captured from the surroundings, and a screen created by an application (hereinafter referred to as an app) executed by the terminal device 10. Regarding audio, the record information creation system 100 synthesizes the audio received by the remote conference app 42 and the audio acquired by the meeting device 60, and includes the synthesized audio in the record information. An outline of this is given below.

[0013] (1) An information recording application 41 and a remote conference application, which will be described later, are running on the terminal device 10. In addition, an application for displaying materials may also be running. The information recording application 41 transmits the audio output by the terminal device 10 (including audio received by the remote conference application from other locations) to the meeting device 60. The meeting device 60 mixes (combines) the audio it has acquired with the audio from the remote conference application.

[0014] (2) The meeting device 60 is equipped with a microphone, and performs processing to extract the speaker from the panoramic image based on the direction from which the voice was captured, thereby creating a speaker image. The meeting device 60 transmits both the panoramic image and the speaker image to the terminal device 10.

[0015] (3) The information recording application 41 running on the terminal device 10 can display the panoramic image 203 and the speaker image 204. The information recording application 41 combines any application screen selected by the user (for example, the screen 103 of a teleconferencing application) with the panoramic image 203 and the speaker image 204. For example, the information recording application 41 combines the panoramic image 203, the speaker image 204, and the application screen 103 so that the panoramic image 203 and the speaker image 204 are arranged on the left and the teleconferencing application screen 103 is arranged on the right (hereinafter referred to as a combined image 105). The application screen is an example of screen information (described later) displayed by each application, such as a teleconferencing application. Since the process of (3) is executed repeatedly, the combined image 105 becomes a moving image (hereinafter referred to as a combined image moving image). The information recording application 41 also combines the synthesized audio with the combined image moving image to create a moving image with audio.

[0016] In this embodiment, an example in which the panoramic image 203, the speaker image 204, and the application screen 103 are combined will be described, but the information recording application 41 may store these separately and arrange them on the screen during playback.

[0017] (4) The information recording application 41 accepts editing operations (cutting out unnecessary parts by the user) and completes the combined image video. The combined image video constitutes part of the recorded information.

[0018] (5) The information recording application 41 transmits the created combined image video (with audio) to the storage service system 70 and stores it there.

[0019] (6) The information recording application 41 also extracts only the audio from the combined image video (the audio before combining may be kept), and transmits the extracted audio to the information processing system 50. The information processing system 50 transmits the audio to the speech recognition service system 80, which converts the audio into text data, and converts the audio into text. The text data also includes data on how many minutes after the start of recording that the speech occurred.

[0020] In the case of real-time text conversion, the meeting device 60 directly transmits the voice to the information processing system 50. The meeting device 60 transmits the text data to the information recording application 41 in real time.

[0021] (7) The information processing system 50 additionally stores the text data in the storage service system 70 that stores the combined image video. The text data constitutes part of the recorded information.

[0022] The information processing system 50 can perform billing processing for the user according to the service used. For example, the billing is calculated based on the amount of text data, the file size of the combined image video, the processing time, etc.

[0023] In this way, the combined image video displays a panoramic image of the surroundings including the user and an image of the speaker, and further displays the screen of an app that was displayed during the remote conference, such as the remote conference app 42. When participants in the remote conference or non-participants view the combined image video as minutes, the situation during the remote conference is reproduced with a sense of realism. [Example]

[0024] <Outline of the process of associating an operator image with an object and displaying it> Next, an outline of a process in which a terminal device 8 at another location associates an operator image with an object and displays it will be described with reference to Fig. 2. Fig. 2 is a diagram illustrating an outline of a process in which a terminal device 8 at another location (an example of a first terminal device) displays an object displayed by the electronic whiteboard 2 together with an operator image 7. In Fig. 2, at least the electronic whiteboard 2 and the terminal device 8 at the other location are participating in the same conference, and the terminal device 8 at the other location shares the object displayed by the electronic whiteboard 2 with the electronic whiteboard 2.

[0025] (1) The meeting device 60 repeatedly captures images of the surroundings and transmits a panoramic image and a speaker image to the terminal device 10 (an example of a second terminal device). It is assumed that the operator 121 appears in the panoramic image.

[0026] (2) When the electronic whiteboard 2 is operated, the electronic whiteboard 2 transmits a notification of the start of operation (a request to cut out the operator image 7) to the meeting device 60 via the terminal device 10. When the meeting device 60 is notified of the start of operation (a request to cut out the operator image 7), it cuts out the electronic whiteboard 2 from the panoramic image. The panoramic image may be captured in response to the start of operation. Since this is the timing when the electronic whiteboard 2 is operated, the operator is visible in the cut-out image. The cut-out image is called the operator image 7.

[0027] (3) The terminal device 10 transmits the operator image 7 to the interactive whiteboard 2.

[0028] (4) The electronic whiteboard 2 transmits the operator image 7 and stroke data to the information processing system 50, for example, for each stroke. If strokes are written continuously (if a pen-down for the next stroke is detected within a certain time period after a pen-up), the electronic whiteboard 2 does not need to transmit the operator image 7.

[0029] (5) The information processing system 50 assigns an object ID to the stroke data, assigns identification information of the operator image 7 to the operator image 7, and associates the stroke data with the identification information of the operator image 7. The information processing system 50 also transmits the stroke data and the identification information of the operator image 7 to a terminal device 8 at another location. The information processing system 50 may transmit the stroke data and the operator image 7 itself to a terminal device 8 at another location.

[0030] (6) The terminal device 8 at the other site receives the stroke data and the identification information of the operator image 7, and requests the operator image 7 from the information processing system 50 by specifying the identification information of the operator image 7.

[0031] (7) In response to this request, the information processing system 50 transmits the operator image 7 associated with the identification information of the operator image 7 to the terminal device 8 at the other location.

[0032] (8) When the terminal device 8 at the other site receives the operator image 7, it displays the operator image 7 in association with the stroke 120.

[0033] As shown in Fig. 2, an operator image 7 is displayed near the stroke at other locations, allowing users at other locations to easily understand who is writing (or has written) the stroke. Also, in the information processing system 50, the operator image 7 is associated with the object data, making it clear who input the object, improving the content of the minutes.

[0034] <Terminology> The operator image is an image that includes an operator operating the interactive whiteboard 2. Since the interactive whiteboard 2 may be included along with the operator, the operator image may also be called a device image.

[0035] A stroke is a series of actions in which a user presses an input tool against a display, moves it continuously, and then releases it from the display. A stroke can include tracking a user's movement without touching the display. In this case, the display device may initiate a stroke by a user gesture, such as using a mouse or pointing device, pressing a button with the user's hand or foot, or in other ways. The user may then terminate the stroke with the same or a different gesture, releasing a button, or using the mouse or pointing device.

[0036] Stroke data is information displayed on a display based on a trajectory of coordinates input by an input means. Stroke data may be interpolated as appropriate. Handwriting data is data having one or more strokes. Handwriting input refers to handwriting data being input by a user. Handwriting input may be performed using a touch interface, a tactile object such as a pen or stylus, or the user's body. Handwriting input may also be performed via other types of input, such as gesture-based input, hand tracking input, or other touch-free input by a user. Although embodiments of the present invention refer to handwriting input and handwriting input data, other forms of handwriting input may be used.

[0037] An object is a display object that is displayed on a display as a result of stroke data. Object means a target, but in this embodiment, it means a display target. Objects converted from handwritten stroke data may include text, as well as stamps displayed as fixed characters such as "Done," shapes such as circles and stars, and straight lines. Text is primarily a character string (character code) that includes one or more characters, and also includes numbers and symbols. Text may also be referred to as a character string.

[0038] The input display device may be any device that displays information. In this embodiment, the input display device is described using the term "electronic whiteboard." The electronic whiteboard may also be called an electronic whiteboard, an electronic information board, or the like. A projector is known as a device equivalent to the electronic whiteboard 2. Alternatively, the input display device may be a digital signage, a television, a display, a video conference terminal, or the like.

[0039] An application (app) is software developed and used for a specific function or purpose. Applications include native apps and web apps. Web apps (cloud-side apps provided by cloud services) can also operate in conjunction with native apps or web browsers.

[0040] A running app is an app that is in a state between when it is launched and when it is closed. The app does not have to be the active app (the foreground app), it just needs to be running in the background.

[0041] Recorded information is information recorded by the information recording application 41, and is information that is stored and saved so that it can be viewed as information linked to the identification information of a particular conference (meeting), and includes, for example, the following information: Video information created based on the screen information displayed by the selected app (such as a remote conferencing app) and image information about the device's surroundings acquired by the device. During a meeting, audio information is acquired and synthesized between the remote conference app (terminal device) and the meeting device at the base. -Text information converted from the captured audio. Other data and images related to the meeting, such as document files used during the meeting, notes added, translations of text data, and images and stroke data created using the cloud electronic whiteboard service during the meeting. For example, when the information recording application 41 records the screen of a remote conference application or the state of a conference at a base, the recorded information may be minutes of the conference that was held. Minutes are an example of recorded information, and the name of the recorded information changes depending on the content of the remote conference or the base, and may be called, for example, a record of communication or a record of the base situation. Furthermore, the recorded information includes files in multiple formats, such as video files (e.g., combined image videos), audio files, text data (text data obtained by voice recognition), document files, image files, and table-format files, and since the files are associated with each other based on the identification information of the conference, they can be viewed collectively or selectively in chronological order.

[0042] A tenant is a group of users (such as a company, a local government, or some of these organizations) who have signed a contract to receive services from a service provider. In this embodiment, the creation of recorded information and conversion to text data are performed because the tenant has signed a contract with the service provider.

[0043] Remote communication refers to communication through audio and video using software and terminal devices with people at physically distant locations. An example of remote communication is a remote conference, which may also be called a meeting, a conference, a consultation, a contract application, a gathering, a get-together, a seminar, a training session, a study group, a seminar, or a workshop.

[0044] A base is a place where various activities are carried out using various network-connected devices. An example of a base is a conference room. A conference room is a room set up primarily for meetings. A base can also be a home, a reception desk, a store, a warehouse, an outdoor site, or any other location or space where terminal devices, etc. can be installed.

[0045] Although voice refers to speech sounds uttered by humans and surrounding sounds, and voice data refers to voice converted into digital form, in this embodiment, a strict distinction will not be made between them. Also, another location refers to a different location away from the home location, such as a different building or a different conference room.

[0046] <System configuration example> Next, the system configuration of the record information creation system 100 will be described with reference to Fig. 3. Fig. 3 shows an example of the configuration of the record information creation system 100. Fig. 3 shows one location (own location 102) among multiple locations where a remote conference is held, and a terminal device 10 at the own location 102 communicates with an information processing system 50, a storage service system 70, and a remote conference service system 90 via a network. A meeting device 60 and an electronic whiteboard 2 are also placed at the own location 102, and the terminal device 10 is connected to the meeting device 60 so as to be able to communicate with it via a USB cable or the like. The terminal device 10 may also communicate with the meeting device 60 via a LAN.

[0047] At least an information recording application 41 and a remote conference application 42 run on the terminal device 10. The remote conference application 42 can communicate with terminal devices 10 at other locations 101 via a remote conference service system 90 on the network, allowing users at each location to hold conferences from remote locations. The information recording application 41 creates record information for the remote conferences held by the remote conference application 42 using the functions of the information processing system 50 and the meeting device 60.

[0048] In this embodiment, an example of creating recorded information during a remote conference will be described, but the conference does not have to be a conference involving communication with a remote location. In other words, the conference may be a conference in which only participants from one location participate. In this case, only the audio collected by the meeting device 60 is saved without synthesis, and the processing of the information recording application 41 remains unchanged.

[0049] The terminal device 10 has a built-in camera with a normal angle of view (or it may be external), which captures a front image including the user 107 operating the terminal device 10. A normal angle of view is an image that is not a panoramic image, and in this embodiment, it is a planar image that is not curved, such as a spherical image. The terminal device 10 also has a built-in microphone (or it may be external), which collects sounds from around the user operating the terminal device 10. Therefore, the user can hold a conventional remote conference using the remote conference app 42 without being aware of the information recording app 41. The information recording app 41 and the meeting device 60 do not affect the remote conference app 42, except for increasing the processing load on the terminal device 10.

[0050] The information recording application 41 is an application that records information by communicating with the meeting device 60 and creating recorded information. The meeting device 60 is a meeting device equipped with a panoramic image capturing device, a microphone, and a speaker. While the camera of the terminal device 10 can only capture a limited range in front of it, the meeting device 60 can capture images of the entire surroundings (not necessarily the entire periphery) of the meeting device 60. The meeting device 60 can always fit multiple participants 106 shown in FIG. 3 within its field of view.

[0051] In addition, the meeting device 60 extracts a speaker image from a panoramic image, synthesizes the audio acquired by the meeting device 60 with the audio output by the terminal device 10 (including the audio received by the remote conference application 42), and so on. The meeting device 60 is not limited to being placed on a desk, a table, or the like, and may be placed anywhere at the home base 102. Since the meeting device 60 can capture spherical images, it may be placed on the ceiling, for example. The meeting device 60 may also be installed at another base or at any base.

[0052] The information recording application 41 displays a list of applications running on the terminal device 10, synthesizes images for the recorded information (creates a combined image video), plays back the combined image video, and accepts editing requests. The information recording application 41 also displays a list of teleconferences that have been held or are scheduled to be held. The list of teleconferences is used for information related to the recorded information, allowing the user to associate the teleconferences with the recorded information.

[0053] The remote conference application 42 is an application that enables a terminal device to remotely communicate with other terminal devices by establishing a communication connection with other terminal devices at other locations 101, transmitting and receiving images and audio, displaying images and outputting audio, etc. The remote conference application can also be called a remote communication application, a remote information common application, etc.

[0054] The information recording application 41 and the remote conference application 42 may be web applications or native applications. A web application is an application that performs processing through collaboration between a program on a web server and a program on a web browser or a native application, and does not need to be installed on the terminal device 10. A native application is an application that is installed on the terminal device 10 and used. In this embodiment, both applications will be described as native applications.

[0055] The terminal device 10 may be, for example, a general-purpose information processing device with a communication function, such as a PC (Personal Computer), a smartphone, or a tablet terminal. The terminal device 10 may also be an electronic whiteboard 2, a game console, a PDA (Personal Digital Assistant), a wearable PC, a car navigation system, industrial machinery, medical equipment, or network home appliances. The terminal device 10 may be any device on which at least the information recording application 41 and the remote conference application 42 run. The terminal device 10 may be any device on which the information recording application 41 and the remote conference application 42 run.

[0056] The electronic whiteboard 2 displays data handwritten on a touch panel using an input means such as a pen or a finger. The electronic whiteboard 2 can communicate with a terminal device 10 or the like via wired or wireless communication, and can capture screens displayed by the terminal device 10 and display them on the display. The electronic whiteboard 2 can convert handwritten data into text data and share information displayed on the display with electronic whiteboards 2 at other locations. The electronic whiteboard 2 may be a simple whiteboard without a touch panel, in which an image is projected by a projector. The electronic whiteboard 2 may also be a tablet terminal, notebook PC (Personal Computer), PDA (Personal Digital Assistant), game console, or the like, equipped with a touch panel.

[0057] The interactive whiteboard 2 can communicate with the information processing system 50. For example, after being powered on, the interactive whiteboard 2 can receive information from the information processing system 50 by polling the information processing system 50. The interactive whiteboard 2 and the meeting device 60 that can capture an image of the interactive whiteboard 2 operate as a display system.

[0058] The information processing system 50 is one or more information processing devices arranged on a network. The information processing system 50 has one or more server applications that perform processing in cooperation with the information recording application 41, and a platform service. The server applications manage remote conference lists, manage recorded information recorded in remote conferences, and manage various settings and storage paths. The platform service performs user authentication, contracts, billing, and other processing. The meeting device 60, the electronic whiteboard 2, and the information processing system 50 function as an equipment system.

[0059] All or part of the functions of the information processing system 50 may exist in a cloud environment or an on-premise environment. The information processing system 50 may be configured with multiple server devices or may be configured with a single information processing device. For example, the server application and the infrastructure service may be provided by separate information processing devices, and each function within the server application may have its own information processing device. The information processing system 50 may be integrated with the storage service system 70 and the voice recognition service system 80 described below.

[0060] The storage service system 70 is a storage means on a network and provides a storage service that accepts the saving of files, etc. Known examples of the storage service system 70 include OneDrive (registered trademark), Google Workspace (registered trademark), and DropBox (registered trademark). The storage service system 70 may also be an on-premise NAS (Network Attached Storage), etc.

[0061] The voice recognition service system 80 provides a service that performs voice recognition on voice data and converts it into text data. The voice recognition service system 80 may be a general-purpose commercial service, or may be part of the functions of the information processing system 50. Furthermore, the voice recognition service system 80 may be set up and used as a different service system for each user, tenant, or conference.

[0062] <Hardware configuration example> The hardware configuration of the information processing system 50 and the terminal device 10 according to this embodiment will be described with reference to FIG.

[0063] <<Information processing system and terminal device>> Fig. 4 is a diagram showing an example of the hardware configuration of an information processing system 50 and a terminal device 10 according to this embodiment. As shown in Fig. 4, the information processing system 50 and the terminal device 10 are constructed by a computer, and include a CPU 501, a ROM 502, a RAM 503, a HD (Hard Disk) 504, an HDD (Hard Disk Drive) controller 505, a display 506, an external device connection I / F (Interface) 508, a network I / F 509, a bus line 510, a keyboard 511, a pointing device 512, an optical drive 514, and a media I / F 516.

[0064] Of these, the CPU 501 controls the overall operation of the information processing system 50 and the terminal device 10. The ROM 502 stores programs, such as an IPL, used to drive the CPU 501. The RAM 503 is used as a work area for the CPU 501. The HD 504 stores various data, such as programs. The HDD controller 505 controls the reading and writing of various data from and to the HD 504 under the control of the CPU 501. The display 506 displays various information, such as a cursor, menus, windows, characters, or images. The external device connection I / F 508 is an interface for connecting various external devices. In this case, external devices include, for example, a USB (Universal Serial Bus) memory or a printer. The network I / F 509 is an interface for data communication using a network. The bus line 510 is an address bus, a data bus, or the like, for electrically connecting the components, such as the CPU 501, shown in FIG. 4.

[0065] The keyboard 511 is a type of input means having multiple keys used to input characters, numbers, various instructions, etc. The pointing device 512 is a type of input means for selecting and executing various instructions, selecting a processing target, moving a cursor, etc. The optical drive 514 controls reading and writing of various data from an optical storage medium 513, which is an example of a removable storage medium. The optical storage medium 513 may be a CD, a DVD, Blu-ray (registered trademark), etc. The media I / F 516 controls reading and writing (storing) of data from a storage medium 515, such as a flash memory.

[0066] <<Meeting Devices>> The hardware configuration of the meeting device 60 will be described using Figure 5. Figure 5 is an example of a hardware configuration diagram of a meeting device 60 capable of capturing 360-degree video. In the following, the meeting device 60 is described as a device that uses an imaging element to capture 360-degree video around the device at a predetermined height, but the imaging element may be one, two, or any number of elements. Furthermore, it does not necessarily have to be a dedicated device; a PC, digital camera, smartphone, etc. may have substantially the same functionality by attaching a 360-degree video imaging unit to it after the fact.

[0067] As shown in Figure 5, the meeting device 60 is composed of an imaging unit 601, an image processing unit 604, an imaging control unit 605, a microphone 608, a sound processing unit 609, a CPU (Central Processing Unit) 611, a ROM (Read Only Memory) 612, an SRAM (Static Random Access Memory) 613, a DRAM (Dynamic Random Access Memory) 614, an operation unit 615, an external device connection I / F 616, a communication unit 617, an antenna 617a, an audio sensor 618, and a concave terminal 621 for Micro USB.

[0068] Of these, the imaging unit 601 includes a wide-angle lens (so-called fisheye lens) 602 with a 360° angle of view for forming a hemispherical image, and imaging elements 603 (image sensors) provided corresponding to each wide-angle lens. The imaging elements 603 include an image sensor such as a CMOS (Complementary Metal Oxide Semiconductor) sensor or a CCD (Charge Coupled Device) sensor that converts the optical image captured by the fisheye lens 602 into image data in the form of an electrical signal and outputs the image data, a timing generation circuit that generates horizontal or vertical synchronization signals and pixel clocks for the image sensor, and a group of registers that set various commands and parameters required for the operation of the imaging elements. The imaging unit 601 may also be a 360° camera, which is an example of imaging means capable of capturing images of the 360° surrounding the meeting device 60.

[0069] The imaging elements 603 (image sensors) of the imaging unit 601 are each connected to the image processing unit 604 via a parallel I / F bus. Meanwhile, the imaging elements 603 of the imaging unit 601 are connected to the imaging control unit 605 via a serial I / F bus (such as an I2C bus). The image processing unit 604, the imaging control unit 605, and the sound processing unit 609 are connected to a CPU 611 via a bus 610. Furthermore, a ROM 612, an SRAM 613, a DRAM 614, an operation unit 615, an external device connection I / F 616, a communication unit 617, an audio sensor 618, and the like are also connected to the bus 610.

[0070] The image processing unit 604 takes in the image data output from the image sensor 603 via a parallel I / F bus, performs predetermined processing on each image data, and creates data for a panoramic image and a speaker image from the fisheye video. Furthermore, the image processing unit 604 synthesizes the panoramic image and the speaker image, etc., and outputs a single video.

[0071] The imaging control unit 605 generally uses the I2C bus to set commands and the like in the registers of the imaging element 603, with the imaging control unit 605 acting as a master device and the imaging element 603 acting as a slave device. Necessary commands and the like are received from the CPU 611. The imaging control unit 605 also uses the I2C bus to retrieve status data and the like from the registers of the imaging element 603 and send it to the CPU 611.

[0072] Furthermore, the imaging control unit 605 instructs the imaging elements 603a and 603b to output image data when the imaging start button on the operation unit 615 is pressed or when an imaging start instruction is received from the PC. Some meeting devices 60 have a preview display function or a function that supports video display on a display (for example, a PC or smartphone display). In this case, the image data is output from the imaging element 603 continuously at a predetermined frame rate (frames per minute).

[0073] As will be described later, the imaging control unit 605 also functions as a synchronization control means that cooperates with the CPU 611 to synchronize the output timing of image data from the imaging element 603. In this embodiment, the meeting device 60 is not provided with a display, but a display unit may be provided.

[0074] The microphone 608 converts sound into sound (signal) data. The sound processing unit 609 takes in the sound data output from the microphone 608 via the I / F bus and performs predetermined processing on the sound data.

[0075] The CPU 611 controls the overall operation of the meeting device 60 and executes necessary processing. The ROM 612 stores various programs for the CPU 611. The SRAM 613 and DRAM 614 are work memories that store programs executed by the CPU 611, data in the middle of processing, etc. In particular, the DRAM 614 stores image data in the middle of processing by the image processing unit 604 and data of processed equirectangular projection images.

[0076] The operation unit 615 is a general term for operation buttons such as an imaging start button 615a. By operating the operation unit 615, the user can start imaging or recording, as well as turn the power on / off, establish communication connections, and input settings such as various imaging modes and imaging conditions.

[0077] The external device connection I / F 616 is an interface for connecting various external devices. In this case, the external device may be, for example, a PC (Personal Computer), a display, a projector, an electronic whiteboard, etc. The external device connection I / F 616 may include, for example, a USB terminal, an HDMI (registered trademark) terminal, etc. The video data and image data stored in the DRAM 614 are transmitted to an external terminal via the external device connection I / F 616 or recorded on external media. Furthermore, by using multiple external device connection I / Fs 616, for example, image information captured by the meeting device 60 may be transmitted to a PC via USB for recording, while video (e.g., screen information displayed by a remote conference app) may be acquired from the PC to the meeting device 60, and the video may then be transmitted from the meeting device 60 via HDMI to another external device (e.g., a display, a projector, an electronic whiteboard) for display.

[0078] The communication unit 617 may communicate with a cloud server via the Internet using a wireless communication technology such as Wi-Fi via an antenna 617a provided in the meeting device 60, and transmit the stored video data and image data to the cloud server. The communication unit 617 may also communicate with nearby devices using a short-range wireless communication technology such as BLE (Bluetooth Low Energy, registered trademark) or NFC.

[0079] The audio sensor 618 is a sensor that acquires 360° audio information to identify from which direction a loud audio is being input within 360° around (horizontal plane) the meeting device 60. The audio processing unit 609 identifies the strongest direction based on the input 360° audio parameters and outputs the audio input direction within 360°.

[0080] Note that other sensors (such as a direction / acceleration sensor or GPS) may calculate direction, position, angle, acceleration, etc., and use this information for image correction or adding location information.

[0081] The image processing unit 604 also performs the following processes.

[0082] The CPU 611 creates a panoramic image in the following way: The CPU 611 performs predetermined camera image processing such as Bayer conversion (RGB interpolation processing) on ​​the RAW data input from the image sensor that inputs spherical images to create a fisheye image (image made up of curved images).The CPU 611 then performs flattening processing such as DeWarp processing (distortion correction processing) on ​​the created fisheye image (curved image) to create a panoramic image (image made up of flat images) that captures 360 degrees around the meeting device 60.

[0083] The CPU 611 creates speaker images in the following way. The CPU 611 creates speaker images by cutting out the speaker from a panoramic image (flat video) that captures 360° of the surrounding area. The CPU 611 cuts out the speaker image from the panoramic image, using the voice input direction identified from the 360° output using the audio sensor 618 and the sound processing unit 609 as the speaker's direction. In this case, the method for cutting out a person's image from the voice input direction is to cut out a 30° area centered on the identified voice direction from the 360°, and perform face detection within that area to cut out the image. The CPU 611 further identifies the speaker images of a specific number of people (e.g., three people) who have spoken most recently from the cut-out speaker images.

[0084] The panoramic image and the one or more speaker images may be sent separately to the information recording application 41, or the meeting device 60 may create a single image from them and send it to the information recording application 41. In this embodiment, the panoramic image and the one or more speaker images are sent separately from the meeting device 60 to the information recording application 41.

[0085] Figure 6 is a diagram illustrating the imaging range of the meeting device 60. As shown in Figure 6(a), the meeting device 60 captures an image in a 360° range in the horizontal direction. As shown in Figure 6(b), the imaging range of the meeting device 60 is 0° in the horizontal direction at the height of the meeting device 60, and a predetermined angle above and below.

[0086] FIG. 7 is a diagram illustrating the cutting out of a panoramic image and a speaker image. As shown in FIG. 7, the image captured by the meeting device 60 forms a portion 110 of a sphere and therefore has a three-dimensional shape. As shown in FIG. 6(b), the meeting device 60 performs perspective projection transformation by dividing the angle of view into predetermined angles in the vertical direction and the horizontal direction. By performing perspective projection transformation across the entire 360° horizontal direction without gaps, a predetermined number of planar images are obtained, and a panoramic image 111 is obtained by connecting the predetermined number of planar images horizontally. In addition, the meeting device 60 performs face detection within a predetermined range from the panoramic image centered on the direction of the voice, and creates a speaker image 112 by cutting out an image 15° left and right from the center of the face (30° overall).

[0087] <<Electronic Whiteboard>> Fig. 8 is a hardware configuration diagram of the electronic whiteboard 2. As shown in Fig. 8, the electronic whiteboard 2 includes a CPU (Central Processing Unit) 401, a ROM (Read Only Memory) 402, a RAM (Random Access Memory) 403, an SSD (Solid State Drive) 404, a network I / F 405, and an external device connection I / F (Interface) 406.

[0088] Of these, the CPU 401 controls the overall operation of the electronic whiteboard 2. The ROM 402 stores the CPU 401 and programs used to drive the CPU 401, such as an IPL (Initial Program Loader). The RAM 403 is used as a work area for the CPU 401. The SSD 404 stores various data, such as programs for the electronic whiteboard 2. The network I / F 405 controls communication with a communication network. The external device connection I / F 406 is an interface for connecting various external devices. In this case, the external devices are, for example, a USB (Universal Serial Bus) memory 430 and external devices (a microphone 440, a speaker 450, and a camera 460).

[0089] The electronic whiteboard 2 also includes a capture device 411, a GPU 412, a display controller 413, a contact sensor 414, a sensor controller 415, an electronic pen controller 416, a short-range communication circuit 419, an antenna 419a of the short-range communication circuit 419, a power switch 422, and selection switches 423.

[0090] Of these, the capture device 411 displays video information as still images or moving images on the display of an external PC (Personal Computer) 470. The GPU (Graphics Processing Unit) 412 is a semiconductor chip that specializes in graphics. The display controller 413 controls and manages the screen display to output the output image from the GPU 412 to the display 480, etc. The contact sensor 414 detects contact of the electronic pen 490, the user's hand H, etc. on the display 480. The sensor controller 415 controls the processing of the contact sensor 414. The contact sensor 414 inputs and detects coordinates using an infrared blocking method. The method for inputting and detecting coordinates will be described below. Two light receiving and emitting devices installed at both ends of the upper side of the display 480 emit multiple infrared rays parallel to the display 480. The light is reflected by a reflecting member installed around the periphery of the display 480, and the two light receiving and emitting devices receive the light returning along the same optical path as the light emitted by the light receiving element.

[0091] The contact sensor 414 outputs the ID of the infrared light emitted by the two light receiving and emitting devices that has been blocked by an object to the sensor controller 415, and the sensor controller 415 identifies the coordinate position of the contact position of the object. The electronic pen controller 416 communicates with the electronic pen 490 to determine whether the pen tip or the pen tail has touched the display 480. The short-range communication circuit 419 is a communication circuit such as NFC (Near Field Communication) or Bluetooth (registered trademark). The power switch 422 is a switch for switching the power of the electronic whiteboard 2 on and off. The selection switches 423 are a group of switches for adjusting, for example, the brightness and color of the display 480.

[0092] Furthermore, the electronic whiteboard 2 includes a bus line 410. The bus line 410 is an address bus, a data bus, or the like for electrically connecting the components such as the CPU 401 shown in FIG.

[0093] The contact sensor 414 is not limited to an infrared blocking type, and may be a capacitive touch panel that identifies the contact position by detecting changes in capacitance. The contact sensor 414 may be a resistive film type touch panel that identifies the contact position by voltage changes across two opposing resistive films. The contact sensor 414 may be an electromagnetic induction type touch panel that identifies the contact position by detecting electromagnetic induction caused by an object touching the display unit. The contact sensor 414 may use various other detection means. Furthermore, the electronic pen controller 416 may determine whether or not the electronic pen 490 is touched not only by the pen tip and pen tail, but also by the part of the electronic pen 490 that the user holds or other parts of the electronic pen.

[0094] <About the function> Next, the functional configuration of the record information creation system 100 will be described with reference to Fig. 9. Fig. 9 is an example of a functional block diagram that explains the functions of the terminal device 10, the meeting device 60, and the information processing system 50 in the record information creation system 100 by dividing them into blocks.

[0095] <<Terminal Device>> The information recording application 41 running on the terminal device 10 includes a communication unit 11, an operation reception unit 12, a display control unit 13, an application screen acquisition unit 14, an audio acquisition unit 15, a device communication unit 16, a recording control unit 17, an audio data processing unit 18, a recording / playback unit 19, an upload unit 20, an edit processing unit 21, a device recognition unit 22, an analysis unit 23, an operator image transmission unit 24, and a cutout request unit 25. Each of these units included in the terminal device 10 is a function or a means for performing the function, which is realized when any of the components shown in FIG. 4 operates in response to an instruction from the CPU 501 in accordance with the information recording application 41 loaded from the HD 504 to the RAM 503. The terminal device 10 also includes a storage unit 1000 implemented by the HD 504 and the like shown in FIG. 4. The storage unit 1000 includes an information storage unit 1001.

[0096] The communication unit 11 communicates various types of information with the information processing system 50 via the network. For example, the communication unit 11 receives a list of remote conferences from the information processing system 50 and transmits a request for recognizing voice data to the information processing system 50. The communication unit 11 also receives an operation start and an operation end from the interactive whiteboard 2.

[0097] The display control unit 13 displays various screens that serve as a user interface in the information recording application 41 in accordance with screen transitions set in the information recording application 41. The operation accepting unit 12 accepts various operations for the information recording application 41.

[0098] The application screen acquisition unit 14 acquires screen information displayed by an application selected by the user or screen information of the desktop screen from an operating system (OS) or the like. If the application selected by the user is a remote conference application 42, screens generated by the remote conference application 42 (images captured by the terminal device cameras at each location, images displayed with shared materials, participant icons, participant names, etc.) are acquired. The screen information displayed by the application (application screen) is information displayed as a window by the running application and acquired as an image by the information recording application. The application window is displayed on a monitor or the like, with the window area depicted as an area within the entire desktop image. The screen information displayed by the application can be acquired by other applications (such as an information recording application) as an image file or a video file consisting of multiple consecutive images via the API of the operating system (OS) or the API of the application that displays it. The screen information of the desktop screen is information composed of an image of the desktop screen generated by the OS and can be similarly acquired as an image file or a video file via the API of the OS. These image files may be in bitmap, PNG, or other formats. The video file format may be MP4 or other formats.

[0099] The audio acquisition unit 15 acquires audio output from the microphone or earphones of the terminal device 10 (including audio data received from the remote conference application 42 during the remote conference). The audio acquisition unit 15 can acquire audio even when the output audio is muted. Regarding audio data, the user does not need to perform any operation, such as selecting the remote conference application 42. The audio acquisition unit 15 can acquire audio that the terminal device 10 can output via the OS or application API (Application Interface). This also acquires audio data received by the remote conference application 42 from other locations 101. If the remote conference application 42 is not running or if a remote conference is not in progress, the information recording application 41 may not be able to acquire audio data. Note that the audio acquired by the audio acquisition unit 15 may not include audio collected by the terminal device 10, and may only include the audio data to be output. This is because the meeting device 60 separately collects audio from the locations.

[0100] The device communication unit 16 communicates with the meeting device 60 using a USB cable or the like. The device communication unit 16 may communicate with the meeting device 60 via wireless LAN, Bluetooth (registered trademark), or the like. The device communication unit 16 receives the panoramic image and the speaker image from the meeting device 60, and transmits the audio data acquired by the audio acquisition unit 15 to the meeting device 60. The device communication unit 16 receives the audio data synthesized by the meeting device 60.

[0101] The recording control unit 17 combines the panoramic image and speaker image received by the device communication unit 16 and the app screen acquired by the app screen acquisition unit 14 to create a combined image. The recording control unit 17 also chronologically connects the repeatedly created combined images to create a combined image video, and combines the synthesized audio data with the combined image video to create a combined image video with audio. The combination of the panoramic image and speaker image may be performed by the meeting device 60. Also, videos consisting of each image, such as the panoramic image, speaker image, app screen, and image consisting of the panoramic image and speaker image, may be stored in the storage service system 70 as separate video files. In this case, the panoramic video, speaker video, video of the app screen, and combined video of the panoramic image and speaker image may be called up and displayed on a single display screen when viewed.

[0102] The audio data processing unit 18 extracts the audio data combined with the combined image video, or requests the information processing system 50 to convert the synthesized audio data received from the meeting device 60 into text data.

[0103] The recording / playback unit 19 plays back the combined image moving image. The combined image moving image is stored in the terminal device 10 while being recorded, and is then uploaded to the information processing system 50.

[0104] When the remote conference ends, the upload unit 20 transmits the combined image moving image to the information processing system 50.

[0105] The editing processing unit 21 executes editing of the combined image moving image (deleting a part, splicing, etc.) in response to a user operation.

[0106] The device recognition unit 22 learns the shape (circumscribing rectangle) of the electronic whiteboard 2 through machine learning and detects the electronic whiteboard 2 from the panoramic image. The device recognition unit 22 may perform recognition simply through pattern matching without using machine learning. The processing performed by the device recognition unit 22 may also be performed by the meeting device 60.

[0107] The analysis unit 23 detects the two-dimensional code included in the panoramic image and analyzes the two-dimensional code to obtain information included in the two-dimensional code, such as device identification information of the interactive whiteboard 2. The processing performed by the analysis unit 23 may be performed by the meeting device 60.

[0108] The operator image transmitting unit 24 transmits the operator image 7 transmitted from the meeting device 60 to the information processing system 50. The information processing system 50 transmits the operator image 7 to terminal devices 8 at other locations participating in the same conference. The processing performed by the operator image transmitting unit 24 may be performed by the meeting device 60.

[0109] When the cut-out request unit 25 receives a start of operation from the electronic whiteboard 2, it requests the meeting device 60 to cut out the operator image 7. When the cut-out request unit 25 receives an end of operation from the electronic whiteboard 2, it requests the meeting device 60 to finish cutting out the operator image 7.

[0110] Fig. 10 shows video recording information stored in the information storage unit 1001. The video recording information includes items such as a conference ID, a recording ID, an update date and time, a title, an upload, and a save destination. When a user logs in to the information processing system 50, the information recording application 41 downloads the conference information from the conference information storage unit 5001 of the information processing system 50. The conference ID and other information included in the conference information are reflected in the video recording information. The video recording information in Fig. 10 is held by a terminal device 10 operated by a certain user.

[0111] The conference ID is identification information for identifying the remote conference that has been held. The conference ID is assigned when a remote conference schedule is registered in the conference management system 9, or is assigned by the information processing system 50 in response to a request from the information recording application 41. The conference management system 9 is a system for registering conference and remote conference schedules, a URL (conference link) for starting a remote conference, reservation information for devices to be used in the conference, and the like, and is a scheduler or the like connected from the terminal device 10 via a network. The conference management system 9 can also transmit the registered schedule, etc. to the information processing system 50.

[0112] The recording ID is identification information for identifying the combined image video recorded during a remote conference. The recording ID is assigned by the meeting device 60, but may also be assigned by the information recording application 41 or the information processing system 50. A different recording ID is assigned to the same conference ID when recording stopped during the remote conference but was resumed for some reason.

[0113] The update date and time is the date and time when the combined image video was updated (recording ended). If the combined image video was edited, this is the edit date and time.

[0114] The title is the name of the conference. It may be set when the conference is registered in the conference management system 9, or may be set arbitrarily by the user.

[0115] The "upload" indicates whether the combined image video has been uploaded to the information processing system 50.

[0116] The destination indicates the location (URL or file path) where the combined image video and text data are saved in the storage service system 70. Therefore, the user can view the uploaded combined image video at will. The combined image video and text data are saved with separate file names, for example, following the URL.

[0117] Terminal devices at other locations 8 Regarding the terminal device 8 at the other location, a function of sharing objects displayed by the electronic whiteboard 2 will be described. The terminal device 8 at the other location may have the same functions as the terminal device 10 at its own location in addition to those shown in the figure. The terminal device 8 at the other location has a communication unit 71, an operation reception unit 72, a display control unit 73, and an operator image acquisition unit 74. Each of these units possessed by the terminal device 8 at the other location is a function or a means for performing the function that is realized when any of the components shown in FIG. 4 operates in response to an instruction from the CPU 501 in accordance with a program (an application for the electronic whiteboard 2 or a web browser) deployed from the HD 504 to the RAM 503.

[0118] The communication unit 71 communicates with the information processing system 50 to receive various information including object data displayed on the electronic whiteboard 2 and the operator image 7. The communication unit 71 also transmits object data handwritten on a terminal device 8 at another location to the information processing system 50.

[0119] The operation receiving unit 72 receives various operations for the terminal devices 8 at other locations. For example, the operation receiving unit 72 may receive input of strokes on the electronic whiteboard 2.

[0120] The display control unit 73 displays the object data and handwritten strokes received by the communication unit 71 on the display of the terminal device 8 at the other site.

[0121] When identification information of the operator image 7 is attached to the object data, the operator image acquisition unit 74 acquires the operator image 7 from the information processing system 50 by specifying the identification information of the operator image 7 via the communication unit 71.

[0122] <<Meeting Devices>> Returning to Fig. 9, the meeting device 60 has a terminal communication unit 61, a panoramic image generation unit 62, a speaker image generation unit 63, a sound collection unit 64, a voice synthesis unit 65, an operator image generation unit 66, and a device direction detection unit 67. Each of these units of the meeting device 60 is a function or a means for performing the function, which is realized when any of the components shown in Fig. 5 operates in response to an instruction from the CPU 611 in accordance with a program loaded from the ROM 612 to the DRAM 614.

[0123] The terminal communication unit 61 communicates with the terminal device 10 using a USB cable or the like. The terminal communication unit 61 may be connected not only by a wired cable but also by wireless LAN, Bluetooth (registered trademark), or the like as long as it can communicate with the terminal device 10.

[0124] The panoramic image generating unit 62 generates a panoramic image. The speaker image generating unit 63 generates a speaker image. The methods for generating these images have been described with reference to Figures 6 and 7. The panoramic image generating unit 62 also serves as an acquisition unit for acquiring image data.

[0125] The sound collection unit 64 converts the audio signal acquired by the microphone of the meeting device 60 into audio data (digital). As a result, the content of speech made by the user or participant at the base on the terminal device 10 side is collected.

[0126] The voice synthesis unit 65 synthesizes the voice transmitted from the terminal device 10 with the voice collected by the voice collection unit 64. As a result, the voices spoken at the other locations 101 and the voices spoken at the own location 102 are combined.

[0127] The operator image generating unit 66 starts cutting out the operator image 7 from the panoramic image in response to the start of operation (cutout request) from the terminal device 10, and ends cutting out the operator image 7 in response to the end of the operation (cutout end request).

[0128] The device orientation detection unit 67 detects a specific sound (sound of a specific frequency) to detect the direction of the electronic whiteboard 2 in the panoramic image. The device orientation detection unit 67 may detect the direction of the electronic whiteboard 2 in the panoramic image based on a preset direction of the electronic whiteboard 2, detection of a two-dimensional code or the like, or the shape of the electronic whiteboard 2. In this embodiment, it is described that the terminal device 10 detects the direction of the electronic whiteboard 2 based on a user setting, detection of a two-dimensional code or the like, or the shape of the electronic whiteboard 2. In this case, the device orientation detection unit 67 receives the direction of the electronic whiteboard 2 from the terminal device 10.

[0129] <<Information Processing System>> The information processing system 50 includes a communication unit 51, an authentication unit 52, a screen generation unit 53, a communication management unit 54, a device management unit 55, a text conversion unit 56, and an object sharing unit 57. Each of these units included in the information processing system 50 is a function or a means for performing the function, which is realized when any of the components shown in FIG. 4 operates in response to an instruction from the CPU 501 in accordance with a program loaded from the HD 504 to the RAM 503. The information processing system 50 also includes a storage unit 5000 implemented by the HD 504 and the like shown in FIG. 4. The storage unit 5000 includes a conference information storage unit 5001, a recording information storage unit 5002, an association information storage unit 5003, an account information storage unit 5004, and an object data storage unit 5005. The object data storage unit 5005 will be described in detail in the description of the electronic whiteboard 2.

[0130] The communication unit 51 transmits and receives various information to and from the terminal device 10. For example, the communication unit 51 transmits a list of remote conferences to the terminal device 10 and receives a request to recognize voice data from the terminal device 10.

[0131] The authentication unit 52 authenticates the user operating the terminal device 10. For example, the authentication unit 52 authenticates the user based on whether or not authentication information (user ID and password) included in an authentication request received by the communication unit 51 matches authentication information stored in advance. Note that the authentication information may be an IC card number, biometric authentication information such as face or fingerprint, or the like. The authentication unit 52 may also perform authentication using an external authentication system or an authentication method such as OAUTH.

[0132] The screen generation unit 53 generates screen information to be displayed by the terminal device 10. When the terminal device 10 executes a native application, the screen information is stored in the terminal device 10, and the information to be displayed is transmitted in XML or the like. When the terminal device 10 executes a web application, the screen information is created using HTML, XML, CSS (Cascade Style Sheet), JavaScript (registered trademark), or the like.

[0133] The communication management unit 54 acquires information about the remote conference from the conference management system 9 using each user's account or a system account assigned to the information processing system 50. The communication management unit 54 stores the conference information of the reserved conference in the conference information storage unit in association with the conference ID. The communication management unit 54 also acquires conference information that users belonging to the tenant have permission to view. Since a conference ID is set for each conference, the remote conference and recorded information are associated with each other by the conference ID.

[0134] When the device management unit 55 receives device identification information of the electronic whiteboard 2 from the terminal device 10, the device management unit 55 associates the conference ID with the electronic whiteboard 2 used in the conference and the meeting device 60, and stores the association information in the association information storage unit 5003. Since the conference ID is also associated with the combined image video, the handwritten data and the combined image video are also associated. Furthermore, when the recording ends (when the conference ends), the device management unit 55 deletes the association from the association information storage unit 5003.

[0135] The text conversion unit 56 converts the voice data requested by the terminal device 10 to text data into text data using an external voice recognition service, or the text conversion unit 56 itself may perform the conversion.

[0136] The object sharing unit 57 shares the input display area, object data, and operator image 7 between the terminal device 10 and the electronic whiteboard 2 participating in the same conference. The object sharing unit 57 provides an input display area, shares the input display area between the electronic whiteboard 2 and the terminal device 10, and accepts object input to the input display area on the electronic whiteboard 2 or the terminal device 10. The terminal device 10 attaches a conference ID to communication, allowing the object sharing unit 57 to identify the conference. In response to object input, the object sharing unit 57 transmits an object transmitted from one electronic whiteboard 2 to another electronic whiteboard 2 (all devices participating in the conference by being linked to the conference), and vice versa. The object sharing unit 57 can also share objects and operator images 7 between the electronic whiteboard 2 and the terminal device 10. The transmitted content is not limited to the object; the entire input display area into which the object has been input may be transmitted and displayed.

[0137] The object sharing unit 57 may be, for example, an application such as an online whiteboard application or an online electronic whiteboard application in which operators write objects to each other and create blackboard writing online (on the cloud) from a plurality of terminal devices 10 and the electronic whiteboard 2. The object sharing unit 57 may also use a web service external to the information processing system 50. The object sharing unit 57 is an example of a sharing unit.

[0138] FIG. 11 shows an example of conference information stored in the conference information storage unit 5001 managed by the communication management unit 54. The communication management unit 54 can use the above account to obtain a list of remote conferences for which the user belonging to the tenant has viewing authority. Viewing authority may be granted to the conference information managed by the communication management unit 54 directly from the information recording application of the terminal device 10. The remote conference information for which the user belonging to the tenant has viewing authority includes information about conferences created by the user and information about conferences for which the user has been granted viewing authority by other users. In this embodiment, a remote conference is used as an example, but the list of remote conferences also includes conferences held in only one conference room.

[0139] The conference information is managed by a conference ID and is associated with the organizer ID, title (conference name), start date and time, end date and time, electronic whiteboard 2, meeting device 60, etc. These are examples of conference information, and the conference information may also include other information.

[0140] The organizer ID is the organizer (host) of the meeting.

[0141] The title item indicates the content of the conference, such as the conference name and overseas agenda.

[0142] The start date and time item is the date and time when the conference is scheduled to start.

[0143] The end date and time item is the date and time when the conference is scheduled to end.

[0144] The item of the electronic whiteboard 2 is identification information of the electronic whiteboard 2 associated with the conference.

[0145] The meeting device 60 item is the identification information of the meeting device 60 used in the conference.

[0146] As shown in FIGS. 10 and 11, the combined image video recorded at the conference is identified by the conference ID.

[0147] The recording information stored in the recording information storage unit 5002 may be the same as that shown in Fig. 10. However, the information processing system 50 has a list of combined image videos recorded by all users belonging to the tenant. The user may input desired storage destination information (path information such as a URL of a cloud storage system) on a user setting screen of the information recording application 41 of the terminal device 10, and the desired storage destination information may be stored in the recording information storage unit 5002.

[0148] 12 shows association information stored in the association information storage unit 5003, in which the conference ID is associated with the device identification information (the electronic whiteboard 2 and the meeting device 60). The association information is held from when the information recording application 41 transmits the device identification information to the information processing system 50 until the recording ends. Because the terminal device 10 is connected to the meeting device 60, the information processing system 50 can also associate and store the identification information (such as an IP address) of the terminal device 10. Alternatively, if the terminal device 10 attaches the device identification information of the meeting device 60 to the communication, it can be identified as the terminal device 10 participating in the same conference as the electronic whiteboard 2.

[0149] 13 is an example of account information stored in the account information storage unit 5004. The account information includes information on not only humans but also non-human users such as the electronic whiteboard 2 and the meeting device 60.

[0150] The user ID is identification information of the users who can participate in the meeting, the electronic whiteboard 2, the meeting device 60, etc.

[0151] The type is the type of each account of the user, the electronic whiteboard 2, or the meeting device 60.

[0152] The name is the name of the user, the name of the electronic whiteboard 2, or the name of the meeting device 60.

[0153] The email address is the email address of the user, the electronic whiteboard 2, the meeting device 60, etc.

[0154] <<Electronic Whiteboard>> 14 is a functional block diagram explaining the functions of the interactive whiteboard 2 by dividing them into blocks. The interactive whiteboard 2 has a contact position detection unit 31, a drawing data generation unit 32, a data recording unit 33, a display control unit 34, a code generation unit 35, a communication unit 36, a voice data generation unit 37, an operation detection unit 38, a user detection unit 39, an object upload unit 40a, and an operator image receiving unit 40b. Each function of the interactive whiteboard 2 is realized by one of the components shown in FIG. 8 operating in response to an instruction from the CPU 401 in accordance with a program loaded from the SSD 404 onto the RAM 403.

[0155] The contact position detection unit 31 detects the coordinates of the position where the electronic pen 490 contacts the contact sensor 414. The drawing data generation unit 32 acquires the coordinates where the pen tip of the electronic pen 490 contacts from the contact position detection unit 31. The drawing data generation unit 32 generates stroke data by connecting this sequence of coordinate points by interpolating them.

[0156] The display control unit 34 displays the handwritten data, character strings converted from the handwritten data, and menus for user operation on the display.

[0157] The data recording unit 33 stores handwritten data written on the electronic whiteboard 2, shapes converted from handwritten data such as circles and triangles, stamps such as "Done", PC screens, files, etc. in the object data storage unit 3002. Handwritten data, character strings (including shapes), images such as PC screens, files, etc. are handled as objects. For handwritten data, a group of stroke data becomes a single object depending on whether it is a time division due to an interruption in handwriting or a distance division due to a difference in handwriting location.

[0158] The communication unit 36 ​​connects to Wi-Fi or a LAN and communicates with the information processing system 50. The communication unit 36 ​​transmits object data to the information processing system 50, receives object data stored in the information processing system 50 from the information processing system 50, and displays the object data on the display 480. The communication unit 36 ​​also transmits an operation start or operation end to the terminal device 10.

[0159] The code generation unit 35 generates a two-dimensional code by encoding the device identification information of the electronic whiteboard 2 stored in the device information storage unit 3001 and information indicating that the device is usable in a conference into a two-dimensional pattern. The code generation unit 35 can also encode the device identification information of the electronic whiteboard 2 and information indicating that the device is usable in a conference into a barcode. The device identification information may be a serial number, a UUID, or the like. The device identification information may also be set by the user.

[0160] The audio data generator 37 generates audio data by sampling a signal of a preset frequency at regular intervals, similar to PCM conversion. The audio data is converted to analog by a D / A converter included in the speaker 450 and output from the speaker 450.

[0161] The operation detection unit 38 detects a user operation on the electronic whiteboard 2. The operation detection unit 38 detects the start or end of an operation, for example, when the contact position detection unit 31 detects that the electronic pen 490, the user's hand H, or the like has come into contact with (or simply approached) the display 480 (touch panel).

[0162] The user detection unit 39 detects when a user approaches or moves away from the interactive whiteboard 2 by using an infrared sensor, an ultrasonic sensor, a camera, or the like.

[0163] The object upload unit 40a uploads the objects displayed on the interactive whiteboard 2 to the information processing system 50 at a predetermined timing, thereby allowing the objects to be shared between locations.

[0164] The operator image receiving unit 40b receives the operator image 7 from the terminal device 10. The operator image receiving unit 40b may receive the operator image 7 from the terminal device 10 via the information processing system 50, or may receive the operator image 7 directly from the terminal device 10.

[0165] The electronic whiteboard 2 also has a storage unit 3000 constructed using the SSD 404 and the like shown in Fig. 8. The storage unit 3000 includes a device information storage unit 3001 and an object data storage unit 3002.

[0166] FIG. 15 shows the device identification information and the like stored in the device information storage unit 3001. The device identification information is the identification information of the electronic whiteboard 2. The IP address is an IP address that allows other devices to connect to the electronic whiteboard 2 via a network. The password is used to authenticate other devices when they connect to the electronic whiteboard 2.

[0167] 16 is a diagram illustrating object data stored in the object data storage unit 3002. The object data is information for managing objects displayed by the electronic whiteboard 2. The object data is transmitted to the information processing system 50 and used as minutes of a meeting. The object is associated with the conference ID notified by the information processing system 50.

[0168] The object ID item is identification information that identifies an object. Objects include strokes, characters, shapes, images, etc. Characters are character strings (character codes) converted from handwritten data. Character strings are sometimes called text data. Shapes are geometric shapes such as triangles and squares converted from handwritten data. Images are image data such as JPEG, PNG, PDF, and TIFF imported from a PC or the Internet.

[0169] The data content items are the sequence of coordinate points that make up the stroke. The data content items are character codes or image file data and their position information.

[0170] The input source field is the identification information of the device where the object was input. When handwriting is done on the electronic whiteboard 2, the identification information of the electronic whiteboard 2 is registered, and when handwriting is done on a terminal device 8 at another location, the ID of the user who wrote the handwriting (logged-in user) is registered.

[0171] The identification information of the operator image 7 associated with the object is registered in the item of the identification information of the operator image 7. The operator image 7 may be stored in the information processing system 50 or the storage service system 70.

[0172] The save destination is the URL or file path where the operator image 7 is saved.

[0173] <Action or Processing> 17, an overall flow will be described in which the meeting device 60 cuts out the operator image 7 and the terminal device 8 at the other location displays the object and the operator image 7. Fig. 17 is a sequence diagram illustrating the process in which the terminal device 8 at the other location displays the strokes handwritten by the user on the electronic whiteboard 2 together with the operator image 7.

[0174] S1: A user participating in a meeting writes by hand using an electronic pen 490 or displays the PC screen on an electronic whiteboard 2.

[0175] S2: The contact position detection unit 31 of the electronic whiteboard 2 detects the contact position of the electronic pen 490, and the operation detection unit 38 detects the start of operation. The user detection unit 39 may detect the approach of a user. Since the electronic whiteboard 2 has been operated, the communication unit 36 ​​of the electronic whiteboard 2 transmits a notification of the start of operation (that the operation has been detected) to the terminal device 10. The method of communication between the electronic whiteboard 2 and the terminal device 10 will be described later (FIGS. 19 to 22).

[0176] S3: The communication unit 11 of the information recording application 41 receives the start of the operation, and the cut-out request unit 25 transmits a cut-out request for the operator image 7 to the meeting device 60 via the device communication unit 16. Note that the cut-out request unit 25 may transmit the cut-out request for the operator image 7 to the meeting device 60 in response to the user inputting an operation to cut out the operator image.

[0177] S4: The terminal communication unit 61 of the meeting device 60 receives the cut-out request, and the operator image generation unit 66 generates (cuts out) the operator image 7 from the panoramic image. A method for detecting the direction of the electronic whiteboard 2 will be described later. Note that the terminal device 10 can also cut out the operator image 7.

[0178] S5: The terminal communication unit 61 of the meeting device 60 transmits the requested operator image 7 to the terminal device 10. The device communication unit 16 of the terminal device 10 receives the operator image 7.

[0179] The meeting device 60 may generate an operator image 7 for each stroke, or may generate an operator image 7 for each object. The meeting device 60 may also generate an operator image 7 periodically.

[0180] S6: A user who has joined the conference ends operation of the electronic whiteboard 2. End of operation can be achieved by lifting the pen or by not operating the electronic whiteboard 2 for a certain period of time since the last operation.

[0181] S7: The operation detection unit 38 of the electronic whiteboard 2 detects the end of the operation. Here, the end of the operation is assumed to be the end of one stroke (pen up). The user detection unit 39 may also detect the user moving away. In response to the end of the operation, the operator image receiving unit 40b of the electronic whiteboard 2 requests the operator image 7 from the terminal device 10 via the communication unit 36. The communication unit 11 of the terminal device 10 transmits the operator image 7 to the electronic whiteboard 2.

[0182] S8: The communication unit 36 ​​of the interactive whiteboard 2 receives the operator image 7, and the object upload unit 40a transmits the operator image 7 to the information processing system 50 via the communication unit 36, specifying the conference ID. The communication unit 51 of the information processing system 50 receives the operator image 7, and the object sharing unit 57 assigns a number to the identification information of the operator image 7. The object sharing unit 57 stores the operator image 7 in association with the identification information. The identification information of the operator image 7 is transmitted to the terminal device 10.

[0183] S9: Next, the object upload unit 40a of the interactive whiteboard 2 transmits the object data (here, stroke data) and the identification information of the operator image 7 to the information processing system 50 via the communication unit 36, specifying the conference ID.

[0184] S10: The communication unit 51 of the information processing system 50 receives the conference ID, the object data, and the identification information of the operator image 7, and the object sharing unit 57 assigns an object ID (the object ID may be assigned by the interactive whiteboard 2). The object sharing unit 57 associates the object ID, the object data, and the identification information of the operator image 7 and stores them. Note that the order of transmitting the object data and the operator image 7 may be reversed.

[0185] S11: When the information processing system 50 receives the object data and the operator image 7, the object sharing unit 57 identifies the terminal devices 8 at other locations participating in the same conference using the conference ID, and the communication unit 51 transmits the object data and the identification information of the operator image 7 to the terminal devices 8 at other locations. The object data and the identification information of the operator image 7 are not transmitted to the electronic whiteboard 2 that sent the data. They may be transmitted to the terminal device 10 at the own location if this terminal device 10 is participating in the same conference.

[0186] S12: The communication unit 71 of the terminal device 8 at the other site receives the object data and the identification information of the operator image 7, and the display control unit 73 displays the object data on the display.

[0187] S13: Furthermore, in response to a user of a terminal device 8 at another location selecting one of the displayed objects, the operator image acquisition unit 74 requests the operator image 7 from the information processing system 50 via the communication unit 71, using the identification information of the operator image 7. The communication unit 51 of the information processing system 50 receives the request, and the object sharing unit 57 transmits the operator image 7 associated with the identification information of the operator image 7 to the terminal device 8 at the other location via the communication unit 51.

[0188] S14: The communication unit 71 of the terminal device 8 at the other site receives the operator image 7, and the display control unit 73 displays the operator image 7 in association with the object transmitted together with the identification information of the operator image 7. Displaying the operator image in association with the object means displaying the operator image 7 near (within a certain distance from) or around (above, below, left, right) the object.

[0189] Through the above process, the terminal devices 8 at other locations can know who wrote the strokes on the electronic whiteboard 2.

[0190] In the above example, the electronic whiteboard 2 transmits the operator image 7 to the information processing system 50 upon completion of the operation, but the electronic whiteboard 2 may transmit the operator image 7 to the information processing system 50 during operation (while writing by hand). Also, although the object data and the operator image 7 are transmitted to the information processing system 50 almost simultaneously, they may be transmitted separately. For example, the electronic whiteboard 2 may transmit the stroke data to the information processing system 50 first, and then perform a process of storing the operator image 7 in the information processing system 50. In other words, only the strokes may be shared first with the terminal device 8 at another location, and then the operator image 7 captured when the strokes were input may be shared.

[0191] <Example of display on a terminal device at another location> 18 shows an example of an object and an operator image 7 displayed by a terminal device 8 at another location. FIG. 18(a) shows an operator 121 writing by hand on the electronic whiteboard 2. An operator image 7 is created showing the operator 121 near a stroke 120 of "a" written by hand on the electronic whiteboard 2. The operator image 7 is the right half of the electronic whiteboard 2 because the stroke was written by hand on the right side of the electronic whiteboard 2, but the entire electronic whiteboard 2 may be cut out, or just the face may be cut out.

[0192] 18(b) shows the stroke 120 and the operator image 7 displayed on the terminal device 8 at another location. The operator image 7 is displayed in a speech bubble or pop-up display near the handwritten "a" on the electronic whiteboard 2. Therefore, the user at the other location can easily understand who wrote the stroke 120.

[0193] In FIG. 18(b), the meeting device 60 intentionally crops the image wider to make it easier for other locations to understand how the user is writing on the electronic whiteboard 2 or to get an impression of the atmosphere, including the electronic whiteboard 2. As a result, the user's hands writing and part of the electronic whiteboard 2 are captured in the image. Note that the display position of the operator image 7 is an example, and the operator image 7 may be displayed around the stroke 120. The display control unit 73 may display the operator image 7 while avoiding existing strokes. Furthermore, the display control unit 73 may display a reduced version of the operator image 7 to avoid existing strokes.

[0194] The meeting device 60 may use known image recognition (face recognition) technology to cut out only the face and its surrounding area into a circle, rectangle, or the like to generate the operator image 7. In FIG. 18(b), the meeting device 60 obtains information from the electronic whiteboard 2 that handwriting has been done on the right side of the electronic whiteboard 2, and generates the right half of the electronic whiteboard 2 as the operator image. Alternatively, the meeting device 60 may detect a person and generate an operator image including the electronic whiteboard 2 and the person.

[0195] Furthermore, the terminal device 8 at the other site can make the operator image 7 disappear after a certain period of time (for example, when the electronic whiteboard 2 detects writing and determines that no writing has been made for several seconds or more), so that the operator image 7 does not become a nuisance. The display control unit 73 may also erase the operator image 7 by user operation even within the certain period of time. Details of the display control of the operator image 7 are shown in FIG. 38.

[0196] <An example of the process until communication between the electronic whiteboard and the terminal device> The following describes the process for communication between the electronic whiteboard and the terminal device. A user registers a new conference in the information processing system 50 using a conference registration screen 250 as shown in Fig. 19. Fig. 19 is an example of the conference registration screen 250 displayed by the terminal device 10.

[0197] The conference name item 241 is the title of the conference.

[0198] The organizer item 242 is information about the organizer who organized the conference, and is usually the person who operates the conference registration screen.

[0199] The date and time item 243 is an input field for the start date and time and the end date and time of the conference. The date and time item 243 can be selected from a calendar.

[0200] The participant item 244 is a participant of the conference, and can be selected from a list of users registered in the account information storage unit 5004.

[0201] The item 245 for the electronic whiteboard 2 is a selection field for the electronic whiteboard 2 to be used in the meeting, and can be selected from the account of type electronic whiteboard 2 in the list of types electronic whiteboard 2 in the account information storage unit 5004.

[0202] The meeting device 60 item 246 is a selection field for the meeting device 60 to be used in the conference, and the type can be selected from a list of meeting devices 60 in the account information storage unit 5004 .

[0203] As shown in FIG. 19, when creating a new meeting, the user can select the electronic whiteboard 2 and meeting device 60 to be used in the meeting.

[0204] Next, as shown in Fig. 20, the user registers the meeting device 60 and the interactive whiteboard 2 in the information processing system 50. Since the meeting has already been created, the user only needs to select the meeting. Fig. 20 is an example of a sequence diagram illustrating the process by which the user registers the meeting device 60 and the interactive whiteboard 2 in the information processing system 50.

[0205] S21: As shown in FIG. 19, the user creates the meeting information.

[0206] S22: The operation reception unit 12 of the terminal device 10 receives the conference information, and the communication unit 11 transmits it to the information processing system 50. The communication unit 51 of the information processing system 50 receives the conference information, and the communication management unit 54 registers it in the conference information storage unit 5001.

[0207] S23: When the user starts the meeting, the user may move to the conference room, and connect the meeting device 60 to the terminal device 10. The meeting device 60 starts up when the USB cable is connected, and communication with the meeting device 60 (transmission of various images) starts when the information recording application 41 is started.

[0208] S24: The user selects a conference to be held from the conference list screen 230 (see FIG. 33) displayed by the information recording application 41.

[0209] S25: The operation reception unit 12 of the information recording application 41 receives the operation, and the communication unit 11 requests registration for the conference by specifying the device identification information and the conference ID of the meeting device 60. The communication unit 51 of the information processing system 50 receives the request for registration for the conference and determines that the meeting device 60 is registered in the conference information. The device management unit 55 associates the device identification information of the meeting device 60 with the conference ID.

[0210] S26: Next, the user selects the upcoming meeting from the meeting list screen 230 displayed on the interactive whiteboard 2.

[0211] S27: The communication unit 36 ​​of the electronic whiteboard 2 requests registration for the conference by specifying the device identification information and the conference ID of the electronic whiteboard 2. The communication unit 51 of the information processing system 50 receives the request for registration for the conference and determines that the electronic whiteboard 2 is registered in the conference information. The device management unit 55 associates the device identification information of the electronic whiteboard 2 with the conference ID.

[0212] In this way, the information processing system 50 associates the meeting device 60 (terminal device 10) with the electronic whiteboard 2, thereby enabling the terminal device 10 and the electronic whiteboard 2 to communicate with each other via the information processing system 50. If the meeting device 60 (terminal device 10) and the electronic whiteboard 2 transmit their respective identification information to the information processing system 50, the information processing system 50 can identify the other device registered with the same conference ID based on the association information.

[0213] In Figures 19 and 20, the user registers the meeting information in advance, but even if the meeting information is not registered, the user can associate the meeting device 60 with the electronic whiteboard 2 and register them in the information processing system 50 at the start of the meeting.

[0214] FIG. 21 is an example of a sequence diagram illustrating a process in which a user registers a meeting device 60 and an interactive whiteboard 2 in the information processing system 50 when no meeting information has been created.

[0215] S31: The user inputs an operation to start registering the association with the meeting device 60 on the electronic whiteboard 2 in the conference room.

[0216] S32: In response to the operation, the interactive whiteboard 2 communicates with the preset information processing system 50 and registers the device identification information and the fact that it can be associated with the conference.

[0217] S33: The code generation unit 35 of the electronic whiteboard 2 placed in the conference room used for the meeting generates a two-dimensional code in which the device identification information of the electronic whiteboard 2 itself and the fact that it is the device identification information are encoded, and the display control unit 34 displays the code. The code may also include a password for the electronic whiteboard 2 to authenticate other devices.

[0218] S34: The user carries the terminal device 10 and the meeting device 60 and enters the conference room where the electronic whiteboard 2 is installed, and connects the terminal device 10 and the meeting device 60 with a USB cable. The meeting device 60 is started up by power supply via the USB cable or by turning on the power. This puts the meeting device 60 into standby mode. The user also starts the information recording application 41 of the terminal device 10. The information recording application 41 starts communication with the meeting device 60, and the meeting device 60 starts capturing images and collecting sound. The panoramic image generation unit 62 of the meeting device 60 generates image data of the surrounding area including the two-dimensional code.

[0219] S35: The terminal communication unit 61 of the meeting device 60 transmits the image data and the device identification information of the meeting device 60 to the terminal apparatus 10.

[0220] S36: The device communication unit 16 of the terminal device 10 receives the image data and the device identification information of the meeting device 60, and the analysis unit 23 detects the two-dimensional code displayed by the interactive whiteboard 2 from the image data. The analysis unit 23 decodes the two-dimensional code, and if it determines that the two-dimensional code contains embedded information indicating that the device can be used in a meeting, it acquires the device identification information of the interactive whiteboard 2 from the two-dimensional code. The communication unit 11 of the information recording application 41 specifies the device identification information of the interactive whiteboard 2 and the device identification information of the meeting device 60 and sends a request to register for the meeting to the information processing system 50. Note that the meeting device 60 can also decode the two-dimensional code.

[0221] S37, S38: When the communication unit 51 of the information processing system 50 receives a request to register for a conference (device identification information, device identification information of the meeting device 60), the communication management unit 54 issues a conference ID. Then, the device management unit 55 associates the device identification information of the electronic whiteboard 2, the device identification information of the meeting device 60, and the conference ID, and stores them in the association information storage unit 5003.

[0222] The communication unit 51 of the information processing system 50 notifies the terminal device 10 and the electronic whiteboard 2 of the conference ID and the completion of registration of the electronic whiteboard 2 to the conference. The communication unit 11 of the terminal device 10 receives and stores the conference ID. Similarly, when the communication unit 36 ​​of the electronic whiteboard 2 receives the conference ID and password, it verifies the password and stores the conference ID if they match. The terminal device 10 receives the conference ID in response to the communication of step S36. The electronic whiteboard 2 receives the conference ID and password in response to polling of the information processing system 50 (requesting association of device identification information with the conference). The electronic whiteboard 2 and the information processing system 50 may communicate using a bidirectional communication method such as WebSocket, which enables push-type communication from the information processing system 50 to the electronic whiteboard 2.

[0223] Thereafter, the terminal device 10 and the interactive whiteboard 2 attach the conference ID (or the identification information of each device) to the data to be transmitted.

[0224] In this way, the information processing system 50 associates the meeting device 60 (terminal device 10) with the interactive whiteboard 2, thereby enabling the terminal device 10 and the interactive whiteboard 2 to communicate with each other via the information processing system 50.

[0225] 19 to 21, the terminal device 10 and the electronic whiteboard 2 can communicate with each other via the information processing system 50, but if the terminal device 10 and the electronic whiteboard 2 are connected to the same network, they can communicate with each other using IP addresses. The same network refers to the case where the network addresses are the same or the destination SSID (Service Set Identifier) ​​is the same.

[0226] FIG. 22 is an example of a sequence diagram illustrating a communication method between the terminal device 10 and the interactive whiteboard 2 when the terminal device 10 and the interactive whiteboard 2 are connected to the same network.

[0227] S41: The user inputs an operation to start association with the meeting device 60 on the electronic whiteboard 2 in the conference room.

[0228] S42: The interactive whiteboard 2 accepts the operation, and the display control unit 34 displays the IP address and password of the interactive whiteboard 2.

[0229] S43: The user sees the displayed IP address and password and inputs the IP address and password into the information recording application 41 of the terminal device 10. The operation reception unit 12 of the terminal device 10 receives the input.

[0230] S44: The communication unit 11 of the terminal device 10 connects to the interactive whiteboard 2 using the IP address and transmits the password. If the password is successfully verified, the terminal device 10 and the interactive whiteboard 2 become able to communicate with each other.

[0231] <Determining the orientation of an electronic whiteboard in a panoramic image> Next, a method for determining the orientation of the electronic whiteboard 2 in a panoramic image will be described. Since the operator image 7 is an image of a participant (user) operating the electronic whiteboard 2, if the meeting device 60 crops the direction of the electronic whiteboard 2, an operator image 7 including the operator can be obtained.

[0232] There are four main methods for determining this: 1. The user selects from the panoramic images at the start of the meeting. 2. The electronic whiteboard 2 displays a specific image (such as a two-dimensional code), which is recognized by the terminal device 10 or the meeting device 60 from a panoramic image captured by the imaging unit 601 of the meeting device 60. 3. The electronic whiteboard 2 outputs a specific sound, which the meeting device 60 recognizes through the microphone 608. 4. An arbitrary information processing device learns the shape of the electronic whiteboard 2 by machine learning, and the terminal device 10 or the meeting device 60 recognizes it from a panoramic image captured by the camera (imaging unit 601) of the meeting device 60.

[0233] <<1. The user selects from the panoramic images at the start of the meeting.>> 23 is a diagram illustrating an operation method for setting the orientation of the interactive whiteboard 2 by pressing the position registration button 207. Pressing the position registration button 207 causes a pop-up panoramic image 203 to be displayed. For example, the user moves a rectangular window 206 on the panoramic image 203 using a pointing device such as a mouse or a touch panel. The user aligns the window 206 with the interactive whiteboard 2, a podium, or the like included in the panoramic image 203.

[0234] 24 shows a screen for confirming the direction set by the user. When the user presses the OK button 208, the direction of the interactive whiteboard 2 relative to the panoramic image is set. The direction set by the user is transmitted to the meeting device 60 and stored by the device direction detection unit 67 of the meeting device 60.

[0235] 2. The electronic whiteboard displays a specific image (such as a two-dimensional code), which is recognized by the terminal device or meeting device from a panoramic image captured by the meeting device's imaging unit. 3. The electronic whiteboard outputs a specific sound, which the meeting device recognizes using its microphone. FIG. 25 is an example of a sequence diagram illustrating a process in which the information recording application 41 determines the direction of the interactive whiteboard 2 based on a specific video or audio.

[0236] S51: The user inputs an operation to the information recording application 41 of the terminal device 10 to determine the orientation of the electronic whiteboard 2. The operation receiving unit 12 receives the operation.

[0237] S52: The communication unit 11 of the information recording application 41 requests the output of a specific image or a specific sound from the interactive whiteboard 2. The communication unit 11 may communicate with the interactive whiteboard 2 via a LAN or via the information processing system 50.

[0238] S53: The communication unit 36 ​​of the interactive whiteboard 2 receives the request, and the code generation unit 35 generates a two-dimensional code as a specific image. The display control unit 34 displays the two-dimensional code on the display 480.

[0239] Furthermore, the communication unit 36 ​​of the interactive whiteboard 2 receives the request, and the audio data generation unit 37 outputs a sound of a specific frequency from the speaker 450. Either the code generation unit 35 or the audio data generation unit 37 needs to operate, but both may also operate.

[0240] S54: The panoramic image generator 62 of the meeting device 60 repeatedly generates panoramic images, so if a two-dimensional code is in the field of view, it automatically generates a panoramic image including the two-dimensional code. The sound collector 64 of the meeting device 60 repeatedly collects sound, so it automatically collects sound of a specific frequency. The device direction detector 67 performs a Fourier transform on the audio data to obtain a frequency spectrum and identifies two directions from which sound of a predetermined frequency above a threshold is coming. The device direction detector 67 of the interactive whiteboard 2 stores the direction of the interactive whiteboard 2 (A° to B° out of 360° horizontally). It is preferable that this sound be in the ultrasonic frequency band, as it is inaudible to the user.

[0241] S55: The terminal communication unit 61 of the meeting device 60 transmits the panoramic image to the terminal device 10.

[0242] S56: The device communication unit 16 of the information recording application 41 receives the panoramic image. The analysis unit 23 detects the two-dimensional code displayed by the interactive whiteboard 2 from within the panoramic image. The analysis unit 23 decodes the two-dimensional code, and if it determines that the code contains information indicating the orientation of the interactive whiteboard 2, it identifies the positions of the left and right ends of the two-dimensional code in the panoramic image. Details will be described with reference to FIG. 26. This process can also be performed by the device orientation detection unit 67 of the meeting device 60.

[0243] S57: The device communication unit 16 of the terminal device 10 transmits the direction of the electronic whiteboard 2 (A° to B° out of 360° in the horizontal direction) to the meeting device 60. The terminal communication unit 61 of the meeting device 60 receives the direction of the electronic whiteboard 2, and the device direction detection unit 67 stores it.

[0244] S58: Now that the direction of the interactive whiteboard 2 has been determined, the communication unit 11 of the information recording application 41 requests the interactive whiteboard 2 to stop outputting a specific image or a specific sound.

[0245] S59: The communication unit 36 ​​of the interactive whiteboard 2 receives the request, and the display control unit 34 ends the display of the two-dimensional code. The audio data generation unit 37 stops outputting the sound of the specific frequency.

[0246] <<Determining direction using a two-dimensional code>> FIG. 26 shows an example of a two-dimensional code 301 displayed as a specific image by the interactive whiteboard 2. In FIG. 26, the interactive whiteboard 2 is shown in a panoramic image, and the interactive whiteboard 2 displays two two-dimensional codes 301. The size (width) of the two-dimensional code 301 is determined by dividing the width of the display of the interactive whiteboard 2 into thirds. Two two-dimensional codes 301 are displayed. One is displayed right-justified on the display 480, and the other is displayed left-justified. Each two-dimensional code 301 includes information indicating whether it is displayed on the right or left. In this embodiment, right and left are determined based on the user's position facing the interactive whiteboard 2.

[0247] 27 is a diagram illustrating a method for determining a direction based on two-dimensional codes 301. The analysis unit 23 detects two two-dimensional codes 301, thereby identifying horizontal position information of each two-dimensional code 301. The analysis unit 23 determines a position 253, which is moved left from the left end 251 of the left-side two-dimensional code 301 by an amount equal to the width of the two-dimensional code 301, as the left edge position of the electronic whiteboard 2. The analysis unit 23 determines a position 254, which is moved right from the right end 252 of the right-side two-dimensional code 301 by an amount equal to the width of the two-dimensional code 301, as the right edge position of the electronic whiteboard 2. The analysis unit 23 determines the distance from the left edge position 253 to the right edge position 254 as the direction of the electronic whiteboard 2.

[0248] <<Determining direction by sound>> 28A and 28B are diagrams illustrating a method for determining the direction of the electronic whiteboard 2 when the electronic whiteboard 2 outputs a specific sound. As shown in Fig. 28A, speakers 450 are installed at the left and right ends of the electronic whiteboard 2. The speakers 450 may be built into the left and right ends.

[0249] 28(b), the audio data generation unit 37 outputs sound from each speaker 450. The microphone 608 of the meeting device 60 has directionality, and identifies the direction from which the sound emitted from each speaker 450 is coming relative to the meeting device 60. Once the direction relative to the meeting device 60 is identified, the direction in the panoramic image 203 is also identified.

[0250] Even if the speaker 450 is directional, it is difficult for the device direction detection unit 67 to determine the center of the speaker 450. Therefore, the device direction detection unit 67 determines the position of the speaker 450 to be within a certain range 261 in the left-right direction centered on the direction of the highest intensity based on the intensity of sound of a specific frequency.

[0251] As shown in FIG. 28(c), the device orientation detection unit 67 determines the area from the left end 262 to the right end 263 of the two ranges 261 as the orientation of the interactive whiteboard 2.

[0252] <<4. An arbitrary information processing device learns the shape of the electronic whiteboard through machine learning, and the terminal device or meeting device recognizes it from a panoramic image captured by the imaging unit of the meeting device.>> 29 is a diagram illustrating a direction for determining the orientation of the electronic whiteboard 2 based on the electronic whiteboard 2 detected by image processing such as machine learning. First, the device recognition unit 22 detects the shape (circumscribed rectangle) of the electronic whiteboard 2 from a panoramic image by machine learning. When the device recognition unit 22 detects the electronic whiteboard 2 from the panoramic image, it divides the width of the electronic whiteboard 2 into thirds and determines the orientation of the electronic whiteboard 2 as the distance from position 271, which is one-third of the width from the left end of the electronic whiteboard 2, to position 272, which is one-third of the width from the right end. One-third is an example.

[0253] The terminal device 10 transmits the orientation of the interactive whiteboard 2 to the meeting device 60, and the device orientation detection unit 67 stores the orientation of the interactive whiteboard 2. Note that the meeting device 60 may perform position detection by image processing.

[0254] <Screen transition> Next, several screens that the terminal device 10 displays during a remote conference will be described with reference to Fig. 30 to Fig. 33. Fig. 30 shows an initial screen 200 after login that is displayed by the information recording application 41 running on the terminal device 10. The user of the terminal device 10 connects the information recording application 41 to the information processing system 50. When the user inputs authentication information and successfully logs in, the initial screen 200 of Fig. 30 is displayed.

[0255] The initial screen 200 has a fixed display button 201, a front change button 202, a panoramic image 203, one or more speaker images 204a to 204c (hereinafter referred to as speaker images 204 when not distinguished), a recording start button 205, and a position registration button 207. If the meeting device 60 is already running and capturing images of the surroundings at the time of login, the panoramic image 203 and speaker image 204 created by the meeting device 60 are displayed on the initial screen 200. Therefore, the user can decide whether to start recording while viewing these. If the meeting device 60 is not running (not capturing images), the panoramic image 203 and speaker image 204 are not displayed.

[0256] The information recording application 41 may display speaker images 204 of all participants based on all faces detected from the panoramic image 203, or may display only the speaker images 204 of the N most recently spoken participants. Fig. 30 shows an example in which speaker images 204 of up to three people are displayed. Until a participant speaks, there may be no speaker image 204 (the speaker images are added one by one as the participant speaks), or three speaker images 204 of participants in a predetermined direction may be displayed (the speaker images are replaced as the participant speaks).

[0257] When no one is speaking, such as immediately after the meeting device 60 is started up, a predetermined direction (0°, 120°, 240°, etc.) within the horizontal 360° is created as the speaker image 204. If a fixed display, which will be described later, is set, the fixed display setting takes priority.

[0258] The fixed display button 201 is a button that allows the user to perform an operation to fix a certain area of ​​the panoramic image 203 in close-up as a speaker image 204 .

[0259] The front change button 202 is a button that allows the user to perform an operation to change the front of the panoramic image 203 (because a panoramic image covers 360° horizontally, the right and left edges are aligned). The user can slide the panoramic image 203 left and right with a pointing device to determine which participant will be shown in the front. The user's operation is transmitted to the meeting device 60, which then changes the angle of the front within the 360° horizontal direction, creates a panoramic image, and transmits it to the terminal device 10.

[0260] The position registration button 207 is a button for the user to perform an operation to set the position (direction) of a device such as the electronic whiteboard 2.

[0261] When the user presses the record start button 205, the information recording application 41 displays the recording setting screen 210 shown in FIG.

[0262] 31 is an example of a recording setting screen 210 displayed by the information recording application 41. On the recording setting screen 210, the user can set whether to record (whether to include in the recording) the panoramic image and speaker image created by the meeting device 60, and the desktop screen or the screen of a running application of the terminal device 10. If the information recording application 41 does not record either the panoramic image and speaker image, or the desktop screen or the screen of a running application, only the audio (audio output by the terminal device 10 + audio collected by the meeting device 60) is recorded.

[0263] The camera toggle button 211 is a button for switching on and off the recording of the panoramic image and the speaker image created by the meeting device 60. The camera toggle button 211 may be set to record the panoramic image and the speaker image separately.

[0264] The PC screen toggle button 212 is a button for switching on and off recording of the desktop screen of the terminal device 10 and the screen of an application running on the terminal device 10. When the PC screen toggle button 212 is in the on state, the desktop screen is recorded.

[0265] If the user wants to record the screen of an application, the user further selects the application in the application selection field 213. The application selection field 213 displays the names of applications currently running on the terminal device 10 in a pull-down menu. The user can select the application to record. The information recording application 41 acquires this application name from the OS. The information recording application 41 can display only those applications that have a UI (screen) among the currently running applications. The selected applications may include the remote conference application 42. Therefore, the information recording application 41 can record videos of materials displayed in the remote conference application 42 and participants at each location. In addition, the applications displayed in the pull-down menu include various applications currently running on the terminal device, such as a presentation application, a word processing application, a spreadsheet application, an application for creating and editing documents or other materials, a cloud electronic whiteboard application, and a web browser application. Therefore, the user can flexibly select the application screens to include in the combined image video.

[0266] When recording on an app basis, the user can select multiple apps, and the information recording app 41 can record the screens of all the selected apps.

[0267] When both the camera toggle button 211 and the PC screen toggle button 212 are off, the recording content confirmation window 214 displays "Only audio will be recorded." This audio includes the audio output by the terminal device 10 (audio received by the remote conference app 42 from other locations 101) and the audio collected by the meeting device 60. In other words, as long as a remote conference is being held, the audio from the remote conference app 42 and the audio from the meeting device 60 are saved regardless of whether images are recorded. However, the user may be able to selectively stop saving the audio from the remote conference app 42 and the audio from the meeting device 60 using user settings.

[0268] The combined image moving image is recorded as follows according to the combination of on and off of the camera toggle button 211 and the PC screen toggle button 212. The combined image moving image is also displayed in real time in the recording content confirmation window 214.

[0269] When the camera toggle button 211 is on and the PC screen toggle button 212 is off, the panoramic image captured by the meeting device 60 and the speaker image are displayed in the recording content confirmation window 214 .

[0270] When the camera toggle button 211 is off and the PC screen toggle button 212 is on (the screen has also been selected), the recording content confirmation window 214 displays the desktop screen or the screen of the selected application.

[0271] When the camera toggle button 211 is on and the PC screen toggle button 212 is on, the panoramic image and speaker image captured by the meeting device 60, as well as the desktop screen and the screen of the selected application, are displayed side by side in the recording content confirmation window 214.

[0272] Therefore, there are cases where the panoramic image, speaker image, and app screen are not recorded, or where the panoramic image, speaker image, and app screen are not recorded at all, but in this embodiment, for convenience, the image created by the information recording application 41 is referred to as a combined image video.

[0273] Furthermore, the recording setting screen 210 has a check box 209 along with the message "Automatically transcribe recording after uploading." The recording setting screen 210 also has a Start Recording Now button 217. When the user checks the check box 209, text data obtained by converting comments made during the remote conference is attached to the recorded video. In this case, after the recording is completed, the information recording application 41 uploads the audio to the information processing system 50 along with a request to convert it into text data. Furthermore, when the user presses the Start Recording Now button 217, the recording in progress screen 220 shown in FIG. 32 is displayed.

[0274] FIG. 32 is an example of a recording screen 220 that the information recording application 41 displays during recording. Note that the explanation of FIG. 32 will mainly focus on the differences from FIG. 30. The recording screen 220 displays a combined image video being recorded in real time under the conditions set by the user on the recording setting screen 210. The recording screen 220 of FIG. 32 is displayed when the camera toggle button 211 is on and the PC screen toggle button 212 is off, and displays a panoramic image 203 and a speaker image 204 (both of which are videos) created by the meeting device 60. The recording screen 220 displays a recording icon 225, a pause button 226, and an end recording button 227.

[0275] The pause button 226 is a button for stopping recording, and after stopping, recording can also be resumed. The end recording button 227 is a button for ending recording. The pause button 226 does not change the recording ID, but the end recording button 227 changes the recording ID. After pausing, when recording is resumed, the user can reset the recording conditions set on the recording setting screen 210. In this case, the information recording application 41 may create multiple video files each time recording is stopped (for example, by pressing the end recording button 227), or may combine multiple files so that they form a single continuous video (for example, by pressing the pause button 226). Furthermore, when the information recording application 41 plays back a combined image video, it may play back multiple video files continuously as a single video.

[0276] Furthermore, the recording in progress screen 220 has an Get Information from Calendar button 221, a conference name field 222, a time field 223, and a location field 224. The Get Information from Calendar button 221 is a button that the user uses to obtain conference information from the conference management system 9. When the Get Information from Calendar button 221 is pressed, the information recording application 41 obtains and displays a list of conferences that the user has permission to view from the information processing system 50. The user selects the upcoming remote conference from the conference list. This causes the conference information to be reflected in the conference name field 222, time field 223, and location field 224. The conference name field 222 reflects the title of the conference information, the time field 223 reflects the start time and end time, and the location field 224 reflects the location. Furthermore, the conference information and recording information in the conference management system are associated with each other by a conference ID.

[0277] Once the remote conference is over and the user stops recording, a combined image video with audio is created.

[0278] FIG. 33 is an example of the conference list screen 230 displayed by the information recording application 41. The conference list screen 230 is a list of conferences, and can display a list of recorded information recorded during remote conferences. It also includes conferences that were held only in a certain conference room, regardless of whether they were remote conferences. The conference list screen 230 displays conference information that the logged-in user has permission to view in the conference information storage unit 5001. Information on video recording information stored in the information storage unit 1001 may be further integrated.

[0279] The conference list screen 230 is displayed when the user selects the conference list tab 231 on the initial screen 200 in FIG. 30. The conference list screen 230 displays a list 236 of recorded information that the user has permission to view. The conference creator (minutes creator) can set viewing permissions for participants. The conference list may be a list of stored recorded information or a list of conference schedules and conference data.

[0280] The conference list screen 230 has the following items: a check box 232 , an update date and time 233 , a title 234 , and a status 235 .

[0281] The checkboxes 232 are used to accept the selection of video files. The checkboxes 232 are used when the user wants to delete video files all at once.

[0282] The update date and time 233 indicates the start and end times of recording the combined image video. If the video has been edited, the edit date and time may be used.

[0283] The title 234 is the title of the meeting (e.g., agenda). It may be transcribed from the meeting information or set by the user.

[0284] The status 235 indicates whether the combined image video has been uploaded to the information processing system 50. If it has not been uploaded, "Local PC" is displayed, and if it has been uploaded, "Uploaded" is displayed. If it has not been uploaded, an upload button is displayed. If there is a combined image video that has not been uploaded, it is recommended that the information recording application 41 automatically uploads it when the user logs in to the information processing system 50.

[0285] When the user selects an arbitrary title or the like from the list 236 of combined image videos with a pointing device, the information recording application 41 displays a recording playback screen, but this is not shown in this embodiment. On the recording playback screen, it is possible to play back the combined image video, etc.

[0286] It is desirable that the user be able to narrow down the conferences based on update date and time, title, keywords, etc. Also, if there are a large number of conferences displayed and it is difficult to find the relevant conference, it is desirable that a search function be provided that allows the user to input a phrase to narrow down the recorded information based on words contained in the remarks made at the conference, the title, etc. The search function allows the user to find the desired recorded information in a short time even when there is a large amount of recorded information. Furthermore, the conference list screen 230 may allow the user to sort the conferences by update date and time or title.

[0287] <Saving combined image video> Next, the process of saving the combined image moving image will be described with reference to Fig. 34. Fig. 34 is an example of a sequence diagram showing the procedure in which the information recording application 41 records the panoramic image, the speaker image, and the application screen.

[0288] S61: The user operates the teleconferencing application 42 to start a teleconferencing. Here, it is assumed that the teleconferencing application 42 at the user's own location 102 and the teleconferencing application 42 at the other location 101 have started a teleconferencing. The teleconferencing application 42 at the user's own location 102 transmits images captured by a camera and audio collected by a microphone of the terminal device 10 to the teleconferencing application 42 at the other location 101. The teleconferencing application 42 at the other location 101 displays the received images on a display and outputs the received audio from a speaker. Similarly, the teleconferencing application 42 at the other location 101 transmits images captured by a camera and audio collected by a microphone of the terminal device 10 to the teleconferencing application 42 at the user's own location 102. The teleconferencing application 42 at the user's own location 102 displays the received images on a display and outputs the received audio from a speaker. Each teleconferencing application 42 repeats this process to realize the teleconferencing.

[0289] S62: The user makes settings related to recording on the recording setting screen 210 of the information recording application 41 shown in Fig. 31. The operation accepting unit 12 of the information recording application 41 accepts the settings. Here, it is assumed that the camera toggle button 211 and the PC screen toggle button 212 are both on.

[0290] If the user has reserved a remote conference in advance, the user can display a list of remote conferences by pressing the Get Information button 221 on the calendar in Fig. 32 and select a remote conference to associate with the recorded video. Since the user has already logged in to the information processing system 50, the information processing system 50 identifies remote conferences that the logged-in user has permission to view. The information processing system 50 transmits a list of the identified remote conferences to the terminal device 10, and the user selects a remote conference that is currently being held or will be held in the future. This determines information about the remote conference, such as the conference ID.

[0291] In addition, even if the user has not reserved a remote conference in advance, the user can create a conference when creating a combined image video. The following describes a case where the information recording application 41 creates a conference when creating a combined image video and acquires a conference ID from the information processing system 50.

[0292] S63: The user instructs the information recording application 41 to start recording (start recording now button 216). The operation reception unit 12 of the information recording application 41 receives the instruction. The display control unit 13 displays the recording screen 220.

[0293] S64: Since the remote conference has not been selected (the conference ID has not been determined), the communication unit 11 of the information recording application 41 transmits a remote conference creation request to the information processing system 50.

[0294] S65: The communication unit 51 of the information processing system 50 receives the remote conference creation request, the communication management unit 54 acquires a unique conference ID assigned by the conference management system 9, and the communication unit 51 transmits the conference ID to the information recording application 41.

[0295] S66: The communication manager 54 also transmits the destination (URL of the storage service system 70) of the combined image moving image to the information recording application 41 via the communication unit 51.

[0296] S67: The communication unit 11 of the information recording application 41 receives the conference ID and the destination to save the video file, and transmits the conference ID to the electronic whiteboard 2. The communication unit 11 may transmit the conference ID via the information processing system 50 or directly.

[0297] S68: When the communication unit 11 of the information recording application 41 receives the conference ID and the destination to save the video file, the recording control unit 17 determines that preparations for recording are complete and starts recording.

[0298] S69: The application screen acquisition unit 14 of the information recording application 41 requests the application screen selected by the user (more specifically, the application screen acquisition unit 14 acquires the application screen via the OS). In FIG. 34, the application selected by the user is the remote conference application 42.

[0299] S70: The recording control unit 17 of the information recording application 41 notifies the meeting device 60 of the start of recording via the device communication unit 16. When notifying, the recording control unit 17 may also notify that the camera toggle button 211 is on (requesting a panoramic image and a speaker image). Regardless of whether a request is made, the meeting device 60 sends the panoramic image and the speaker image to the information recording application 41.

[0300] S71: When the terminal communication unit 61 of the meeting device 60 receives a recording start command, it assigns a unique recording ID and returns the recording ID to the information recording application 41. The recording ID may be assigned by the information recording application 41 or may be obtained from the information processing system 50.

[0301] S72: The voice acquisition unit 15 of the information recording application 41 acquires the voice data output by the terminal device 10 (voice data received by the remote conference application 42).

[0302] S73: The device communication unit 16 transmits the voice data acquired by the voice acquisition unit 15 and a synthesis request to the meeting device 60.

[0303] S74: The terminal communication unit 61 of the meeting device 60 receives the voice data and the synthesis request, and the voice synthesis unit 65 synthesizes the received voice data with the surrounding voice data collected by the sound collection unit 64. For example, the voice synthesis unit 65 adds the two pieces of voice data together. Because clear voices from around the meeting device 60 are recorded, the accuracy of converting voices from around the meeting device 60 (conference room side) into text is improved.

[0304] This audio synthesis can also be performed by the terminal device 10. However, by distributing the recording function to the terminal device 10 and the audio processing to the meeting device 60, the load on the terminal device 10 and the meeting device 60 can be reduced. The recording function may be distributed to the meeting device 60 and the audio processing may be distributed to the terminal device 10.

[0305] S75: Furthermore, the panoramic image generator 62 of the meeting device 60 generates a panoramic image, and the speaker image generator 63 generates a speaker image.

[0306] S76: The device communication unit 16 of the information recording application 41 repeatedly acquires the panoramic image and the speaker image from the meeting device 60. The device communication unit 16 also repeatedly requests and acquires the synthesized audio data from the meeting device 60. These acquisitions may be performed by the device communication unit 16 making a request to the meeting device 60. Alternatively, the meeting device 60 may automatically send the panoramic image and the speaker image upon receiving a notification that the camera toggle button 211 is on. The meeting device 60 may automatically send the synthesized audio data to the information recording application 41 upon receiving a request to synthesize audio data.

[0307] S77: The recording control unit 17 of the information recording application 41 creates a combined image by arranging the application screen, panoramic image, and speaker image acquired from the remote conference application 42. The recording control unit 17 repeatedly creates combined images and creates a combined image video by specifying each combined image as a frame that makes up the video. The recording control unit 17 also saves the audio data received from the meeting device 60.

[0308] The information recording application 41 repeats the above steps S72 to S77. During steps S72 to S77, the process of Fig. 17 (generation and transmission of the operator image 7) is performed as needed.

[0309] S78: When the remote conference ends and recording is no longer necessary, the user instructs the information recording application 41 to end recording (for example, by pressing the recording end button 227). The operation reception unit 12 of the information recording application 41 receives the instruction.

[0310] S79: The device communication unit 16 of the information recording application 41 notifies the meeting device 60 that recording has ended. The meeting device 60 continues creating panoramic images and speaker images and synthesizing audio. However, the meeting device 60 may change the processing options, such as changing the resolution or fps, depending on whether recording is in progress.

[0311] S80: The recording control unit 17 of the information recording application 41 combines the audio data with the combined image moving image to create a combined image moving image with audio.

[0312] S81: Furthermore, if the user checks the checkbox 215 associated with "Automatically transcribe recording after uploading" on the recording setting screen 210, the audio data processing unit 18 requests the information processing system 50 to convert the audio data into text data. In detail, the audio data processing unit 18 specifies the URL of the save destination via the communication unit 11, and transmits a request to the information processing system 50 to convert the audio data combined into the combined image video together with the conference ID and recording ID.

[0313] S82: The communication unit 51 of the information processing system 50 receives a request to convert the voice data, and the text conversion unit 56 converts the voice data into text data using the voice recognition service system 80. The communication unit 51 saves the text data in the same storage destination (the URL of the storage service system 70) as the storage destination of the combined image video. In the recorded video information storage unit 5002, the text data is associated with the combined image video by the conference ID and the recorded video ID. The text data may be managed by the communication management unit 54 of the information processing system 50 and stored in the storage unit 5000. Alternatively, the terminal device 10 may request voice recognition from the voice recognition service system 80 and store the text data obtained from the voice recognition service system 80 in a storage destination. The voice recognition service system 80 returns the converted text data to the information processing system 50, or may directly send it to the URL of the storage destination. The voice recognition service system 80 may select or switch from multiple services depending on setting information set by the user in the information processing system 50.

[0314] S83: The upload unit 20 of the information recording application 41 also saves the combined image moving image in the destination for saving the combined image moving image via the communication unit 11. The combined image moving image is associated with the conference ID and the recording ID in the recording information storage unit 5002. The combined image moving image is recorded as having been uploaded.

[0315] S84: The user inputs the end of the meeting into the electronic whiteboard 2. The user may input the end of the meeting into the terminal device 10, and the end of the meeting may be transmitted from the terminal device 10 to the electronic whiteboard 2. In this case, the end of the meeting may be transmitted to the electronic whiteboard 2 via the information processing system 50.

[0316] S85: The communication unit 36 ​​of the electronic whiteboard 2 specifies the conference ID and transmits saved object data displayed (for example, handwritten) during the conference to the information processing system 50. The object data itself is shared in real time by the information processing system 50. The communication unit 36 ​​may also transmit device identification information of the electronic whiteboard 2 to the information processing system 50. In this case, the conference ID is identified by the association information.

[0317] S86: The information processing system 50 stores the object data in the same storage location as the combined image video, etc., based on the conference ID.

[0318] The save destination is notified to the user, so the user can share the combined image video with participants by informing them of the save destination by email, etc. Even if the combined image video, audio data, text data, and object data are created using different devices, they can be collected and stored in a single storage location, making them easy for users to view later.

[0319] The processing of steps S72 to S77 does not have to be performed in the order shown in FIG. 34, and the synthesis of the audio data and the creation of the combined image may be performed in the reverse order.

[0320] <Major Effects> According to this embodiment, the operator image 7 is displayed near the stroke, so that users at other locations can easily understand who is writing (or has written) the stroke. Also, in the information processing system 50, the operator image 7 is associated with the object data, so that it becomes clear who input the object, improving the content of the minutes. [Example]

[0321] In this embodiment, a case will be described in which the electronic whiteboard 2 and the terminal device 10 do not communicate directly with each other, but rather send and receive information via the information processing system 50. Even if the electronic whiteboard 2 and the terminal device 10 cannot communicate directly with each other, the terminal device 8 at another location can display the operator image 7.

[0322] In this embodiment, it is assumed that the hardware configuration diagrams of FIGS. 4, 5, and 8 and the functional block diagram shown in FIG. 9 described in the above embodiments can be used.

[0323] 35 is a sequence diagram illustrating a process in which a terminal device 8 at another location displays handwritten strokes on the electronic whiteboard 2 by a user together with an operator image 7. The explanation of FIG. 35 will mainly focus on the differences from FIG. 17.

[0324] S91, S92: A user who has participated in the meeting writes by hand with the electronic pen 490 or displays the PC screen on the electronic whiteboard 2. Since the electronic whiteboard 2 has been operated, the communication unit 36 ​​of the electronic whiteboard 2 transmits a notification that the operation has started together with the device identification information of the electronic whiteboard 2 to the information processing system 50.

[0325] S93: The communication unit 51 of the information processing system 50 transmits a command to start operation to the meeting device 60 (i.e., the terminal device 10) associated with the device identification information of the interactive whiteboard 2. The terminal device 10 repeatedly polls the information processing system 50 together with the conference ID and receives communication from the information processing system 50. Alternatively, the terminal device 10 may communicate with the information processing system 50 via two-way communication such as WebSocket.

[0326] The processes in S94 to S96 may be the same as those in S3 to S5 in FIG.

[0327] S97: The user finishes writing.

[0328] S98: The operation detection unit 38 of the electronic whiteboard 2 detects the end of the operation (a request to save the cropped image of the electronic whiteboard direction). In response to the end of the operation, the operator image receiving unit 40b of the electronic whiteboard 2 requests the operator image 7 from the information processing system 50 via the communication unit 36, specifying the device identification information of the electronic whiteboard 2.

[0329] S99: The communication unit 51 of the information processing system 50 receives the device identification information of the electronic whiteboard 2 and a request for the operator image 7, and the object sharing unit 57 requests the operator image 7 from the terminal device 10 connected to the meeting device 60 associated with the electronic whiteboard 2 via the communication unit 51.

[0330] S100: The communication unit 11 of the terminal device 10 receives the request for the operator image 7, and the operator image transmission unit 24 transmits the operator image 7 to the information processing system 50 via the communication unit 11.

[0331] S101: The communication unit 51 of the information processing system 50 receives the operator image 7, and the object sharing unit 57 assigns a number to the identification information of the operator image 7. The object sharing unit 57 associates the identification information of the operator image 7 with the operator image 7 and stores them. In addition, the object sharing unit 57 transmits the identification information of the operator image 7 via the communication unit 51 to the interactive whiteboard 2 that transmitted the operation end.

[0332] S102: The communication unit 36 ​​of the electronic whiteboard 2 has received the identification information of the operator image 7, so the object upload unit 40a transmits the conference ID, object data, and the identification information of the operator image 7 to the information processing system 50 via the communication unit 36.

[0333] S103: The communication unit 51 of the information processing system 50 receives the conference ID, the object data, and the identification information of the operator image 7, and the object sharing unit 57 associates the object data, the identification information of the operator image 7, and the operator image 7 and stores them. The object ID may be assigned by the electronic whiteboard 2 or the information processing system 50.

[0334] S104: The object sharing unit 57 also transmits the object data and the identification information of the operator image 7 to the terminal devices 8 at other locations participating in the same conference identified by the conference ID via the communication unit 51. The subsequent process is the same as in FIG.

[0335] <Major Effects> According to this embodiment, in addition to the effects of the first embodiment, even if the terminal device 10 and the interactive whiteboard 2 cannot communicate directly, the operator image 7 is transmitted from the terminal device 10 to the interactive whiteboard 2 via the information processing system 50. [Example]

[0336] In this embodiment, a case will be described in which a user writes strokes consecutively, and then a different user writes strokes. When consecutive processes are assumed as in this embodiment, the timing of image acquisition, etc. differs from those in the first and second embodiments.

[0337] In this embodiment, it is assumed that the hardware configuration diagrams of FIGS. 4, 5, and 8 and the functional block diagram shown in FIG. 9 described in the above embodiments can be used.

[0338] 36 is a sequence diagram illustrating a process in which, when a user 1 writes handwriting on the electronic whiteboard 2 and then a different user 2 writes handwriting, a terminal device 8 at another location displays the strokes together with an operator image 7. First, the processes in steps S111 to S116 are the same as those in steps S91 to S96 in FIG.

[0339] Furthermore, the processing of steps S117 to S120 may be similar to S98 to S101 in Figure 35, but in Figure 36, it is assumed that strokes will be processed continuously, so the electronic whiteboard 2 acquires the identification information of the operator image 7 without waiting for the operation to end.

[0340] S121: The communication unit 36 ​​of the interactive whiteboard 2 receives the identification information of the operator image 7, and stores it in the object data storage unit 3002. The identification information of the operator image 7 is assumed to be 1 here for the sake of convenience.

[0341] S122: The user finishes writing.

[0342] S123: The operation detection unit 38 of the interactive whiteboard 2 detects the end of the operation, and the object upload unit 40a transmits the identification information of the operator image 7 and the object data to the information processing system 50 via the communication unit 36.

[0343] S124: The communication unit 51 of the information processing system 50 receives the identification information and object data of the operator image 7, and the object sharing unit 57 assigns an object ID (which may be assigned by the electronic whiteboard 2) and stores it in the object data storage unit 5005 in association with the identification information of the operator image 7.

[0344] S125: The object sharing unit 57 also transmits the object data and the identification information of the operator image 7 to the terminal devices 8 at other locations participating in the same conference via the communication unit 51. The process thereafter is the same as in FIG.

[0345] S126: Next, user 1 starts writing stroke 2. Because the input starts (pen down) within a certain time after the input of stroke 1 (the last stroke) ends (pen up), the operator image receiving unit 40b determines that stroke 2 was written by the same user and does not acquire the operator image 7. Because the operator image receiving unit 40b holds the ID=1 acquired for stroke 1, it can use the same operator image 7.

[0346] S127 to S130: User 1 finishes writing stroke 2. This process is the same as steps S122 to S125. For the identification information of the operator image 7, ID=1 is used.

[0347] S131: User 2 starts writing stroke 3. Because this input start (pen down) occurs a certain amount of time after the input end (pen up) of stroke 2 (the last stroke), the operator image receiving unit 40b determines that user 2 is a different user and acquires the identification information of the operator image 7. The operator image receiving unit 40b determines that the person writing may have been replaced by another user, and therefore determines that the image needs to be updated.

[0348] In addition to the operator image receiving unit 40b detecting a change in operator based on whether a certain amount of time has passed between pen-up and pen-down, the change in operator may also be detected by image processing. Although the face of the user operating the interactive whiteboard 2 is not captured on the meeting device 60, the operator image receiving unit 40b can detect a change in operator based on the characteristics of the back view (height, hairstyle, clothing, etc.).

[0349] In steps S132 to S141, the same processes as in steps S112 to S121 are performed, except that a new number is assigned to the identification information of the operator image 7, and ID=2 is assigned.

[0350] S142 to S145: User 2 finishes writing stroke 3. This process is the same as steps S122 to S125. For the identification information of the operator image 7, ID=2 is used.

[0351] In this way, according to this embodiment, the interactive whiteboard 2 can estimate that the user has been replaced and acquire the identification information of the operator image 7 each time the user has been replaced.

[0352] Figure 37 shows an operator image 7 displayed by a terminal device 8 at another location when two users 1 and 2 are operating the electronic whiteboard 2. Figure 37(a) shows two users 1 and 2 operating the electronic whiteboard 2. Figure 37(b) shows an operator image 7 displayed by a terminal device 8 at another location. As explained in Figure 36, while user 1 is handwriting the stroke 120 for "a," the terminal device 8 at the other location displays the operator image 7 only once, and then erases it after a certain period of time has passed.

[0353] Next, when the user 2 starts to handwrite the stroke 122 "i", the terminal device 8 at the other location acquires and displays a new operator image 7. If the operator image 7 associated with "a" is currently being displayed, the display control unit 73 erases the currently displayed operator image 7 (FIG. 36(b)) and displays the operator image 7 associated with "i" (FIG. 36(c)).

[0354] <Processing of terminal devices at other locations> Next, a process in which the terminal device 8 at the other location displays the operator image 7 will be described with reference to Fig. 38. Fig. 38 is a flowchart illustrating a process in which the terminal device 8 at the other location receives the operator image 7 and displays the operator image 7. The following process is performed by the terminal device 8 at the other location.

[0355] First, the display control unit 73 displays the strokes and the operator image 7 received from the information processing system 50 (S201).

[0356] Next, the communication unit 71 determines whether the next stroke data and the identification information of the operator image 7 have been received (S202). The stroke data is accompanied by the identification information of the operator image 7, but as described above, the identification information of the operator image 7 may or may not be the same. If the determination in step S202 is No, the process proceeds to step S206.

[0357] If the determination in step S202 is Yes, the operator image obtaining unit 74 determines whether the identification information of the operator image 7 is the same (S203).

[0358] If the identification information of the operator image 7 is the same as that of the currently displayed operator image 7 (Yes in S203), there is no need to acquire the operator image 7, and the process proceeds to step S205.

[0359] If the identification information of the operator image 7 is not the same as that of the currently displayed operator image 7 (No in S203), the display control unit 73 switches the operator image 7 (S204). That is, the operator image acquisition unit 74 acquires the operator image 7 from the information processing system 50 by specifying the identification information of the operator image 7 via the communication unit 71. Note that even if the identification information of the operator image 7 is different, the operator image 7 linked to the newly added stroke may show the same operator as in the operator image 7 before switching, or may show a different operator.

[0360] Furthermore, when a certain time has elapsed since the display of the operator image 7 (Yes in S205), the display control unit 73 erases the operator image 7 (S206).

[0361] The terminal device 8 at the other site keeps the operator image 7 linked to the object data even after erasing the operator image 7 that was once displayed, so that the operator image 7 can be displayed again. In this way, for example, when a user touches a stroke, the image of the operator can be displayed again.

[0362] FIG. 39 shows a mark 130 indicating that an operator image 7 is attached to a stroke 120 displayed by a terminal device 8 at another location. As described above, the operator image 7 is erased after a certain time has passed since it was displayed, but there may be cases where the user wants to display the operator image 7 again. When there is a stroke 120 for which the user wants to display the operator image 7, the user displays the mark 130 by pressing it with a pointing device 131 such as a mouse, pen, or finger. When the user presses the mark 130 with the pointing device 131, the display control unit displays the operator image 7 associated with the stroke 120. This operator image 7 is stored in the terminal device 8 at the other location if a conference is in progress. The user can check the operator of an object at any time.

[0363] Note that the display control unit 73 may display the operator image 7 again when the user simply presses the stroke 120.

[0364] After the conference ends, if a terminal device 8 at another location displays an object acquired from the information processing system 50, the operator image 7 may not be transmitted to the terminal device 8 at the other location together with the object data because the operator image 7 has a large capacity. In this case, the operator image acquisition unit 74 can acquire the operator image 7 from the information processing system 50 based on the identification information of the operator image 7.

[0365] The mark 130 is an example, and the stroke 120 associated with the operator image 7 may blink or change color. The mark 130 may also be displayed by hovering the mouse over the pointing device 131 instead of pressing it.

[0366] <Input display area> 40 is a diagram illustrating an input display area. The object sharing unit 57 shares the input display area, object data, and operator image 7 between the terminal devices 8 and the electronic whiteboards 2 participating in the same conference. As shown in FIGS. 40(a) and 40(b), the object sharing unit 57 shares the input display area 270 between the electronic whiteboard 2 and the terminal device 8, and accepts object input to the input display area 270 on the electronic whiteboard 2 or the terminal device 8. The object sharing unit 57 transmits an object transmitted from one electronic whiteboard 2 to the other electronic whiteboard 2 in response to the object input, and vice versa.

[0367] <Other application examples> The best mode for carrying out the present invention has been described above using examples, but the present invention is not limited to these examples in any way, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention.

[0368] For example, the range of the operator image in the panoramic image may be determined by detecting, through image processing, a person near the electronic whiteboard 2. Alternatively, a meeting device, a terminal device, or the like may use image processing to detect that a person near the electronic whiteboard 2 is writing by hand on the electronic whiteboard 2.

[0369] The terminal device 10 and the meeting device 60 may be integrated. The meeting device 60 may be externally attached to the terminal device 10. The meeting device 60 may be an omnidirectional camera, a microphone, and a speaker connected by a cable.

[0370] Furthermore, a meeting device 60 may also be placed at another location 101. The other location 101 separately uses the meeting device 60 to create combined image video and text data. Furthermore, multiple meeting devices 60 may be placed at one location. In this case, multiple pieces of record information are created for each meeting device 60.

[0371] The layout of the panoramic image 203, the speaker image 204, and the app screen in the combined image video used in this embodiment is merely an example. The panoramic image 203 may be on the bottom and the speaker image 204 on top, or the user may change the layout, or may individually switch between displaying and hiding the panoramic image 203 and the speaker image 204 during playback.

[0372] Furthermore, the configuration examples in Figure 9 and the like are divided according to main functions to facilitate understanding of the processing by the terminal device 10, the meeting device 60, and the information processing system 50. The method of dividing the processing units and the names thereof do not limit the present invention. The processing by the terminal device 10, the meeting device 60, and the information processing system 50 can be divided into even more processing units depending on the processing content. Furthermore, the processing can also be divided so that one processing unit includes even more processes.

[0373] Additionally, the devices described in the examples are merely illustrative of one of several computing environments for implementing the embodiments disclosed herein. In one embodiment, information processing system 50 includes multiple computing devices, such as a server cluster, configured to communicate with each other via any type of communication link, including a network, shared memory, etc., and to perform the processes disclosed herein.

[0374] Furthermore, the information processing system 50 can be configured to share the disclosed processing steps, such as those shown in FIG. 17, in various combinations. For example, a process executed by a specific unit can be executed by multiple information processing devices included in the information processing system 50. Furthermore, the information processing system 50 may be integrated into a single server device, or may be divided into multiple devices.

[0375] Each function of the above-described embodiments can be realized by one or more processing circuits. Here, the term "processing circuit" in this specification includes a processor programmed to perform each function by software, such as a processor implemented by an electronic circuit, as well as devices such as an ASIC (Application Specific Integrated Circuit), a DSP (Digital Signal Processor), an FPGA (Field Programmable Gate Array), and a conventional circuit module designed to perform each of the above-described functions. [Explanation of symbols]

[0376] 10 Terminal Equipment 50 Information Processing Systems 60 Meeting Devices 100 Recording Information Creation System [Prior art documents] [Patent documents]

[0377] [Patent Document 1] Japanese Patent Application Laid-Open No. 2016-122226

Claims

1. an imaging unit that captures an image of the surroundings; a detection unit that detects a predetermined behavior of a user; an image generation unit that generates an image of a partial area from the surrounding image based on a specific image included in the surrounding image in response to the predetermined action; a display unit that displays an image of the partial area; the image generation unit generates an image of the partial region based on the ranges indicated by the plurality of specific images; An equipment system characterized by:

2. the plurality of specific images represent ranges of the right and left edges of the image of the partial region, the image generation unit generates an image of the partial area including the plurality of specific images; 2. The equipment system according to claim 1.

3. The plurality of specific images are two-dimensional codes.

2. The equipment system according to claim 1.

4. the plurality of specific images are displayed on an object included in an image of the partial region generated based on the plurality of specific images; 2. The equipment system according to claim 1.

5. The detection unit detects a predetermined action of the user with respect to the object.

5. The equipment system according to claim 4.

6. the detection unit detects a user instruction to terminate generation of an image of the partial area based on the plurality of specific images included in the surrounding image; 2. The equipment system according to claim 1.

7. A device connected to a terminal device having a display unit, an imaging unit that captures an image of the surroundings; a detection unit that detects a predetermined behavior of a user; an image generation unit that generates an image of a partial area from the surrounding image based on a specific image included in the surrounding image in response to the predetermined action; a transfer unit that transfers the image of the partial area to the terminal device. the image generation unit generates an image of the partial region based on the ranges indicated by the plurality of specific images; A device characterized by:

8. the plurality of specific images represent ranges of the right and left edges of the image of the partial region, the image generation unit generates an image of the partial area including the plurality of specific images; 8. The device according to claim 7 .

9. The plurality of specific images are two-dimensional codes.

8. The device according to claim 7 .

10. the plurality of specific images are displayed on an object included in an image of the partial region generated based on the plurality of specific images; 8. The device of claim 7.

11. The detection unit detects a predetermined action of the user with respect to the object.

11. The device of claim 10.

12. the detection unit detects a user instruction to terminate generation of an image of the partial area based on the plurality of specific images included in the surrounding image; 8. The device of claim 7.

13. A device connected to a terminal device having a display unit, an imaging unit that captures an image of the surroundings; a detection unit that detects a predetermined behavior of a user; an image generation unit that generates a cut-out image by cutting out the surrounding image based on a specific image included in the surrounding image in response to the predetermined action; a transfer unit that transfers the extracted image to the terminal device. the image generation unit generates the clipped image based on ranges indicated by the plurality of specific images; A device characterized by:

14. the plurality of specific images represent ranges between the right and left ends of the cut-out image; the image generation unit generates the clipped image including a plurality of the specific images.

14. The device of claim 13.

15. The plurality of specific images are two-dimensional codes.

14. The device of claim 13.

16. the plurality of specific images are displayed on an object included in the cut-out image generated based on the plurality of specific images.

14. The device of claim 13.

17. The detection unit detects a predetermined action of the user with respect to the object.

17. The device of claim 16.

18. the detection unit detects a user instruction to terminate generation of the clipped image based on the plurality of specific images included in the surrounding image; 14. The device of claim 13.

19. A terminal device including a display unit, an acquisition unit that acquires the surrounding image captured by the imaging unit; an image generation unit that generates a cut-out image by cutting out the surrounding image based on a specific image included in the surrounding image in response to a predetermined action of the user; a control unit that displays the cut-out image on the display unit; the image generation unit generates the clipped image based on ranges indicated by the plurality of specific images; A terminal device characterized by:

20. An image generation method performed by a device connected to a terminal device having a display unit, A process in which an imaging unit captures a surrounding image; A process in which a detection unit detects a predetermined behavior of a user; an image generation unit generating an image of a partial area from the surrounding image based on a specific image included in the surrounding image in accordance with the predetermined action; a transfer unit for transferring the image of the partial area to the terminal device; 、 the image generation unit generates an image of the partial region based on the ranges indicated by the plurality of specific images; An image generating method comprising:

21. A device connected to a terminal device having a display unit, an imaging unit that captures an image of the surroundings; a detection unit that detects a predetermined behavior of a user; an image generation unit that generates an image of a partial area from the surrounding image based on a specific image included in the surrounding image in response to the predetermined action; a transfer unit that transfers the image of the partial area to the terminal device; the image generation unit generates an image of the partial region based on the ranges indicated by the plurality of specific images; A program characterized by:

Images