Communication system, voice processing method, imaging apparatus, program, and communication processing system

Abstract

To provide a communication system, voice processing method, imaging apparatus, program, and communication processing system for performing appropriate voice processing on sound information of a transmission source.SOLUTION: A communication processing system 5 is a communication system having a plurality of bases for transmitting / receiving images and voices and is a server computer. The communication processing system includes: a communication unit for receiving a wide field image and sound information acquired using an imaging apparatus at a first base such as a construction field; a parameter determination unit for determining a voice processing parameter made to correspond to a utilization scene determined on the basis of the wide field image received by the communication unit; and a voice processing unit for performing voice processing on the sound information using the voice processing parameter determined by the parameter determination unit. A communication terminal 7 at the first base includes a sound output control unit for outputting the sound information on which the voice processing is performed by the voice processing unit to a communication terminal 9 of a second base.SELECTED DRAWING: Figure 14

Description

[Technical Field]

[0001] The present invention relates to a communication system, an audio processing method, an imaging device, a program, and a communication processing system. [Background technology]

[0002] There is a known communication system in which images and audio are transmitted in real time from one location to one or more other locations, enabling remote communication between users in remote locations using images and audio. For example, a user at one location can remotely check the specific status of work being done at another location based on the images and audio.

[0003] There is known a technology for subjecting audio to audio processing suited to the environment of the audio source (see, for example, Patent Document 1). Patent Document 1 discloses a technology for associating types of background noise, such as the sound of a running vehicle, an emergency vehicle, a passing train, a railroad crossing, a construction site, or a coffee shop, with a noise canceling method to be applied in that environment, and for processing audio using a noise canceling method suited to the environment. Summary of the Invention [Problem to be solved by the invention]

[0004] However, conventional technologies remove background sounds from the audio source using a system that assumes a typical environment, which can result in the removal of necessary sounds when broadcasting from a site where on-site sounds are required. First, the sound information from the source includes background sounds in addition to human voices. For example, on-site sounds may be important when a remote user checks the specific status of work at a construction site. Conventional technologies determine that construction site noise is an extremely common noise and should be removed, resulting in the construction site sounds not being played back at the remote location even if the remote user determines that they are necessary. While disabling the noise removal function allows the remote user to check the sounds at the construction site, it is desirable to remove noise unrelated to sounds that represent the specific status of work, and disabling this function is hardly a solution.

[0005] In view of the above-mentioned problems, the present invention provides a technology for performing appropriate audio processing on sound information from a transmission source. [Means for solving the problem]

[0006] In view of the above problems, the present invention provides a communication system for transmitting and receiving images and audio between multiple locations, comprising: a communication unit that receives a wide-field image and audio information acquired at a first location; a parameter determination unit that determines audio processing parameters associated with a usage scene determined based on the wide-field image received by the communication unit; an audio processing unit that processes the audio information using the audio processing parameters determined by the parameter determination unit; and an audio output control unit that outputs the audio information that has been audio-processed by the audio processing unit to a second location. [Effects of the Invention]

[0007] The present invention can perform audio processing appropriate for the sound information of the sender. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating an outline of voice processing by a communication processing system. [Figure 2]1A is a left side view of the imaging device, FIG. 1B is a front view of the imaging device, and FIG. 1C is a plan view of the imaging device. [Figure 3] FIG. 1 is a conceptual diagram of the imaging device in use. [Figure 4] (a) is a hemispherical image (front) captured by an imaging device, (b) is a hemispherical image (back) captured by an imaging device, and (c) is an image represented by the Mercator projection. [Figure 5] (a) A conceptual diagram showing how a sphere is covered with a Mercator image, and (b) a diagram showing a spherical image. [Figure 6] FIG. 10 is a diagram showing the positions of a virtual camera and a predetermined area when the celestial sphere image is a three-dimensional sphere. [Figure 7] (a) is a three-dimensional oblique view of Figure 6, (b) is a diagram showing the predetermined area image in the state of (a) displayed on the display, (c) is a diagram showing the predetermined area after the viewpoint of the virtual camera IC in (a) has been changed, and (d) is a diagram showing the predetermined area image in the state of (c) displayed on the display. [Figure 8] FIG. 1 illustrates points in three-dimensional Euclidean space in spherical coordinates. [Figure 9] FIG. 10 is a conceptual diagram showing the relationship between a predetermined area and a point of interest. [Figure 10] 1 is a schematic diagram of a communication system according to an embodiment of the present invention; [Figure 11] FIG. 2 is a diagram illustrating a hardware configuration of the imaging device. [Figure 12] FIG. 2 is a hardware configuration diagram of a relay device. [Figure 13] FIG. 2 is a hardware configuration diagram of a communication processing system and a communication terminal. [Figure 14] FIG. 2 is a diagram illustrating the functional configuration of the communication system. [Figure 15] FIG. 1 is a conceptual diagram of a user device management DB. [Figure 16] FIG. 10 is a conceptual diagram of a virtual room management DB. [Figure 17] FIG. 10 is a conceptual diagram of a viewpoint information management DB. [Figure 18] FIG. 10 is a conceptual diagram of a usage scene information DB. [Figure 19] FIG. 2 is a conceptual diagram of a voice processing parameter information DB. [Figure 20] FIG. 10 is a diagram showing an example of conference information registered in a conference scheduler. [Figure 21] FIG. 10 is a sequence diagram showing a communication process of a wide-field image and each viewpoint information in the communication system. [Figure 22] 10 is a flowchart showing a process in which the communication terminal of the observer displays a predetermined area image. [Figure 23] 10 is an explanatory diagram illustrating a case where a predetermined area T2 of a predetermined area image currently being displayed on another communication terminal 9 is superimposed on a predetermined area image of a predetermined area T1 currently being displayed on a communication terminal 7. FIG. [Figure 24] FIG. 10 is a diagram showing a predetermined area image on the observer side, displaying a viewpoint display area indicating the predetermined area being viewed by each viewer. [Figure 25] FIG. 10 is a diagram showing a predetermined area image on the observer side, displaying a viewpoint display area indicating the predetermined area being viewed by each viewer. [Figure 26] FIG. 10 is a diagram showing a predetermined area image on the observer side, displaying a viewpoint display area indicating the predetermined area being viewed by each viewer. [Figure 27] FIG. 10 is a diagram showing a predetermined area image on the observer side, displaying a viewpoint display area indicating the predetermined area being viewed by each viewer. [Figure 28] FIG. 10 is a diagram showing a predetermined area image on the observer side, displaying a viewpoint display area indicating the predetermined area being viewed by each viewer. [Figure 29] FIG. 1 is a diagram illustrating an example of the configuration of a discrimination model for usage scenes using CNN. [Figure 30] FIG. 2 is an example of a functional block diagram of a learning unit. [Figure 31] 10 is an example of a sequence diagram illustrating a process in which the communication processing system determines a usage scene based on a communication status of image data or sound information and performs audio processing. [Figure 32] 10 is an example of a sequence diagram illustrating a process in which the communication processing system determines a usage scene based on a voice instruction from a participant and performs voice processing. [Figure 33]10 is an example of a sequence diagram illustrating a process in which the communication processing system determines a usage scene based on a communication status of image data or sound information and performs audio processing. [Figure 34] 19 is an example of a flowchart illustrating a communication processing system performing voice processing using the voice processing parameters of FIG. 18. [Figure 35] 10 is a sequence diagram illustrating an example of a case in which the imaging device determines a usage scene and performs audio processing. [Figure 36] 10 is an example of a sequence diagram in which the imaging device determines the usage scene and the communication processing system performs audio processing. DETAILED DESCRIPTION OF THE INVENTION

[0009] A communication system and a voice processing method performed in the communication system will be described below as an example of an embodiment of the present invention.

[0010] <Outline of voice processing> The communication processing system of this embodiment changes voice processing parameters depending on the usage scenario. Even sounds that are generally considered noise may be necessary sounds at construction sites or inspection sites. By changing the voice processing parameters depending on the usage scenario of the communication system, even sounds that are generally considered noise are not removed, so that participants in remote locations can hear necessary sounds at construction sites or inspection sites. Therefore, the quality of remote communications such as remote conferences (e.g., teleconferences, television conferences, video conferences) can be improved.

[0011] FIG. 1 is a diagram illustrating an overview of voice processing by a communication processing system. The communication processing system determines a usage scenario for the communication system (S1). For example, in the case of remote communication that is conducted periodically, previous usage scenarios can be used. The communication processing system can also determine the usage scenario based on image data. If both participants (which may be on-site personnel) at two remote locations are speaking, the communication processing system can determine that the usage scenario is a conversation. If there is non-speech sound from the on-site side, such as a construction site, and there is speech from the other base communicating with the on-site, the communication processing system can determine that the usage scenario is listening to background sounds. The communication system then determines voice processing parameters that are pre-assigned to each usage scenario.

[0012] The communication processing system 5 processes the sound information transmitted from the imaging device using sound processing parameters that are pre-assigned to each usage scenario of the communication system (S2). The main sound processing is noise reduction. Therefore, the sound processing parameters can be the strength of the noise suppressor and the threshold value of the noise gate.

[0013] <Terminology> A base is a place where activities are based. In this embodiment, a conference room, a place where participants are located, etc. are used as an example of a base. A conference room is a room set up primarily for use in meetings. A conference is also called a get-together, a meeting, a consultation, a gathering, a get-together, a gathering, etc.

[0014] The viewpoint information is parameter information that specifies which specific area of ​​the wide-field-of-view image to display on the display. In this embodiment, the radius vector, polar angle, and azimuth angle corresponding to the center of the wide-field-of-view image displayed on the display are described as examples of viewpoint information, but the viewpoint information may be specified by other parameter information such as the coordinates of the diagonal vertices.

[0015] A wide-field-of-view image refers to an image having a wide viewing angle that is wider than the display range that can be displayed on a display at one time in a specified display method. A wide-field-of-view image has a display range of up to 360 degrees vertically and 360 degrees horizontally. However, even if the display range is less than 360 degrees vertically or horizontally, an image having a wide viewing angle that is wider than the display range that can be displayed on a display at one time is included in the wide-field-of-view image. For example, an image having a display range wider than the range that a human can see at one time is also included in the wide-field-of-view image. Note that even if an image can be displayed on a display at one time depending on the display method, the image is included in the wide-field-of-view image if it has a wide viewing angle in a specified display method. Note that in this embodiment, an equirectangular omnidirectional image will be described as an example of a wide-field-of-view image, but omnidirectional images, hemispherical images, 3D panoramic images, 2D panoramic images, and VR (Virtual Reality) images are also included in the wide-field-of-view image.

[0016] An image with a normal angle of view is an image that is not a wide-field image, but in this embodiment, it will be described as an image that is not a wide-field image (planar image).

[0017] Users at each location (on-site staff and participants) communicate remotely between remote locations. Remote communication is a meeting held at remote locations. A meeting is when people get together for consultation, discussion, etc. Meetings can take various forms, such as customer service, conferences, rallies, meetings, study sessions, classes, seminars, and presentations. Communication does not necessarily have to be two-way. Therefore, a virtual room can also be called a virtual conference room.

[0018] Sound information is information that includes human speech and other sounds. Voice is treated as voice data in information processing, and sound is treated as sound data in information processing.

[0019] [Overview of spherical images] A method for generating a spherical image will be described with reference to Fig. 2 to Fig. 9. A spherical image is also called a spherical panoramic image or a 360° panoramic image, and is an example of a wide-field video with a wide viewing angle. Wide-field images also include simple panoramic images of about 180°.

[0020] First, the appearance of the imaging device 10 will be described with reference to Fig. 2. The imaging device 10 is a digital camera for obtaining captured images that are the basis for creating a spherical image. Fig. 2(a) is a left side view of the imaging device, Fig. 2(b) is a front view of the imaging device, and Fig. 2(c) is a plan view of the imaging device.

[0021] As shown in Fig. 2(a), the imaging device 10 is small enough to be held in one hand. Also, as shown in Figs. 2(a), (b), and (c), an imaging element 103a is provided on the front side (front side) of the upper portion of the imaging device 10, and an imaging element 103b is provided on the rear side (rear side). Also, as shown in Fig. 2(b), an operation unit 115 such as a shutter button is provided on the front side of the imaging device 10.

[0022] Next, a usage state of the imaging device 10 will be described with reference to FIG. 3. Note that FIG. 3 is a conceptual diagram of the imaging device in use. As shown in FIG. 3, the imaging device 10 is communicably connected to a relay device 3 installed on some kind of stand, and is used to capture surrounding subjects, scenery, and the like. In this case, two hemispherical images can be obtained by capturing images of subjects around the user using the imaging element 103a and the imaging element 103b shown in FIG. 2. Note that if the omnidirectional image captured by the imaging device 10 is not to be transmitted to other communication terminals or systems, the relay device 3 is not necessary.

[0023] Next, an outline of processing until a celestial sphere image is created from an image captured by the imaging device 10 will be described with reference to Figs. 4 and 5. Fig. 4(a) is a diagram showing a hemispherical image (front side) captured by the imaging device, Fig. 4(b) is a diagram showing a hemispherical image (rear side) captured by the imaging device, and Fig. 4(c) is a diagram showing an image expressed by equirectangular projection (hereinafter referred to as "equirectangular projection image"). An image expressed by Mercator projection or the like (hereinafter referred to as "Mercator image") may also be used. Fig. 5(a) is a conceptual diagram showing a state in which a sphere is covered with an equirectangular projection image, and Fig. 5(b) is a diagram showing a celestial sphere image. The "equirectangular projection image" is an equirectangular celestial sphere image as an example of the wide-field-of-view image.

[0024] As shown in Fig. 4(a), the image obtained by the image sensor 103a becomes a hemispherical image (front side) curved by a wide-angle lens 102a such as a fisheye lens, which will be described later. Also, as shown in Fig. 4(b), the image obtained by the image sensor 103b becomes a hemispherical image (rear side) curved by a wide-angle lens 102b such as a fisheye lens, which will be described later. Then, the image capturing device 10 combines the hemispherical image (front side) with a hemispherical image (rear side) flipped 180 degrees to create an equirectangular projection image EC as shown in Fig. 4(c).

[0025] The imaging device 10 then uses software such as OpenGL ES (Open Graphics Library for Embedded Systems) to apply an equirectangular projection image EC to cover the spherical surface as shown in FIG. 4( a), thereby creating a celestial sphere image CE as shown in FIG. 4( b). In this way, the celestial sphere image CE is represented as an image in which the equirectangular projection image EC faces the center of the sphere. OpenGL ES is a graphics library used to visualize 2D (two-dimensional) and 3D (three-dimensional) data. OpenGL ES is merely an example of software that performs image processing, and the celestial sphere image CE may be created by other software. The celestial sphere image CE may be a still image or a video. While the imaging device 10 has been described as generating a celestial sphere image as an example, similar image processing or part of the image processing steps may be performed by the communication processing system 5 or the communication terminals 7 and 9.

[0026] Then, by using OpenGL ES (Open Graphics Library for Embedded Systems), the Mercator image is pasted to cover the spherical surface as shown in FIG. 5(a), and a spherical image as shown in FIG. 5(b) is created. In this way, the spherical image is expressed as an image in which the Mercator image faces the center of the sphere. Note that OpenGL ES is a graphics library used to visualize 2D (2-Dimensions) and 3D (3-Dimensions) data. Note that the spherical image may be a still image or a video.

[0027] As described above, the spherical image CE is an image pasted to cover the spherical surface, which gives a sense of incongruity to people. Therefore, the communication terminals 7 and 9 can display a predetermined region of the spherical image (hereinafter referred to as a "predetermined region image") as a flat image with little curvature, thereby enabling a display that does not give a sense of incongruity to people. This will be described with reference to FIGS. 6 to 9.

[0028] Fig. 6 is a diagram showing the positions of a virtual camera and a predetermined area when a celestial sphere image is a three-dimensional sphere. The virtual camera IC corresponds to the position of a virtual viewpoint of a user viewing a celestial sphere image CE displayed as a three-dimensional sphere. In Fig. 7, (a) is a diagram showing the predetermined area T, (b) is a diagram showing the predetermined area image in the state of (a) displayed on the display, (c) is a diagram showing the predetermined area after the viewpoint of the virtual camera IC in (a) is changed, and (d) is a diagram showing the predetermined area image in the state of (c) displayed on the display.

[0029] If the celestial sphere image CE generated in this way is a three-dimensional sphere CS, the virtual camera IC is located inside the celestial sphere image CE as shown in Fig. 6. A predetermined area T in the celestial sphere image CE is an imaging area of ​​the virtual camera IC, and is specified by viewpoint information (also referred to as "predetermined area information") that indicates the imaging direction and angle of view of the virtual camera IC in a three-dimensional virtual space including the celestial sphere image CE.

[0030] Furthermore, zooming of the predetermined region T can also be expressed by moving the virtual camera IC closer to or farther away from the celestial sphere image CE. The predetermined region image Q is an image of the predetermined region T in the celestial sphere image CE. Therefore, the predetermined region T can be specified by the angle of view α and the distance f from the virtual camera IC to the celestial sphere image CE.

[0031] Furthermore, when the virtual viewpoint of the virtual camera IC is moved (also referred to as "changed") from the state of Fig. 7(a) to the right (left as one faces the drawing) as shown in Fig. 7(c), the predetermined area T in the omnidirectional image CE is accordingly moved to a predetermined area T', and the predetermined area image Q displayed on the predetermined display is changed to the predetermined area image Q'. As a result, the image shown in Fig. 7(b) is changed to the image shown in Fig. 7(d) and displayed on the display.

[0032] Next, the relationship between viewpoint information and the image of a predetermined area T will be described with reference to Fig. 8 and Fig. 9. Fig. 8 is a diagram showing points in a three-dimensional Euclidean space using spherical coordinates. Fig. 9 is a conceptual diagram showing the relationship between a predetermined area and a point of interest (center point).

[0033] Here, the coordinates of an arbitrary position when the center point CP shown in Fig. 8 is expressed in a spherical polar coordinate system are (r, θ, φ). (r, θ, φ) are the radius vector, polar angle, and azimuth angle, respectively. The radius vector r is the distance from the origin of the three-dimensional virtual space including the omnidirectional image to an arbitrary point (the center point CP in Fig. 9), and is equal to the distance f shown in Fig. 9.

[0034] Furthermore, as shown in FIG. 9, when the center of a predetermined area T, which is the imaging area of ​​the virtual camera IC, is considered to be the center point CP in FIG. 8, the trigonometric function shown in the following (Equation 1) generally holds. (L / f) = tan(α / 2) (Equation 1) Note that f is the distance from the virtual camera IC to the center point CP. L is the distance between any vertex of the predetermined area T and the center point CP (2L is the diagonal). α is the angle of view. In this case, the viewpoint information for identifying the predetermined area T can be expressed by pan(θ), tilt(φ), and fov(α). Note that zooming of the predetermined area T can be expressed by widening or narrowing the range (arc) of the angle of view α.

[0035] [Communication system overview] Next, an overview of a communication system according to an embodiment of the present invention will be described with reference to Fig. 10. Fig. 10 is a schematic diagram of a communication system according to an embodiment of the present invention.

[0036] 10, the communication system 1 of this embodiment is configured with an imaging device 10, a relay device 3, a communication terminal 7, and communication terminals 9a, 9b, and 9c, and transmits and receives images and audio to and from multiple locations. The imaging device 10, the communication terminal 7, and the communication terminals 9a, 9b, and 9c hold a remote conference via a communication network 100. The communication terminals 9a, 9b, and 9c are collectively referred to as "communication terminal 9." The communication terminals may also be referred to as "display terminals" that display images, etc.

[0037] Of these, the imaging device 10 is a digital camera for obtaining a spherical image as described above. The relay device 3 is an example of a cradle that charges the imaging device 10 and transmits and receives data to and from the imaging device 10. The relay device 3 can perform data communication with the imaging device 10 via a contact point, and can also perform data communication with the communication processing system 5 via the communication network 100. The communication network 100 includes, for example, the Internet, a LAN (Local Area Network), a (wireless) router, etc.

[0038] Furthermore, the communication processing system 5 is, for example, a server computer, and can perform data communication with the relay device 3 and the communication terminals 7 and 9 via the communication network 100. The communication terminals 7 and 9 are, for example, laptop PCs (Personal Computers), and can perform data communication with the communication processing system 5 via the communication network 100. OpenGL ES is installed in the communication terminals 7 and 9, and they create a predetermined area image (see FIG. 6 ) from the omnidirectional image received from the communication processing system 5. Note that the communication processing system 5 may be configured by a single server computer or may be configured by multiple server computers.

[0039] Furthermore, the imaging device 10 and the relay device 3 are installed at predetermined locations by a site manager X at a construction site or the like, which is a first base. The communication terminal 7 is operated by the site manager X at the first base. The communication terminal 9a set up at the second base is operated by a participant A who is a remote customer or the like. Similarly, the communication terminals 9b and 9c are operated by participants B and C who are remote customers or the like, respectively.

[0040] The communication processing system 5 transmits (distributes) the spherical image obtained from the imaging device 10 via the relay device 3 to the communication terminal 7 at the first base and the communication terminal 9 at the second base. Furthermore, the communication processing system 5 receives, from each communication terminal 9, each piece of viewpoint information for specifying a predetermined area of ​​the predetermined area image being displayed on each communication terminal 9 (being viewed by each of the participants A, B, and C), and transmits each piece of viewpoint information to the communication terminal 7. Then, the communication terminal 7 displays a viewpoint display area based on each piece of viewpoint information on the predetermined area image, which is a predetermined area of ​​the spherical image received from the communication processing system 5. This allows the on-site person in charge X to know which predetermined area of ​​the spherical image the remote participants A, B, and C are focusing on when viewing.

[0041] [Hardware configuration of the embodiment] Next, the hardware configurations of the imaging device 10, relay device 3, and communication terminals 7 and 9 of this embodiment will be described in detail with reference to FIGS.

[0042] <Hardware configuration of imaging device> Fig. 11 is a hardware configuration diagram of an imaging device 10. As shown in Fig. 11, the imaging device 10 includes an imaging unit 101, an image processing unit 104, an imaging control unit 105, a microphone 108, a sound processing unit 109, a CPU (Central Processing Unit) 111, a ROM (Read Only Memory) 112, an SRAM (Static Random Access Memory) 113, a DRAM (Dynamic Random Access Memory) 114, an operation unit 115, an input / output I / F 116, a short-range communication circuit 117, an antenna 117a of the short-range communication circuit 117, an electronic compass 118, a gyro sensor 119, an acceleration sensor 120, and a network I / F 121.

[0043] Of these, the imaging unit 101 is equipped with wide-angle lenses 102a and 102b (hereinafter referred to as lenses 102 when there is no need to distinguish between them) that are each capable of capturing an image with a field of view of 180° or more to form a hemispherical image, and two imaging elements 103a and 103b that are provided corresponding to the lenses 102a and 102b, respectively.

[0044] Furthermore, the imaging elements 103a, 103b include an image sensor such as a CMOS (Complementary Metal Oxide Semiconductor) sensor or a CCD (Charge Coupled Device) sensor that converts optical images captured by the lenses 102a, 102b, etc. into image data in the form of electrical signals and outputs the image data, a timing generation circuit that generates horizontal or vertical synchronization signals and pixel clocks for the image sensors, and a group of registers in which various commands or parameters required for the operation of the imaging elements are set. Note that the configuration in which the imaging unit 101 has two wide-angle lenses is merely an example, and the imaging unit 101 may have only one, or three or more.

[0045] The imaging elements 103a and 103b of the imaging unit 101 are each connected to the image processing unit 104 via a parallel I / F bus. On the other hand, the imaging elements 103a and 103b of the imaging unit 101 are each connected to the imaging control unit 105 via a serial I / F bus (such as an I2C bus).

[0046] The image processing unit 104, the imaging control unit 105, and the sound processing unit 109 are connected to a CPU 111 via a bus 110. Furthermore, the bus 110 is also connected to a ROM 112, an SRAM 113, a DRAM 114, an operation unit 115, an input / output I / F 116, a short-range communication circuit 117, an electronic compass 118, a gyro sensor 119, an acceleration sensor 120, a network I / F 121, and the like.

[0047] The image processing unit 104 takes in the image data output from the image sensors 103a and 103b via a parallel I / F bus, performs predetermined processing on each piece of image data, and then synthesizes the image data to create data for an equirectangular projection image (an example of a wide-field image), which will be described later.

[0048] The imaging control unit 105 generally sets commands and the like in the registers of the imaging elements 103a and 103b using an I2C bus, with the imaging control unit 105 acting as a master device and the imaging elements 103a and 103b acting as slave devices. Necessary commands and the like are received from the CPU 111. The imaging control unit 105 also uses the I2C bus to retrieve status data and the like from the registers of the imaging elements 103a and 103b and send it to the CPU 111.

[0049] Furthermore, the imaging control unit 105 instructs the imaging elements 103a and 103b to output image data when the shutter button of the operation unit 115 is pressed. Some imaging devices 10 have a preview display function or a function for displaying moving images on a display (for example, a display of an external terminal such as a smartphone that performs short-range communication with the imaging device 10 using the short-range communication circuit 117). In this case, the image data is output continuously from the imaging elements 103a and 103b at a predetermined frame rate (frames / minute).

[0050] As will be described later, the imaging control unit 105 also functions as a synchronization control means that cooperates with the CPU 111 to synchronize the output timing of image data from the imaging elements 103a and 103b. In this embodiment, the imaging device 10 is not provided with a display unit, but a display unit may be provided. The microphone 108 converts sound into sound (signal) data. The sound processing unit 109 receives the sound data output from the microphone 108 via an I / F bus and performs predetermined processing on the sound data.

[0051] The CPU 111 controls the overall operation of the imaging device 10 and executes necessary processing. The ROM 112 stores various programs for the CPU 111. The SRAM 113 and DRAM 114 are work memories that store programs executed by the CPU 111, data in the middle of processing, etc. In particular, the DRAM 114 stores image data in the middle of processing by the image processing unit 104 and data of processed equirectangular projection images.

[0052] The operation unit 115 is a general term for various operation buttons, a power switch, a shutter button, a touch panel that combines display and operation functions, etc. A user operates the operation unit 115 to input various imaging modes, imaging conditions, etc.

[0053] The input / output I / F 116 is a general term for an interface circuit (such as a USB I / F) with an external medium such as an SD card or a personal computer. The input / output I / F 116 may be wireless or wired. The data of the equirectangular projection image stored in the DRAM 114 is recorded on an external medium via the input / output I / F 116, or transmitted to an external terminal (device) via the input / output I / F 116 as needed.

[0054] The short-range communication circuit 117 communicates with an external terminal (device) by short-range wireless communication technology such as NFC (Near Field Communication), Bluetooth (registered trademark), or Wi-Fi via an antenna 117a provided in the imaging device 10. The short-range communication circuit 117 can transmit data of the equirectangular projection image to the external terminal (device).

[0055] The electronic compass 118 calculates the direction of the image capture device 10 from the Earth's magnetism and outputs direction information. This direction information is an example of related information (metadata) according to Exif, and is used for image processing such as image correction of the captured image. The related information also includes data such as the image capture date and time and the data size of the image data.

[0056] The gyro sensor 119 is a sensor that detects changes in angle (roll angle, pitch angle, yaw angle) that accompany the movement of the image capture device 10. The changes in angle are an example of related information (metadata) according to Exif, and are used for image processing such as image correction of captured images.

[0057] The acceleration sensor 120 is a sensor that detects acceleration in three axial directions. The imaging device 10 calculates the attitude (angle with respect to the direction of gravity) of the imaging device 10 itself (the imaging device 10) based on the acceleration detected by the acceleration sensor 120. By providing the imaging device 10 with the acceleration sensor 120, the accuracy of image correction is improved.

[0058] The network I / F 121 is an interface for performing data communication using a communication network 100 such as the Internet via a router or the like. The hardware configuration of the imaging device 10 is not limited to that shown here, and may be any configuration that can realize the functional configuration of the imaging device 10. At least a part of the hardware configuration may be present on the relay device 3 or the communication network 100.

[0059] <Hardware configuration of relay device> Fig. 12 is a diagram showing the hardware configuration of the relay device 3. Note that Fig. 12 shows the hardware configuration when the relay device 3 is a cradle having a wireless communication function.

[0060] As shown in FIG. 12, the relay device 3 includes a CPU 301, a ROM 302, a RAM 303, an EEPROM 304, a CMOS sensor 305, a bus line 310, a communication unit 313, an antenna 313a, a GPS receiving unit 314, and an input / output I / F 316.

[0061] Of these, the CPU 301 controls the overall operation of the relay device 3. The ROM 302 stores programs such as an IPL (Initial Program Loader) used to drive the CPU 301. The RAM 303 is used as a work area for the CPU 301.

[0062] An EEPROM (Electrically Erasable and Programmable ROM) 304 reads or writes data under the control of the CPU 301. The EEPROM 304 stores an operating system (OS) executed by the CPU 301, other programs, and various data.

[0063] The CMOS (Complementary Metal Oxide Semiconductor) sensor 305 is a solid-state image sensor that captures an image of a subject under the control of the CPU 301 and obtains image data.

[0064] The communication unit 313 communicates with the communication network 100 by using a wireless communication signal via an antenna 313a.

[0065] The GPS receiving unit 314 receives a GPS signal including the position information (latitude, longitude, and altitude) of the relay device 3 via a GPS (Global Positioning Systems) satellite or an IMES (Indoor Messaging System) as an indoor GPS.

[0066] The input / output I / F 316 is an interface circuit (such as a USB I / F) electrically connected to the input / output I / F 116 of the imaging device 10. The input / output I / F 316 may be wireless or wired.

[0067] The bus line 310 is an address bus, a data bus, etc. for electrically connecting the above-mentioned components.

[0068] <Hardware configuration of communication processing system and communication terminal> 13 shows the hardware configuration of the communication processing system 5. The hardware configuration of the communication terminals 7 and 9 is the same as that of the communication processing system 5, and therefore a description thereof will be omitted.

[0069] As shown in FIG. 13, the communication processing system 5 is a computer and includes a CPU 501, a ROM 502, a RAM 503, an SSD 504, an external device connection I / F 505, a network I / F 506, a display 507, an operation unit 508, a media I / F 509, a bus line 510, a CMOS sensor 511, and a speaker 512.

[0070] Of these, the CPU 501 controls the overall operation of the communication processing system 5. The ROM 502 stores programs such as IPL used to drive the CPU 501. The RAM 503 is used as a work area for the CPU 501.

[0071] The SSD 504 reads or writes various data under the control of the CPU 501. If the communication terminals 7 and 9 are smartphones or the like, the SSD 504 may not be provided. Also, the computer may have an HDD (Hard Disk Drive) instead of the SSD 504.

[0072] The external device connection I / F 505 is an interface for connecting various external devices, such as a display, a speaker, a keyboard, a mouse, a USB memory, and a printer.

[0073] The network I / F 506 is an interface for performing data communication via the communication network 100 .

[0074] The display 507 is a type of display unit such as a liquid crystal display or organic electroluminescence (EL) display that displays various images.

[0075] An operation unit 508 is an input means for selecting and executing various instructions such as various operation buttons, a power switch, a shutter button, and a touch panel, selecting a processing target, moving a cursor, and the like.

[0076] The media I / F 509 controls reading and writing (storing) of data from and to a recording medium 509m such as a flash memory, etc. The recording medium 509m includes DVDs, Blu-ray Discs (registered trademarks), etc.

[0077] The CMOS sensor 511 is a type of imaging means that captures an image of a subject and obtains image data under the control of the CPU 501. The computer may use a CCD sensor instead of the CMOS sensor 511.

[0078] The speaker 512 is a circuit that converts electrical signals into physical vibrations to produce sounds such as music and voice.

[0079] The bus line 510 is an address bus, a data bus, etc. for electrically connecting the components such as the CPU 501 shown in FIG.

[0080] [Functional configuration of the embodiment] Next, the functional configuration of this embodiment will be described with reference to FIGS.

[0081] <Functional configuration of the imaging device> 14, the imaging device 10 has a reception unit 12, an imaging unit 16, a sound collection unit 17, a connection unit 18, and a storage / readout unit 19. Each of these units is a function or means realized by any of the components shown in FIG. 11 operating in response to an instruction from the CPU 111 in accordance with an imaging storage program loaded from the SRAM 113 onto the DRAM 114.

[0082] The imaging device 10 also includes a storage unit 1000 configured by a ROM 112, an SRAM 113, and a DRAM 114 shown in FIG.

[0083] (Functional configuration of the imaging device) The reception unit 12 of the imaging device 10 is realized by the processing of the operation unit 115 for the CPU 111, and receives operation input from the user.

[0084] The imaging section 16 is mainly realized by processing from the CPU 111 to the imaging unit 101, the image processing unit 104, the imaging control unit 105, and the CPU 111, and captures images of scenery and the like to obtain captured images.

[0085] The sound collection unit 17 is mainly realized by processing of the sound processing unit 109 from the CPU 111, and collects sounds around the imaging device .

[0086] The connection unit 18 is mainly realized by processing from the CPU 111 to the input / output I / F 116, and performs data communication with the relay device 3.

[0087] The storage / readout unit 19 is mainly realized by the processing of the CPU 111 , and stores various data (or information) in the storage unit 1000 and reads out various data (or information) from the storage unit 1000 .

[0088] <Functional configuration of relay device> 14, the relay device 3 has a communication unit 31 and a connection unit 38. Each of these units is a function or means realized when any of the components shown in FIG. 12 operates in response to an instruction from the CPU 301 in accordance with the program for the relay device 3 loaded from the EEPROM 304 onto the RAM 303.

[0089] (Functional configuration of relay device 3) The communication unit 31 of the relay device 3 is mainly realized by processing from the CPU 301 shown in FIG. 12 to the communication unit 313, and performs data communication between the imaging device 10 and the communication processing system 5 via the communication network 100.

[0090] The connection unit 38 is mainly realized by processing from the CPU 301 to the input / output I / F 316, and performs data communication with the imaging device 10.

[0091] <Functional configuration of communication processing system> Next, each functional configuration of the communication processing system 5 will be described in detail with reference to Fig. 14. The communication processing system 5 has a communication unit 51, a reception unit 52, a parameter determination unit 53, a voice processing unit 54, and a storage / readout unit 59. Each of these units is a function or means realized when any of the components shown in Fig. 13 operates in response to an instruction from the CPU 501 in accordance with the program for the communication processing system 5 loaded from the SSD 504 onto the RAM 503.

[0092] 13, the communication processing system 5 also has a storage unit 5000 constructed by a RAM 503 and an SSD 504. In this storage unit 5000, a user device management DB 5001, a virtual room management DB 5002, a viewpoint information management DB 5003, a usage scene information DB 5004, and a voice processing parameter information DB 5005 are constructed.

[0093] (User / Device Management DB) 15 is a conceptual diagram of the user device management DB. The user device management DB 5001 is configured in a table format, and stores and manages user IDs (or device IDs), names, and IP addresses in association with each other.

[0094] Of these, the user ID is an example of user identification information for identifying users (site staff member X, participants A, B, C). The device ID is an example of device identification information for identifying devices such as the imaging device 10. Note that in FIG. 10, if a head-mounted display or the like is used in addition to the imaging device 10, the head-mounted display or the like is also treated as a device.

[0095] The name is the name of the user or device.

[0096] The IP address is an example of destination identification information for a communication terminal used by a user and a device such as the imaging device 10.

[0097] (Virtual room management DB) 16 is a conceptual diagram of the virtual room management DB 5002. The virtual room management DB 5002 is configured in a table format, and stores and manages the virtual room ID, virtual room name, device ID, on-site staff ID, participant ID, and storage (information on the storage location of image data) in association with each other.

[0098] Of these, the virtual room ID is an example of virtual room identification information for identifying a virtual room.

[0099] The virtual room name is the name of the virtual room and is given by the user or the like.

[0100] The device ID is the same as the device ID in FIG. 15, and is the ID of the device that has participated in the virtual room indicated by the virtual room ID of the same record.

[0101] The site staff member ID is an example of identification information for identifying the site staff member ID among the user IDs in FIG. 15, and is the ID of the site staff member who participated in the virtual room indicated by the virtual room ID of the same record.

[0102] The participant ID is an example of participant identification information for identifying the participant ID among the user IDs in FIG. 15, and is the ID of a participant who has participated in the virtual room indicated by the virtual room ID of the same record.

[0103] The storage is an example of storage location information indicating the location where the wide-field image or the like is stored, and specifically indicates a URL, a file path, or the like.

[0104] (Viewpoint Information Management DB) 17 is a conceptual diagram of the viewpoint information management DB 5003. The viewpoint information management DB 5003 is configured in a table format, and associates and stores and manages participant IDs, IP addresses, viewpoint information (pan, tilt, fov), and timestamps.

[0105] Among these, the participant ID is the same as the participant ID in FIG.

[0106] The IP addresses are the same as those in FIG.

[0107] The viewpoint information (pan, tilt, fov) is viewpoint information sent from the communication terminal of the participant indicated by the participant ID of the same record.

[0108] The timestamp indicates the time when the viewpoint information of the same record was sent.

[0109] (Usage Scene Information DB5004) 18 is a conceptual diagram of the usage scene information DB 5004. The usage scene information DB 5004 is configured in a table format, and stores and manages No., usage scene, and voice processing parameter ID in association with each other. Of these, No. is an example of identification information for identifying a usage scene. Usage scenes are various situations in which sound information can be used. When sound information acquired by the imaging device 10 in remote communication is transmitted to the communication processing system 5, this usage scene is determined. The voice processing parameter ID is identification information that identifies the voice processing parameter associated with the usage scene.

[0110] 18(a) shows usage scenes that are determined by inputting image data into a usage scene discrimination model. Each usage scene is associated with an optimal voice processing parameter ID. The parameter determination unit 53 (more specifically, the determination unit 55) determines the usage scene from the image data using the usage scene discrimination model described below, and determines the voice processing parameter ID associated with the usage scene in the usage scene information DB 5004.

[0111] In the case of a final inspection, a voice processing parameter ID of "3" is specified. The voice processing parameter ID of "3" is a voice processing parameter that is suitable for conversation, as shown in FIG. 19. This is because conversation takes place during a final inspection. The voice processing parameter ID of "3" is a voice processing parameter that makes it easier to hear the sounds that are the target of listening during a final inspection.

[0112] In the case of foundation work, a voice processing parameter ID of "4" is specified. The voice processing parameter ID of "4" is a voice processing parameter that emphasizes background sounds, as shown in FIG. 19. This is because the background sounds of construction are important in foundation work. In the case of scaffolding work, a voice processing parameter ID of "4" is specified. This is because the background sounds of construction are important in scaffolding work. The voice processing parameter of "4" is a voice processing parameter that makes it easier to hear sounds that are the target of attention during foundation work and scaffolding work.

[0113] In the case of interior construction work, a voice processing parameter ID of "1" is specified. As shown in FIG. 19, a voice processing parameter ID of "1" does not have noise cancellation. Since there are various types of interior construction work, the voice processing parameter is not uniquely specified. The voice processing parameter of "1" is a voice processing parameter that makes it easier to hear sounds that are the target of attention during interior construction work.

[0114] In the case of evidence imaging, an audio processing parameter ID of "2" is specified. The audio processing parameter ID of "2" is the default audio processing parameter, as shown in FIG. 19. This is because evidence imaging is used in situations such as imaging a construction schedule board every morning, and general audio processing parameters are effective. The audio processing parameter of "2" is an audio processing parameter that makes it easier to hear the sounds that are the subject of listening when capturing evidence.

[0115] In the case of a hammering test, a voice processing parameter ID of "5" is specified. The voice processing parameter ID of "5" is a voice processing parameter suitable for a hammering test, as shown in Figure 19. In a hammering test, participants judge cavities and the like by the sound of hitting a wall, etc., so a voice processing parameter suitable for the hammering test is used. The voice processing parameter of "5" is a voice processing parameter that makes it easier to hear the target sound during the hammering test.

[0116] FIG. 18(b) shows a usage scene that is determined based on the type of sound information transmitted and received between base stations. According to the usage scene information DB 5004 in FIG. 18(b), for example, when both the site worker and the participants at the construction site are speaking, the usage scene is determined to be "speech from both sides," and a voice processing parameter ID of "3" is identified. The voice processing parameter ID of "3" is a voice processing parameter that is suitable for conversation, as shown in FIG. 19. When there is sound that is not speech from the construction site and speech from the participants, the usage scene is determined to be "sound other than speech from the site," and a voice processing parameter ID of "4" is identified. The voice processing parameter ID of "4" is a voice processing parameter that prioritizes background sound, as shown in FIG. 19.

[0117] The determination unit 55 of the parameter determination unit 53 can determine whether the sound information contains speech by performing a Fourier transform on the sound information. The frequency spectrum of the sound is obtained by performing a Fourier transform. Since the frequency range of human voices is fixed, if the intensity in that frequency range is equal to or greater than a threshold, it can be determined that the sound information contains speech. Similarly, the determination unit 55 can determine whether the sound information contains sounds other than speech by performing a Fourier transform on the sound information. If the intensity in a frequency range specific to construction sites is equal to or greater than a threshold, the determination unit 55 can determine that the sound information contains sounds other than speech. Alternatively, the determination unit 55 may determine whether the sound information contains sounds other than speech using a model that has learned a frequency spectrum specific to construction sites.

[0118] According to the usage scene information DB 5004 of Figures 18(a) and (b), the determination unit 56 of the parameter determination unit 53 can determine voice processing parameters in real time (for example, during a meeting) and switch to voice processing parameters that correspond to the usage scene.

[0119] 18(c) shows a usage scene determined based on other information. For example, the determination unit 55 can determine the usage scene before the start of the meeting based on the meeting information. If the meeting name in the meeting information includes, for example, site inspection, structural work, exterior work, equipment work, exterior work, demolition work, completion inspection, foundation work, scaffolding work, interior work, evidence imaging, hammering inspection, etc., the determination unit 56 can determine the voice processing parameters in the same way as in FIG. 18(a). In this case, it is assumed that the meeting name and the usage scene are associated in advance.

[0120] As the conference information, in addition to the conference name, participants, summary and materials may be used, and the determining unit 55 may determine whether or not a similar construction name is included.

[0121] Furthermore, in the case of a conference that is held periodically, the communication processing system 5 can use the voice processing parameters used in the previous conference. Furthermore, when a remote conference (hereinafter simply referred to as a conference) is registered in the scheduler, the determination unit 55 acquires the conference name, participants, summary, etc. registered in the scheduler and searches past conference information to identify conferences similar to the current conference. The determination unit 56 may adopt the voice processing parameters used in the past conference that is determined to be similar.

[0122] The explicit specification in Fig. 18(c) means that the participant manually sets the usage scenario or the voice processing parameters from a menu. The voice recognition in Fig. 18(c) means that the voice processing parameter ID is determined by voice recognition when the participant specifies the usage scenario by voice. For example, if the participant utters "perform a tapping test," the determination unit 55 performs voice recognition and determines that the usage scenario is a tapping test.

[0123] (Speech processing parameter information DB5005) 19 is a conceptual diagram of the audio processing parameter information DB 5005. The audio processing parameter information DB 5005 is configured in a table format, and stores and manages IDs, audio processing parameter names, noise suppressor strengths, and noise gate thresholds in association with each other. Note that the noise suppressor strengths and noise gate thresholds are examples of audio processing parameters. Among these, ID is a voice processing parameter ID, and is an example of identification information for identifying a voice processing parameter. The voice processing parameter name is a name given to the voice processing parameter, and is given so that the person who manages the voice processing parameter information DB 5005 can easily manage the voice processing parameters. A noise suppressor is an audio process that reduces the volume of a certain frequency band when the volume for that band becomes low. Generally, the frequency band for which the volume is reduced and the threshold volume are also set. The noise suppressor indicates the noise reduction effect with a value between 0 and 1; for example, 0.7 reduces the volume of that band by 70% when the volume becomes low. In the example in Figure 19, the audio processing parameter "conversation" is set to reduce the noise band by 90% (making the voice clearer), and the audio processing parameter "background sound emphasis" is set to reduce the noise band by 30% (making it more difficult to eliminate environmental sounds). Noise gate is an audio processing method that shuts out all sounds when they fall below a certain volume (threshold). In Figure 19, the threshold for prioritizing background sound is smaller than that for conversation, so when conversation occurs, environmental sounds other than conversation are shut out, making the voice clearer. When prioritizing background sound, only quieter sounds are shut out, making it harder for environmental sounds to be shut out, making it easier for environmental sounds to be heard.

[0124] Audio processing parameters with the name "Not Applied" are audio processing parameters that do not perform audio processing using a noise suppressor. Audio processing parameters with the name "Default" are set to general-purpose audio processing parameters that are used in situations where the environment is not specified. Audio processing parameters with the name "Conversation" are audio processing parameters that perform noise removal suitable for conversation. Audio processing parameters with the name "Background Sound Emphasis" are audio processing parameters that capture construction sounds that are removed because they are background noise in conversations, etc., without removing them. Audio processing parameters with the name "Hammering Inspection" are audio processing parameters that are suitable for an inspection method that determines the internal condition of a building by the sound made when the building is hammered.

[0125] Note that there are various other types of voice processing parameters, such as echo cancellers and automatic gain control, and they are not limited to those shown in Figure 19. Echo cancellers are processes that reduce the phenomenon in which a speaker's voice is output from the other party's speaker and input to the other party's microphone, resulting in the voice being heard from the speaker. The communication processing system 5 strengthens the echo canceller to make the conversation clearer during conversation, and weakens the echo canceller to make the surrounding environmental sounds clearer when prioritizing background sounds. Automatic gain control is a voice processing method that adjusts both loud and soft sounds picked up by the microphone to an optimal volume and reduces volume differences at the other end of the call caused by differences in distance from the microphone and voice volume. This automatic gain control has a noise reduction function (changing the amplification rate between the noise band and other bands) and can amplify only human voices in noisy environments. Therefore, the communication processing system 5, for example, sets the automatic gain control to remove noise to make the conversation clearer during conversation, and to weaken noise removal and amplify environmental sounds when prioritizing background sounds.

[0126] Additionally, the communication processing system 5 may change voice processing parameters such as noise reduction and dereverberation.

[0127] (Meeting Information) Figure 20 shows an example of conference information registered in a conference scheduler. A conference scheduler is, for example, a conference room reservation system or scheduling system. Conference information has fields for conference room name, time column, participant column, conference name, summary, and materials. The conference room name is the conference room used when the conference is face-to-face. In this embodiment, this corresponds to a virtual room. The time period determines the start and end time of the conference. Participants are those who plan to attend the conference. The conference name is the name, title, agenda, etc. of the conference. The summary is the main points of the conference. Minutes of the conference may be registered in the summary. Materials are materials that are used or will be used in the conference. Materials can be of any type, but materials related to a construction site include schedules and drawings.

[0128] (Functional configuration of the communication processing system) Next, each functional configuration of the communication processing system 5 will be described in detail with reference to Fig. 14. The communication unit 51 of the communication processing system 5 is mainly realized by processing from the CPU 501 shown in Fig. 13 to the network I / F 506, and performs data communication with other devices (relay device 3, communication terminals 7 and 9) via the communication network 100.

[0129] The reception unit 52 is realized by the processing of the operation unit 508 for the CPU 501, and receives operation input from a user (here, a system administrator or the like).

[0130] 18 , the parameter determination unit 53 determines the usage scene based on the usage scene determination information DB, and determines the audio processing parameter ID associated with the usage scene. The parameter determination unit 53 has a determination unit 55 and a determination unit 56. The determination unit 55 determines the usage scene by inputting the wide-field image into a discrimination model of the usage scene that associates the wide-field image with the usage scene. The determination unit 56 determines the audio processing parameters that are associated in advance with the usage scene determined by the determination unit 55.

[0131] The audio processing unit 54 performs audio processing on the sound information of the transmission source using the audio processing parameter ID determined by the determination unit 56 .

[0132] The storage / readout unit 59 is mainly realized by the processing of the CPU 501 , and stores various data (or information) in the storage unit 5000 and reads out various data (or information) from the storage unit 5000 .

[0133] <Functional configuration of communication terminal 7> Next, the functional configuration of the communication terminal 7 will be described in detail with reference to Fig. 14. The communication terminal 7 has a communication unit 71, a reception unit 72, a display control unit 74, a sound output control unit 75, and a storage / readout unit 79. Each of these units is a function or means realized when any of the components shown in Fig. 13 operates in response to an instruction from the CPU 501 in accordance with the program for the communication terminal 7 loaded from the SSD 504 onto the RAM 503.

[0134] 13, the communication terminal 7 has a storage unit 7000 constructed by the RAM 503 and the SSD 504. A viewpoint information management DB 7003 is constructed in the storage unit 7000. The viewpoint information management DB 7003 stores and manages one record of viewpoint information etc. sent from the communication processing system 5, and has the same configuration as the viewpoint information management DB 7003, so a description thereof will be omitted.

[0135] (Functional configuration of communication terminal 7) Next, each functional configuration of the communication terminal 7 will be described in detail with reference to FIG.

[0136] The communication unit 71 of the communication terminal 7 is mainly realized by processing from the CPU 501 shown in FIG. 13 to the network I / F 506, and performs data communication with other devices (communication processing system 5) via the communication network 100.

[0137] The reception unit 72 is realized by the processing of the operation unit 508 for the CPU 501, and receives operation input from the user (in this case, the site worker X).

[0138] The display control unit 74 is realized by the processing of the CPU 501, and performs control for displaying various images on the display 507 of the communication terminal 7 or an external display connected to the external device connection I / F 505.

[0139] The sound output control unit 75 is realized by the processing of the CPU 501, and performs control to output sound from the speaker 512 of the communication terminal 7 or an external speaker connected to the external device connection I / F 505.

[0140] The storage / readout unit 79 is realized by the processing of the CPU 501 , and stores various data (or information) in the storage unit 7000 and reads out various data (or information) from the storage unit 7000 .

[0141] <Functional configuration of communication terminal 9> Next, each functional configuration of the communication terminal 9 will be described in detail with reference to FIG.

[0142] The communication unit 91 of the communication terminal 9 is realized by processing from the CPU 501 shown in FIG. 13 to the network I / F 506, and performs data communication with other devices (communication processing system 5) via the communication network 100.

[0143] The reception unit 92 is realized by the processing of the operation unit 508 on the CPU 501, and receives operation input from users (here, participants A, B, and C).

[0144] The display control unit 94 is realized by the processing of the CPU 501, and performs control for displaying various images on the display 507 of the communication terminal 9 or an external display connected to the external device connection I / F 505.

[0145] The sound output control unit 95 is realized by the processing of the CPU 501, and performs control to output sound from the speaker 512 of the communication terminal 9 or an external speaker connected to the external device connection I / F 505.

[0146] The storage / readout unit 99 is realized by the processing of the CPU 501 , and stores various data (or information) in the storage unit 9000 and reads out various data (or information) from the storage unit 9000 .

[0147] [Processing or Operation of the Embodiment] Next, the processing or operation of this embodiment will be described with reference to Figures 21 to 28. Note that the following processing is processing that is performed after the imaging device 10 and the communication terminals 7 and 9 have already participated in the same virtual room.

[0148] <Transmission and reception processing of wide-field image and sound information in communication systems> First, the transmission and reception process of wide-field images and sound information in a communication system will be described using Fig. 21. Fig. 21 is a sequence diagram showing the communication process of wide-field images and each viewpoint information in a communication system. Note that processes S11 to S22 in Fig. 21 are repeated, for example, about 30 or 60 times per second.

[0149] S11: In the imaging device 10, the imaging unit 16 captures an image of the construction site and obtains a wide-field image, and then the connection unit 18 sends the wide-field image to the relay device 3. At the same time, in the imaging device 10, the sound collection unit 17 collects sounds from the construction site and obtains sound information, and then the connection unit 18 sends the sound information to the relay device 3. In this case, the connection unit 18 also sends a virtual room ID for identifying the virtual room in which the imaging device 10 is participating and a device ID for identifying the imaging device 10. As a result, the connection unit 38 of the relay device 3 acquires the wide-field image, sound information, virtual room ID, and device ID.

[0150] S12: In the relay device 3, the communication unit 31 transmits the information (wide-field image, sound information, virtual room ID, and device ID) acquired by the connection unit 38 in process S11 to the communication processing system 5 via the communication network 100. As a result, in the communication processing system 5, the communication unit 51 receives the information (wide-field image, sound information, virtual room, and device ID).

[0151] S13: In the communication processing system 5, the storage and reading unit 59 searches the virtual room management DB 5002 based on the virtual room ID received in process S12, thereby reading out the site staff IDs and participant IDs participating in the same virtual room as the imaging device 10. The storage and reading unit 59 also searches the user device management DB 5001 based on the read site staff ID and participant ID, thereby reading out the IP address of the communication terminal 7 of the corresponding site staff member X and the IP addresses of the communication terminals 9a, 9b, and 9c of the participants A, B, and C. The communication unit 51 then references the IP address of the communication terminal 7 and transmits the wide-field image and sound information received in process S12 to the communication terminal 7. As a result, the communication unit 71 of the communication terminal 7 receives the wide-field image and sound information.

[0152] S14: The communication unit 51 of the communication processing system 5 references the IP address of the communication terminal 9a and transmits the wide-field image and sound information received in step S12 to the communication terminal 9a. As a result, the communication unit 91 of the communication terminal 9a receives the wide-field image and sound information.

[0153] S15: Similarly, the communication unit 51 of the communication processing system 5 references the IP address of the communication terminal 9b and transmits the wide-field image and sound information received in step S12 to the communication terminal 9b. As a result, the communication unit 91 of the communication terminal 9b receives the wide-field image and sound information.

[0154] S16: Similarly, the communication unit 51 of the communication processing system 5 references the IP address of the communication terminal 9c and transmits the wide-field image and sound information received in step S12 to the communication terminal 9c. As a result, the communication unit 91 of the communication terminal 9c receives the wide-field image and sound information.

[0155] S17: Next, in the communication terminal 9a, the display control unit 94 displays a predetermined area image showing a predetermined area of ​​the wide-field image received in process S14, and the sound output control unit 95 outputs sound based on the sound information received in process S14. Furthermore, when the reception unit 92 receives a screen operation from participant A, the display control unit 94 changes the predetermined area T (see FIG. 7(a)) and displays a predetermined area image showing a predetermined area T' (see FIG. 7(c)) in which a product or the like in which participant A is interested is displayed. Then, the communication unit 91 of the communication terminal 9a transmits viewpoint information for identifying the predetermined area T' in the wide-field image to the communication processing system 5. In this case, the viewpoint information includes a participant ID for identifying participant A, who is the sender. Thus, the communication unit 51 of the communication processing system 5 receives the viewpoint information. Furthermore, the storage / readout unit 59 associates the viewpoint information (pan, tilt, fov) received in process S17 with the participant ID and stores them as one record in the viewpoint information management DB 5003. In this case, the storage / reading unit 59 also stores the time information (timestamp) indicating the time when the viewpoint information was received in process S17 and the participant ID of participant A, which is the user ID managed in the user / device management DB 5001, in association with the same record.

[0156] S18: The storage / readout unit 59 of the communication processing system 5 reads out the viewpoint information etc. for one record stored in the viewpoint information management DB 5003 in processing S17, and the communication unit 51 transmits this viewpoint information etc. to the communication terminal 7. As a result, the communication unit 71 of the communication terminal 7 receives the viewpoint information etc. Note that the communication processing system 5 searches the virtual room management DB 5002 based on the virtual room ID identified in processing S12 to read out the corresponding on-site staff ID (here, "100x"), and then searches the user device management DB 5001 based on this on-site staff ID to read out the corresponding IP address. As a result, the communication unit 51 transmits the viewpoint information etc. to the communication terminal 7 of only on-site staff X among on-site staff X and participants A, B, and C.

[0157] The processing of the communication terminal 9b and the communication processing system 5 (S19, S20) and the processing of the communication terminal 9c and the communication processing system 5 (S21, S22) are similar to the above processing of S17, S18, and therefore description thereof will be omitted.

[0158] <Display control process in communication terminal 7> Next, the communication terminal 7 performs the processing shown in Fig. 22. Fig. 22 is a flowchart showing the processing by the communication terminal 7 of the on-site person in charge of displaying the predetermined area image. Fig. 23 is an explanatory diagram of a case where a predetermined area T2 of a predetermined area image currently being displayed on another communication terminal 9 is superimposed on the predetermined area image of the predetermined area T1 currently being displayed on the communication terminal 7. Figs. 24 to 28 are diagrams showing the predetermined area image on the on-site person's side, displaying viewpoint display areas indicating the predetermined areas currently being viewed by each participant.

[0159] S31: First, the display control unit 74 displays a predetermined area image 750 of a predetermined area on the display 507 (an example of a display unit) as shown in Fig. 24. This predetermined area image 750 displays viewpoint display areas 750a, 750b, and 750c that indicate the respective predetermined areas of the respective predetermined area images being displayed on the respective communication terminals 9a, 9b, and 9c, based on viewpoint information transmitted from the respective communication terminals 9a, 9b, and 9c via the communication processing system 5. Furthermore, the viewpoint display areas 750a, 750b, and 750c display the respective participant IDs transmitted by processes S18, S20, and S22.

[0160] In steps S18, S20, and S22, the communication processing system 5 may transmit information on each name managed in the user device management DB 5001 instead of or in addition to each participant ID. In this case, in Fig. 24, the participant name is displayed instead of or in addition to each participant ID.

[0161] Furthermore, the viewpoint display areas 750a, 750b, and 750c may be indicated by solid-line frames, dashed-line frames, or only the four corners of the frames, as shown in Fig. 24. The inside of the frames may be masked semi-transparently.

[0162] Furthermore, a mark m1 is displayed at the bottom right of the specified area image 750, indicating that the specified area image 750 can be changed (see Figures 7(b) to 7(d)) ​​by changing the specified area within the wide-field image (see Figures 7(a) to 7(c)).

[0163] 23, a process will be described in which the display control unit 74 superimposes and displays viewpoint display areas 740a, 740b, and 740c on the predetermined area image 740. Fig. 23 is an explanatory diagram of a case in which a predetermined area T2 of a predetermined area image being displayed on another communication terminal 9 is superimposed on a predetermined area image of a predetermined area T1 being displayed on the communication terminal 7.

[0164] Figure 23 shows a case where, when the communication terminal 7 displays a predetermined area image of the predetermined area T1 (θ1, φ1, α1) similar to that of Figure 9, a viewpoint display area indicating the predetermined area T2 (θ2, φ2, α2) identified by the viewpoint information sent from the communication terminal 9 is superimposed and displayed.

[0165] In this case, the display control unit 74 calculates the respective display areas based on the viewpoint information that identifies the predetermined area T1 in the communication terminal 7 and the viewpoint information sent from the communication terminal 9.

[0166] S32: The display control unit 74 determines whether or not there is a viewpoint display area at least a part of which is outside the predetermined area image being displayed.

[0167] S33: If there is no viewpoint display area at least a portion of which is outside the specified area image being displayed in process S32 (S32; NO), the display control unit 74 determines whether or not viewpoint information whose content has been changed has been received from the communication processing system 5 by processes S18, S20, and S22.

[0168] S34: In the process S33, if viewpoint information whose content has been changed is received, the display control unit 74 changes the predetermined viewpoint display area, and then the process returns to the above-mentioned process S32.

[0169] For example, if the content of the viewpoint information sent from the communication terminal 9a via the communication processing system 5 changes by enlarging a specified area on the communication terminal 9a side to enlarge and display the materials of interest (S33; YES), the display control unit 74 displays a viewpoint display area 751a that is reduced from the viewpoint display area 750a (S34), as shown in Figure 25.

[0170] Furthermore, for example, when the content of the viewpoint information sent from communication terminal 9a via communication processing system 5 changes as a result of moving the predetermined area on communication terminal 9a to display another material of interest (S33; YES), the display control unit 74 displays viewpoint display area 751b moved from viewpoint display area 750b (S34), as shown in Fig. 26. However, in this case, at least a part of viewpoint display area 751b is outside the displayed predetermined area image 750 and is not displayed.

[0171] S35: In process S33, if viewpoint information with changed content has not been received (there is no change in the received viewpoint information) (S33; NO), the reception unit 72 determines whether or not a change to the specified area of ​​the specified area image being displayed has been received from the field staff member X.

[0172] S36: In the process S35, if a change of the predetermined area is accepted (S35; YES), the display control unit 36 ​​displays the predetermined area image of the changed predetermined area. After that, the process returns to the process S32.

[0173] Furthermore, for example, if at least a part of the viewpoint display area 751b is out of the predetermined area image 750 shown in Fig. 26 and is not displayed, and it is difficult for the site person X to understand what material or the like the participant A is paying attention to, the display control unit 74 displays the predetermined area image 751 obtained by moving the predetermined area from the predetermined area image 750 by manual screen operation on the communication terminal 7 (S35; YES), as shown in Fig. 27 (S36). However, in this case, the entire viewpoint display area 751b is displayed, but at least a part of the other viewpoint display areas 751a, 750c is out of the predetermined area image 751 being displayed and is not displayed.

[0174] S37: Therefore, the display control unit 74 reduces the predetermined area image so that all of the viewpoint display areas are fully displayed (S37). For example, as shown in Fig. 27, if the site worker X manually operates the screen to prioritize displaying the entire area of ​​the viewpoint display area 751b, and therefore at least a portion of the other viewpoint display areas 751a, 750c are outside the predetermined area image 751 shown in Fig. 27 and are not displayed, the display control unit 74 reduces the predetermined area of ​​the predetermined area image 751 so that all of the viewpoint display areas 751a, 751b, 750c are fully displayed, so that the predetermined area image 752 is displayed, as shown in Fig. 28.

[0175] S38: The reception unit 72 determines whether or not an operation to exit the virtual room has been received from the site staff member X. If the site staff member has not exited (S38; NO), the process returns to the above-described process 32. On the other hand, if the site staff member has exited (S38; YES), the process shown in FIG. 22 ends.

[0176] This completes the description of the processing or operation.

[0177] <Determining usage scenarios using wide-field images> By utilizing the fact that the wide-field-of-view image simultaneously captures the task and the background, the communication terminal 7 can determine the usage scene from the wide-field-of-view image. CNN (Convolutional Neural Network) is known as a method of determination using images.

[0178] FIG. 29 shows an example of the configuration of a discrimination model for a usage scene using CNN 160. As an example, CNN 160 has convolutional layers 162 and 164, pooling layers 163 and 165, and a fully connected layer 170. Input image 161 is image data of a construction site. Since a wide-field-of-view image captures a 360-degree surrounding area, input image 161 is either the wide-field-of-view image itself or a predetermined area T displayed on display 507 of communication terminal 7 or 9. In the latter case, parameter determination unit 53 can change the voice processing parameters according to the predetermined area T displayed by communication terminal 7.

[0179] An input image 161 is processed in the order of a convolutional layer 162, a pooling layer 163, a convolutional layer 164, a pooling layer 165, and a fully connected layer 170.

[0180] The convolutional layers 162 and 164 convert lattice-like numerical data called a kernel (or filter) and the numerical data of a partial image (called a window) of the same size as the kernel into a single numerical value by calculating the sum of the products of each element. The convolutional layers 162 and 164 convert this conversion process into small lattice-like numerical data (i.e., tensors) by shifting the window slightly. The lattice-like numerical data is activated by an activation function and input to the pooling layers 163 and 165.

[0181] Pooling layers 163 and 165 are used to create a single numerical value from activated numerical data. Examples include maximum pooling, which selects the maximum value within a window, and average pooling, which selects the average value within a window. Convolutional layers 162 and 164 extract features from the image data, and pooling layers 163 and 165 blur the precise location of the object. Activation functions are functions that nonlinearly transform (activate) the input (e.g., ReLU, tanh, sigmoid, etc.).

[0182] The output of the pooling layer 165 is input to the fully connected layer 170. The fully connected layer 170 is called a neural network. In a neural network, L layers are fully connected from the nodes in the input layer 166 to the nodes in the output layer 168. A neural network with a deep hierarchy is called a DNN. The layer between the input layer 166 and the output layer 168 is called an intermediate layer 167. The number of intermediate layers 167 and the number of nodes in each layer are merely examples.

[0183] Weights are set for the connections between nodes, and the output from a node multiplied by the weight is transmitted to the node in the next layer. The node in the next layer receives the output of all nodes in the previous layer, so the node in the next layer sums the output of all nodes in the previous layer. The node in the next layer activates the summed output using an activation function and transmits it to the next node. This process is repeated until the value is transmitted to the output layer.

[0184] In this embodiment, since it is desired to distinguish between usage scenes, a classification model is generated (another model is a regression model). Therefore, in the model for distinguishing between usage scenes, the output layer 168 is provided with nodes equal to the number of usage scenes to be distinguished. For example, if it is desired to distinguish between five types of usage scenes, the number of nodes in the output layer 168 is five.

[0185] In a classification model, it is common for each node in the output layer 168 to output the probability of being classified into that node. Therefore, in Fig. 29, the output layer 168 outputs the probability of a usage scene associated with each node, such as node 171 being the "probability of completion inspection" and node 172 being the "probability of foundation work." Probabilities of voice processing parameters may be output instead of the probability of a usage scene.

[0186] In the model learning phase, image data for which usage scenarios are known is prepared, so the training data is a vector in which only the node corresponding to the relevant usage scenario is "1" and the other nodes are zero. For example, in the case of image data showing the completion inspection, only node 171 is "1" and the other nodes are "0". The learning unit, which will be described later, calculates the difference between the output (probability) of each node in the output layer 168 and the training data using a loss function, and transmits this to the input layer side using the backpropagation method. The weights between the nodes are learned using the backpropagation method, and gradually nodes 171 to 173 in the output layer begin to output the correct probabilities.

[0187] In the inference phase using the model, for example, when image data that can be considered to be a completion inspection is input, it is expected that node 171 in output layer 168 corresponding to the completion inspection will output a probability close to "1," and nodes 172 and 173 in output layer 168 corresponding to other usage scenarios will output probabilities close to "0." The judgment unit 55 included in the parameter determination unit 53 judges (infers) that the usage scenario corresponds to the node with the highest probability. Note that when image data of a usage scenario that the model has not learned is input, each node in output layer 168 outputs a similar probability, and therefore, when the highest probability is equal to or less than a threshold, the judgment unit 55 judges that the usage scenario is an unclassified usage scenario.

[0188] 29, the output layer 168 outputs a usage scene, but it is also possible to learn so that the output layer 168 outputs a voice processing parameter ID. In this case, the voice processing parameter ID can be used as training data.

[0189] 30 is a functional block diagram of the learning unit 624. The learning unit 624 has a function of generating a discrimination model for a usage scene. Learning may be performed by any computer separate from the communication processing system 5, but the communication processing system 5 may also be used in the learning phase.

[0190] The learning unit 624 has a learning data acquisition unit 641, a learning data storage unit 642, and a model generation unit 643. The learning data acquisition unit 641 acquires learning data. The learning data is as follows. - Usage scenario discrimination model Image data (input data) with known usage scenarios, usage scenarios (teaching data) The learning data acquisition unit 641 acquires learning data and stores it in the learning data storage unit 642. The learning data is a set of input data and teacher data, and multiple sets are prepared (for example, 500 sets).

[0191] The learning data storage unit 642 stores the learning data acquired by the learning data acquisition unit 641. The model generation unit 643 learns the learning data using various machine learning algorithms to generate a discrimination model for the usage scenario. The discrimination model for the usage scenario can also be expressed as correspondence information that associates image data with the usage scenario. The model of this embodiment is a classification model that classifies image data. Note that classification models used in supervised learning include support vector machines, logistic regression, decision trees, random forests, etc. in addition to CNN.

[0192] <Speech processing using speech processing parameters> FIG. 31 is a sequence diagram illustrating the process in which the communication processing system 5 determines the usage scene based on the communication status of image data or sound information and performs audio processing.

[0193] S101: When the conference starts, the connection unit 18 of the imaging device 10 repeatedly transmits the wide-field image and sound information to the relay device 3. The connection unit 38 of the relay device 3 receives the wide-field image and sound information and transmits them to the communication processing system 5. Note that if the imaging device 10 and the communication terminal 7 are connected, the communication terminal 7 may transmit the wide-field image and sound information to the communication processing system 5.

[0194] S102: The communication unit 51 of the communication processing system 5 receives the wide-field image and sound information. The determination unit 55 of the communication processing system 5 determines the usage scene based on the usage scene information DB 5004 of FIGS.

[0195] S102-2: The determination unit 56 determines the voice processing parameter ID associated with the usage scene. Details will be described later.

[0196] S103: Next, the audio processing unit 54 acquires audio processing parameters associated with the audio processing parameter ID determined by the determination unit 56 from the audio processing parameter information DB 5005. The audio processing unit 54 performs audio processing on the sound information using the audio processing parameters. The audio processing is, for example, noise removal.

[0197] S104: The communication unit 51 of the communication processing system 5 transmits the wide-field image and the audio-processed sound information to the communication terminal 9.

[0198] S105: The communication unit 91 of the communication terminal 9 receives the wide-field image and the audio-processed sound information, the display control unit 94 displays the wide-field image on the display 507, and the sound output control unit 95 outputs the audio-processed sound information from the speaker 512.

[0199] FIG. 32 is a sequence diagram illustrating the process in which the communication processing system 5 determines the usage scene based on the voice instructions of the participants and performs voice processing.

[0200] S201: During a conference, a participant operating a communication terminal 9 can optionally give a voice instruction regarding a usage scene. The reception unit 92 receives the voice instruction (perform a tapping test). The communication unit 91 transmits the voice instruction (perform a tapping test) to the communication processing system 5.

[0201] S202: The connection unit 18 of the imaging device 10 repeatedly transmits the wide-field image and sound information to the relay device 3. The connection unit 38 of the relay device 3 receives the wide-field image and sound information and transmits them to the communication processing system 5. Note that if the imaging device 10 and the communication terminal 7 are connected, the communication terminal 7 may transmit the wide-field image and sound information to the communication processing system 5.

[0202] S203: The communication unit 51 of the communication processing system 5 receives a voice instruction (to perform a hammering inspection). The communication unit 51 receives the wide-field image and sound information. The judgment unit 55 of the communication processing system 5 performs existing voice recognition, compares the recognition result with a dictionary, and determines whether the usage scene matches or is similar to a registered usage scene (completion inspection, foundation work, scaffolding work, interior work, evidence imaging, hammering inspection, interior work, etc.).

[0203] S203-2: The determination unit 56 determines the voice processing parameter ID associated with the determined usage scene.

[0204] S204: Next, the audio processing unit 54 acquires audio processing parameters associated with the audio processing parameter ID determined by the determination unit 56 from the audio processing parameter information DB 5005. The audio processing unit 54 performs audio processing on the sound information using the audio processing parameters. The audio processing is, for example, noise removal.

[0205] S205: The communication unit 51 of the communication processing system 5 transmits the wide-field image and the audio-processed sound information to the communication terminal 9.

[0206] S206: The communication unit 91 of the communication terminal 9 receives the wide-field image and the audio-processed sound information, the display control unit 94 displays the wide-field image on the display 507, and the sound output control unit 95 outputs the audio-processed sound information from the speaker 512.

[0207] 33 is a sequence diagram illustrating the process of performing audio processing by the communication processing system 5 by determining the usage scene based on the communication status of image data or sound information. The explanation of FIG. 33 will mainly focus on the differences from FIG.

[0208] S106: The reception unit 92 receives the screen operation from the participant A, and displays a predetermined area image showing the predetermined area T (see FIG. 7) displaying the construction details, etc., that interest the participant A. Then, the communication unit 91 of the communication terminal 9 transmits viewpoint information for identifying the predetermined area T in the wide-field image to the communication processing system 5.

[0209] The subsequent processing may be the same as in Fig. 33. However, in step S102, the determination unit 55 of the communication processing system 5 determines the usage scene only from the predetermined area T. This improves the accuracy of determining the usage scene. Since the usage scene can be determined from the predetermined area T, which can be changed during a meeting, it is possible to respond to changes in the usage scene.

[0210] <<Details on determining usage scenarios>> Fig. 34 is an example of a flowchart illustrating how the communication processing system 5 performs voice processing using the voice processing parameters of Fig. 18. The process of Fig. 34 starts at the start of the conference (at the start of communication), and steps S53 to S55 are repeatedly executed during the conference.

[0211] At the start of a conference or the like, the parameter determination unit 53 of the communication processing system 5 acquires advance information (S51). The advance information is information for identifying a usage scene and voice processing parameters as follows.

[0212] The determination unit 55 acquires the name of the conference registered in the conference scheduler, determines the usage scene that is pre-associated with this conference name, and the determination unit 56 determines the voice processing parameters that are pre-associated with the usage scene.

[0213] The determination unit 55 compares conference information, such as the conference name registered in the conference scheduler, with the conference information of the current conference to search for similar conferences. The audio processing unit 54 stores the usage scenarios or audio processing parameters used in conferences with similar conference information, or the usage scenarios or audio processing parameters most used during the conference. For example, if the conference information is a participant, the audio processing parameters determined by the determination unit 56 in a conference held by the same participants are used. The parameter determination unit 53 stores, for example, the usage scenario or audio processing parameters determined at the end of the previous conference, or the usage scenario or audio processing parameters most used during the conference. It is desirable for all multiple participants to be the same, but if there is no conference with all multiple participants, partial matches are sufficient. Alternatively, the conference information may be the conference name.

[0214] The voice processing unit 54 sets the voice processing parameters acquired in step S51 as initial settings. After the conference starts, the voice processing unit 54 processes voice using these voice processing parameters until the determination unit 55 determines the usage scene in real time. However, even after the conference starts, the determination unit 56 may fix the initially set voice processing parameters.

[0215] When the conference starts, the determination unit 55 determines the usage scene based on the communication status of image data and sound information, voice instructions, or manual settings, and the determination unit 56 determines the current voice processing parameters (S53). (1) The determination unit 55 first inputs the wide-field image (or the predetermined region T') into a discrimination model of the use scene, and determines the use scene. (2) If the probability of any of the usage scenes is less than the threshold, the determination unit 55 determines the usage scene based on the communication status of the sound information (whether there is speech from both sides, or whether there is sound other than speech from the site). (3) The determination unit 55 also determines whether there is a voice instruction or whether the usage scene has been explicitly set manually, and in this case, determines the usage scene by giving priority to (1) and (2).

[0216] Next, the audio processing unit 54 executes audio processing using the noise suppressor strength and noise gate threshold associated with the audio processing parameter ID (S54). The audio processing is, for example, noise removal.

[0217] Next, the communication unit 51 of the communication processing system 5 transmits the wide-field image and the audio-processed sound information to the communication terminal 9 (S55).

[0218] In the description of Fig. 34, the determination unit 55 prioritizes image data over the communication status of sound information, but the priorities may be reversed. For example, if it is not possible to determine that there is speech from both parties and there is no sound other than speech from the construction site, the determination unit 55 determines the usage scene based on the image data. Alternatively, participants may set which of (1) and (2) they would like to prioritize.

[0219] <Determining usage scenarios and variations in devices that perform voice processing> In FIG. 34 etc., the communication processing system 5 determines the usage scene and processes the audio, but the usage scene determination and audio processing may be performed by any of the imaging device 10, the communication terminal 7, the relay device 3, and the communication terminal 9.

[0220] Fig. 35 shows, as an example, a sequence diagram when the imaging device 10 determines a usage scene and performs audio processing. In Fig. 35, the imaging device 10 determines a usage scene (step S301), determines audio processing parameters (S301-2), and performs audio processing (step S302). Note that the imaging device 10 does not receive audio information from the communication terminal 9, and therefore image data, conference information, audio instructions, or manual settings are used to determine the usage scene. When the imaging device 10 receives audio information from the communication terminal 9, the usage scene may be determined based on the communication status of the audio information.

[0221] Furthermore, as shown in Fig. 36, the device that determines the usage scene and the device that performs the audio processing may be different. Fig. 36 shows, as an example, a sequence diagram in which the imaging device 10 determines the usage scene and determines the audio processing parameters, and the communication processing system 5 performs the audio processing. In Fig. 36, the imaging device 10 determines the usage scene (step S401) and determines the audio processing parameters (S401-2), and the communication processing system 5 performs the audio processing (step S403).

[0222] Alternatively, the voice processing may be performed by the communication terminal 9. In this case, the communication processing system 5 may transmit voice processing parameters or voice processing parameter IDs to the communication terminal 9. The communication terminal 9 performs voice processing on the received sound information using the transmitted voice processing parameters or voice processing parameters identified by the voice processing parameter ID.

[0223] <Major Effects> The communication system 1 of this embodiment performs voice processing with appropriate voice processing parameters depending on the usage scenario of the communication system 1, so that participants in remote locations can hear the sounds that are important in remote communication, thereby improving the quality of remote communication.

[0224] [supplement] Although the embodiments have been described above, the present invention is not limited to these embodiments, and various modifications and substitutions can be made without departing from the scope of the present invention.

[0225] For example, the configuration example in Fig. 14 and the like is divided according to main functions to facilitate understanding of the processing by the communication processing system 5, the imaging device 10, and the communication terminals 7 and 9. The present invention is not limited by the manner in which the processing units are divided or the names of the processing units. The processing by the communication processing system 5, the imaging device 10, and the communication terminals 7 and 9 can be divided into even more processing units depending on the processing content. Furthermore, the processing units can also be divided so that even more processes are included in one processing unit.

[0226] Each function of the above-described embodiments can be realized by one or more processing circuits. Here, the term "processing circuit" in this specification includes a processor programmed to perform each function by software, such as a processor implemented by an electronic circuit, as well as devices such as an ASIC (Application Specific Integrated Circuit), a DSP (Digital Signal Processor), an FPGA (Field Programmable Gate Array), or a conventional circuit module designed to perform each function described above.

[0227] Additionally, the above-described devices represent only one of several computing environments for implementing the embodiments disclosed herein. For example, communication processing system 5 may include multiple computing devices, such as a server cluster, configured to communicate with each other via any type of communication link, including a network, shared memory, or the like, and to perform the processes disclosed herein.

[0228] Furthermore, the communication processing system 5 may be integrated into one server device, or may be divided into multiple devices.

[0229] Furthermore, the programs of the above-described embodiments can be provided domestically or internationally as a (non-transitory) recording medium such as a DVD-ROM on which the programs are stored, or as a program product.

[0230] Furthermore, there may be multiple CPUs 111, 301, and 501 serving as processors. [Explanation of symbols]

[0231] 1. Communication Systems 3. Relay Device 5. Communication Processing System 7. Communication terminals 9. Communication terminals 10. Imaging device

Prior Technical Literature

Charter Documents

[0232] [Patent Document 1] Patent Gazette No. 6201279

Claims

1. A communication system for transmitting and receiving images and audio between multiple locations, a communication unit that receives the wide-field image and sound information acquired at the first location; a parameter determination unit that determines audio processing parameters associated with a usage scene determined based on the wide-field image received by the communication unit; a sound processing unit that performs sound processing on the sound information using the sound processing parameters determined by the parameter determination unit; a sound output control unit that outputs the sound information that has been sound-processed by the sound processing unit to a second location; A communication system having:

2. The parameter determination unit a determination unit that determines the usage scene from the wide-field image received by the communication unit based on a discrimination model of the usage scene in which the wide-field image and the usage scene are associated with each other; a determination unit that determines the voice processing parameters that are associated in advance with the usage scene determined by the determination unit; 2. The communication system of claim 1, comprising:

3. When the sound information of the first location includes speech and the sound information of the second location includes speech, The communication system according to claim 1 , wherein the parameter determination unit determines the voice processing parameters suitable for conversation.

4. When the sound information of the first location includes a sound that is not speech and the sound information of the second location includes a speech, The communication system according to claim 1 , wherein the parameter determination unit determines the audio processing parameters suitable for outputting background sound.

5. the parameter determination unit performs speech recognition on the sound information of the second location, and when a recognition result includes information indicating the speech processing parameter, 2. The communication system according to claim 1, wherein the voice processing parameter is determined based on information indicating the voice processing parameter.

6. the information indicating the sound processing parameters indicates a hammering test; The communication system according to claim 5 , wherein the parameter determination unit determines the voice processing parameters suitable for the hammering test.

7. When the communication system is used for remote conferencing, The communication system according to claim 2 , wherein the parameter determination unit determines the usage scene that is associated in advance with a conference name related to the remote conference.

8. When the communication system is used for remote conferencing, the parameter determination unit searches for conference information similar to the conference information regarding the remote conference; The communication system according to claim 1 , wherein the voice processing parameters used in the remote conference of the conference information determined to be similar are used.

9. The communication system according to claim 2, wherein the usage scenario is a site survey, structural work, exterior work, equipment work, exterior construction work, demolition work, completion inspection, foundation work, scaffolding work, interior construction work, evidence imaging, or hammering inspection at a construction site.

10. The parameter determination unit When the usage scene is the completion inspection, determining the voice processing parameters that make it easier to hear the sound that is the target of listening during the completion inspection; When the usage scene is the foundation work, the voice processing parameters are determined so that sounds that are the subject of the foundation work can be easily heard; When the usage scene is the scaffolding work, the voice processing parameters are determined so that sounds that are the target of listening during the scaffolding work can be easily heard; When the usage scene is the interior construction work, the voice processing parameters are determined so that sounds to be heard during the interior construction work can be easily heard; When the usage scene is capturing evidence, determining the audio processing parameters that make it easier to hear a sound that is a target for listening when capturing evidence; The communication system according to claim 9 , wherein when the usage scene is the hammering test, the voice processing parameters are determined so that a sound to be heard during the hammering test becomes easier to hear.

11. When the communication terminal at the second location receives an operation to display a predetermined area of ​​the wide-field image on a display, The communication system according to claim 2 , wherein the parameter determination unit determines the usage scene by inputting the predetermined region of the wide-field image received by the communication unit into a discrimination model of the usage scene.

12. A voice processing method performed by a communication system that transmits and receives images and voices at multiple locations, comprising: receiving wide-field images and sound information acquired at a first location; determining audio processing parameters associated with the usage scene determined based on the received wide-field image; a process of performing audio processing on the sound information using the determined audio processing parameters; outputting the processed sound information to a second location; An audio processing method.

13. A communication processing system that sends and receives images and audio at multiple locations. a communication unit that receives the wide-field image and sound information acquired at the first location; a parameter determination unit that determines audio processing parameters associated with a usage scene determined based on the wide-field image received by the communication unit; a sound processing unit that performs sound processing on the sound information using the sound processing parameters determined by the parameter determination unit; The communication unit is a program that transmits the sound information that has been subjected to sound processing by the sound processing unit to a second location.

14. An imaging device that acquires a wide-field image and sound information at a first location, a parameter determination unit that determines audio processing parameters associated with a usage scene determined based on the wide-field image; a sound processing unit that performs sound processing on the sound information using the sound processing parameters determined by the parameter determination unit; a connection unit that transmits the sound information that has been processed by the sound processing unit to a second location via a network; An imaging device having the above configuration.

15. A communication processing system for transmitting and receiving images and audio between multiple locations, a communication unit that receives the wide-field image and sound information acquired at the first location; a parameter determination unit that determines audio processing parameters associated with a usage scene determined based on the wide-field image received by the communication unit; a sound processing unit that performs sound processing on the sound information using the sound processing parameters determined by the parameter determination unit, The communication unit transmits the sound information that has been subjected to sound processing by the sound processing unit to a second location.

Images