Multimedia communication system, work support method, and program

Abstract

To reduce a processing load on a terminal used on-site for work support, in a multimedia communication system.SOLUTION: A multimedia communication system of an embodiment includes: a terminal that is used by a worker performing work at a predetermined work site; and a server that is connected to the terminal via a communication network capable of allocating bandwidth of network resources. The terminal includes: a display unit that displays information; an audio input unit that inputs ambient audio; and an audio output unit that outputs audio. The server includes: an overlay video processing unit that performs predetermined overlay video processing on video acquired from a photographing unit that photographs the work site to create overlay video; and a communication processing unit that processes communication services using multiple information media including at least video and audio with the terminal.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a multimedia communication system, a work support method, and a program. [Background technology]

[0002] In the past, when on-site work such as construction or inspection was performed by inexperienced or new personnel, a veteran would stand by and provide support by giving instructions, etc. In such cases, there was a problem of an increase in human resources, as it was necessary to have another support person accompany the worker at all times.

[0003] In addition, when supporters provide support at multiple remote work sites, it is necessary to incur time and financial costs for the supporters to travel to each site. Furthermore, in recent years, with the increasing risk of infectious diseases, there has been an increasing need to "control the flow of people" and "reduce the number of people working together."

[0004] Due to these circumstances, currently, remote support for on-site workers is mainly provided by voice instructions over the telephone, which makes it difficult for the supporter to grasp the situation on-site and provide appropriate support.

[0005] To address this issue, remote work support systems have been gaining attention in recent years. These systems use camera-equipped VR (Virtual Reality) / MR (Mixed Reality) glasses, allowing remote supporters to grasp the on-site situation in detail in real time and send support video to on-site workers, thereby enabling smooth work support. The technology that combines VR and MR is called XR (Extended Reality).

[0006] Furthermore, low-latency, wideband communications such as 5G (5th Generation: 5th generation mobile communications system) and local 5G are becoming more widespread, and particularly at work sites such as factories, there is growing consideration of building work support systems using local 5G, which has a low possibility of radio interference and can be adjusted according to the radio wave environment and purpose of the site.

[0007] In XR, on-site video is analyzed using camera footage and geographic information, and 3D (dimensional) objects are laid out in appropriate positions on the on-site video and displayed as overlays. Multimedia communication systems that comprehensively exchange audio, video, and the ability to overlay 3D objects on such on-site video, such as work support systems using XR, enable more sophisticated information transmission than conventional communication systems, and are therefore attracting attention as a means of communication for work support. [Prior art documents] [Patent documents]

[0008] [Patent Document 1] Patent Publication No. 2021-51519 [Patent Document 2] Japanese Patent Application Publication No. 2020-149138 Summary of the Invention [Problem to be solved by the invention]

[0009] However, in order to support advanced XR functions, multimedia communication systems used for work support require highly functional on-site terminals, and the significant barriers to implementation due to the hardware requirements are a problem.

[0010] Therefore, an object of the embodiments of the present invention is to provide a multimedia communication system, a work support method, and a program that are used for work support and can reduce the processing load on terminals used on-site. [Means for solving the problem]

[0011] A multimedia communication system according to an embodiment includes a terminal used by a worker performing work at a predetermined work site, and a server connected to the terminal via a communication network capable of allocating bandwidth for network resources, wherein the terminal includes a display unit for displaying information, an audio input unit for inputting ambient audio, and an audio output unit for outputting audio. The server includes an overlay video processing unit that performs predetermined overlay video processing on video acquired from a camera unit that captures the work site to create an overlay video, and a communication processing unit that processes communication services using multiple information media including at least video and audio between the terminal and the server. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 is a diagram showing the overall configuration of a multimedia communication system according to the first embodiment. [Figure 2] FIG. 2 is an explanatory diagram of a processing sequence in the multimedia communication system of the first embodiment. [Figure 3] FIG. 3 is an explanatory diagram regarding the restriction of the transmission band by monitoring the network load. [Figure 4] FIG. 4 is an explanatory diagram regarding restrictions on display objects according to transmission bands. [Figure 5] FIG. 5 is a diagram showing the overall configuration of a multimedia communication system according to the second embodiment. [Figure 6] FIG. 6 is an explanatory diagram of a processing sequence in the multimedia communication system of the second embodiment. [Figure 7] FIG. 7 is a diagram showing the overall configuration of a multimedia communication system according to the third embodiment. [Figure 8]FIG. 8 is an explanatory diagram of a processing sequence in the multimedia communication system of the third embodiment. [Figure 9] FIG. 9 is a diagram showing an example of the configuration of a conventional multimedia communication system. DETAILED DESCRIPTION OF THE INVENTION

[0013] Hereinafter, embodiments (first to third embodiments) of a multimedia communication system, a work support method, and a program of the present invention will be described with reference to the accompanying drawings.

[0014] (Prior Art) To facilitate understanding of the embodiments, the prior art will be explained again. Fig. 9 is a diagram showing an example of the configuration of a conventional multimedia communication system. The terminals of the multimedia communication system are devices used by workers who perform work at a specific work site.

[0015] The terminal includes a microphone, a speaker, a camera, a storage unit, a display unit, and a processing unit. The processing unit also includes a video analysis processing unit, an overlay video processing unit, an image processing unit, and a communication processing unit.

[0016] The microphone inputs the sounds around the worker, etc. The speaker outputs sound based on audio data from a server (not shown). The camera captures the area to be captured (work site) and outputs video data. The memory unit stores various types of information. The display unit displays various types of information such as video.

[0017] The video analysis processing unit analyzes the video captured by the camera, and the overlay video processing unit performs predetermined overlay video processing on the video captured by the camera to create an overlay video.

[0018] The image processing unit performs video analysis. The communication processing unit processes communication services between the terminals using multiple information media including at least video and audio.

[0019] In this way, in conventional technology, all functions required to build a multimedia communication system are realized within a single device. Note that the display unit is an organic electroluminescence (EL) display or the like onto which images are projected if the device is an XR glass, and is equivalent to a touch panel display if the device is a smartphone or tablet.

[0020] However, in this case, high-performance on-site terminals are required, and the hardware requirements create significant barriers to implementation. Therefore, below we will explain a technology that can reduce the processing load on terminals used for work support in the field in a multimedia communication system, thereby suppressing the required functions and hardware performance.

[0021] (First embodiment) 1 is an overall configuration diagram of a multimedia communication system S according to a first embodiment. The multimedia communication system S includes a terminal 1 used by a worker performing work at a predetermined work site, and an application server 2 (server) connected to the terminal 1 via a communication network N capable of allocating network resource bandwidth with low latency. Note that the communication network N may be, for example, a low-latency wired communication network, or a low-latency wireless communication network such as 5G or local 5G.

[0022] 1, terminals 1 and application servers 2 are in one-to-one correspondence, but this is not limiting, and multiple terminals 1 may be associated with one application server 2. In this case, a single application server 2 can provide multimedia communication services to multiple remote terminals 1 (details will be described later).

[0023] The terminal 1 is, for example, XR glasses worn by a worker. The terminal 1 includes a microphone 11 (audio input unit), a speaker 12 (audio output unit), a camera 13 (photographing unit), a storage unit 14, a display unit 15, and a processing unit 16.

[0024] The microphone 11 inputs the sounds around the worker, etc. The speaker 12 outputs sounds based on the audio data from the application server 2. The camera 13 captures the area to be photographed (specifically, the subject (real-world object) located in front of the worker's face) and outputs the video data.

[0025] The storage unit 14 is configured by, for example, a random access memory (RAM), a read only memory (ROM), a flash memory, etc., and stores various types of information.

[0026] The display unit 15 displays various information such as images. The display unit 15 corresponds to, for example, the eyeglass lens portion located in front of the eyes of the worker when the worker wears the XR glasses, and has an optically transparent display for the right eye and a display for the left eye.

[0027] The processing unit 16 corresponds to a so-called processor and is configured, for example, by a CPU (Central Processing Unit), an MPU (Micro Processing Unit), a GPU (Graphics Processing Unit), etc., and executes various processes. The processing unit 16 may also be configured, for example, by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0028] The processing unit 16 includes a communication relay processing unit 161. The communication relay processing unit 161 relays various communications with external devices such as the application server 2.

[0029] The terminal 1 also includes various sensors such as an acceleration sensor and an angular velocity sensor, a communication interface for communicating with an external device such as the application server 2, and the like, as necessary.

[0030] Furthermore, the terminal 1 is not limited to XR glasses, but may also be a smartphone, a tablet terminal, a PC (Personal Computer) equipped with microphone and speaker functions, etc. The microphone 11, speaker 12, and display unit 15 may be provided outside the terminal 1 as an interface.

[0031] The application server 2 includes a storage unit 21 , a processing unit 22 , a microphone 23 , a speaker 24 , and a display unit 25 .

[0032] The microphone 23 inputs the surrounding voice of the user of the application server 2. The speaker 24 outputs voice based on the voice data from the terminal 1.

[0033] The storage unit 21 is configured by, for example, a RAM, a ROM, a solid state drive (SSD), a hard disk drive (HDD), etc., and stores various types of information.

[0034] The display unit 25 is configured by, for example, an LCD (Liquid Crystal Display) or the like, and displays various information such as images. The microphone 23, speaker 24, and display unit 25 may be provided outside the application server 2 as an interface.

[0035] The processing unit 22 corresponds to a so-called processor and is configured, for example, by a CPU, an MPU, a GPU, etc., and executes various processes. The processing unit 22 may also be configured, for example, by an integrated circuit such as an ASIC or an FPGA.

[0036] The processing unit 22 includes a communication relay processing unit 221 , an image processing unit 222 , a video analysis processing unit 223 , an overlay video processing unit 224 , a communication processing unit 225 , and a network monitoring processing unit 226 .

[0037] The communication relay processing unit 221 transmits and receives at least video, audio, and control signals to and from the terminal 1 .

[0038] When the terminal 1 does not have an interface for inputting information by the worker (mouse, keyboard, etc.), the image processing unit 222 performs processing to determine that input of information associated with the set action (for example, start or end of a specified function of the terminal 1 (work support function, etc.), zooming in or out of the image, etc.) has been accepted when the action (for example, hand or finger movement (gesture)) of the worker obtained from the camera 13 matches a predetermined set action.

[0039] The video analysis processing unit 223 analyzes the video captured by the camera 13 .

[0040] The overlay image processor 224 performs predetermined overlay image processing on the image acquired from the camera 13 capturing the work site to create an overlay image. In this case, for example, the overlay image processor 224 analyzes the image, determines object information (e.g., the conveyor belt and robot arm in FIG. 4(b)) to be displayed on the display unit 15 of the terminal 1, calculates the position of the object information to be displayed from spatial information including at least one of the position and direction of the terminal 1, and creates the overlay image. Note that the object information may or may not be present at the work site.

[0041] Furthermore, for example, when there is a shortage of available communication capacity, the overlay video processing unit 224 executes at least one of limiting the information of objects (3D objects) to be displayed in an overlay manner, lowering the video frame rate, reducing the amount of displayed information, and narrowing the display range. This will be explained using FIG. 4.

[0042] Figure 4 is an explanatory diagram regarding the restriction of displayed objects according to the transmission bandwidth. When restricting the display of 3D objects, there are several methods for suppressing throughput, one of which is to limit the object display range shown in Figure 4(a). For example, when a worker wearing XR glasses moves around and works within a work site building, a 3D object is displayed on the XR glasses and is visually recognized by the worker.

[0043] When the requested throughput is suppressed, the overlay video processing unit 224 of the application server 2 adjusts the display range of the 3D object to limit (suppress) the required communication volume. Specifically, under normal conditions, objects within a certain range R1 centered on the terminal 1 are displayed on the XR glasses, but when the bandwidth is low, only objects within a range R2 narrower than the range R1 are displayed. In this way, the display range of the 3D object can be narrowed as needed, making it possible to flexibly respond to network loads.

[0044] Furthermore, in terms of the method of limiting 3D objects, in addition to changing the display range, it is also possible to address bandwidth limitations by lowering the quality of the 3D object displayed on the XR glasses. Specifically, as shown in (b), when a 3D object is displayed on the XR glasses, a high-resolution 3D object is normally displayed as shown in (b1), but when the bandwidth is low, the required communication volume is limited by changing the object to a simpler one by reducing the number of vertices, as shown in (b2). Note that specific methods for changing to such a simpler object include, for example, preparing a simpler object in advance, or creating a temporary object by lowering the resolution of the 3D object.

[0045] Furthermore, with regard to the method of restricting a 3D object, the display range restriction and the object quality restriction can be applied individually or in combination, and there are no limitations on the restriction method.

[0046] Returning to FIG. 1, the communication processing unit 225 executes processing for communication services between the terminal 1 and the terminal 1 using a plurality of information media including at least video and audio.

[0047] Furthermore, there may be cases where the application server 2 is connected to other terminals in addition to the terminal 1 via the communication network N. In such cases, the communication processing unit 225 may provide the communication service provided to the terminal 1 to the other terminals in parallel.

[0048] The network monitoring processor 226 monitors the traffic volume in the communication network N, and adjusts (increases or decreases) the bandwidth resources according to the traffic volume by changing the settings of the communication network N. For example, when changing the settings of the communication network, the network monitoring processor 226 changes the settings by at least one of changing the ratio of upstream communication / downstream communication, changing the frame format in TDD (Time Division Duplex), changing the modulation method in wireless communication, changing the communication resources, and changing the allocation of virtual communication resources (details will be described later).

[0049] Furthermore, when there is a shortage of available communication capacity, the network monitoring processor 226 controls the quality of each application according to the priority of each application of the information medium, and allocates bandwidth resources to secure the transmission bandwidth.

[0050] For example, when controlling the quality of each application according to priority, the network monitoring processor 226 changes at least one of the audio compression method, the video compression method, and the upper limit communication capacity. The processing of the network monitoring processor 226 will be described with reference to FIG.

[0051] FIG. 3 is an explanatory diagram regarding the restriction of transmission bandwidth by network load monitoring. The communication network N is a network capable of allocating network resources to the communication of each application. For example, as typified by a technology called network slicing in 5G and local 5G, it is possible to guarantee or restrict communication quality by allocating each communication data to a virtual communication resource called a slice. In this embodiment, application to a communication network capable of such bandwidth restriction is assumed.

[0052] The network monitoring processor 226 of the application server 2 constantly monitors the load status of the communication network N. If the usage status has room for the communication bandwidth, no bandwidth adjustment is necessary, and if the network usage status becomes constrained, bandwidth adjustment is performed. The network monitoring processor 226 manages the priority, quality, and required network resources of each multimedia communication, and allocates an appropriate communication bandwidth as a network slicing resource based on these items and the load status of the communication network N. In addition to communications in multimedia communication, the network load is also expected to include bandwidth loads caused by other applications using the same communication network N. The network monitoring processor 226 monitors the comprehensive load status of the communication network N, including other applications.

[0053] Figure 4 shows an example of allocation of multimedia communication types: voice calls, video, 3D objects, and procedure manual downloads. Voice calls have the highest priority and communication interruptions cannot be tolerated, so highly reliable communication is ensured by allocating the necessary amount of network slicing resources to the requested network resources.

[0054] Video communication, which has the next highest priority, requires large network resources but has a lower priority than voice calls, so when network usage becomes constrained, the quality is lowered and resource allocation is limited to prioritize it over voice calls, thereby reducing the load on the communication network N.

[0055] For 3D objects and procedure manual downloads, which are set to the lowest priority level, when network usage becomes constrained, the load on the communication network N is reduced by limiting resource allocation to the minimum level that allows communication.

[0056] When quality is reduced in accordance with resource limitations for various multimedia communications due to transmission bandwidth restrictions, for example, for audio and video communications, throughput is reduced by switching to a communication method with a high compression rate. For procedure manual downloads, throughput is also reduced by limiting network resources and reducing throughput, thereby accepting longer download times. For 3D objects, the required communication volume is reduced by limiting the display objects mentioned above (Figure 4), among other measures.

[0057] Furthermore, if the communication network N is capable of changing the communication bandwidth through network settings, the network monitoring processing unit 226 can calculate the amount of required resources and change the network settings. For example, in local 5G, multiple frame formats with different upstream / downstream communication ratios are provided as communication frame formats, and changing the frame format makes it possible to change the upstream / downstream communication ratio. Therefore, for example, if the application usage status indicates a high demand for upstream communication but there is room for downstream communication, it is possible to ensure stable communication by setting a frame format with a larger upstream communication bandwidth for the communication path. Such changes in network settings are not limited to local 5G and can be applied to other communication networks if similar changes are possible.

[0058] In this way, it is possible to ensure quality while distributing the functions of multimedia communication between the terminal 1 and the application server 2 and suppressing congestion in the communication band of the communication network N. Note that the above explanation shows an example of transmission band restriction, and band restriction using other methods is also possible.

[0059] Next, a processing sequence in the multimedia communication system S will be described with reference to Fig. 2. Fig. 2 is an explanatory diagram of a processing sequence in the multimedia communication system S of the first embodiment.

[0060] During work assistance, voice data input through the microphone 11 of the terminal 1 is transmitted to the application server 2 via the communication relay processing unit 161, and then transmitted to the communication processing unit 225 via the communication relay processing unit 221 of the application server 2 (S1). Then, the voice is output from the speaker 24.

[0061] Furthermore, the voice data input by the microphone 23 of the application server 2 is transmitted to the terminal 1 via the communication processing unit 225 and the communication relay processing unit 221, and is then sent to the communication relay processing unit 161 of the terminal 1. Then, the voice is output from the speaker 12 (S2).

[0062] Furthermore, the video captured by the camera 13 of the terminal 1 is transmitted to the application server 2 via the communication relay processing unit 161, and then transmitted to the video analysis processing unit 223 via the communication relay processing unit 221 of the application server 2 (S3). The video may also be displayed on the display unit 25.

[0063] In addition, the video analysis processing unit 223 performs image analysis using the functions of the image processing unit 222, generates a video object (3D object) to be projected in multimedia communication, determines the display position of the video object, and communicates this to the overlay video processing unit 224 (S4).

[0064] The overlay video processing unit 224 overlays the generated video object on the actual video to generate an overlay video, and transmits the overlay video to the terminal 1 via the communication relay processing unit 221. The display unit 15 of the terminal 1 displays the overlay video (S5).

[0065] In addition, the image processing unit 222 of the application server 2 can analyze the images captured by the camera 13 of the terminal 1 through image processing, and if the terminal 1 is a lightweight, simple XR glass or the like and does not have an input interface such as a mouse or keyboard, it can also substitute the input function with the movements (gestures) of the hands and fingers captured in the camera image.

[0066] In this way, according to the first embodiment of the multimedia communication system S, by distributing functions between the terminal 1 and the application server 2, the processing load on the terminal 1 is reduced and the functions and hardware performance required for the terminal 1 can be reduced.

[0067] In other words, by distributing the XR functions between the terminal 1 used by the on-site worker and the remote application server 2 using a low-latency communication network N, it is possible to lower the hardware specifications required for the terminal 1. Also, by moving the XR functions from the on-site terminal 1 to the application server 2, it becomes easier to manage and control the application functions. Furthermore, while distributing the functions in this way, the terminal 1 and application server 2 can be operated virtually as a single device.

[0068] Furthermore, depending on the communication volume in the communication network N, bandwidth resources are adjusted by changing the settings of the communication network N by changing the ratio of uplink communication / downlink communication, changing the frame format in TDD, changing the modulation method in wireless communication, changing communication resources (e.g., changing the number of allocated wireless channels, etc.), changing the allocation of virtual communication resources (e.g., virtual CPUs, virtual base stations, etc.), etc. This makes it possible to suppress quality degradation. Note that increasing communication efficiency may also increase the error rate, so it is preferable to take this into consideration as well.

[0069] Furthermore, when there is a shortage of available communication capacity, the quality of each application on the information medium can be controlled according to the priority of each application, and bandwidth resources can be allocated to secure the transmission bandwidth (Fig. 3). In this case, specific examples include changing the audio compression method, changing the video compression method, or changing the upper limit of communication capacity.

[0070] Furthermore, when overlay video is used and there is a shortage of available communication capacity, the overlay video can be displayed on the display unit 15 of the terminal 1 while suppressing degradation in quality by limiting the object information to be overlaid and displayed, lowering the video frame rate, reducing the displayed information, narrowing the display range, etc. (Figure 4).

[0071] Furthermore, even if the terminal 1 does not have an interface for the worker to input information, by enabling information input using the actions of the worker shown in the video, a simpler terminal 1 can be used.

[0072] Furthermore, when other terminals are used in addition to terminal 1, the application server 2 can provide the communication services provided to terminal 1 to the other terminals in parallel. This allows, for example, a worker wearing terminal 1 and a worker wearing another terminal to share the same video and audio, making work support more effective.

[0073] (Second embodiment) Next, a second embodiment will be described. Fig. 5 is a diagram showing the overall configuration of a multimedia communication system S according to the second embodiment. Explanations of matters similar to those in the first embodiment will be omitted where appropriate.

[0074] The multimedia communication system S includes a signal repeater 3 and a signal repeater 4 installed between a terminal 1 and an application server 2. The signal repeater 3 includes a communication relay processing unit 31.

[0075] The terminal 1 also includes an I / F unit 17. The application server 2 also includes an I / F unit 26. The I / F unit 17 and the I / F unit 26 may be any connection unit capable of transmitting video and audio signals, such as a Universal Serial Bus (USB) or a High-Definition Multimedia Interface (HDMI) (registered trademark), and the type of the connection unit is not limited.

[0076] In this case, the terminal 1 does not need to have the communication relay processing unit 161 as shown in Fig. 1. Furthermore, the application server 2 does not need to have the communication relay processing unit 221 as shown in Fig. 1. The communication relay processing unit 31 of the signal repeater 3 takes over the function of the communication relay processing unit 161, and the signal repeater 4 takes over the function of the communication relay processing unit 221. The signal repeater 3 and the signal repeater 4 are connected by a communication network N that has low delay and is capable of limiting bandwidth.

[0077] Fig. 6 is an explanatory diagram of a processing sequence in the multimedia communication system S of the second embodiment. S11 to S15 in Fig. 6 correspond to S1 to S5 in Fig. 1, respectively, and are the same as Fig. 1 except that they go through signal repeater 3 and signal repeater 4, so detailed explanation will be omitted.

[0078] Thus, according to the second embodiment, in addition to the same effects as the first embodiment, the multimedia communication system S can be realized using a simpler terminal 1 that does not have a communication relay processing unit 161.

[0079] (Third embodiment) Next, a second embodiment will be described. Fig. 7 is a diagram showing the overall configuration of a multimedia communication system S according to the third embodiment. Explanations of matters similar to those in the second embodiment will be omitted where appropriate.

[0080] 5, an external camera 5 is provided. The application server 2 also includes a video reception processing unit 227.

[0081] Fig. 8 is an explanatory diagram of a processing sequence in the multimedia communication system S of the third embodiment. In S23, the video captured by the external camera 5 is transmitted to the application server 2 via the communication network N, and then transmitted to the video analysis processing unit 223 via the video reception processing unit 227 of the application server 2. Furthermore, S21, S22, S24, and S25 in Fig. 8 correspond to S11, S12, S14, and S12 in Fig. 6, respectively, and are the same as Fig. 6, so detailed description thereof will be omitted.

[0082] Thus, according to the third embodiment, in addition to the same effects as the second embodiment, it is possible to realize a multimedia communication system S using a simpler terminal 1 that does not have a camera 13.

[0083] The external camera 5 may be communicatively connected to the application server 2 via the signal repeaters 3 and 4.

[0084] The programs executed by the multimedia communication system S of this embodiment can be provided by being recorded in an installable or executable file format on a computer-readable recording medium such as a CD (Compact Disc)-ROM (Read Only Memory), a flexible disk (FD), a CD-R (Recordable), or a DVD (Digital Versatile Disk).The programs may also be provided or distributed via a network such as the Internet.

[0085] Although several embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be embodied in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included within the scope and spirit of the invention, and are also included in the scope of the invention and its equivalents as defined in the claims.

[0086] For example, the communication network to which the present invention is applied is not limited to 5G (including local 5G), but may be 6G or later or other communication networks as long as the communication network is capable of allocating bandwidth for network resources.

[0087] 1 and other figures show the application server 2 as including the microphone 23, the speaker 24, and the display unit 25, but this is not limiting. Some or all of the microphone 23, the speaker 24, and the display unit 25 may be external to the application server 2. [Explanation of symbols]

[0088] 1...terminal, 2...application server, 3...signal repeater, 4...signal repeater, 5...external camera, 11...microphone, 12...speaker, 13...camera, 14...storage unit, 15...display unit, 16...processing unit, 17...I / F unit, 21...storage unit, 22...processing unit, 23...microphone, 24...speaker, 25...display unit, 26...I / F unit, 31...communication relay processing unit, 221...communication relay processing unit, 222...image processing unit, 223...video analysis processing unit, 224...overlay video processing unit, 225...communication processing unit, 226...network monitoring processing unit, 227...video reception processing unit, N...communication network, S...multimedia communication system

Claims

1. A multimedia communication system comprising: a terminal used by a worker performing work at a predetermined work site; and a server connected to the terminal via a communication network capable of allocating bandwidth of network resources, the terminal comprises a display unit that displays information, an audio input unit that inputs surrounding audio, and an audio output unit that outputs audio; The server an overlay image processing unit that performs predetermined overlay image processing on the image acquired from the image capturing unit that captured the work site, thereby creating an overlay image; a communication processing unit that executes processing for communication services using a plurality of information media including at least video and audio between the terminals;

2. 2. The multimedia communication system according to claim 1, wherein said server further comprises a communication relay processing unit for transmitting and receiving at least video, audio and control signals to and from said terminal.

3. 2. The multimedia communication system according to claim 1, wherein the server further comprises a network monitoring processing unit that monitors communication volume in the communication network and adjusts bandwidth resources by changing settings of the communication network in accordance with the communication volume.

4. The multimedia communication system of claim 3, wherein when changing the settings of the communication network, the network monitoring processing unit changes the settings by at least one of changing the ratio of upstream communication / downstream communication, changing the frame format in TDD (Time Division Duplex), changing the modulation method in wireless communication, changing communication resources, and changing the allocation of virtual communication resources.

5. The multimedia communication system of claim 3, wherein the network monitoring processing unit, when there is a shortage of available communication capacity, controls the quality of each application according to the priority of each application of the information medium, and allocates bandwidth resources to secure transmission bandwidth.

6. The multimedia communication system described in claim 5, wherein the network monitoring processing unit, when controlling the quality of each application according to the priority, performs at least one of changing the audio compression method, changing the video compression method, and changing the upper limit communication capacity.

7. The multimedia communication system of claim 1, wherein when the overlay image processing unit performs the specified overlay image processing on the image acquired from the imaging unit to create an overlay image, the overlay image processing unit analyzes the image, determines object information to be displayed on the display unit of the terminal, calculates the position of the object information to be displayed from spatial information including at least one of the position and orientation of the terminal, and creates the overlay image.

8. The multimedia communication system described in claim 7, wherein when the overlay image processing unit performs the specified overlay image processing on the image acquired from the shooting unit to create an overlay image, if there is insufficient free communication capacity, it performs at least one of limiting the object information to be overlaid and displayed, lowering the image frame rate, reducing the displayed information, and narrowing the display range.

9. The multimedia communication system of claim 1, wherein the server further includes an image processing unit that, when the terminal does not have an interface for inputting information by the worker, performs processing to determine that input of information associated with the setting action has been accepted when the action matches a predetermined setting action based on video including the worker's action acquired from the imaging unit.

10. the server is connected to other terminals in addition to the terminal via the communication network; 2. The multimedia communication system according to claim 1, wherein said communication processing unit provides said communication service provided to said terminal to said other terminals in parallel.

11. 2. The multimedia communication system according to claim 1, further comprising a signal repeater installed between said terminal and said server for repeating communication between said terminal and said server.

12. A work support method using a multimedia communication system including a terminal used by a worker performing work at a predetermined work site and a server connected to the terminal via a communication network capable of allocating bandwidth for network resources, comprising: the terminal includes a display unit that displays information, an audio input unit that inputs surrounding audio, and an audio output unit that outputs audio; In the server, an overlay image processing unit performing predetermined overlay image processing on the image acquired from the image capturing unit that captured the work site to create an overlay image; a step in which a communication processing unit executes processing of a communication service using a plurality of information media including at least video and audio between the terminal and the communication processing unit.

13. A program to be executed by a computer that is a server in a multimedia communication system that includes a terminal used by a worker performing work at a predetermined work site and a server connected to the terminal via a communication network that can allocate bandwidth for network resources, the program comprising: the terminal includes a display unit that displays information, an audio input unit that inputs surrounding audio, and an audio output unit that outputs audio; The computer an overlay image processing unit that performs predetermined overlay image processing on the image acquired from the image capturing unit that captured the work site, thereby creating an overlay image; A program for causing the terminal to function as a communication processing unit that executes processing of communication services using multiple information media including at least video and audio between the terminal and the terminal.

Images