Moving image transmission system and moving image transmission method

Abstract

[Issue] The range of high-quality moving images that can be directly transmitted by the transmitting terminal can be arbitrarily selected on the receiving end side. [Solution] In a moving image transmission system including a transmitting terminal 10 and a receiving terminal 20 connected via a communication line, the transmitting terminal 10 transmits a moving image obtained by a camera unit to the receiving terminal 20. The receiving terminal 20 specifies a specific area of ​​interest from the moving image received by the transmitting terminal 10 and transmits information related to the area of ​​interest to the transmitting terminal 10. If the transmitting terminal 10 receives information related to the area of ​​interest from the receiving terminal 20, a modified moving image is generated based on the information related to the area of ​​interest and transmitted to the receiving terminal 20. The modified moving image is processed by cutting out the moving image within the area of ​​interest from the original moving image obtained by the camera unit or reducing the quality of the moving image outside the area of ​​interest from the original moving image.

Description

Technical Field

[0001] The present invention relates to a system for transmitting a moving image from a transmitting terminal to a receiving terminal via a communication line, the transmitting terminal itself, a program for the transmitting terminal, and a method for transmitting the moving image. Background Art

[0002] In recent years, the DX (digital transformation) of field operations has become a pressing issue to address various challenges, including the increasing complexity of field operations, a declining workforce, and the need to pass on skills held by skilled workers. As a solution to these issues, a remote support system has been proposed that connects a wearable device worn by field workers and a computer operated by a supporter via the internet, allowing them to share audio and visual information (Patent Document 1).

[0003] In this remote support system, a wearable device (transmitting terminal) captures moving images of the worker's surroundings using a camera and transmits them to a computer operated by the supporter (receiving terminal). In recent years, small cameras have become capable of capturing high-resolution, high-frame-rate moving images. However, due to bandwidth limitations on the communication lines connecting the transmitting and receiving terminals, it is sometimes impossible to directly transmit moving images captured by the transmitting terminal's camera to the receiving terminal. In such cases, the transmitting terminal typically optimizes the moving image quality to match the communication line bandwidth before transmitting it to the receiving terminal.

[0004] Patent Document 2 discloses a technology primarily related to transmitting moving images captured by surveillance cameras, aiming to reduce the data volume and bit rate of the moving images. Specifically, the invention described in Patent Document 2 uses a user-defined, framed area within a moving image to transmit this specific area at a first frame rate, while transmitting the remaining, non-specific areas at a second frame rate. This technique is believed to efficiently reduce the data volume and bit rate of the moving images while maintaining the resolution of the specific area.

[0005] Prior art literature

[0006] Patent Literature

[0007] Patent Document 1: Japanese Patent No. 7023022

[0008] Patent Document 2: Japanese Patent Application Laid-Open No. 2021-13145 Summary of the Invention

[0009] Problems to be solved by the invention

[0010] Furthermore, the invention described in Patent Document 2 includes a setting unit for receiving user-specified specific areas within the surveillance camera itself that captures moving images. Therefore, according to the invention of Patent Document 2, it is possible to set a specific area on the surveillance camera and transmit that specific area to an electronic device or device at a high frame rate. However, Patent Document 2 does not contemplate transmitting moving images captured by the surveillance camera to a remote computer via the internet, for example.

[0011] On the other hand, if a remote support system for sharing information between the aforementioned operator and supporter is considered, the technology of patent document 2 can also be used as a reference to set a specific area to a high frame rate on the wearable device (transmitting terminal) side and set other non-specific areas to a low frame rate. However, in this case, the area set to a high frame rate on the transmitting terminal side is not necessarily consistent with the area that the supporter at the remote end wants to focus on. That is, the operator wears a wearable device to capture the environment around him, but the supporter sometimes wants to observe a part of the moving image sent by the operator with high quality (high resolution and / or high frame rate). In addition, the supporter also has the following requirement: within the range allowed by the bandwidth of the communication line connecting the transmitting terminal and the receiving terminal, he wants to observe a part of the moving image with the highest possible quality. In this way, it is more convenient if the area to be transmitted with the highest possible quality can be arbitrarily selected on the receiving terminal side operated by the supporter from the moving image transmitted from the transmitting terminal.

[0012] Therefore, a main object of the present invention is to enable a transmission system for moving images including a transmitting terminal and a receiving terminal to directly transmit high-quality moving images, mainly by allowing the receiving terminal to arbitrarily select a transmission terminal.

[0013] Technical solutions to problems

[0014] A first aspect of the present invention relates to a moving image transmission system. The moving image transmission system of the present invention includes a transmitting terminal and a receiving terminal connected via a communication line. The transmitting terminal and the receiving terminal may be connected directly via a wireless LAN or the Internet, or via an external cloud server providing a video communication tool such as a web conferencing service. The transmitting terminal includes a camera unit that captures moving images and a transmission unit that transmits the moving images to the receiving terminal. The transmitted moving images may be reduced in quality to accommodate the bandwidth of the communication line. Either or both the transmitting terminal and the receiving terminal include a region of interest designation unit that specifies a specific region of interest within the moving images. In particular, the receiving terminal preferably includes a receiving unit that receives the moving images from the transmitting terminal, a region of interest designation unit that specifies a specific region of interest within the moving images, and a transmission unit that transmits information related to the region of interest to the transmitting terminal. In this case, the transmitting terminal only needs to include a receiving unit that receives information related to the region of interest from the receiving terminal. The region of interest designation can include, for example, specifying an arbitrary shaped region within the moving images, such as a rectangle or a circle, or specifying a specific object within the moving images. The transmitting terminal further has: an image processing unit, which corrects the motion image based on information related to the area of ​​interest. Specifically, the image processing unit performs a process of cutting out the motion image within the area of ​​interest from the original motion image obtained by the camera unit, or reduces the quality of the motion image outside the area of ​​interest from the original motion image, based on the information related to the area of ​​interest. In the present specification, the image obtained by performing any of these processes on the original motion image obtained by the camera unit is referred to as a "corrected motion image." Moreover, in the present specification, the so-called "quality" of the motion image is a concept that includes both the resolution and frame rate of the motion image. That is, the process of reducing the quality of the motion image includes both or either the process of reducing the resolution of the motion image and the process of reducing the frame rate. The transmitting terminal transmits the corrected motion image obtained in this way to the receiving terminal through the transmitting unit.

[0015] As described above, by enabling the receiving terminal to specify a specific region of interest within a moving image transmitted from the transmitting terminal, a supporter, for example, remotely supporting a field worker, can freely specify a desired region of interest within the worker's surroundings. Furthermore, the present invention preferably does not process the moving image transmitted from the transmitting terminal at the receiving terminal, but rather performs image processing corresponding to the region of interest specified at the receiving terminal. Consequently, the moving image transmitted from the transmitting terminal to the receiving terminal is reduced in data volume by deleting areas outside the region of interest or reducing its quality, enabling efficient transmission of moving image data even within limited communication bandwidths.

[0016] In the system of the present invention, both or either of the transmitting terminal and the receiving terminal preferably further includes: a bandwidth measuring unit that measures the bandwidth of the communication line. In addition, the so-called "bandwidth" is the maximum amount of data that can be transmitted or received per unit time, also known as the communication capacity. In this case, the image processing unit of the transmitting terminal creates a corrected image based on the bandwidth of the communication line. Specifically, when the process of cutting out the moving image within the area of ​​interest from the original moving image is performed, it is sufficient to adjust the range of the cut moving image or adjust the quality of the moving image cut out according to the bandwidth of the communication line. Moreover, in the case of reducing the quality of the moving image outside the area of ​​interest from the original moving image, it is sufficient to adjust the degree of quality reduction according to the bandwidth of the communication line. In this way, the bandwidth of the communication line can be measured in real time while the moving image within the area of ​​interest can be transmitted to the receiving terminal with maximum utilization of the bandwidth.

[0017] In the system of the present invention, the receiving terminal preferably further includes a support unit. This support unit provides a user interface (UI) that supports the designation of a region of interest based on the bandwidth of the communication line. For example, when cropping a moving image within the region of interest, the support unit can simply frame and display the maximum area of ​​the original moving image quality that can be directly transmitted based on the bandwidth of the communication line, allowing the user to select the portion to be cropped from the frame as the region of interest. Furthermore, when reducing the quality of moving images outside the region of interest, the support unit can simply frame and display the maximum area of ​​the original moving image quality that can be directly transmitted based on the bandwidth of the communication line, allowing the operator of the receiving terminal to select the portion to be designated from the frame as the region of interest. In this way, even if the maximum area of ​​the original moving image quality that can be directly transmitted varies based on the bandwidth of the communication line, the operator can easily designate the region of interest by displaying the maximum area to the operator of the receiving terminal through the UI as described above.

[0018] A second aspect of the present invention relates to a program for a transmission terminal. This program enables a mobile information terminal (PDA) equipped with a camera to function as a transmission terminal in the system of the first aspect. This program can be downloaded to the mobile information terminal via the Internet, pre-installed in the mobile information terminal, or stored on a recording medium such as a CD-ROM readable by the mobile information terminal.

[0019] A third aspect of the present invention relates to a transmission terminal for moving images. The transmission terminal of the present invention is connected to a receiving terminal via a communication line. The transmission terminal has a camera unit, a transmission unit, a receiving unit, and an image processing unit. The camera unit obtains a moving image. The transmission unit transmits the moving image to a receiving device. The receiving unit receives information related to a specific area of ​​interest specified from the moving image from the receiving terminal. Alternatively, the transmission terminal may also have an area of ​​interest designation unit for designating a specific area of ​​interest from the moving image instead of the receiving unit. The image processing unit creates a corrected moving image based on the information related to the area of ​​interest, wherein the corrected moving image has been processed to cut out the moving image within the area of ​​interest from the original moving image obtained by the camera unit or to reduce the quality of the moving image outside the area of ​​interest from the original moving image. The transmission unit further transmits the corrected moving image to the receiving terminal.

[0020] A fourth aspect of the present invention relates to a method for transmitting a motion image from a transmitting terminal to a receiving terminal via a communication line. In the method of the present invention, first, the transmitting terminal obtains a motion image and transmits the motion image to the receiving terminal (first step). Then, the transmitting terminal or the receiving terminal specifies a specific area of ​​interest from the motion image (second step process). Then, the transmitting terminal creates a corrected motion image based on information related to the area of ​​interest, and transmits the corrected motion image to the receiving terminal. The corrected motion image has been processed by cutting out the motion image within the area of ​​interest from the original motion image obtained by the camera unit or reducing the quality of the motion image outside the area of ​​interest from the original motion image (third step).

[0021] Effects of the Invention

[0022] According to the present invention, it is mainly possible on the receiving terminal side to arbitrarily select a range in which the transmitting terminal directly transmits high-quality moving images. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] [ Figure 1 ] Figure 1 This is an overall diagram of a system showing one embodiment of the present invention.

[0024] [ Figure 2 ] Figure 2 An overview of information processing of a system according to one embodiment of the present invention is shown.

[0025] [ Figure 3 ] Figure 3 A neck-hanging wearable device is shown as an example of a transmission terminal.

[0026] [ Figure 4 ] Figure 4 This is a block diagram showing the functional configuration of a system according to one embodiment of the present invention.

[0027] [ Figure 5 ] Figure 5 This figure shows an example of a support user interface provided by the receiving terminal.

[0028] [ Figure 6 ] Figure 6 A flowchart showing a process executed by a system according to one embodiment of the present invention.

[0029] [ Figure 7 ] Figure 7 This shows another example of a method for specifying a region of interest. DETAILED DESCRIPTION

[0030] The specific embodiments of the present invention will be described below using the accompanying drawings. The present invention is not limited to the embodiments described below, but also includes embodiments appropriately modified from the embodiments below within the scope readily apparent to those skilled in the art.

[0031] Figure 1 The overall structure of a system 100 according to one embodiment of the present invention is schematically shown. The system 100 according to this embodiment is suitable for applications such as a supporter remotely supporting a worker who is working on-site via the Internet. In this system 100, a worker shoots a moving image using a transmitting terminal 10 and transmits the moving image to a receiving terminal 20 operated by the supporter via the Internet. Furthermore, the remote supporter can also view the moving image transmitted from the worker's transmitting terminal 10 or communicate with the transmitting terminal 10 via the receiving terminal 20. Furthermore, the system 100 may include an external cloud server 30 that provides a video conversation tool to the transmitting terminal 10 and the receiving terminal 20. In this case, the moving image is transmitted from the transmitting terminal 10 to the receiving terminal 20 via this cloud server 30.

[0032] Figure 2 1 shows an overview of the system 100 of the present invention. Figure 2 As shown, the transmitting terminal 10 and the receiving terminal 20 are connected via a communication line such as the Internet, but the bandwidth of this communication line is limited. Therefore, even if the transmitting terminal 10 obtains a high-quality (high-resolution and high-frame-rate) original motion image, if the amount of data per unit time of this original motion image is greater than the bandwidth of the communication line, the original motion image cannot be directly transmitted to the receiving terminal 20 at the quality at which it was obtained. Therefore, the transmitting terminal 10 reduces the overall quality of the original motion image according to the bandwidth of the communication line and transmits it to the receiving terminal 20. For example, as a process for reducing the quality of the original motion image, the screen resolution (image quality) of the animation or the frame rate of the animation can be reduced. The receiving terminal 20 receives the motion image whose quality has been reduced by the transmitting terminal 10 and displays this motion image on the display unit in real time.

[0033] Here, the supporter operating the receiving terminal 20 may want to view a portion of the moving image received from the transmitting terminal 10 with higher quality. For example, when the transmitting terminal 10 obtains a wide range of moving images of the front side of the operator on site and transmits it to the receiving terminal 20, the supporter may want to recognize small text displayed in the moving image or confirm in detail the equipment operated by the operator. In this case, the supporter operates the receiving terminal 20 to specify the area of ​​interest in the moving image. For example, Figure 2 As shown, a partial rectangular area in the moving image displayed on the screen can be designated as the target area. Information related to the target area designated by the receiving terminal 20 is transmitted to the transmitting terminal 10 via the Internet.

[0034] When the transmitting terminal 10 receives information related to the region of interest from the receiving terminal 20, it performs image processing on the original moving image based on this information. Specifically, the transmitting terminal 10 cuts out only the area designated as the region of interest from the original moving image, maintaining the quality of the cutout area while deleting the moving image data outside the area designated as the region of interest. Alternatively, the transmitting terminal 10 maintains the quality of the area designated as the region of interest from the original moving image while reducing the quality of the moving image outside the area designated as the region of interest. In this way, the transmitting terminal 10 creates a modified moving image that maintains the quality of the original moving image in the area designated as the region of interest and deletes or degrades the moving image data in the area outside the area of ​​interest. The data volume per unit time of this modified moving image is set within the bandwidth of the communication line, and the modified moving image is transmitted to the receiving terminal 20. As a result, the receiving terminal 20 can directly display the quality of the original moving image obtained by the transmitting terminal 10 for the area designated as the region of interest.

[0035] Next, the configuration of the system 100 according to one embodiment of the present invention will be described in more detail. Figure 3 1 is a perspective view showing an example of the appearance of the transmission terminal 10. Figure 4 An example of hardware elements of the transmission terminal 10 is shown in FIG. Figure 3 As shown, the transmission terminal 10 in this embodiment is a neck-hanging wearable device. It includes a left arm, a right arm, and a main body that connects these arms behind the wearer's neck. To wear the transmission terminal 10, simply place the main body in contact with the back of the wearer's neck, and let the left and right arms hang from the wearer's neck toward the chest, allowing the entire device to be worn around the neck. The housing of the transmission terminal 10 houses various electronic components.

[0036] The left arm and the right arm are respectively provided with a plurality of sound collecting parts 14 (microphones). The sound collecting parts 14 are mainly configured for the purpose of acquiring the sounds around the wearer and the voices of the wearer and the interlocutor. In order to widely collect the sounds generated around the wearer, it is preferable to use an omnidirectional (non-directional) microphone as the sound collecting part 14. As the sound collecting part 14, any known microphone such as a dynamic microphone, a condenser microphone, a MEMS (Micro-Electrical-Mechanical Systems) microphone, etc. can be used. The sound collecting part 14 converts the sound into an electrical signal, amplifies the electrical signal through an amplifying circuit, converts it into digital information through an A / D conversion circuit, and outputs it to the control part 11. The sound signal acquired by the sound collecting part 14 is transmitted to the control part 11 provided in the shell. In addition, in this embodiment, the sound signal acquired by the sound collecting part 14 can be transmitted to the receiving terminal 20 via the Internet through the communication part 13. As a result, the sound acquired by the on-site operator through the transmitting terminal 10 is also shared with the receiving terminal 20 of the remote supporter.

[0037] The left arm is further provided with an imaging unit 15. Specifically, the imaging unit 15 is provided on the front end of the left arm, and is used to capture moving images of the wearer's front side. Furthermore, the imaging unit 15 can also capture still images. The moving images and still images (hereinafter collectively referred to as "images") captured by the imaging unit 15 are transmitted to the control unit 11 within the housing and stored as image data. A conventional digital camera can be used as the imaging unit 15. The imaging unit 15 may be composed, for example, of a photographic lens, a mechanical shutter, a shutter driver, a photoelectric conversion element such as a charge-coupled device image sensor, a digital signal processor (DSP) that reads the charge from the photoelectric conversion element and generates image data, and an IC memory. The image data captured by the imaging unit 15 is supplied to the control unit 11 and stored in the storage unit 12. Furthermore, predetermined image analysis processing may be performed on the image data. The moving images or still images captured by the imaging unit 15 are then transmitted to the receiving terminal 20 via the Internet via the communication unit 13. As a result, the worker on site can also share the moving images or still images acquired by the transmitting terminal 10 with the receiving terminal 20 of the remote supporter.

[0038] The right arm is further provided with a non-contact gesture sensor 16. This sensor is primarily designed to detect the wearer's hand movements in front of the transmission terminal 10 and is located on the front end of the right arm. For example, the gesture sensor 16 detects the movement or shape of the wearer's fingers. An example of a gesture sensor 16 is an optical sensor that radiates light from an infrared LED toward an object and detects changes in the reflected light using a photodetector. The sensing information obtained by the gesture sensor 16 is transmitted to the control unit 11 and used primarily to control the camera unit 15 or the sound player 18. Specifically, the sensing information from the gesture sensor 16 is used to control the start and stop of the camera unit 15 or the sound player 18. For example, the gesture sensor 16 can control the camera unit 15 by detecting the proximity of an object such as the wearer's hand to the gesture sensor 16, or it can control the camera unit 15 by detecting the wearer performing a predetermined gesture within the gesture sensor 16's sensing range. Furthermore, the positions of the camera unit 15 and the gesture sensor 16 can be interchanged. Furthermore, the gesture sensor 16 can also be replaced with a proximity sensor. The proximity sensor, for example, senses when the wearer's finger approaches a predetermined range. Known proximity sensors such as optical, ultrasonic, magnetic, electrostatic capacitive, or temperature sensors can be used as proximity sensors.

[0039] A sound emitting unit (speaker) 18 is provided on the outside of the main body (on the side opposite the wearer) behind the wearer's neck. In this embodiment, the sound emitting unit 18 is configured to output sound toward the outside of the main body. By emitting sound directly behind the wearer's neck, the sound output from the sound emitting unit 18 is less likely to be directly transmitted to the interlocutor located directly in front of the wearer. This makes it easier for the interlocutor to distinguish between the wearer's own voice and the sound emitted by the sound emitting unit 18 of the transmitting terminal 10. The sound emitting unit 18 is an audio device that converts electrical signals into physical vibrations (i.e., sound). An example of the sound emitting unit 18 is a conventional speaker that transmits sound to the wearer through air vibrations. Alternatively, the sound emitting unit 18 may be a bone conduction speaker that transmits sound to the wearer by vibrating the wearer's bones. In this case, the sound emitting unit 18 can be arranged on the inside of the main body (on the wearer's side) so that the bone conduction speaker contacts the bones (cervical vertebrae) behind the wearer's neck. Furthermore, in this embodiment, voice signals input to the receiving terminal 20 are transmitted to the transmitting terminal 10 via the Internet. The transmitting terminal 10 converts the voice signals received from the receiving terminal 20 into sound via the sound emitting unit 18 and outputs the sound. This allows the voice of the supporter operating the receiving terminal 20 to be transmitted to the worker wearing the transmitting terminal 10.

[0040] like Figure 4As shown, the control unit 11 of the transmission terminal 10 performs computational processing to control the other components of the transmission terminal 10. A processor such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit) can be used as the control unit 11. The control unit 11 essentially reads a program stored in the storage unit 12, expands it into the main memory, and executes predetermined computational processing in accordance with the program. Furthermore, the control unit 11 can write or read computational results obtained in accordance with the program into or from the storage unit 12 as appropriate.

[0041] Furthermore, the control unit 11 of the transmission terminal 10 includes a bandwidth measurement unit 11a and an image processing unit 11b as functional blocks. These functional blocks 11a and 11b are implemented by the control unit 11 executing a predetermined program.

[0042] The bandwidth measurement unit 11a measures the bandwidth of the communication line used to connect to the receiving terminal 20. As described above, the transmitting terminal 10 primarily transmits moving images to the receiving terminal 20. The bandwidth measurement unit 11a simply measures the bandwidth of the communication line used for transmission (upload) of these moving images. Bandwidth measurement can be performed using known methods. For example, the bandwidth measurement unit 11a of the transmitting terminal 10 repeatedly processes test data packets transmitted to the receiving terminal 20 while gradually shortening the interval between packet transmissions. The bandwidth measurement unit 11a then calculates the transmission rate (data volume / transmission time) at the time when test data packet loss occurs as the available bandwidth at that time. Furthermore, if packet loss occurs while transmitting moving image data to the receiving terminal 20 at a predetermined transmission rate, the bandwidth measurement unit 11a may determine that the moving image cannot be transmitted at that transmission rate and determine to reduce the transmission rate.

[0043] The image processing unit 11b performs predetermined image processing on the original moving image obtained by the imaging unit 15. For example, based on the bandwidth of the communication line measured by the bandwidth measurement unit 11a, the image processing unit 11b optimizes the overall quality (resolution and / or frame rate) of the original moving image obtained by the imaging unit 15 so that the moving image can be transmitted to the receiving terminal 20 within this bandwidth. Furthermore, based on a region of interest specified by the receiving terminal 20, the image processing unit 11b creates a modified moving image by cropping a portion corresponding to the region of interest from the original moving image obtained by the imaging unit 15 or reducing the quality of areas outside the region of interest.

[0044] The storage unit 12 of the transmission terminal 10 is an element for storing information used in the calculations and other processing performed by the control unit 11, as well as its calculation results. The storage function of the storage unit 12 can be implemented by non-volatile memory such as an HDD or SDD. Furthermore, the storage unit 12 can also function as main memory, used to write and read intermediate steps of the calculations performed by the control unit 11. The storage function of the storage unit 12 can be implemented by volatile memory such as RAM or DRAM. Furthermore, the storage unit 12 can also store unique ID information of the user who owns it. Furthermore, the storage unit 12 can also store the IP address, which is the identification information of the transmission terminal 10 on the network.

[0045] The communication unit 13 of the transmitting terminal 10 is an element for wireless communication with the receiving terminal 20 or the cloud server 30. To communicate with the receiving terminal 20 or the cloud server 30 via the Internet, the communication unit 13 can simply employ a communication module that performs wireless communication using known mobile communication standards such as 3G (W-CDMA), 4G (LTE / LTE-Advanced), or 5G, or a wireless LAN method such as Wi-Fi (registered trademark). Furthermore, the transmitting terminal 10 can also communicate with other transmitting terminals 10. In this case, to directly communicate with other transmitting terminals 10, the communication unit 13 preferably employs a communication module for short-range wireless communication using methods such as Bluetooth (registered trademark) or NFC.

[0046] The sensors 17 of the transmitting terminal 10 include sensor devices for sensing, for example, the movement or usage of the transmitting terminal 10, or the biometric information of the wearer. Sensors 17 can be sensor modules typically found on mobile information terminals or wearable devices. For example, sensors 17 include gyroscopes, accelerometers, geomagnetic sensors, and battery sensors. Sensors 17 may also include biometric sensors such as body temperature sensors, heart rate sensors, blood oxygen concentration sensors, blood pressure sensors, and electrocardiogram sensors to sense the wearer's biometric information.

[0047] The position information acquisition unit 19 of the transmission terminal 10 is an element for acquiring the current position information of the transmission terminal 10. Specifically, the position information acquisition unit 19 has a function of positioning using GPS (Global Positioning System). The position information acquisition unit 19 measures the time required to receive each radio wave based on the information on the radio wave transmission time contained in the radio waves transmitted from multiple GPS satellites, and transmits the time information representing the time to the control unit 11. The control unit 11 can calculate information related to the latitude and longitude of the location of the transmission terminal 10 based on the acquired time information. In addition, the position information acquisition unit 19 can also acquire the current position information by scanning the radio waves or beacon signals emitted from wireless base stations such as Wi-Fi (registered trademark) access points.

[0048] In addition, by Figure 3 and Figure 4 As will be apparent, in this embodiment, the transmitting terminal 10 lacks a display device such as a monitor or display. Therefore, while the operator can use gesture sensors 16 and other devices to perform relatively simple operations such as turning hardware components on and off, complex operations such as operating applications are difficult. When using a transmitting terminal 10 without a display, remote control of the transmitting terminal 10 via the internet using a receiving terminal 20, as in the system of the present invention, is particularly effective.

[0049] Figure 4 The following further illustrates examples of hardware components of the receiving terminal 20. The receiving terminal 20 can be implemented using a general personal computer (PC). The receiving terminal 20 can be either a desktop PC or a notebook PC. Furthermore, the receiving terminal 20 can be a smartphone or a tablet terminal. The receiving terminal 20 includes a control unit 21, a storage unit 22, a communication unit 23, a display unit 24, an operation unit 25, a sound collection unit 26, and a sound playback unit 27. These components can be implemented using a general PC and its associated peripheral devices.

[0050] The control unit 21 of the receiving terminal 20 is composed of a processor such as a CPU or GPU. The control unit 21 basically reads a program stored in the storage unit 22, expands it into the main memory, and executes predetermined computations according to the program. Furthermore, the control unit 21 writes or reads computation results obtained according to the program into or from the storage unit 22 as appropriate. In this manner, the control unit 21 performs control processing for each element 22-27 according to the program stored in the storage unit 22.

[0051] The control unit 21 of the receiving terminal 20 includes a frequency band measurement unit 21a, a support unit 21b, and a region of interest designation unit 21c as functional blocks. These functional blocks 21a to 21c are implemented by the control unit 21 executing a predetermined program.

[0052] The bandwidth measurement unit 21a measures the bandwidth of the communication line used to connect to the transmitting terminal 10. The receiving terminal 20 primarily receives moving images from the transmitting terminal 10, and the bandwidth measurement unit 21a simply measures the bandwidth of the communication line used to receive (download) these moving images when receiving them. Bandwidth measurement can be performed using known methods. For example, the bandwidth measurement unit 21a of the receiving terminal 20 repeatedly receives test data packets from the transmitting terminal 10 while gradually shortening the intervals between packet receptions. The bandwidth measurement unit 21a then calculates the reception rate (data volume / reception time) at the time when test data packet loss occurs as the available bandwidth at that time. Furthermore, if packet loss occurs while receiving moving image data from the transmitting terminal 10 at a predetermined reception rate, the bandwidth measurement unit 21a may determine that the moving image cannot be transmitted at that reception rate and determine to reduce the reception rate.

[0053] The support unit 21 b provides a UI for supporting the designation of a region of interest based on the bandwidth of the communication line measured by the bandwidth measurement unit 21 a . Figure 5 (a) shows an example of a support UI. Figure 5 In the example shown in (a), the support unit 21b displays a frame F on the display screen of the display unit 24, indicating the maximum area of ​​the quality of the original moving image that can be directly transmitted, based on the quality of the original moving image received from the transmission terminal 10 and the bandwidth of the communication line (particularly the download line). In other words, the area of ​​this frame F increases or decreases depending on the quality of the original moving image and the bandwidth of the communication line. For example, the higher the quality (higher image quality, higher frame rate) of the original moving image captured by the camera unit 15 of the transmission terminal 10, the greater the amount of data per unit area of ​​the original moving image, and the narrower the range of the original moving image quality that can be directly transmitted to the receiving terminal 20 via the communication line. Therefore, the area of ​​the frame F becomes narrower. Conversely, the lower the quality of the original moving image captured by the camera unit 15 of the transmission terminal 10, the wider the area of ​​the frame F. On the other hand, the narrower the bandwidth of the communication line, the smaller the amount of data per unit time that can be transmitted by the communication line of the original moving image captured by the camera unit 15 of the transmission terminal 10, and therefore the narrower the area of ​​the frame F. Conversely, the wider the bandwidth of the communication line, the wider the area of ​​the frame F. Thus, the support unit 21b simply calculates the maximum area that can directly transmit the quality of the original moving image transmitted from the transmission terminal 10 and the bandwidth of the communication line, and displays the frame F corresponding to this area on the display unit 24.

[0054] And, as Figure 5As shown in (a), once the area of ​​frame F is determined, the frame F can be freely moved on the display screen of the moving image. The operation of moving frame F can be performed simply through the operating unit 25 of the receiving terminal 20. Furthermore, if the area of ​​frame F is fixed, the shape of frame F can be freely deformed to a certain extent within the range of the display screen of the moving image. Figure 5 The frame F shown in (a) is a horizontally long rectangle, but may be formed in a square, formed in a vertically long shape, or formed in a further horizontally long shape. Furthermore, the shape of the frame F is not limited to a rectangle, and may be formed in a circular or elliptical shape.

[0055] The region of interest designation unit 21c receives a specific region of interest designated by the user (supporter) of the receiving terminal 20 in the moving image received from the transmitting terminal 10. In this embodiment, as described above, the support unit 21b displays a frame F representing the maximum area of ​​quality that can directly transmit the original moving image on the display screen. In this case, the region of interest designation unit 21c receives a specific region of interest designated by the user within the range of this frame F. Specifically, as shown in FIG. Figure 5 As shown in (a) of FIG. 1 , the user determines the position and shape of the frame F on the display screen by operating the operation unit 25. Then, if the position and shape of the frame F are fixed, as shown in FIG. Figure 5 As shown in (b), the user specifies the region of interest R within the range of the frame F by operating the operation unit 25. This region of interest R is configured so as not to exceed the frame F. In addition, the frame F and the region of interest R may also be consistent. The region of interest designation unit 21c temporarily stores information related to the region of interest R specified by the user in this manner in the storage unit 22. As information related to the region of interest R, for example, when the region of interest R is rectangular, it is sufficient to pre-store the coordinate information (xy coordinates) on the moving image of two diagonally opposite vertices (opposite vertices). The control unit 21 of the receiving terminal 20 transmits the information related to the region of interest R obtained in this manner to the transmitting terminal 10 via the communication unit 23. As described above, the information related to the region of interest R is used for processing in the image processing unit 11b in the control unit 11 of the transmitting terminal 10.

[0056] The storage unit 22 of the receiving terminal 20 is used to store information used for calculations and other processing by the control unit 21, as well as its calculation results. The storage function of the storage unit 22 can be implemented by non-volatile memory such as an HDD or SDD. Furthermore, the storage unit 22 can also function as main memory for writing and reading intermediate steps of calculations performed by the control unit 11. The storage function of the storage unit 22 can be implemented by volatile memory such as RAM or DRAM.

[0057] The communication unit 23 of the receiving terminal 20 is an element for communicating with the receiving terminal 20 or the cloud server 30. The communication method of the communication unit 23 can be either wired or wireless. For example, the receiving terminal 20 can be connected to the Internet via a wired optical line or telephone line, or it can be connected to the Internet via a wireless LAN method such as Wi-Fi (registered trademark).

[0058] The display unit 24 of the receiving terminal 20 is an element for displaying images. In the system of the present invention, the moving images received from the transmitting terminal 10 are primarily displayed on the display unit 24. A known display such as a liquid crystal display or an organic EL display can be used as the display unit 24. Alternatively, the display unit 24 may be a projector that projects image light onto a screen.

[0059] The operating unit 25 of the receiving terminal 20 is an element for allowing the user (supporter) to input predetermined operational information into the receiving terminal 20 (particularly the control unit 21). Operating unit 25 may utilize known devices such as a touch screen, a mouse, a keyboard, a touchpad, a stylus pen, or a graphics tablet. Furthermore, a touch screen display can be constructed by overlaying a touch screen on a display screen.

[0060] The sound collecting unit 26 of the receiving terminal 20 is mainly used to obtain the voice of the user (supporter). As the sound collecting unit 26, a directional or omnidirectional (non-directional) microphone can be used. As the microphone, a known microphone such as a dynamic microphone, a condenser microphone, a MEMS microphone, etc. can be used. In this embodiment, the voice signal obtained by the sound collecting unit 26 of the receiving terminal 20 can be transmitted to the transmitting terminal 10 via the Internet through the communication unit 23. In this way, the voice of the supporter is output from the transmitting terminal 10 of the operator working on the scene.

[0061] The sound output unit 27 of the receiving terminal 20 is primarily used to output elements of the audio received from the transmitting terminal 10. A standard speaker, earphones, or headphones that transmit sound through air vibrations can be used as the sound output unit 27. In this embodiment, the audio signal input to the transmitting terminal 10 is transmitted to the receiving terminal 20 via the Internet. The receiving terminal 20 converts the audio signal received from the transmitting terminal 10 into audio through the sound output unit 27 and outputs it. This allows the audio received by the transmitting terminal 10 to be transmitted to the supporter operating the receiving terminal 20.

[0062] The cloud server 30 provides video conversation tools, such as web conferencing, to the transmitting terminal 10 and the receiving terminal 20. The cloud server 30 is composed of one or more server devices 31. For example, when establishing a communication connection between the transmitting terminal 10 and the receiving terminal 20, the cloud server 30 requires each user (operator and supporter) of the transmitting terminal 10 and the receiving terminal 20 to log in to an account for the video conversation tool. A dedicated application for the video conversation tool is installed in each of the transmitting terminal 10 and the receiving terminal 20. The transmitting terminal 10 and the receiving terminal 20 execute this application, access the cloud server 30, and log in to their respective accounts. If the transmitting terminal 10 and the receiving terminal 20 successfully log in and authenticate, the cloud server 30 begins to relay the transmission and reception of image data or audio data between these two terminals. For use as a video conversation tool, generally available commercial tools can be appropriately utilized.

[0063] Next, refer to Figure 6 The flow of a process for transmitting a moving image from the transmitting terminal 10 to the receiving terminal 20 will be described.

[0064] First, the transmitting terminal 10 activates the camera unit 15 and starts acquiring moving images (step S1). In this embodiment, it is assumed that the moving images acquired by the transmitting terminal 10 are transmitted in real time to the receiving terminal 20. Therefore, the transmitting terminal 10 continues to acquire moving images.

[0065] Next, the bandwidth measurement unit 11 a of the transmission terminal 10 measures the bandwidth of the communication line used to transmit the moving image to the reception terminal 20 (step S2 ).

[0066] Next, the image processing unit 11b of the transmitting terminal 10 optimizes the quality of the original moving image captured by the camera unit 15 based on the bandwidth measured by the bandwidth measurement unit 11a (step S3). Typically, bandwidth is limited, making it difficult to directly transmit the original moving image to the receiving terminal 20 at the desired quality. Therefore, the image processing unit 11b typically performs image processing on the entire original moving image, such as reducing the resolution or frame rate, to lower the quality.

[0067] Next, the transmitting terminal 10 uses the communication unit 13 to transmit the moving image optimized by the image processing unit 11b to the receiving terminal 20 via the communication line (step S4). Furthermore, the transmitting terminal 10 may transmit metadata related to the quality of the original moving image to the receiving terminal 20 along with the moving image. This metadata includes the resolution, frame rate, and data volume per unit time of the original moving image. The receiving terminal 20 receives the moving image and metadata from the transmitting terminal 10 via the communication unit 23 (step S5). Alternatively, the moving image or metadata may be transmitted from the transmitting terminal 10 to the receiving terminal 20 via the cloud server 30.

[0068] Next, the receiving terminal 20 begins displaying the moving image received from the transmitting terminal 10 on the display unit 24 (step S6). In this embodiment, it is assumed that the moving image obtained by the transmitting terminal 10 is displayed in real time on the display unit 24 of the receiving terminal 20. Therefore, the receiving terminal 20 continues to display the moving image. Furthermore, typically, at this stage, the moving image displayed on the display unit 24 is of lower quality than the original moving image obtained by the camera unit 15 of the transmitting terminal 10.

[0069] Next, the control unit 21 of the receiving terminal 20 determines whether the user (supporter) has input an operation to start the attention mode (step S7). This input operation is performed via the operation unit 25. If the attention mode is not started, the process (steps S2 to S6) is repeated until the transmitting terminal 10 optimizes the moving image according to the bandwidth of the communication line and transmits it to the receiving terminal 20. On the other hand, if the attention mode is started, the process proceeds to the next step S7.

[0070] When the attention mode starts, the bandwidth measurement unit 21a of the reception terminal 20 measures the bandwidth of the communication line for receiving the moving image from the transmission terminal 10 (step S8).

[0071] Next, the support unit 21b of the receiving terminal 20 calculates the maximum area that can directly transmit the quality of the original moving image based on the bandwidth measured by the bandwidth measurement unit 21a and the meta-information related to the quality of the original moving image received from the transmitting terminal 10, and places the frame F (see FIG. Figure 5 ) is displayed on the display screen of the motion image currently being displayed (step S9). As mentioned above, the user can arbitrarily set the position and shape of this frame F through the operation unit 25.

[0072] Next, the region of interest designation unit 21c of the receiving terminal 20 accepts a user-specified region of interest within the frame F displayed by the support unit 21b (step S10). Information (coordinate information) related to the region of interest specified by the user is transmitted from the receiving terminal 20 to the transmitting terminal 10 (step S11). Furthermore, the transmitting terminal 10 receives information related to the region of interest from the receiving terminal 20 (step S12).

[0073] Next, the image processing unit 11b of the transmitting terminal 10 corrects the original moving image captured by the camera unit 15 based on the information about the region of interest received from the receiving terminal 20 (step S13). As described above, the image processing unit 11b may also cut out only the region of interest from the original moving image while maintaining the quality of the original moving image. Furthermore, the image processing unit 11b may maintain the quality of the original moving image for the region of interest while lowering the quality of other regions based on the bandwidth of the communication line. The transmitting terminal 10 transmits the corrected moving image obtained in this manner to the receiving terminal 20 (step S14). The receiving terminal 20 then receives the corrected moving image from the transmitting terminal 10 (step S15).

[0074] Next, the receiving terminal 20 begins displaying the corrected moving image received from the transmitting terminal 10 (step S16), replacing the moving image previously displayed on the display unit 24 (step S6). The process of displaying the corrected moving image focused on the user-specified area of ​​interest (steps S8-S16) continues until the focus mode ends (step S7). Once the focus mode ends, the process returns to displaying an image that optimizes the entire moving image received by the transmitting terminal 10 (steps S2-S6). The moving image display process continues until communication between the transmitting terminal 10 and the receiving terminal 20 is disconnected or the camera unit 15 of the transmitting terminal 10 ceases recording.

[0075] Figure 7 Another example of a method for specifying a region of interest is schematically shown. In the above embodiment, the user (supporter) of the receiving terminal 20 specifies a region of interest by specifying an arbitrary rectangular region on a moving image. Figure 7 In the example shown, the user specifies an object included in a moving image and thereby specifies the display area of ​​the object as the target area.

[0076] Specifically speaking, Figure 7 (a) shows that the overall quality of the moving image displayed on the receiving terminal 20 is lower than that of the original moving image obtained by the transmitting terminal 10. Figure 7 In the example shown in (a), three people are included as objects in the moving image.

[0077] then, Figure 7(b) shows a situation where the user specifies an object included in a moving image. In this case, the region of interest designation unit 21c of the receiving terminal 20 analyzes the moving image, determines the display area of ​​the object specified by the user through known object recognition, and designates the display area of ​​the object as the region of interest R. Information regarding the region of interest R specifying this object is transmitted from the receiving terminal 20 to the transmitting terminal 10. Similar to the receiving terminal 20, the transmitting terminal 10 analyzes the original moving image from the imaging unit 15 and determines the display area of ​​the object specified by the user through known object recognition. Then, the image processing unit 11b of the transmitting terminal 10 maintains the quality of the original moving image for the display area of ​​the object specified as the region of interest R while reducing the resolution or frame rate for other areas to adapt to the bandwidth of the communication line. In this way, the corrected moving image, which has undergone image processing by the image processing unit 11b, is transmitted from the transmitting terminal 10 to the receiving terminal 20.

[0078] then, Figure 7 (c) shows the situation where the corrected moving image is displayed on the receiving terminal 20. As shown in this figure, the resolution or frame rate of the original moving image is maintained for the person designated as the focus area R, while the overall quality is reduced for other areas or people. Therefore, the person designated as the focus area R is displayed on the display unit 24 of the receiving terminal 20 with high precision and smooth movements. On the other hand, the data volume is reduced by reducing the quality of other areas or people. Therefore, the data volume of the entire moving image is reduced, so even if the person designated as the focus area R is maintained at high quality, the moving image can be transmitted from the transmitting terminal 10 to the receiving terminal 20 via the communication line.

[0079] In the above description, in order to express the content of the present invention, the following Figure 1 However, the present invention is not limited to the above-described embodiments, and includes obvious changes or improvements that can be made by those skilled in the art based on the matters described in this specification.

[0080] Description of Reference Numerals

[0081] 10: Transmission terminal

[0082] 11: Control Department

[0083] 11a: Band measurement unit

[0084] 11b: Image processing unit

[0085] 12: Storage

[0086] 13: Communication unit (receiving unit, transmitting unit)

[0087] 14: Sound Collection

[0088] 15: Camera Department

[0089] 16: Gesture sensor

[0090] 17: Sensor

[0091] 18: Playback Department

[0092] 19: Location information acquisition unit

[0093] 20: Receiving terminal

[0094] 21: Control Department

[0095] 21a: Band measurement unit

[0096] 21b: Support Department

[0097] 21c: Focus Area Designation Department

[0098] 22: Storage

[0099] 23: Communication unit (receiving unit, transmitting unit)

[0100] 24: Display unit

[0101] 25: Operation Department

[0102] 26: Sound Collection

[0103] 27: Playback Department

[0104] 30: Cloud Server

[0105] 31: Server device

[0106] 100: System

Claims

1. A system comprising a transmitting terminal and a receiving terminal connected via a communication line, wherein: The transmitting terminal has: a camera unit that acquires a moving image; and a transmitting unit, which transmits the moving image to the receiving terminal, The receiving terminal has: a region of interest designating unit for designating a specific region of interest from the moving image; The transmission terminal further includes an image processing unit configured to generate a modified moving image based on information related to the target area, wherein the modified moving image is processed by cutting out the moving image within the target area from the original moving image obtained by the imaging unit, or by reducing the quality of the moving image outside the target area from the original moving image. and transmitting the corrected moving image to the receiving terminal through the transmitting unit, Both or either of the transmitting terminal and the receiving terminal further includes: a bandwidth measuring unit that measures the bandwidth of the communication line; The receiving terminal further includes: a support unit, which provides a user interface, and the user interface displays a frame of the maximum area representing the quality of the original motion image that can be directly transmitted on the display screen according to the bandwidth of the communication line, and supports the designation of the area of ​​interest. The area of ​​interest designation unit accepts the specific area of ​​interest specified by the user within the range of the frame.

2. The system according to claim 1, wherein: The receiving terminal has: a receiving unit configured to receive the moving image from the transmitting terminal; a region of interest designating unit that designates a specific region of interest from the moving image; and a transmitting unit, configured to transmit information related to the area of ​​interest to the transmitting terminal, The transmitting terminal further includes a receiving unit configured to receive information related to the area of ​​interest from the receiving terminal.

3. The system according to claim 1, wherein: Both or either of the transmitting terminal and the receiving terminal further includes: a bandwidth measuring unit that measures the bandwidth of the communication line; The image processing unit creates the corrected moving image according to a bandwidth of the communication line.

4. A method for transmitting a moving image from a transmitting terminal to a receiving terminal via a communication line, characterized in that: The following steps are included: The transmitting terminal acquires a motion image through a camera unit and transmits the motion image to the receiving terminal; The receiving terminal specifies a specific area of ​​interest from the moving image; The transmitting terminal creates a modified moving image based on the information related to the region of interest, and transmits the modified moving image to the receiving terminal, wherein the modified moving image is processed by cutting out the moving image within the region of interest from the original moving image obtained by the imaging unit, or by reducing the quality of the moving image outside the region of interest from the original moving image; A step of measuring a bandwidth of the communication line by either or both of the transmitting terminal and the receiving terminal; as well as The receiving terminal provides a user interface, which displays a frame of the maximum area representing the quality of the original motion image that can be directly transmitted on the display screen based on the bandwidth of the communication line, and supports a user interface for specifying the area of ​​interest. The area of ​​interest specifying unit accepts the step of specifying the specific area of ​​interest by the user within the range of the frame.

Images