Information processing terminal and image transmission system

Abstract

To facilitate an acquisition of high quality still picture data corresponded to an optional timing of moving image data.SOLUTION: An information processing terminal comprises: an image sensor 15 that generates an image signal; a moving encoder 11c that converts an image signal into moving data by a moving encode; a memory 11f; and a still picture encoder 11e that converts the image signal into still picture data by a still picture encode, and corresponds this still picture data with the moving data, and holds into the memory 11f.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present invention relates to an information processing terminal, an image transmission system, and a program. More specifically, the present invention relates to a technique for acquiring high-quality still image data together with video data in an information processing terminal, and transmitting the still image data to a receiving terminal in addition to the video data as necessary. [Background technology]

[0002] In recent years, digital transformation (DX) of on-site work has become an urgent issue in order to deal with various problems such as the increasing complexity of on-site work, the decline in the working population, and the transfer of skills of skilled workers. As one solution to these problems, a remote support system has been proposed in which a wearable device worn by on-site workers is connected to a computer operated by a supporter via the Internet, etc., to share audio and visual information (Patent Document 1).

[0003] In such a remote support system, a wearable device (transmitting terminal) captures video data of the worker's surroundings using a camera and transmits it to a computer (receiving terminal) operated by a supporter. At that time, the image signal captured by the image sensor of the camera is converted into video data of a predetermined format in the transmitting terminal by video encoding, which converts the format into a predetermined video format and compresses the data volume. In recent years, even small cameras have become capable of capturing high-resolution and high-frame-rate video images, but since there is a certain limit to the bandwidth of the communication line connecting the transmitting terminal and the receiving terminal, there are cases where the video captured by the camera of the transmitting terminal cannot be transmitted to the receiving terminal as is. In this case, the transmitting terminal may optimize the quality of the video image to match the bandwidth of the communication line before transmitting it to the receiving terminal.

[0004] Patent Document 2 discloses a remote conference system for remote medical care. In this remote system, a first terminal (transmission terminal) acquires a high-quality video signal with a high bit rate and converts it into a low-quality video signal with a low bit rate. The first terminal then transmits the low-quality video signal together with an audio signal to a remote conference server in real time, while transmitting the high-quality video signal together with the audio signal asynchronously to a symptom recognition server. This allows for a real-time examination to be performed using the first terminal operated by a patient and the remote conference server operated by a doctor, while enabling the symptom recognition server to more accurately diagnose symptoms of the patient's illness. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Patent No. 7023022 [Patent Document 2] International Publication WO2019 / 207392 Brochure Summary of the Invention [Problem to be solved by the invention]

[0006] Incidentally, in a remote support system aimed at digitalization of on-site work, when a worker uses a wearable device (transmitting terminal) to transmit images of the work site, etc., in real time to the supporter's computer (receiving terminal), a situation may arise in which the supporter wants to pay attention to the image at a certain timing and examine the image at that timing in detail. For example, when a worker takes a picture of the detailed structure or wiring of a device at hand, or a document or manual, it is assumed that the supporter accurately grasps the contents of the structure or document, etc., and gives instructions to the worker. At this time, the wearable device generates video data by encoding the image signal acquired by the image sensor for video. However, since the video data is compressed in data capacity, even if a specific frame is extracted from the video data, the resolution of the frame is lower than that of the original image signal. For this reason, when a supporter wants to grasp the detailed structure or the contents of a document at the worker's hand, even if a frame at a specific timing is extracted from the video data, the image quality of the frame is rough, and the supporter cannot accurately grasp the contents.

[0007] In addition, as described above, in the system described in Patent Document 2, a high-quality video signal with a high bit rate and a low-quality video signal with a low bit rate are acquired in the first terminal (transmission terminal). However, in this system, it is assumed that the high-quality video signal is transmitted to the symptom recognition server asynchronously, and the low-quality video signal is transmitted to the remote conference server in real time. For this reason, in the system described in Patent Document 2, no process is performed to associate the timing at which the high-quality video signal is acquired with the timing at which the low-quality video signal is acquired. For this reason, even if one tries to view a high-quality video signal corresponding to a video at a certain timing in the low-quality video signal, it is difficult to identify the video at that timing from the high-quality video signal.

[0008] Therefore, a main object of the present invention is to make it possible to easily obtain high-quality still image data that corresponds to any timing of moving image data. [Means for solving the problem]

[0009] As a result of intensively studying means for solving the above problems, the inventors of the present invention have found that, in addition to a path of a video encoder that converts an original image signal acquired by an image sensor into video data, a path of a still image encoder that converts the same image signal into still image data is added, and this still image data is stored in memory in association with a frame of the video data, so that still image data corresponding to any timing in the video data can be easily acquired. Based on the above findings, the inventors have come up with the idea that the problems of the conventional technology can be solved, and have completed the present invention. Specifically, the present invention has the following configuration.

[0010] A first aspect of the present invention relates to an information processing terminal. The information processing terminal according to the present invention includes an image sensor, a video encoder, a memory, and a still image encoder. The image sensor generates an image signal. The video encoder converts the image signal into video data by video encoding (mainly compression and format conversion). The video encoding includes a compression / conversion process for combining a plurality of frames constituting the video data into one video data, and a compression / conversion process for each frame. In addition, in the case where an audio signal is acquired together with the image signal, the video encoding also includes a compression / conversion process for combining the video data and the audio data into one file. The memory is preferably a non-volatile memory, but a volatile memory may be used. The still image encoder converts the image signal into still image data by still image encoding. Unlike video encoding, still image encoding only performs compression / conversion process for each frame. In addition, in still image encoding, it is basically unnecessary to associate still image data with audio data. The still image encoder associates still image data with video data and stores it in the memory. As a specific means, for example, a still image encoder may add synchronization data to still image data for chronologically associating it with video data and store it in a memory. For example, in the present invention, video data and still image data are generated from the same image signal, and the same synchronization data (in other words, a time stamp) is added to a certain frame and a certain still image data constituting the video data. However, the specific means for associating still image data with video data is not limited to this. As a result, when one frame is arbitrarily designated from among a large number of frames constituting the video data, it becomes possible to easily distinguish still image data acquired at the same timing as the designated frame.

[0011] In the information processing terminal according to the present invention, the moving image encoder and / or the still image encoder may be implemented in software as part of the functions executed by a processor such as a CPU or a GPU. The moving image encoder and / or the still image encoder may be implemented in hardware by an LSI (large scale integrated circuit) different from the processor. In particular, it is preferable that one of the moving image encoder and the still image encoder is implemented in software and the other is implemented in hardware. In this way, the same image signal can be efficiently converted into moving image data and still image data by driving the moving image encoder and the still image encoder in parallel on separate paths.

[0012] The information processing terminal according to the present invention preferably includes a communication unit and a control unit. The communication unit is an element for communicating with a receiving terminal via a communication line such as the Internet. The information processing terminal (transmitting terminal) and the receiving terminal may be configured to be able to communicate with each other via a server device or the like, or may be configured to be able to communicate directly by Peer-to-Peer. When the control unit receives a still image request signal including a designation of a specific timing in the video data from the receiving terminal while transmitting video data to the receiving terminal, the control unit executes a process of reading still image data corresponding to the timing from the memory and transmitting the still image data to the receiving terminal. This embodiment assumes that, for example, a worker at the site wears an information processing terminal (transmitting terminal) to capture a video and transmits the video data to a receiving terminal of a remote supporter. In this case, when the supporter designates an arbitrary timing in the video data, high-quality still image data corresponding to the designated timing can be transmitted from the information processing terminal to the receiving terminal.

[0013] In the information processing terminal according to the present invention, the communication unit may further be capable of communicating with a server device. In this case, the control unit further executes a process of uploading still image data stored in the memory to the server device after the transmission of the moving image data to the receiving terminal is completed. At this time, the information processing terminal may upload the moving image data to the server device together with the still image data. In this way, if the information processing terminal uploads the still image data to the server device, the receiving terminal can obtain or view any still image data by accessing the server device, even if the still image data is deleted from the memory of the information processing terminal, for example.

[0014] In the information processing terminal according to the present invention, the control unit may analyze the video data, and when it detects that a specific image is included in the video data, drive the still image encoder to convert the image signal into still image data and control the still image encoder to store the still image data in the memory. For example, even if a two-dimensional code or a three-dimensional code (QR Code (registered trademark)) is included in the video data, the information processing terminal (transmitting terminal) or the receiving terminal may not be able to accurately read these codes due to poor image quality of the video data. In this case, by using the inclusion of such a three-dimensional code or the like in the video data as a trigger to start the still image encoder and obtain still image data, it becomes possible to accurately read the three-dimensional code or the like because the image quality of the still image data is better than that of the video data.

[0015] The information processing terminal according to the present invention preferably includes one or more sensors selected from a gesture sensor, an acceleration sensor, a gyro sensor, a geomagnetic sensor, and a biosensor. In this case, the control unit of the information processing terminal may control the still image encoder so that, when the sensor detects a specific signal, the encoder is driven to convert the image signal into still image data and store the still image data in the memory. In this way, when the sensor detects a specific movement of the wearer of the information processing terminal, such as an action of looking at one's hand, an action of focusing on a point in front of the wearer for a predetermined period of time or more, or an action of falling, still image data of higher image quality is automatically generated. As a result, for example, before a designation is made by a supporter at a remote location, still image data that the supporter will likely pay attention to can be predicted and prepared in advance. This improves the convenience of the information processing terminal according to the present invention.

[0016] The information processing terminal according to the present invention is preferably a wearable device to be worn by a worker on-site. In the information processing terminal according to the present invention, the image signal is preferably a signal of an image captured by the worker during on-site work.

[0017] A second aspect of the present invention relates to an image transmission system. The image processing system according to the present invention includes the information processing device (transmission terminal) according to the first aspect described above, and a reception terminal configured to be able to receive moving image data and still image data from the information processing device.

[0018] The third aspect of the present invention relates to a program for causing a mobile information terminal to function as the information processing terminal according to the first aspect. This program may be downloadable via the Internet or the like, or may be pre-installed in the mobile information terminal. This program may also be recorded on a recording medium such as a CD-ROM. Effect of the Invention

[0019] According to the present invention, still image data corresponding to any timing of moving image data can be easily obtained. [Brief description of the drawings]

[0020] [Figure 1] FIG. 1 shows an overall view of a system according to an embodiment of the present invention. [Diagram 2] FIG. 2 shows an overview of the generation process of moving image data and still image data in the transmitting terminal. [Diagram 3] FIG. 3 shows a neck-worn wearable device as an example of a transmitting terminal. [Figure 4] FIG. 4 is a block diagram showing the functional configuration of the system according to the present invention. [Diagram 5] FIG. 5 is a block diagram showing more specific components regarding the image processing function in the control unit of the transmitting terminal. [Figure 6] FIG. 6 shows the correspondence between moving image data and still image data, as well as an overview of the process of transmitting moving image data and still image data from a transmitting terminal to a receiving terminal. [Figure 7] FIG. 7 shows an example of a process flow for transmitting moving image data and still image data from a transmitting terminal to a receiving terminal, and illustrates a case in which still image data is periodically saved while the moving image data is being transmitted. [Figure 8] FIG. 8 shows an example of a process flow for transmitting video data and still image data from a transmitting terminal to a receiving terminal, and illustrates a case in which the still image data is saved when a predetermined trigger is detected during the transmission of the video data. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0021] Hereinafter, the embodiments of the present invention will be described with reference to the drawings. The present invention is not limited to the embodiments described below, and includes appropriate modifications of the embodiments described below within the scope obvious to those skilled in the art.

[0022] FIG. 1 is a schematic diagram showing the overall configuration of a system 100 according to an embodiment of the present invention. The system 100 according to the present embodiment is preferably used in applications where a supporter remotely supports a worker performing work on site via the Internet. In the present system 100, the worker captures video and still images using a transmission terminal 10, and transmits these data to a receiving terminal 20 operated by the supporter via the Internet. In addition, a supporter in a remote location can view the video and still images sent from the worker's transmission terminal 10 through the receiving terminal 20, and can also exchange voices with the transmission terminal 10. In addition, the present system 100 may include an external cloud server 30 that provides a video conversation tool for the transmission terminal 10 and the receiving terminal 20. In this case, the video and still image data is transmitted from the transmission terminal 10 to the receiving terminal 20 via the cloud server 30.

[0023] 2 is a schematic diagram showing an image transmission process by a transmitting terminal 10 (information processing terminal) constituting a system 100 according to the present invention. The transmitting terminal 10 is equipped with a digital camera, and converts an image signal acquired by an image sensor of this camera into video data composed of a plurality of frames that are successive over time. The transmitting terminal 10 is also equipped with a microphone, and an audio signal acquired by this microphone is also associated with the video data. Such video data with audio is transmitted in real time to the receiving terminal 20 via, for example, a cloud server 30. Such transmission of video data is already known from a video conference system, etc.

[0024] The transmitting terminal 10 of the present invention further converts the image signal acquired by the image sensor of the camera into still image data consisting of a single frame. At this time, the video data and the still image data are based on the image signal acquired by the same image sensor. Therefore, a certain frame constituting the video data (hereinafter referred to as a "video frame") and a frame constituting the still image data (hereinafter referred to as a "still image frame") can be synchronized in terms of timing. Also, still image data, unlike video data, does not require combining frames or attaching audio signals, so the data capacity of the still image frame can be made larger than that of the video frame, and the resolution (density) of each frame can be made higher. Also, this still image data is not a frame extracted from the video data, but is generated from the original image signal separately from the video frame, so it may also include information that was discarded when the image signal was converted into video data. Therefore, it can be said that the still image frame captures the image of the subject more clearly.

[0025] The still image data obtained in this manner is assigned the same synchronization data (codes t1, t2, ...) as the video frame that has the same timing of acquiring the image signal among the many frames constituting the video data, and is stored in the memory of the transmitting terminal 10. Here, for example, assume that the receiving terminal 20, which is displaying a video transmitted from the transmitting terminal 10, requests the transmission of a clearer image by specifying a transmission frame at a certain timing (e.g., t1). In this case, the transmitting terminal 10 reads out from the memory the still image data corresponding to the timing (e.g., t1) specified by the receiving terminal 20, and transmits this still image data to the receiving terminal 20. In this manner, by generating high-quality still image data separately from the video data and storing it at least temporarily in the memory, it becomes possible to transmit high-quality still image data in response to a request from the receiving terminal 20.

[0026] Next, the configuration of the system 100 according to an embodiment of the present invention will be described in more detail. FIG. 3 is an external perspective view showing an example of the transmission terminal 10. FIG. 4 shows an example of hardware elements of the transmission terminal 10. As shown in FIG. 3, the transmission terminal 10 in this embodiment is a neck-hanging type wearable device. The transmission terminal 10 includes a left arm section, a right arm section, and a main body section that connects them at the back of the neck of the wearer. When wearing the transmission terminal 10, the main body section is brought into contact with the back of the neck of the wearer, and the left arm section and the right arm section are hung from the side of the neck toward the chest side of the wearer, and the entire device is hung around the neck. Various electronic components are stored inside the housing of the transmission terminal 10.

[0027] The left arm and the right arm are each provided with a plurality of sound collection units 14 (microphones). The sound collection units 14 are arranged mainly for the purpose of acquiring sounds around the wearer and the voices of the wearer and the person speaking to him / her. It is preferable to adopt an omnidirectional (non-directional) microphone as the sound collection unit 14 so that sounds generated around the wearer can be widely collected. As the sound collection unit 14, a well-known microphone such as a dynamic microphone, a condenser microphone, or a MEMS (Micro-Electrical-Mechanical Systems) microphone may be adopted. The sound collection unit 14 converts sounds into electrical signals, amplifies the electrical signals by an amplifier circuit, converts the electrical signals into digital information by an A / D conversion circuit, and outputs the digital information to the control unit 11. The sound signals acquired by the sound collection unit 14 are transmitted to the control unit 11 provided in the housing. In this embodiment, the sound signals acquired by the sound collection unit 14 can be transmitted to the receiving terminal 20 via the Internet through the communication unit 13. As a result, the sounds acquired by the on-site worker by the transmitting terminal 10 are also shared with the receiving terminal 20 of the supporter in a remote location.

[0028] The left arm is further provided with an imaging unit 15 including an image sensor. Specifically, the imaging unit 15 is provided on the tip surface of the left arm, and the imaging unit 15 can capture an image of the front side of the wearer. The image sensor forms an image of light (including reflected light) emitted from an object on the light receiving surface of the image sensor through an optical system, and photoelectrically converts the brightness of the image into an amount of electric charge to obtain an image signal. The image signal acquired by the image sensor of the imaging unit 15 is transmitted to the control unit 11 in the housing and converted into video data and / or still image data by encoding for video and / or encoding for still images. A general digital camera may be used as the imaging unit 15. The imaging unit 15 is composed of, for example, a photographing lens, a mechanical shutter, a shutter driver, a photoelectric conversion element such as a CCD image sensor unit, a digital signal processor (DSP) that reads the amount of electric charge from the photoelectric conversion element and generates image data, and an IC memory. The image data acquired by the imaging unit 15 is supplied to the control unit 11 and stored in the storage unit 12. In addition, a predetermined image analysis process may be performed on the image data. As described above, the video and still images captured by the imaging unit 15 are transmitted to the receiving terminal 20 via the Internet through the communication unit 13. This allows the video and still images captured by the on-site worker at the transmitting terminal 10 to be shared with the receiving terminal 20 of the supporter in a remote location.

[0029] A non-contact gesture sensor 16 is further provided on the right arm. The gesture sensor 16 is arranged on the distal end surface of the right arm mainly for the purpose of detecting the movement of the wearer's hand in front of the transmission terminal 10. The gesture sensor 16 detects, for example, the movement and shape of the wearer's fingers. An example of the gesture sensor 16 is an optical sensor that irradiates light from an infrared emitting LED toward an object and detects the movement and shape of the object by capturing the change in the reflected light with a light receiving element. Detection information by the gesture sensor 16 is transmitted to the control unit 11 and is mainly used to control the imaging unit 15 and the sound emitting unit 18. Specifically, the detection information by the gesture sensor 16 is used to control the start and stop of the imaging unit 15 and the sound emitting unit 18. For example, the gesture sensor 16 may detect that an object such as the wearer's hand approaches the gesture sensor 16 and control the imaging unit 15, or may detect that the wearer makes a predetermined gesture within the detection range of the gesture sensor 16 and control the imaging unit 15. The positions of the imaging unit 15 and the gesture sensor 16 can be interchanged. The gesture sensor 16 may also be replaced with a proximity sensor. The proximity sensor detects, for example, that the wearer's finger approaches within a predetermined range. As the proximity sensor, a known type such as an optical type, an ultrasonic type, a magnetic type, a capacitance type, or a thermal type can be used.

[0030] A sound emitting unit (speaker) 18 is provided on the outside (opposite side of the wearer) of the main body unit located on the back of the neck of the wearer. In this embodiment, the sound emitting unit 18 is arranged to output sound toward the outside of the main body unit. By emitting sound from the back of the neck of the wearer toward the rear in this manner, the sound output from the sound emitting unit 18 is less likely to reach the interlocutor who is in front of the wearer. This makes it easier for the interlocutor to distinguish between the voice emitted by the wearer himself and the sound emitted from the sound emitting unit 18 of the transmission terminal 10. The sound emitting unit 18 is an acoustic device that converts an electric signal into physical vibrations (i.e., sound). An example of the sound emitting unit 18 is a general speaker that transmits sound to the wearer by air vibration. In addition, the sound emitting unit 18 may be a bone conduction speaker that transmits sound to the wearer by vibrating the bones of the wearer. In this case, the sound emitting unit 18 may be provided on the inside (wearer side) of the main body unit, and the bone conduction speaker may be configured to contact the bones (cervical vertebrae) at the back of the neck of the wearer. In this embodiment, the audio signal input to the receiving terminal 20 is transmitted to the transmitting terminal 10 via the Internet. The transmitting terminal 10 converts the audio signal received from the receiving terminal 20 into sound by the sound emitting unit 18 and outputs the sound. This allows the voice of the supporter operating the receiving terminal 20 to be delivered to the worker wearing the transmitting terminal 10.

[0031] As shown in Fig. 4, the control unit 11 of the transmitting terminal 10 performs arithmetic processing to control other elements included in the transmitting terminal 10. A processor such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit) can be used as the control unit 11. The processor constituting the control unit 11 basically reads a program stored in the storage unit 12, expands it in a main memory, and executes a predetermined arithmetic processing according to the program. The control unit 11 can also write and read the results of the arithmetic processing according to the program to and from the storage unit 12 as appropriate. The control unit 11 may also include an LSI (Large Scale Integration) for image processing. Details of the control unit 11 of the transmitting terminal 10 will be described later with reference to Fig. 5.

[0032] The memory unit 12 of the transmitting terminal 10 is an element for storing information used for arithmetic processing and the like in the control unit 11 and the results of the arithmetic processing. The storage function of the memory unit 12 can be realized by a non-volatile memory such as an HDD and an SDD. The memory unit 12 may also have a function as a main memory for writing or reading intermediate progress of arithmetic processing by the control unit 11. The memory function of the memory unit 12 can be realized by a volatile memory such as a RAM or a DRAM. The memory unit 12 may also store ID information specific to the user who owns it. The memory unit 12 may also store an IP address, which is identification information on the network of the transmitting terminal 10.

[0033] The communication unit 13 of the transmitting terminal 10 is an element for wirelessly communicating with the receiving terminal 20 and the cloud server 30. In order to communicate with the receiving terminal 20 and the cloud server 30 via the Internet, the communication unit 13 may employ a communication module for wireless communication using a known mobile communication standard such as 3G (W-CDMA), 4G (LTE / LTE-Advanced), or 5G, or a wireless LAN method such as Wi-Fi (registered trademark). The transmitting terminal 10 can also wirelessly communicate directly with the receiving terminal 20 without going through the cloud server 30. In this case, the communication unit 13 preferably employs a communication module for close-proximity wireless communication using a method such as Bluetooth (registered trademark) or NFC in order to directly communicate with the receiving terminal 20.

[0034] The sensors 17 of the transmitting terminal 10 include, for example, a sensor device for detecting the operation or usage status of the transmitting terminal 10, or the biological information of the wearer. As the sensors 17, a sensor module mounted on a general mobile information terminal or a wearable device may be adopted. For example, the sensors 17 include a gyro sensor, an acceleration sensor, a geomagnetic sensor, and a battery sensor. The sensors 17 may also include a biological sensor for detecting the biological information of the wearer, such as a body temperature sensor, a heart rate sensor, a blood oxygen concentration sensor, a blood pressure sensor, and an electrocardiogram sensor.

[0035] The location information acquisition unit 19 of the transmitting terminal 10 is an element for acquiring current location information of the transmitting terminal 10. Specifically, the location information acquisition unit 19 has a function of performing positioning using a global positioning system (GPS). Based on information on radio wave transmission times included in radio waves transmitted from a plurality of GPS satellites, the location information acquisition unit 19 measures the time required to receive each radio wave, and transmits time information indicating the time to the control unit 11. Based on the acquired time information, the control unit 11 can calculate information on the latitude and longitude of the location of the transmitting terminal 10. The location information acquisition unit 19 may also acquire current location information by scanning radio waves and beacon signals transmitted from wireless base stations such as Wi-Fi (registered trademark) access points.

[0036] As is clear from Fig. 3 and Fig. 4, in this embodiment, the transmitting terminal 10 does not have a display device such as a display or a monitor. Therefore, although an operator can perform relatively simple operations such as turning on / off each hardware element using the gesture sensor 16 or the like, it is difficult for the operator to perform complex operations such as operating an application program. When using a transmitting terminal 10 that does not have such a display device, it is particularly effective to remotely control the transmitting terminal 10 via the Internet using the receiving terminal 20 or the like, as in the system according to the present invention.

[0037] FIG. 4 further illustrates an example of hardware elements of the receiving terminal 20. The receiving terminal 20 can be realized by a general personal computer (PC). The receiving terminal 20 may be either a desktop PC or a laptop PC. The receiving terminal 20 may also be a smartphone or a tablet terminal. The receiving terminal 20 has a control unit 21, a storage unit 22, a communication unit 23, a display unit 24, an operation unit 25, a sound collection unit 26, and a sound emission unit 27. A general PC and peripheral devices used therewith can be used as these elements.

[0038] The control unit 21 of the receiving terminal 20 is configured with a processor such as a CPU or a GPU. The control unit 21 basically reads out a program stored in the storage unit 22, loads it into the main memory, and executes a predetermined calculation process according to the program. The control unit 11 can also write and read out the calculation results according to the program to and from the storage unit 12 as appropriate. In this way, the control unit 21 performs control processing of each of the elements 22 to 27 according to the program stored in the storage unit 22.

[0039] The memory unit 22 of the receiving terminal 20 is an element for storing information used for arithmetic processing and the like in the control unit 21 and the results of the arithmetic processing. The storage function of the memory unit 22 can be realized by a non-volatile memory such as an HDD or an SDD. The memory unit 22 may also have a function as a main memory for writing or reading intermediate progress of arithmetic processing by the control unit 11. The memory function of the memory unit 22 can be realized by a volatile memory such as a RAM or a DRAM.

[0040] The communication unit 23 of the receiving terminal 20 is an element for communicating with the transmitting terminal 10 and the cloud server 30. The communication method of the communication unit 23 may be either wired or wireless. For example, the receiving terminal 20 may be connected to the Internet via a wired optical line or telephone line, or may be connected to the Internet via a wireless LAN method such as Wi-Fi (registered trademark). As described above, the receiving terminal 20 can also wirelessly communicate directly with the transmitting terminal 10 without going through the cloud server 30.

[0041] The display unit 24 of the receiving terminal 20 is an element for displaying images. In the system according to the present invention, mainly moving images and / or still images received from the transmitting terminal 10 are displayed on the display unit 24. As the display unit 24, a known device such as a liquid crystal display or an organic EL display can be used. The display unit 24 may also be a projector that projects image light onto a screen.

[0042] The operation unit 25 of the receiving terminal 20 is an element for the user (supporter) to input predetermined operation information to the receiving terminal 20 (particularly the control unit 21). As the operation unit 25, known devices such as a touch panel, a mouse, a keyboard, a track pad, a stylus pen, and a pen tablet can be used. It is also possible to configure a touch panel display by overlaying a touch panel on a display screen.

[0043] The sound collection unit 26 of the receiving terminal 20 is an element for mainly acquiring the voice of the user (supporter). A directional or omnidirectional (non-directional) microphone may be adopted as the sound collection unit 26. As the microphone, a well-known microphone such as a dynamic microphone, a condenser microphone, or a MEMS microphone may be adopted. In this embodiment, the voice signal acquired by the sound collection unit 26 of the receiving terminal 20 can be transmitted to the transmitting terminal 10 via the communication unit 23 and the Internet. As a result, the voice of the supporter is output from the transmitting terminal 10 of the worker working at the site.

[0044] The sound emitting unit 27 of the receiving terminal 20 is an element for mainly outputting the sound received from the transmitting terminal 10. As the sound emitting unit 27, a general speaker, earphone, or headphone that transmits sound by air vibration may be adopted. In this embodiment, the sound signal input to the transmitting terminal 10 is transmitted to the receiving terminal 20 via the Internet. The receiving terminal 20 converts the sound signal received from the transmitting terminal 10 into sound by the sound emitting unit 27 and outputs the sound. In this way, the sound acquired by the transmitting terminal 10 can be delivered to the supporter operating the receiving terminal 20.

[0045] The cloud server 30 provides a video conversation tool such as a Web conference to the transmitting terminal 10 and the receiving terminal 20. The cloud server 30 is composed of one or more server devices 31. When establishing a communication connection between the transmitting terminal 10 and the receiving terminal 20, the cloud server 30 requests each user (worker and supporter) of the transmitting terminal 10 and the receiving terminal 20 to log in to an account for the video conversation tool. A dedicated application program for the video conversation tool is installed in each of the transmitting terminal 10 and the receiving terminal 20. The transmitting terminal 10 and the receiving terminal 20 execute the program to access the cloud server 30 and log in to their respective accounts. When the transmitting terminal 10 and the receiving terminal 20 are successful in the login authentication, the cloud server 30 starts relaying the transmission and reception of image data and sound data between the transmitting terminal 10 and the receiving terminal 20. A commercially available tool that is generally available can be appropriately used as the video conversation tool.

[0046] Next, the encoding process of video data and still image data in the transmitting terminal 10 will be specifically described with reference to Fig. 5. As shown in Fig. 5, an image signal acquired by an image sensor of the imaging unit 15 is input to the control unit 11. This image signal is also called a RAW image, which is in principle unprocessed sensor data acquired from an image sensor, and is the data with the largest amount of information before encoding for video or still images. In a RAW image, the numerical values ​​acquired by the red, green, and blue sensors are recorded as they are, and the color of a pixel is expressed by an RGB value. In the example shown in Fig. 5, the RAW image has a maximum image size of 3264 x 2448.

[0047] When an image signal (RAW image) is input to the control unit 11, it is basically input to a pipeline for moving images. The pipeline for moving images includes a buffer memory 11a, an image processing engine 11b for moving images, and an encoder 11c for moving images.

[0048] In the video pipeline, since image processing must be performed continuously, the RAW image is first temporarily stored in the buffer memory 11a. The buffer memory 11a can hold several frames of RAW images from the image sensor. Note that the buffer memory 11a has a plurality of storage areas secured as fixed areas on the main memory, and also holds information on which buffer area the RAW image from the image sensor is written to. For example, the buffer memory 11a is in the form of a circular ring buffer in which the storage area at the end and the storage area at the beginning are logically linked, and a certain amount of data can always be accumulated by overwriting the oldest data with the newest data.

[0049] The video image processing engine 11b reduces the RAW image read from the buffer memory 11a to a size that can be received by the video encoder 11c at the subsequent stage, and converts the color format. For example, the image processing engine 11b reduces the image size of the RAW image from 3264 x 2448 to 1920 x 1080. In addition, the colors of the RAW image are originally expressed in RGB values, but the image processing engine 11b converts them to YUV values. YUV expresses the color of a pixel using brightness (Y), the difference between brightness and blue (U), and the difference between brightness and red (V), and can be expressed with a smaller amount of data than RGB.

[0050] The video encoder 11c encodes the image data, which has been subjected to size reduction and color format conversion by the video image processing engine 11b, into a video format. Specifically, the video encoder 11c performs compression / conversion processing for combining multiple frames into one video data, and compression / conversion processing for each frame. In addition, when an audio signal is acquired together with an image signal, compression / conversion processing for combining the video data and the audio data into one file is also performed. In addition, the video encoder 11c varies the compression ratio according to the data transmission rate (communication bandwidth) between the transmitting terminal 10 and the receiving terminal 20. Specifically, when the data transmission rate is low, the image compression ratio may be increased. This allows video data in which multiple frames are combined over time to be obtained. Note that the video data may be encoded based on a known video compression standard. In the example shown in FIG. 5, H.264 (MPEG-4 AVC) is adopted as the video compression standard, but the encoding method is not limited to this.

[0051] Furthermore, the moving image encoder 11c assigns a timestamp (synchronization data) to the frames constituting the moving image data. This timestamp is used to synchronize still image data and moving image data, which will be described later. The moving image encoder 11c may assign a unique timestamp to all frames. Alternatively, when generating still image data, the moving image encoder 11c may assign a unique timestamp only to the frames corresponding to the still image data, as will be described later.

[0052] The video data created by the video pipelines 11a to 11c is transmitted in real time to the receiving terminal 20 and / or cloud server 30 via the Internet by the communication unit 13 of the transmitting terminal 10. The receiving terminal 20 can view the video data received from the transmitting terminal 10 in real time. The cloud server 30 may also store (back up) the video data received from the transmitting terminal 10 in cloud storage.

[0053] The transmitting terminal 10 of the present invention includes a pipeline for still images in addition to the above-mentioned pipeline for moving images. As shown in FIG. 5, the pipeline for still images includes an image processing engine 11d for still images, a still image encoder 11e, and a memory 11f. The buffer memory 11a can also be interpreted as being shared by the pipeline for moving images and the pipeline for still images. The pipeline for still images is for converting image signals from an image sensor into still image data in parallel with the pipeline for moving images. Therefore, by providing this pipeline for still images, it is possible to generate still image data in parallel with the generation process of moving image data by the pipeline for moving images, as described above.

[0054] The image processing engine 11d for still images reads out the RAW image stored in the buffer memory 11a and mainly performs a process of converting the color format. As described above, the buffer memory 11a temporarily holds the original RAW image acquired by the image sensor for a predetermined frame. When a processor such as a CPU constituting the control unit 11 receives a predetermined still image output instruction, the processor reads out the RAW image from the storage area immediately before the storage area of ​​the buffer memory 11a to which the current image is written, and transfers it to the image processing engine 11d for still images. The predetermined still image output instruction may be generated periodically, or may be generated when a predetermined trigger condition (described later) is satisfied. That is, the still image data may be generated periodically or at any timing. In the example shown in FIG. 5, the image processing engine 11d only performs color format conversion (RBG to YUV) of the RAW image, and does not reduce the image size. This makes it possible to obtain still image data with the highest image quality. However, the image processing engine 11d for still images may also reduce the image size in addition to color format conversion. In this case, however, it is preferable to maintain the image size of the still image data at least larger than that of each frame of the moving image data.

[0055] The still image encoder 11e encodes the image data, which has been subjected to color format conversion mainly by the image processing engine 11d for still images, into a still image format. In encoding for still images, since it is not necessary to combine a plurality of frames over time, only encoding within each frame is performed. Specifically, the still image encoder 11e may encode the image based on a known image compression standard for still images, such as JPEG. In addition, it is also possible to adopt an image compression method such as PNG or GIF. In this way, still image data is obtained. Note that this still image data is generated by the still image encoder 11e periodically or at an arbitrary timing, but not all of it is transmitted to the receiving terminal 20 or the like via the Internet. As such, since the still image data is not intended to be transmitted to the receiving terminal 20, it is not necessary to vary the compression rate according to the data transmission rate (communication band) between the transmitting terminal 10 and the receiving terminal 20. Therefore, it is preferable to compress the still image data at a fixed value that is not affected by such a data transmission rate. This allows the still image data to always maintain a high image quality.

[0056] The still image data generated by the still image encoder 11e is stored at least temporarily in the memory 11f. This memory 11f may use a part of the storage area of ​​the main memory, or a volatile or non-volatile memory may be provided separately from the main memory. The amount and time of storing the still image data in the memory 11f may be adjusted appropriately. For example, a limit may be set on the capacity of the memory 11f to store still image data, and when the capacity is exceeded, the oldest still image data may be deleted and new still image data may be stored. Also, the still image data may be stored in the memory 11f only while the transmitting terminal 10 is transmitting video data to the receiving terminal 20, and the still image data in the memory 11f may be deleted when the transmission of the video data is completed.

[0057] Furthermore, when storing still image data in the memory 11f, the still image encoder 11e assigns a timestamp (synchronization data) to each still image data. Specifically, in this embodiment, a frame of moving image data and a frame of still image data can be generated from one RAW image acquired by the image sensor. In this way, the same timestamp is assigned to each frame of moving image data and still image data generated from the same RAW image. This allows the moving image data being transmitted in real time and the still image data temporarily stored in the memory 11f to be synchronized by the timestamp.

[0058] In the example shown in FIG. 5, the image processing engine 11b for moving images and the encoder 11c for moving images are preferably implemented in hardware by an LSI different from a processor such as a CPU. On the other hand, the image processing engine 11d for still images and the encoder 11e for still images are preferably implemented in software, as functions are realized by executing a program on a processor such as a CPU. As described above, image processing of moving images is frequently performed in real time, so that hardware implementation can speed up processing and reduce power consumption. Image processing of still images is less frequent than that of moving images, so software implementation can be sufficient. However, as described above, still image data has high image quality and a large data capacity, so if still image data is generated frequently, the load on the processor becomes large. In this case, the image processing engine 11d for still images and the encoder 11e for still images can also be implemented in hardware. In this case, the image processing engine 11b for moving images and the encoder 11c for moving images may be implemented in software.

[0059] The still image data stored in the memory 11f is basically transmitted from the transmitting terminal 10 to the receiving terminal 20 in response to a request from the receiving terminal 20. An example of a process for transmitting such still image data from the transmitting terminal 10 to the receiving terminal 20 will be described in detail below.

[0060] FIG. 6 shows an overview of information exchanged between the transmitting terminal 10 and the receiving terminal 20. As shown in FIG. 6, the transmitting terminal 10 generates video data including a plurality of frames and still image data consisting of a single frame by the image processing described above. In this example, the still image data is generated periodically, and the frames of the video data generated at the same timing as the still image data are given the same timestamp as the still image data. In the illustrated example, timestamps t1, t2, t3, and t4 are given to the (1st), (5th), (9th), and (13th) frames of the video data, and the same timestamps are also given to the still image data corresponding to each frame. The video data is transmitted from the transmitting terminal 10 to the receiving terminal 20 in real time, but the still image data is first stored in the memory of the transmitting terminal 10.

[0061] Here, when the receiving terminal 20 receives the video data transmitted by the transmitting terminal 10, the receiving terminal 20 displays the video on a display unit such as a liquid crystal display. This video continues to be displayed on the display unit of the receiving terminal 20 as long as the transmitting terminal 10 continues to transmit the video data. Here, it is assumed that the supporter operating the receiving terminal 20 wishes to view a clearer image of a video frame displayed at a certain timing while watching the video. In this case, the supporter operates an operation unit such as a mouse or a touch panel to pause the playback of the video or click a predetermined UI icon to specify the timing at which a clearer image is desired. In this way, the video frame for which the supporter desires a clearer image is specified from the video data. Then, the receiving terminal 20 specifies the video frame that is closest to the video frame specified by the supporter among the video frames that have timestamps in the video data. For example, in the example shown in FIG. 6, it is assumed that the supporter specifies the (4)th frame among the video frames that make up the video data. In this case, the video frame that is closest to the (4)th video frame among the video frames that have timestamps is the (5)th video frame, and the (5)th video frame is given a timestamp of "t2". The receiving terminal 20 transmits information indicating that the timestamp "t2" has been specified to the transmitting terminal 10. The transmitting terminal 10 then reads out from memory the still image data to which the timestamp "t2" has been added, and transmits this still image data to the receiving terminal 20 in addition to the video data. When the receiving terminal 20 receives the still image data from the transmitting terminal 10, it displays the received still image data on the display unit together with or in place of the video data. This allows the supporter to obtain and view higher quality still image data from the transmitting terminal 10 while watching the video.

[0062] In the above example, the video frame closest to the video frame specified by the supporter is specified among the video frames in the video data to which a timestamp has been added. However, it is also possible to specify the video frame immediately before or immediately after the video frame specified by the supporter among the video frames in the video data to which a timestamp has been added. For example, in the example shown in FIG. 6, it is assumed that the supporter specifies the (4)th frame among the video frames constituting the video data. In this case, among the video frames to which a timestamp has been added, the frame immediately before the (4)th video frame is the (1)th video frame, and this (1)th video frame is assigned a timestamp of "t1". The receiving terminal 20 may transmit information indicating that the timestamp of "t1" has been specified to the transmitting terminal 10.

[0063] Fig. 7 shows a specific example of the flow of information processing performed by the transmitting terminal 10 and the receiving terminal 20 described above. Fig. 7 mainly shows an example of a case where still image data is generated periodically in the transmitting terminal 10. As shown in Fig. 7, this flow starts with the transmitting terminal 10 acquiring a RAW image using the image sensor of the imaging unit 15 (step S1-1). When the image sensor acquires the RAW image, the image processing engine 11b for moving images and the encoder 11c for moving images perform encoding for moving images to obtain moving image data (step S1-3), as described with reference to Fig. 5. This moving image data is transmitted in real time from the transmitting terminal 10 to the receiving terminal 20 (step S1-3).

[0064] During the transmission of the moving image data, the transmitting terminal 10 performs encoding for still images using the image processing engine 11d for still images and the still image encoder 11e as described with reference to Fig. 5 to obtain still image data (step S1-4). The obtained still image data is given a time stamp for synchronizing with the frames of the moving image data, and is temporarily stored in the memory 11f (step S1-5). In the embodiment shown in Fig. 7, still image encoding is performed periodically, and still image data with a time stamp is accumulated in the memory 11f.

[0065] On the other hand, when the receiving terminal 20 receives the video data from the transmitting terminal 10, it displays it on a display or the like (step S1-6). This display of the video by the receiving terminal 20 continues until the transmitting terminal 10 finishes acquiring images. Here, it is assumed that the user (supporter) operating the receiving terminal 20 specifies an arbitrary frame from the video being displayed (step S1-7). For example, the frame is specified by pausing the video or clicking a specific UI icon while the video is being displayed. The receiving terminal 20 identifies, from among the videos to which timestamps have been added, the frame closest to the frame specified by the user, for example, and transmits the timestamp of the specified frame to the transmitting terminal 10 as specification information (step S1-8).

[0066] When the transmitting terminal 10 receives the designation information from the receiving terminal 20, it reads out still image data to which the same timestamp as that timestamp is added from the memory (step S1-9). Then, the transmitting terminal 10 returns the still image data read out here to the receiving terminal 20 (step S1-10). When the receiving terminal 20 receives the still image data from the transmitting terminal 10, it displays it on the display (step S1-11). For example, the receiving terminal 20 may display the still image data instead of the video data, or may display the video data in one window while displaying the still image data in another window. The receiving terminal 20 may also save the still image data in a storage or the like. The receiving terminal 20 may simply save the still image data in a storage or the like without displaying it on the display. The receiving terminal 20 may also request the still image data many times while the video data is being transmitted from the transmitting terminal 10.

[0067] The above-mentioned real-time transmission process of moving image data and the sequential transmission process of still image data when a request is received are performed until the transmitting terminal 10 finishes image acquisition. When the transmitting terminal 10 finishes image acquisition (i.e., the image sensor is turned off), the transmitting terminal 10 uploads the still image data stored in the memory to the cloud server 30 (steps S1-12, S1-13). As a result, even if the transmitting terminal 10 deletes the still image data from the memory, the still image data is stored in the cloud server 30 for a certain period of time. Therefore, the receiving terminal 20 can also download the still image data from the cloud server 30 instead of the transmitting terminal 10.

[0068] Fig. 8 shows another specific example of the flow of information processing performed by the transmitting terminal 10 and the receiving terminal 20. In the example shown in Fig. 8, the processing (steps S2 to S4) from when the transmitting terminal 10 acquires an image, to when the transmitting terminal 10 encodes the image and transmits the video data to the receiving terminal 20, to when the receiving terminal 20 displays the video data, is the same as the example shown in Fig. 7.

[0069] On the other hand, in the example shown in Fig. 8, the control unit of the transmitting terminal 10 judges whether or not a predetermined trigger condition is satisfied during the transmission of the video data (step S2-5). If no particular trigger is detected, the process returns to step S2-1 and the transmission of the video data is continued. If it is judged that the predetermined trigger condition is satisfied, the RAW image acquired by the image sensor is encoded for a still image (step S2-6).

[0070] The trigger condition is not particularly limited, and a condition suitable for generating high-quality still image data may be set. For example, as described above, the transmitting terminal 10 includes various sensors such as a gesture sensor, an acceleration sensor, a gyro sensor, a geomagnetic sensor, and a biosensor. When these sensors detect a specific signal, it may be determined that the trigger condition is satisfied. For example, when a specific motion of a worker wearing the transmitting terminal 10 is detected by the acceleration sensor and the gyro sensor, it may be determined that the trigger condition is satisfied. Examples of the specific motion of the worker include a motion of the worker leaning forward to check his / her hands, a motion of the worker continuing to look at the same place for a predetermined time (e.g., 5 to 10 seconds) or more, a motion of the worker looking up, or a motion of the worker suddenly falling down.

[0071] Also, for example, when the control unit analyzes video data and detects that a specific image is included in the video data, it may determine that the trigger condition is satisfied. For example, when it is detected that a specific two-dimensional code or three-dimensional code (QR code (registered trademark)) is included in the video data, this may be used as a trigger to start a still image encoder and acquire still image data. This makes it possible to accurately read three-dimensional codes, etc., using high-quality still image data.

[0072] The transmitting terminal 10 stores the still image data in memory at least temporarily (step S2-7). Thereafter, the transmitting terminal 10 notifies the receiving terminal 20 that the still image data has been stored (step S2-8). Upon receiving this notification, the receiving terminal 20 displays the notification on a display or the like to confirm with the user (supporter) whether or not he or she wishes to obtain the still image data (step S2-9). If there is no request for still image data, the receiving terminal 20 continues to display the video data transmitted from the transmitting terminal 10 as usual until image acquisition by the transmitting terminal 10 is completed. On the other hand, if a request for still image data is made, the receiving terminal 20 transmits request information for the still image data to the transmitting terminal 10 (step S2-10).

[0073] When the transmitting terminal 10 receives still image data request information from the receiving terminal 20, it reads out the still image data related to the request from the memory (step S2-11) and returns it to the receiving terminal 20 (step S2-12). When the receiving terminal 20 receives the still image data from the transmitting terminal 10, it displays it on the display (step S2-13). Such real-time transmission processing of moving image data and sequential transmission processing of still image data when a request is received are performed until the transmitting terminal 10 finishes image acquisition. When the transmitting terminal 10 finishes image acquisition (i.e., the image sensor is turned off), the transmitting terminal 10 uploads the still image data stored in the memory to the cloud server 30 (steps S2-14, S2-15). This allows the receiving terminal 20 to download the still image data from the cloud server 30 even if the transmitting terminal 10 deletes the still image data from the memory.

[0074] It should be noted that the specific examples shown in Figures 7 and 8 are not alternatives, and both can be adopted as processes for transmitting video data and still image data from the transmitting terminal 10 to the receiving terminal 20. In other words, the transmitting terminal 10 can generate still image data periodically according to the specific example of Figure 7, and can also generate still image data when a predetermined trigger condition is satisfied according to the specific example of Figure 8. On the other hand, it is of course possible to adopt either the specific example shown in Figure 7 or Figure 8.

[0075] In the above, in order to express the contents of the present invention, the present specification has described the embodiments of the present invention with reference to the drawings. However, the present invention is not limited to the above-mentioned embodiments, and includes modifications and improvements that are obvious to a person skilled in the art based on the matters described in the present specification. [Explanation of symbols]

[0076] 10: transmitting terminal 11: control unit 11a...Buffer memory 11b...Video image processing engine 11c…Video Encoder 11d…Image Processing Engine for Still Images 11e…Still image encoder 11f…Memory 12...Memory section 13...Communication section 14... Sound collecting unit 15... Imaging unit 16...Gesture sensor 17...Sensors 18...Sound emitting section 19...Position information acquisition section 20: Receiving terminal 21: Control unit 22: Memory section 23: Communication section 24...Display section 25...Operation section 26...Sound collecting section 27...Sound emitting section 30... Cloud server 31... Server device 100…System

Claims

1. an image sensor for generating an image signal; a video encoder for converting the image signal into video data by encoding for video; Memory, a still image encoder that converts the image signal into still image data by encoding for still images and stores the still image data in the memory in association with the moving image data; An information processing terminal that is a wearable device.

2. The still image encoder adds synchronization data for chronologically associating the still image data with the video data, and stores the still image data in the memory.

2. The information processing terminal according to claim 1.

3. A communication unit capable of communicating with a receiving terminal via a communication line; a control unit that, when receiving a still image request signal including a designation of a specific timing in the moving image data from the receiving terminal while transmitting the moving image data to the receiving terminal, executes a process of reading still image data corresponding to the timing from the memory and transmitting the still image data to the receiving terminal.

2. The information processing terminal according to claim 1.

4. The communication unit is further capable of communicating with a server device, The control unit further executes a process of uploading the still image data stored in the memory together with the video data to the server device after the transmission of the video data to the receiving terminal is completed.

4. The information processing terminal according to claim 3.

5. a control unit that analyzes the moving image data, and controls the still image encoder so that, when it is detected that a specific image is included in the moving image data, the still image encoder is driven to convert the image signal into still image data and store the still image data in the memory.

2. The information processing terminal according to claim 1.

6. One or more sensors selected from a gesture sensor, an acceleration sensor, a gyro sensor, a geomagnetic sensor, and a biosensor; a control unit that controls the still image encoder so that, when the sensor detects a specific signal, the still image encoder is driven to convert the image signal into still image data and store the still image data in the memory.

2. The information processing terminal according to claim 1.

7. The information processing terminal is a wearable device that is worn by workers on-site.

2. The information processing terminal according to claim 1.

8. The image signal is a signal of an image taken by the worker during on-site work.

8. An information processing terminal according to claim 7.

9. The information processing terminal is a neck-hanging wearable device.

2. The information processing terminal according to claim 1.

10. An information processing terminal according to claim 3; the receiving terminal configured to be able to receive the moving image data and the still image data from the information processing terminal Image transmission system.

Images