Camera terminal, instruction terminal, communication system, work instruction method and program

Abstract

To provide an imaging terminal which can properly display an instruction from an instruction terminal by utilizing an image captured by the imaging terminal.SOLUTION: An imaging terminal comprises: an imaging unit which performs imaging in time series to acquire a plurality of frame images; a recording unit which records first time stamp information indicating a first time point at which each of the plurality of frame images is acquired in association with motion information indicating movement of the imaging terminal at the first time point; an instruction information reception unit which receives instruction information associated with instruction content information indicating instruction contents about the target frame image among the plurality of frame images and second time stamp information indicating a second time point at which the target frame image is acquired; and a generation unit which generates display contents to be displayed on a display unit on the basis of the instruction content information and the motion information of movement determination associated with the first time stamp information of the movement determination indicating a time point subsequent to the second time point.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present disclosure relates to a photographing terminal, an instruction terminal, a communication system, a work instruction method, and a program.

Background Art

[0002] Patent Document 1 discloses a technique for estimating the position and orientation of a worker from an image captured by a work-side terminal and displaying the estimated position and orientation of the worker on an instruction-side terminal. In the technique disclosed in Patent Document 1, the instruction-side terminal instructs the work-side terminal of the next work direction from the virtual screen, and the work-side terminal displays an image indicating the next work direction instructed from the instruction-side terminal.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the technique disclosed in Patent Document 1, there is a possibility that a display delay time may occur due to the communication time between the instruction-side terminal and the work-side terminal. In that case, in the technique disclosed in Patent Document 1, there is a possibility that the position and orientation of the worker may change after the work-side terminal transmits the captured image until it receives information indicating the direction to the next work position. In that case, in the image displayed by the work-side terminal, the direction to the next work position may be different from the direction intended by the work instructor. Therefore, an aspect of the present disclosure aims to provide a photographing terminal, an instruction terminal, a communication system, a work instruction method, and a program that can appropriately display an instruction from an instruction terminal using an image captured by the photographing terminal.

Means for Solving the Problems

[0005] A shooting terminal according to one embodiment of the present disclosure includes: a shooting unit that takes pictures in a time series and acquires multiple frame images; a recording unit that records motion information indicating the movement of the shooting terminal at a first time, associated with a first timestamp information indicating a first time when each of the multiple frame images was acquired; an instruction information receiving unit that receives instruction information associated with instruction content information indicating an instruction content for a target frame image among the multiple frame images and a second timestamp information indicating a second time when the target frame image was acquired; and a generation unit that generates display content to be displayed on a display unit based on the instruction content information and motion information for movement determination associated with a first timestamp information for movement determination indicating a time after the second time.

[0006] An instruction terminal according to one embodiment of the present disclosure includes: a video receiving unit that receives each frame video from a plurality of frame videos from a shooting terminal, associating it with a first timestamp information indicating a first time point in time when each frame video was acquired; and an instruction information transmitting unit that transmits instruction information to the shooting terminal, associating it with instruction content information indicating an instruction content for a target frame video from the plurality of frame videos, and a second timestamp information indicating a second time point in time when the target frame video was acquired, and causing the shooting terminal to display the instruction content based on the movement of the shooting terminal after the second time point.

[0007] A communication system according to one embodiment of the present disclosure includes a shooting terminal and an instruction terminal capable of communicating with the shooting terminal. The shooting terminal includes a shooting unit that performs shooting in a time series and acquires a plurality of frame images; a recording unit that records motion information indicating the movement of the shooting terminal at a first time, associated with a first timestamp information indicating a first time when each of the plurality of frame images was acquired; an instruction information receiving unit that receives instruction information associated with instruction content information indicating instruction content for a target frame image among the plurality of frame images and a second timestamp information indicating a second time when the target frame image was acquired; a generation unit that generates display content to be displayed on a display unit based on the instruction content information and motion information for movement determination associated with a first timestamp information for movement determination indicating a time after the second time; and a video transmission unit that transmits the instruction information to the instruction terminal, associating each of the frame images with the first timestamp information. The instruction terminal includes an instruction information transmission unit that transmits the instruction information to the shooting terminal.

[0008] A work instruction method according to one embodiment of the present disclosure includes the steps of: taking photographs in a time series to acquire multiple frame images; recording motion information indicating the movement of the shooting terminal at the first time, associated with a first timestamp information indicating the time when each frame image among the multiple frame images was acquired; receiving instruction information associated with instruction content information indicating the instruction content for a target frame image among the multiple frame images and a second timestamp information indicating the second time when the target frame image was acquired; and generating display content to be displayed on the display unit based on the instruction content information and motion information for movement determination associated with the first timestamp information for movement determination indicating a time after the second time.

[0009] A program according to one embodiment of the present disclosure causes a computer to perform the following functions: a function to capture images in a time series and acquire multiple frame images; a function to record motion information indicating the movement of the shooting terminal at a first time, associated with a first timestamp information indicating the first time when each of the multiple frame images was acquired; a function to receive instruction information associated with instruction content information indicating the content of an instruction for a target frame image among the multiple frame images, and a second timestamp information indicating the second time when the target frame image was acquired; and a function to generate display content to be displayed on a display unit based on the instruction content information and motion information for movement determination associated with the first timestamp information for movement determination indicating a time after the second time. [Brief explanation of the drawing]

[0010] [Figure 1] This is a diagram showing an example of a communication system configuration. [Figure 2] This is a block diagram showing an example of the configuration of a camera terminal. [Figure 3] This is a block diagram showing an example of the configuration of the control unit of the shooting terminal according to the first embodiment. [Figure 4] This figure shows an example of information where motion information and first timestamp information are associated and stored in the motion information storage area. [Figure 5] This is a block diagram showing an example of the configuration of a control terminal. [Figure 6] This is a block diagram showing an example of a control unit for an instruction terminal. [Figure 7] This flowchart shows an example of the operation of the shooting terminal according to the first embodiment. [Figure 8] Figure 7 shows a flowchart illustrating an example of the operation of the camera terminal. [Figure 9] This flowchart shows an example of the operation of the instruction terminal according to the first embodiment. [Figure 10] This figure shows an example of the displayed content when the camera terminal according to the first embodiment is moved. [Figure 11]This is a diagram showing an example of display content when there is no movement of the imaging terminal according to the first embodiment. [Figure 12] This is a block diagram showing an example of the configuration of the control unit of the imaging terminal according to the second embodiment. [Figure 13] This is a flowchart showing an example of the operation of the imaging terminal according to the second embodiment. [Figure 14] This is a flowchart showing an example of the operation of the imaging terminal following FIG. 13. [Figure 15] This is a diagram showing an example of display content of the imaging terminal according to the second embodiment. [Figure 16] This is a flowchart showing an example of the operation of the imaging terminal according to the third embodiment. [Figure 17] This is a flowchart showing an example of the operation of the instruction terminal according to the third embodiment. [Figure 18] This is a diagram showing an example of display content of the imaging terminal according to the third embodiment. [Figure 19] This is a diagram showing an example of display content of the imaging terminal following FIG. 18.

Embodiments for Carrying Out the Invention

[0011] (First Embodiment) FIG. 1 is a diagram showing an example of the configuration of the communication system 100. The communication system 100 includes an imaging terminal 101 and an instruction terminal 102. The imaging terminal 101 and the instruction terminal 102 are connected via a network 103. The network 103 is composed of a mobile communication network such as LTE (Long Term Evolution), 5G, or a LAN (Local Area Network) such as Wi-Fi (registered trademark).

[0012] The imaging terminal 101 is an information processing device used by an operator. The operator uses the imaging terminal 101 to capture images of the scene where the operator is working in chronological order. The imaging terminal 101 transmits the captured frame images to the instruction terminal 102 existing in a remote location. The imaging terminal 101 is, for example, a smartphone, a tablet terminal, or the like.

[0013] The instruction terminal 102 is an information processing device used by an instructor. The instructor is a person who gives instructions to an operator. And the instruction terminal 102 is, for example, a PC (Personal Computer) or the like. The instruction terminal 102 reproduces the frame images transmitted from the imaging terminal 101 in time series. For example, the instruction terminal 102 streams and plays back a moving image composed of the frame images transmitted from the imaging terminal 101. And the instruction terminal 102 acquires instruction content regarding the target frame image being reproduced based on the operations of the instructor. The instruction content includes an instruction position within the target frame image, various instructions, specification of a photographing object, etc. The various instructions include an instruction indicating approaching the specified object, an instruction indicating the position and direction of the operator's movement.

[0014] FIG. 2 is a block diagram showing an example of the configuration of the imaging terminal 101. It includes a communication unit 201, an audio output unit 202, an audio input unit 203, a touch panel unit 204, an imaging unit 205, a motion sensor 206, a storage unit 207, a memory 208, and a control unit 209.

[0015] The communication unit 201 is an interface for connecting to the network 103 for voice calls and packet communications.

[0016] The audio output unit 202 outputs audio. For example, the audio output unit 202 outputs audio based on the audio signal transmitted from the instruction terminal 102. The audio output unit 202 is composed of a speaker and a receiver.

[0017] The audio input unit 203 generates an audio signal when audio is input. The audio input unit 203 generates an audio signal when the audio spoken by the operator is input. The audio input unit 203 is composed of a microphone.

[0018] The touch panel unit 204 displays frame images and accepts input operations via touch operation by the operator. The touch panel unit 204 comprises a display unit 211 and a touch sensor unit 212. The display unit 211 displays information. The display unit 211 is composed of a liquid crystal panel, an organic EL (Electro-Luminescence), etc. The touch sensor unit 212 is positioned at the location of the display unit 211 and accepts user input operations by detecting the contact position. For example, the touch sensor unit 212 detects the contact position using a capacitive method.

[0019] Camera unit 205 captures multiple frames of video footage in chronological order.

[0020] The motion sensor 206 measures the position and orientation of the imaging terminal 101. For example, the motion sensor 206 consists of an accelerometer and an angular velocity sensor, and acquires output values ​​from the accelerometer and the angular velocity sensor. The output value from the accelerometer indicates the acceleration of the imaging terminal 101 in the horizontal X direction, the vertical Y direction, and the depth Z direction. The output value from the angular velocity sensor indicates the angular velocity of the imaging terminal 101 with respect to the three rotation axes of the X, Y, and Z directions. The X, Y, and Z directions are orthogonal to each other.

[0021] The storage unit 207 is a recording medium capable of recording various data, programs, etc. For example, the storage unit 207 is composed of flash ROM (Read Only Memory) or the like. The storage unit 207 includes a program storage area 213 and a motion information storage area 214. The program storage area 213 stores a program for operating the shooting terminal 101. The motion information storage area 214 stores motion information 322 (see Figure 3) associated with first timestamp information 323 (see Figure 3). The first timestamp information 323 indicates the first point in time when each frame of the multiple frame images was acquired. The motion information 322 indicates the movement of the shooting terminal 101 at the first point in time.

[0022] Memory 208 temporarily stores the program read by the control unit 209 and stored in the storage unit 207, as well as the processing results from the control unit 209.

[0023] The control unit 209 executes various processes according to the programs and data stored in the memory unit 207. The control unit 209 is composed of a processor, such as a CPU (Central Processing Unit).

[0024] Figure 3 is a block diagram showing an example of the configuration of the control unit 209. The control unit 209 includes a video acquisition unit 301, a motion information acquisition unit 302, a video encoding unit 303, a timestamp information acquisition unit 304, a recording unit 305, a motion history acquisition unit 306, a video transmission unit 307, an instruction information reception unit 308, a movement determination unit 309, an instruction position calculation unit 310, a generation unit 311, and an instruction response transmission unit 312. Note that the program storage area 213 is not shown in Figure 3.

[0025] The video acquisition unit 301 acquires the frame video 321 acquired by the shooting unit 205.

[0026] The motion information acquisition unit 302 acquires motion information 322. The motion information 322 indicates the movement of the shooting terminal 101 at a first time point. For example, the motion information 322 indicates the position of the shooting terminal 101 and the direction in which the shooting terminal 101 is held at a first time point.

[0027] The video encoding unit 303 encodes the frame video 321. Specifically, the video encoding unit 303 encodes the frame video 321 using the encoding method of a predetermined video compression standard. The video encoding unit 303 then adds a first timestamp information 323 to the encoded frame video 321, indicating the first time point in time when the frame video 321 was acquired by the shooting unit 205, to create stream data in mpeg2ts format or the like. For example, the predetermined video compression standard may be H.264 format, H.265 format, etc.

[0028] The timestamp information acquisition unit 304 acquires first timestamp information 323. The first timestamp information 323 indicates the first point in time when the frame video 321 was acquired. For example, the first timestamp information 323 may indicate a timestamp showing the elapsed time from the time when the acquisition of the frame video 321 started until the time when each frame video 321 was acquired, or it may indicate a frame number.

[0029] The recording unit 305 associates the motion information 322 with the first timestamp information 323 and records it in the motion information storage area 214.

[0030] The motion history acquisition unit 306 reads and acquires the motion information 322 associated with the specified first timestamp information 323 from the motion information storage area 214.

[0031] The video transmission unit 307 associates each video frame 321 with the first timestamp information 323 and transmits it to the instruction terminal 102. Specifically, the video transmission unit 307 transmits the stream created by the video encoding unit 303 to the instruction terminal 102 via the communication unit 201 using a streaming protocol. Examples of streaming protocols include SRT (Secure Reliable Transport) and RTP (Real-time Transport Protocol).

[0032] The instruction information receiving unit 308 receives instruction information 325, which is associated with instruction content information 326 and second timestamp information 327. The instruction content information 326 indicates the instruction content for the target frame video among a plurality of frame videos 321. The instruction content information 326 includes the instruction content and instruction coordinate information. The instruction coordinate information indicates the coordinates of the instruction position within the target frame video. The second timestamp information 327 indicates the second time point in time when the target frame video was acquired by the imaging unit 205. For example, the instruction content may include approaching the instruction position, specifying the object to be worked on, or moving to the instruction position or in the instruction direction.

[0033] The movement determination unit 309 determines, based on the motion information 324 for movement determination, whether or not the shooting terminal 101 has moved since the second time point. The motion information 324 for movement determination is motion information 322 associated with the first timestamp information 323 for movement determination. Specifically, the motion information 324 for movement determination is motion information 322 associated with the first timestamp information 323, which is used to determine whether or not the shooting terminal has moved. The first timestamp information 323 for movement determination is the first timestamp information 323 used for movement determination, and the one corresponding to the second time point and beyond is referenced.

[0034] The instruction position calculation unit 310 corrects the coordinates of the instruction position based on the motion information 324 of the movement determination and calculates the corrected coordinates.

[0035] The generation unit 311 generates display content 328 to be displayed on the display unit 211 based on the instruction content information 326 and the motion information 324 for movement determination. Specifically, the generation unit 311 determines whether or not the instruction position in the target frame image is present in the display frame image displayed on the display unit 211, based on the instruction coordinate information and the motion information 324 for movement determination. Then, the generation unit 311 generates display content 328 to be displayed on the display unit 211 according to whether or not the shooting terminal 101 has moved since the second time point and whether or not the instruction position is present in the display frame image. More specifically, if the shooting terminal 101 has moved since the second time point and the instruction position is not present in the display frame image, the generation unit 311 includes a movement request in the display content 328 that requests the shooting terminal 101 to move backward from the movement indicated by the motion information 324 for movement determination.

[0036] The instruction response transmission unit 312 transmits instruction response information 329 to the instruction terminal 102. The instruction response information 329 indicates the processing result related to the instruction information 325, or a request for re-instruction to the instruction terminal 102.

[0037] Figure 4 shows an example of information stored in the motion information storage area 214, in which motion information 322 and first timestamp information 323 are associated. For example, as illustrated in Figure 4, terminal time information, terminal position, terminal direction, and timestamp are stored in association with each other in the motion information storage area 214.

[0038] The terminal time information indicates the time when the motion information 322 was acquired. The terminal position indicates the position of the shooting terminal 101. The terminal direction indicates the orientation in which the user of the shooting terminal 101 is holding the terminal. For example, the terminal direction is shown as a vector in the X, Y, and Z directions. For example, the motion information 322 shows the terminal position and terminal direction as illustrated in Figure 4.

[0039] The timestamp indicates the elapsed time from the start of frame video acquisition 321 to the time each frame video 321 has been acquired. For example, the timestamp shown in Figure 4 shows the elapsed time from "16:00:01:000", the time when frame video acquisition 321 started, to the time each frame video 321 has been acquired. For example, the first timestamp information 323 shows the timestamp shown in Figure 4.

[0040] Figure 5 is a block diagram showing an example of the configuration of the instruction terminal 102. The instruction terminal 102 comprises a communication unit 501, an audio output unit 502, an audio input unit 503, a display unit 504, an operation unit 505, a storage unit 506, a memory 507, and a control unit 508.

[0041] The communication unit 501 is an interface that connects to the network 103 for voice calls and packet communication.

[0042] The audio output unit 502 outputs sound. For example, the audio output unit 502 outputs sound based on an audio signal transmitted from the camera terminal 101. The audio output unit 502 consists of a speaker.

[0043] The voice input unit 503 receives voice input and generates an audio signal. The voice input unit 503 receives voice input from the person giving the instruction and generates an audio signal. The voice input unit 503 consists of a microphone.

[0044] The display unit 504 displays frame images. The display unit 504 is composed of a liquid crystal panel, an organic EL, etc.

[0045] The control unit 505 accepts input operations from the user giving the instruction. Specifically, the control unit 505 accepts input operations to select the instruction content and the instruction location. The control unit 505 consists of a mouse, keyboard, touch sensor, etc.

[0046] The storage unit 506 is a recording medium capable of storing various data, programs, etc. For example, the storage unit 506 is composed of an HDD (Hard Disk Drive) or the like. The storage unit 506 includes a program storage area 511. The program storage area 511 stores a program that operates the instruction terminal 102.

[0047] Memory 507 temporarily stores the program stored in the storage unit 506, which has been read from the program storage area 511 by the control unit 508, as well as the processing results from the control unit 508.

[0048] The control unit 508 executes various processes according to the programs and data stored in the program storage area 511. For example, the control unit 508 is composed of a processor such as a CPU.

[0049] Figure 6 is a block diagram showing an example of the configuration of the control unit 508. The control unit 508 comprises a video receiving unit 601, a video decoding unit 602, a video generation unit 603, an instruction operation processing unit 604, a timestamp information acquisition unit 605, an instruction information creation unit 606, an instruction information transmission unit 607, an instruction response receiving unit 608, and an instruction result output unit 609.

[0050] The video receiving unit 601 receives the frame video 611 and the timestamp information 612 in association. The timestamp information 612 indicates the time when the frame video 611 was acquired. Specifically, the video receiving unit 601 receives the stream created by the video encoding unit 303 using a streaming protocol via the communication unit 501. The video receiving unit 601 then generates packet data in a format such as mpeg2ts. For example, streaming protocols include SRT and RTP.

[0051] The video decoding unit 602 interprets the mpeg2ts format data generated by the video receiving unit 601 and decodes the frame video 611 in H.264 and H.265 formats, and obtains the decoded frame video 611. Furthermore, the video decoding unit 602 obtains timestamp information 612 associated with the decoded frame video 611.

[0052] The video generation unit 603 displays the decoded frame video 611 on the display unit 504 at a timing based on the timestamp information 612 associated with it.

[0053] The instruction operation processing unit 604 receives input operations acquired by the operation unit 505.

[0054] The timestamp information acquisition unit 605 acquires timestamp information 612 associated with the frame video 611 displayed on the display unit 504.

[0055] The instruction information creation unit 606 generates instruction content information 326 and second timestamp information 327. The instruction content information 326 indicates the instruction content related to the target frame video. The target frame video is the frame video displayed on the display unit 504 when the input operation is received by the operation unit 505, and is also the frame video acquired by the shooting unit 205 at a second time point. The second timestamp information 327 indicates the timestamp information 612 associated with the target frame video, which is acquired by the timestamp information acquisition unit 605. In other words, the second time point indicated by the second timestamp information 327 indicates the time point indicated by the timestamp information 612 associated with the target frame video.

[0056] The instruction information transmission unit 607 transmits instruction information 325, created from the instruction content information 326 and the second timestamp information 327, to the camera terminal 101. The instruction response receiving unit 608 receives the instruction response information 329. The instruction result output unit 609 outputs a message indicating the instruction response information 329.

[0057] Figure 7 is a flowchart showing an example of the operation of the shooting terminal 101 according to this embodiment. When the shooting unit 205 is activated and shooting begins, the control unit 209 starts the process of step S701 as illustrated in Figure 7. For example, when the touch sensor unit 212 detects an operation to press the button displayed on the display unit 211 to start shooting, the control unit 209 starts the process of step S701 as illustrated in Figure 7.

[0058] In step S701, the motion information acquisition unit 302 starts acquiring motion information 322. Specifically, the motion information acquisition unit 302 starts acquiring the output value obtained by the motion sensor 206. When the motion information acquisition unit 302 acquires the output value obtained by the motion sensor 206, it calculates the position of the shooting terminal 101 and the holding direction of the shooting terminal 101 from the output value obtained by the motion sensor 206. The motion information acquisition unit 302 then acquires the calculated position and direction as motion information 322.

[0059] In step S702, the video acquisition unit 301 acquires the frame video 321 acquired by the shooting unit 205. When the video acquisition unit 301 acquires the frame video 321, the generation unit 311 includes the frame video 321 in the display content 328. In other words, when the video acquisition unit 301 acquires the frame video 321, the generation unit 311 displays the frame video 321 on the display unit 211.

[0060] In step S703, the timestamp information acquisition unit 304 acquires first timestamp information 323. Specifically, the first timestamp information 323 indicates the first point in time when the frame video 321 acquired in step S702 was acquired.

[0061] In step S704, the video encoding unit 303 encodes the frame video 321. Specifically, the video encoding unit 303 encodes the frame video 321 using the encoding method of a predetermined video compression standard. Then, the video encoding unit 303 adds the first timestamp information 323 obtained in step S703 to the encoded frame video 321 to create stream data such as mpeg2ts format. For example, the predetermined video compression standard may be H.264 format, H.265 format, etc.

[0062] In step S705, the video transmission unit 307 associates the frame video 321 with the first timestamp information 323 acquired in step S703 and transmits it to the instruction terminal 102. Specifically, the video transmission unit 307 transmits the stream created by the video encoding unit 303 to the instruction terminal 102 via the communication unit 201 using a streaming protocol. Examples of streaming protocols include SRT (Secure Reliable Transport) and RTP (Real-time Transport Protocol).

[0063] In step S706, the recording unit 305 associates the motion information 322 acquired in step S701 with the first timestamp information 323 and records it in the motion information storage area 214.

[0064] In step S707, the instruction information receiving unit 308 determines whether or not it has received instruction information 325. If instruction information 325 has not been received in step S707, the control unit 209 returns to step S701. On the other hand, if instruction information 325 has been received in step S707, the control unit 209 proceeds to step S801, as illustrated in Figure 8.

[0065] The operation of the shooting terminal 101 will now be explained, referring to Figure 8.

[0066] In step S801, the instruction information receiving unit 308 acquires the second timestamp information 327 contained in the instruction information 325 received in step S707 as illustrated in Figure 7. The instruction information 325 includes instruction content information 326 and the second timestamp information 327.

[0067] In step S802, the motion history acquisition unit 306 acquires motion information 324 for movement determination from the second time point onward, indicated by the second timestamp information 327 acquired in step S801. Specifically, the motion history acquisition unit 306 reads and acquires the motion information 324 for movement determination from the motion information storage area 214. The motion information 324 for movement determination is motion information 322 associated with the first timestamp information 323 for movement determination. Specifically, the motion information 324 for movement determination is motion information 322 associated with the first timestamp information 323, which is used to determine whether the shooting terminal has moved. The first timestamp information 323 for movement determination that corresponds to the second time point onward is referenced.

[0068] In step S803, the movement determination unit 309 determines whether or not the shooting terminal 101 has moved since the second time point, based on the motion information 324 of the movement determination acquired in step S802. Specifically, the movement determination unit 309 determines whether or not the position and direction indicated by the motion information 324 of the movement determination indicate that the shooting terminal 101 has moved.

[0069] If it is determined in step S803 that the shooting terminal 101 has not moved since the second time point, in step S804 the generation unit 311 specifically includes the target frame video in the display content 328 displayed on the display unit 211, and superimposes the instruction content onto the coordinates of the instruction position within the display frame video displayed on the display unit 211. The coordinates of the instruction position are indicated by the instruction coordinate information included in the instruction content information 326. Then the control unit 209 moves the process to step S809.

[0070] On the other hand, if it is determined in step S803 that the shooting terminal 101 has moved at a second time point or later, in step S805 the instruction position calculation unit 310 corrects the coordinates of the instruction position based on the motion information 324 of the movement determination. Specifically, the instruction position calculation unit 310 corrects the coordinates of the instruction position by moving the coordinates of the instruction position based on the position and direction indicated by the motion information 324 of the movement determination.

[0071] In step S806, the instruction position calculation unit 310 determines whether or not the instruction position is within the display frame image. Specifically, if the coordinates of the instruction position corrected in step S805 are not included within the display frame image, the instruction position calculation unit 310 determines that the instruction position is not within the display frame image. On the other hand, if the corrected coordinates of the instruction position are included within the display frame image, the instruction position calculation unit 310 determines that the instruction position is within the display frame image.

[0072] If it is determined in step S806 that there is no indicated position within the displayed frame image, in step S807 the generation unit 311 displays a movement request on the display unit 211. Specifically, the generation unit 311 includes a movement request in the display content 328 that requests the camera terminal 101 to move by going back in time to the movement indicated by the motion information 324 of the movement determination. Then the control unit 209 returns to step S805.

[0073] After the control unit 209 returns the process to step S805, it executes the processes from steps S702 to S706 in parallel with the processes from step S805 onward. Therefore, after the control unit 209 returns the process to step S805, the display unit 211 updates the display frame image with the new frame image acquired by the video acquisition unit 301. The generation unit 311 includes the movement request in the display content 328 until the display frame image displayed on the display unit 211 includes the indicated position within the target frame image. For example, the message indicating the movement request is superimposed on the display frame image and continues to be displayed on the display unit 211 until the display frame image displayed on the display unit 211 includes the indicated position within the target frame image.

[0074] In accordance with the movement request displayed in step S807, the operator moves the camera terminal 101 to reverse the movement indicated by the motion information 324 of the movement determination. Specifically, the operator moves the camera terminal 101 so that the indicated position in the target frame image is included in the display frame image displayed on the display unit 211.

[0075] On the other hand, if it is determined in step S806 that the instruction position exists within the displayed frame image, in step S808 the generation unit 311 includes the instruction content in the display content 328 and superimposes the instruction content onto the coordinates of the instruction position corrected in step S805 within the displayed frame image. In other words, if the instruction position in the target frame image becomes present in the new displayed frame image displayed on the display unit 211 after the generation unit 311 has included the movement request in the display content 328, it includes the instruction content in the display content 328.

[0076] In step S809, the instruction response transmission unit 312 transmits instruction response information 329 to the instruction terminal 102. If the instruction content included in the instruction content information 326 is displayed on the display unit 504, the instruction response information 329 indicates an instruction completion notification. The instruction completion notification indicates that the instruction has been completed. If, after a predetermined time has elapsed since the movement request was displayed on the display unit 211 in step S807, the instruction response transmission unit 312 may transmit instruction response information 329 indicating a request for re-instruction to the instruction terminal 102. Then, the control unit 209 terminates processing related to the instruction information 325 transmitted from the instruction terminal 102.

[0077] Figure 9 is a flowchart showing an example of the operation of the instruction terminal 102 according to this embodiment. When the instruction terminal 102 starts communicating with the imaging terminal 101, the control unit 508 starts the process of step S901 as illustrated in Figure 9.

[0078] In step S901, the video receiving unit 601 begins waiting for the frame video 611 to be received. In step S902, the video receiving unit 601 receives the frame video 611 and the timestamp information 612 associated with the frame video 611. Specifically, the video receiving unit 601 receives the stream transmitted from the video transmitting unit 307 via the communication unit 501 using a streaming protocol. The video receiving unit 601 then generates packet data for the stream data, such as in mpeg2ts format.

[0079] In step S903, the video decoding unit 602 decodes the frame video 611 received in step S902. Specifically, the video decoding unit 602 interprets the stream data such as mpeg2ts format generated in step S901 and decodes the frame video 611 in H.264 format or H.265 format.

[0080] In step S904, the video generation unit 603 displays the frame video 611 decoded in step S903 on the display unit 504. Specifically, the video generation unit 603 displays the frame video 611 on the display unit 504 at a timing based on the timestamp information 612 associated with the frame video 611 decoded in step S903.

[0081] In step S905, the instruction operation processing unit 604 determines whether or not an instruction operation has been received. For example, if the operation unit 505 receives an operation to specify a position within the display frame image currently displayed on the display unit 504, the instruction operation processing unit 604 determines that an instruction operation has been received.

[0082] If no instruction is received in step S905, the control unit 508 returns to step S902. On the other hand, if an instruction is received in step S905, the control unit 508 moves the process to step S906. After the control unit 508 moves the process to step S906, it may execute the processes from step S902 onwards in parallel with executing the processes from step S906 onwards.

[0083] In step S906, when an instruction operation is received in step S905, the instruction information creation unit 606 determines the frame image 611 currently displayed on the display unit 504 as the target frame image and generates instruction content information 326. For example, the instruction information creation unit 606 generates instruction content information 326 based on an operation that specifies a position within the target frame image. The instruction coordinate information included in the instruction content information 326 indicates the coordinates of the instruction position specified by the instruction operation within the target frame image.

[0084] For example, the instruction information generation unit 606 generates instruction content information 326 that instructs the operator to approach the object in the image at the specified location within the target frame video. In other words, the operator specifies the location of the object in the frame video 611 displayed on the display unit 504 that they want the operator to approach. In this case, the instruction content indicates to approach the object in the image at the specified location.

[0085] In step S907, the instruction information creation unit 606 acquires second timestamp information 327 associated with the target frame video currently displayed on the display unit 504.

[0086] In step S908, the instruction information transmission unit 607 generates instruction information 325, which is an association between the instruction content information 326 generated in step S906 and the second timestamp information 327 acquired in step S907.

[0087] In step S909, the instruction information transmission unit 607 transmits instruction information 325 to the shooting terminal 101 via the communication unit 501.

[0088] In step S910, the instruction response receiving unit 608 determines whether or not it has received the instruction response information 329. If the instruction response information 329 has not been received in step S910, the control unit 508 returns to step S910. In other words, the control unit 508 repeats the process in step S910 until the instruction response information 329 is received. Note that in step S909, after the instruction information transmitting unit 607 transmits the instruction information 325 to the imaging terminal 101 via the communication unit 501, the control unit 508 may execute the processes from step S902 onward and the process in step S910 in parallel until the instruction response information 329 is received.

[0089] On the other hand, if instruction response information 329 is received in step S910, the instruction result output unit 609 outputs the instruction response information 329 in step S911. For example, if the instruction response information 329 indicates an instruction completion notification, the instruction result output unit 609 displays characters or the like indicating that the instruction has been completed on the display unit 504. For example, the instruction result output unit 609 displays the characters "Instruction received" on the display unit 504. Alternatively, if the instruction response information 329 indicates an instruction completion notification, the instruction result output unit 609 may output an audio message indicating that the instruction has been completed to the audio output unit 502.

[0090] Alternatively, if the instruction response information 329 indicates a request for re-instruction, the instruction result output unit 609 will display characters or other information indicating a request for re-instruction on the display unit 504. For example, the instruction result output unit 609 will display the words "Please execute the instruction again" on the display unit. Alternatively, if the instruction response information 329 indicates a request for re-instruction, the instruction result output unit 609 will output an audio message indicating a request for re-instruction to the audio output unit 502.

[0091] Figure 10 shows an example of the displayed content 328 when the camera terminal 101 moves after the second time point.

[0092] The display screens 1011 to 1015 illustrated in Figure 10 show the display content 328 displayed on the display unit 211 of the shooting terminal 101 from time t1001 to time t1005. The display content 328 shown in the display screens 1011 to 1015 shows frame images 321 associated with the first timestamp information 323 indicating timestamps TIMESTAMP1 to TIMESTAMP5. Timestamps TIMESTAMP1 to TIMESTAMP5 are timestamps at time t1001 to time t1005. The motion information associated with the first timestamp information 323 indicating timestamps TIMESTAMP2 and TIMESTAMP3 is (x,y,z)=(2,0,0). This indicates that the shooting terminal 101 moved in the +x direction at time t1002 and time 1003.

[0093] Furthermore, the display screen 1021 shows the target frame video and instruction content information 326 displayed on the display unit 504 of the instruction terminal 102. The target frame video shown on the display screen 1021 is the frame video 321 associated with the timestamp TIMESTAMP1 indicating time t1001.

[0094] The instruction information 326 shown on the display screen 1021 indicates an instruction to approach object A at the coordinates (300,800) within the target frame video. Specifically, the instruction information creation unit 606 displays an arrow at the coordinates (300,800) within the target frame video, thereby displaying the instruction information 326 on the display screen 1021.

[0095] The instruction information creation unit 606 displays the display screen 1021 on the display unit 504 at a time after time t1001. In other words, after the shooting terminal 101 displays the frame video 321 associated with the timestamp TIMESTAMP1 on the display unit 211 at time t1001 in step S701, the instruction terminal 102 displays the frame video 321 associated with the timestamp TIMESTAMP1 on the display unit 504 at a time after time t1001 in step S904. This is because there is a communication delay and a display delay between the time the frame video 321 transmitted from the shooting terminal 101 and the time the instruction terminal 102 receives it. Then, for example, at time t1004, the shooting terminal 101 receives instruction information 325 associated with the timestamp TIMESTAMP1 indicating a second time point, transmitted from the instruction terminal 102.

[0096] Since the shooting terminal 101 moves in the +x direction at the second time point and beyond, the target frame video associated with the second timestamp information 327, which indicates the timestamp TIMESTAMP1 indicating the second time point, is different from the display frame video displayed on the display unit 211 at time point t1004. For example, on the display screen 1021, the indicated position indicated by the indicated coordinate information included in the received instruction information 325 indicates the position of object A. On the other hand, the display screen 1014 does not include an image of object A. In other words, the indicated position is not included in the display frame video displayed on the display unit 211 at time point t1004.

[0097] Therefore, in step S807, the generation unit 311 includes in the display content 328 a movement request that requests the camera terminal 101 to move backward from the movement indicated by the motion information 324 of the movement determination. Specifically, the generation unit 311 includes in the display content 328 a movement request that requests the camera terminal 101 to move in the -x direction, which is backward from the movement in the +x direction. For example, as illustrated in the display screen 1014, the generation unit 311 includes in the display content 328 an arrow indicating movement in the -x direction. Then, when the camera terminal 101 moves to a position where the indicated position is included in the displayed frame image, in step S808, as illustrated in the display screen 1015, the generation unit 311 includes the instruction content in the display content 328 and superimposes the instruction content on the corrected coordinates of the indicated position in the displayed frame image. As a result, the camera terminal 101 can appropriately display instructions from the instruction terminal 102 even if the camera terminal 101 moves after it has transmitted the frame image 321 to the instruction terminal 102. Therefore, the communication system 100 can appropriately display instructions from the instruction terminal using the video captured by the camera terminal 101.

[0098] Figure 11 shows an example of the display content 328 when the camera terminal 101 does not move after the second time point. Display screens 1111 to 1114 shown in Figure 11 show the display content 328 displayed on the display unit 211 of the camera terminal 101 from time point t1101 to time point 1104.

[0099] The display content 328 shown on display screens 1111 to 1114 shows frame images 321 associated with the first timestamp information 323, which indicates timestamps TIMESTAMP1 to TIMESTAMP4. Timestamps TIMESTAMP1 to TIMESTAMP4 are timestamps at time points t1101 to t1104.

[0100] The motion information associated with the first timestamp information 323, which indicates timestamps TIMESTAMP1 to TIMESTAMP4, is (x,y,z)=(0,0,0). This indicates that there was no movement of the shooting terminal 101 between time points t1101 and t1104. The display screen 1121 shows the target frame video and instruction content information 326 displayed on the display unit 504 of the instruction terminal 102. The target frame video shown on the display screen 1121 is the frame video 321 associated with timestamp TIMESTAMP1, which indicates the second time point.

[0101] The instruction information creation unit 606 displays the display screen 1121 on the display unit 504 at a time after time t1101. In other words, after the shooting terminal 101 displays the frame video 321 associated with the timestamp TIMESTAMP1 on the display unit 211 at time t1101 in step S701, the instruction terminal 102 displays the frame video 321 associated with the timestamp TIMESTAMP1 on the display unit 504 at a time after time t1101 in step S904. This is because there is a communication and display delay between the time the frame video 321 transmitted from the shooting terminal 101 and the time the instruction terminal 102 receives it. Then, for example, at time t1104, the shooting terminal 101 receives instruction information 325 associated with the timestamp TIMESTAMP1 indicating a second time point, transmitted from the instruction terminal 102.

[0102] Since the shooting terminal 101 does not move at the second time point and beyond, the target frame video and the display frame video displayed at the second time point and beyond are identical. For example, the instruction position is included in the display frame video displayed at time point t1104. Therefore, as illustrated in display screen 1114, the generation unit 311 includes the instruction content in the display content 328 and superimposes the instruction content onto the display frame video displayed at time point t1104. Consequently, even if a display delay occurs between the shooting terminal 101 and the instruction terminal 102, the instruction from the instruction terminal 102 can be displayed appropriately.

[0103] (Variation 1) As a modification 1 of the shooting terminal 101 according to this embodiment, when the motion information acquisition unit 302 acquires motion information 322 in step S701, it may calculate the position of the shooting terminal 101 and motion information 322 indicating the direction in which the shooting terminal 101 has moved, based on the frame video 321 acquired by the shooting unit 205. For example, the motion information acquisition unit 302 may calculate the position of the shooting terminal 101 and motion information 322 indicating the direction of the shooting terminal 101 from the time-series change in the position of the object image included in the time-series frame video.

[0104] (Second embodiment) The second embodiment will be described with reference to Figures 12 to 15. In the drawings, the same or equivalent elements are denoted by the same reference numerals, and redundant explanations are omitted. Configurations and processes that have substantially the same function as the other embodiments are referred to by the same reference numerals and their explanations are omitted, while the differences from the other embodiments will be explained.

[0105] Figure 12 is a block diagram showing an example of the configuration of the control unit 209 of the shooting terminal 101 according to this embodiment. The difference between the control unit 209 illustrated in Figure 12 and the control unit 209 illustrated in Figure 3 is that the control unit 209 illustrated in Figure 12 includes a recording unit 1201, a video decoding unit 1202, and a video playback unit 1203.

[0106] The recording unit 1201 records each of the multiple frame images 321. Specifically, the recording unit 1201 records the frame images 321 encoded by the video encoding unit 303 in the storage unit 207 in a stream data format such as mpeg2ts format, associated with the first timestamp information 323.

[0107] The video decoding unit 1202 decodes the frame video 321 associated with the specified first timestamp information 323 from among the multiple frame video 321 recorded by the recording unit 1201.

[0108] The video playback unit 1203 plays back the decoded frame video 321 associated with the specified first timestamp information 323.

[0109] In this embodiment, if there is movement of the shooting terminal 101 after the second time point, the generation unit 311 includes the target frame video played back by the video playback unit 1203 from among the multiple frame videos 321 recorded in the storage unit 207 in the display content 328, and superimposes the instruction content onto the coordinates of the instruction position within the target frame video.

[0110] Figure 13 is a flowchart illustrating an example of the operation of the imaging terminal 101 according to this embodiment. The processes of steps S1301 to S1304 illustrated in Figure 13 are the same as the processes of steps S701 to S704 illustrated in Figure 7, so a detailed explanation is omitted.

[0111] In step S1304, the video encoding unit 303 encodes the frame video 321. Then, in step S1305, the recording unit 1201 records the frame video 321 acquired in step S1302, associated with a first timestamp information 323 indicating the first point in time when the frame video 321 was acquired. Specifically, the recording unit 1201 records the frame video 321 encoded by the video encoding unit 303 in the storage unit 207 in the format of stream data such as mpeg2ts, associated with the first timestamp information 323. The control unit 209 then proceeds to step S1306. The processing in steps S1306 to S1308 is the same as the processing in steps S705 to S707, so a detailed explanation is omitted. Furthermore, after executing the processing in step S1308, the control unit 209 proceeds to step S1401, as illustrated in Figure 14.

[0112] Next, we will explain the operation of the shooting terminal 101, referring to Figure 14.

[0113] In step S1401, the instruction information receiving unit 308 acquires the second timestamp information 327 contained in the instruction information 325 received in step S1308 as illustrated in Figure 13. The processing in step S1401 is the same as the processing in step S801 as illustrated in Figure 8, so a detailed explanation is omitted.

[0114] In step S1402, the video decoding unit 1202 decodes the target frame video stored in the storage unit 207, which is associated with the second timestamp information 327. In step S1403, the video playback unit 1203 plays back the target frame video decoded in step S1402. In step S1404, the generation unit 311 overlays the instruction content onto the coordinates of the instruction position in the target frame video played back in step S1403 and displays it on the display unit 211.

[0115] In step S1405, the instruction response transmission unit 312 transmits instruction response information 329 to the instruction terminal 102. The process in step S1405 is the same as the process in S809 illustrated in Figure 8, so a detailed explanation is omitted. Then, the control unit 209 finishes processing the instruction information 325 transmitted from the instruction terminal 102.

[0116] Figure 15 shows an example of the display content 328 of the camera terminal 101 according to this embodiment. Display screens 1511 to 1515 illustrated in Figure 15 show the display content 328 displayed on the display unit 211 of the camera terminal 101 from time t1501 to time t1505. Timestamps TIMESTAMP1 to TIMESTAMP5 are timestamps at time t1501 to t1505.

[0117] Similar to Figure 10, since the shooting terminal 101 has moved in the +x direction at time 1501 and later, the target frame video associated with the second timestamp information 327 indicating the timestamp TIMESTAMP1 indicating the second time point is different from the display frame video displayed on the display unit 211 at time t1504. Therefore, in step S1402, the video decoding unit 1202 decodes the target frame video stored in the storage unit 207, which is associated with the timestamp TIMESTAMP1 indicating the second time point. Then, in step S1403, the video playback unit 1203 plays back the decoded target frame video. Then, in step S1404, the generation unit 311 includes the instruction content in the display content 328 at time t1505 and superimposes the instruction content onto the played-back target frame video. As a result, the shooting terminal 101 can display the frame video 321 corresponding to the instruction from the instruction terminal 102 on the display unit 211, and appropriately display the instruction from the instruction terminal 102. Therefore, the communication system 100 can appropriately display instructions from the instruction terminal using the video captured by the camera terminal 101.

[0118] (Third embodiment) A third embodiment will be described with reference to Figures 16 to 19. In the drawings, the same or equivalent elements are denoted by the same reference numerals, and redundant explanations are omitted. Configurations and processes that have substantially the same function as the other embodiments are referred to by the same reference numerals and their explanations are omitted, while the differences from the other embodiments will be explained.

[0119] In this embodiment, if the shooting terminal 101 moves after the second time point and the indicated position is not within the displayed frame image, the generation unit 311 includes a still request in the display content 328 that requests the shooting terminal 101 to stop moving. For example, if the shooting terminal 101 moves after the second time point and the indicated position is not within the displayed frame image, the generation unit 311 in this embodiment causes the display unit 211 to display a message indicating a still request.

[0120] In this embodiment, if the camera capture terminal 101 moves after the second time point and the instruction position is not within the displayed frame image, the instruction response transmission unit 312 transmits instruction response information 329 indicating that instruction is not possible to the instruction terminal 102. Furthermore, if the camera capture terminal 101 comes to a stop after the stop request is displayed on the display unit 211, the instruction response transmission unit 312 transmits a re-instruction request to the camera capture terminal 101 requesting new instruction information 325.

[0121] In this embodiment, the instruction result output unit 609 outputs a message indicating that instruction is not possible when it receives instruction response information 329 from the shooting terminal 101 indicating that instruction is not possible.

[0122] Figure 16 is a flowchart showing an example of the operation of the shooting terminal 101 according to this embodiment. If the control unit 209 determines in step S806, as illustrated in Figure 8, that there is no indicated position within the displayed frame image, it starts the process of step S1601.

[0123] In step S1601, the generation unit 311 includes the stop request in the display content 328. That is, the generation unit 311 displays the stop request on the display unit 211. For example, the generation unit 311 displays the text "Please stop the terminal" on the display unit 504. Then, in step S1602, the instruction response transmission unit 312 transmits instruction response information 329 indicating that the instruction is not possible to the instruction terminal 102.

[0124] In step S1603, the movement determination unit 309 determines whether the imaging terminal 101 has come to a standstill. Specifically, the movement determination unit 309 determines whether the position and direction indicated by the newly acquired motion information 322 indicate that the imaging terminal 101 has come to a standstill. If the position and direction indicated by the newly acquired motion information 322 indicates that the imaging terminal 101 has moved, the movement determination unit 309 determines that the imaging terminal 101 is not come to a standstill. On the other hand, if the position and direction indicated by the newly acquired motion information 322 does not indicate that the imaging terminal 101 has moved, the movement determination unit 309 determines that the imaging terminal 101 has come to a standstill. The movement determination unit 309 may also determine that the imaging terminal 101 has come to a standstill if the position and direction indicated by the newly acquired motion information 322 does not indicate that the imaging terminal 101 has moved for a predetermined period of time.

[0125] If it is not determined in step S1603 that the imaging terminal 101 has come to a standstill, the control unit 209 returns to step S1603 and repeats the process. In other words, the movement determination unit 309 repeatedly determines whether or not the imaging terminal 101 has moved until the position and direction indicated by the motion information 322 indicates that the imaging terminal 101 has come to a standstill. On the other hand, if it is determined in step S1603 that the imaging terminal 101 has come to a standstill, the instruction response transmission unit 312 transmits instruction response information 329 indicating a request for re-instruction to the instruction terminal 102 in step S1604.

[0126] In step S1605, the instruction information receiving unit 308 determines whether or not it has received instruction information 325. If instruction information 325 has not been received in step S1605, the control unit 209 returns to step S1605. In other words, the control unit 209 includes the quiescent request in the display content 328 and repeats the process in S1605 until it receives instruction information 325. On the other hand, if instruction information 325 has been received in step S1605, the generation unit 311 removes the quiescent request from the display content 328 in step S1606. In other words, the generation unit 311 hides the quiescent request in the display unit 211. Then, the control unit 209 returns to step S801, as illustrated in Figure 8.

[0127] Figure 17 is a flowchart showing an example of the operation of the instruction terminal 102 according to this embodiment.

[0128] In step S909, illustrated in Figure 9, the instruction information transmission unit 607 transmits instruction information 325 to the imaging terminal 101 via the communication unit 501. In step S1701, the instruction response receiving unit 608 determines whether or not it has received instruction response information 329. If instruction response information 329 has not been received in step S1701, the control unit 508 returns to step S1701. On the other hand, if instruction response information 329 has been received in step S1701, the instruction result output unit 609 outputs instruction response information 329 in step S1702.

[0129] In step S1703, the instruction result output unit 609 determines whether the instruction response information 329 indicates that instruction is not possible. If the instruction response information 329 does not indicate that instruction is not possible in step S1703, the control unit 508 terminates the process of transmitting instruction information 325. On the other hand, if the instruction response information 329 indicates that instruction is not possible in step S1703, the instruction response receiving unit 608 determines in step S1704 whether or not it has received instruction response information 329 indicating a request for re-instruction.

[0130] If the instruction response information 329 indicating a request for re-instruction is not received in step S1704, the control unit 508 returns the process to step S1704. In other words, the instruction terminal 102 outputs a message indicating that it cannot be instructed until the instruction response information 329 indicating a request for re-instruction is received. Furthermore, the control unit 508 repeats the process of steps S902 to S904 illustrated in Figure 9 until the instruction response information 329 indicating a request for re-instruction is received. In other words, the instruction terminal 102 continues to output a message indicating that it cannot be instructed until the instruction response information 329 indicating a request for re-instruction is received, while receiving a new frame image from the capture terminal 101 and displaying it on the display unit 504.

[0131] On the other hand, if instruction response information 329 indicating a request for re-instruction is received in step S1704, the instruction result output unit 609 outputs a message indicating that instruction is possible in step S1705. For example, the instruction result output unit 609 displays characters or the like indicating that instruction is possible on the display unit 504. Alternatively, the instruction result output unit 609 may output an audio message indicating that instruction is possible to the audio output unit 502.

[0132] In step S1706, the instruction operation processing unit 604 determines whether or not it has received an instruction operation. The process in step S1706 is the same as the process in step S905 illustrated in Figure 9, so a detailed explanation is omitted.

[0133] If the instruction operation is not accepted in step S1706, the control unit 508 returns to step S1706. In other words, the control unit 508 repeats the process of step S1706 until the instruction operation is accepted. On the other hand, if it is determined that the instruction operation has been accepted in step S1706, the control unit 508 proceeds to step S906, which is illustrated in Figure 9.

[0134] Figure 18 shows an example of the display content 328 of the shooting terminal 101 according to this embodiment. Display screens 1811 to 1814 illustrated in Figure 18 show the display content 328 displayed on the display unit 211 of the shooting terminal 101 from time t1801 to time t1804. Timestamps TIMESTAMP1 to TIMESTAMP4 show the timestamps at time t1801 to time t1804. In addition, display screens 1821 to 1823 show the target frame video and instruction content information displayed on the display unit 504 of the instruction terminal 102.

[0135] At time point t1801 or later, the camera terminal 101 has moved in the +x direction. Furthermore, the display frame image shown at time point t1804 does not include the instruction position. Therefore, in step S1601, the generation unit 311 includes the still request in the display content 328, as exemplified in the display screen 1814, and displays the still request on the display unit 211. This allows the camera terminal 101 to request the operator to keep the camera terminal 101 still. Furthermore, as exemplified in the display screen 1823, the video generation unit 603 outputs a message to the display unit 504 indicating that instructions are not possible until a request for further instructions is received. This allows the instruction terminal 102 to show the instructioner that it is in a state where it cannot give instructions properly.

[0136] Figure 19, following Figure 18, shows an example of the display content 328 of the shooting terminal 101 according to this embodiment. Display screens 1911 to 1914 illustrated in Figure 19 show the display content displayed on the display unit 211 of the shooting terminal 101 from time t1901 to time t1904. Display screens 1921 to 1922 show the target frame video and instruction content information displayed on the display unit 504 of the instruction terminal 102. Timestamps TIMESTAMP 5 to TIMESTAMP 8 show the timestamps at time t1901 to time t1904.

[0137] If, at time t1901, the movement determination unit 309 determines that the shooting terminal 101 has come to a standstill, in step S1604, the instruction response transmission unit 312 transmits instruction response information 329 indicating a request for further instruction to the instruction terminal 102.

[0138] At time t1902, if the instruction terminal 102 receives instruction response information 329 indicating a request for further instruction, in step S1705, the instruction result output unit 609 displays the message "Instruction OK" on the display unit 504 at a time after time t1902, as illustrated in display screens 1921 and 1922. This allows the instruction terminal 102 to prompt the user when the instructions from the instruction terminal 102 can be appropriately displayed using the video captured by the camera terminal 101. Then, at time t1903, the instruction terminal 102 transmits instruction information 325 associated with the timestamp TIMESTAMP 5 to the camera terminal 101. For example, the instruction information 325 includes instruction content information 326 indicating "move left".

[0139] As described above, if the displayed frame image does not include the instruction position, the shooting terminal 101 displays a stationary request on the display unit 211 until it is determined that the shooting terminal 101 is stationary. The instruction terminal 102 then outputs a message to the display unit 504 indicating that instructions are not possible until it receives a request for further instructions. After receiving the request for further instructions, the instruction terminal 102 accepts the instruction operation. As a result, the shooting terminal 101 can generate display content 328 that includes the instruction position. Consequently, the shooting terminal 101 can appropriately display instructions from the instruction terminal 102 using the image captured by the shooting terminal 101.

[0140] This disclosure is not limited to the embodiments described above, and various modifications are possible within the scope of the claims. Embodiments obtained by appropriately combining the technical means disclosed in different embodiments are also included in the technical scope of the present invention. Furthermore, new technical features can be formed by combining the technical means disclosed in each embodiment. [Explanation of Symbols]

[0141] 100 Communication system, 101 Shooting terminal, 102 Instruction terminal, 103 Network, 201 Communication unit, 202 Audio output unit, 203 Audio input unit, 204 Touch panel unit, 205 Shooting unit, 206 Motion sensor, 207 Storage unit, 208 Memory, 209 Control unit, 211 Display unit, 212 Touch sensor unit, 213 Program storage area, 214 Motion information storage area, 301 Video acquisition unit, 302 Motion information acquisition unit, 303 Video encoding unit, 304 Timestamp information acquisition unit, 305 Recording unit, 306 Motion history acquisition unit, 307 Video transmission unit, 308 Instruction information reception unit, 309 Movement determination unit, 310 Instruction position calculation unit, 311 Generation unit, 312 Instruction response transmission unit, 321 Frame video, 322 Motion information, 323 1st timestamp information, 324 Motion information for movement determination, 325 Instruction information, 326 Instruction content information, 327 2nd timestamp information, 328 Display content, 329 Instruction response information, 501 Communication unit, 502 Audio output unit, 503 Audio input unit, 504 Display unit, 505 Operation unit, 506 Storage unit, 507 Memory, 508 Control unit, 511 Program storage area, 601 Video reception unit, 602 Video decoding unit, 603 Video generation unit, 604 Instruction operation processing unit, 605 Timestamp information acquisition unit, 606 Instruction information creation unit, 607 Instruction information transmission unit, 608 Instruction response reception unit, 609 Instruction result output unit, 611 Frame video, 612 Timestamp information, 1201 Recording unit, 1202 Video decoding unit, 1203 Video playback unit

Claims

1. The shooting unit captures multiple frames of video footage in chronological order, A recording unit records motion information indicating the movement of the shooting terminal at the first time, associated with a first timestamp information indicating the first time when each frame image among the plurality of frame images was acquired. An instruction information receiving unit receives instruction information which is associated with instruction content information indicating the instruction content for a target frame video among the plurality of frame videos, and second timestamp information indicating a second time point in time when the target frame video was acquired. A generation unit generates display content to be displayed on the display unit based on the instruction content information and motion information of a movement determination associated with the first timestamp information of the movement determination indicating a time after the second time point, Equipped with, The instruction content information includes instruction coordinate information indicating the coordinates of the instruction position within the target frame video, If the generating unit has moved the shooting terminal since the second time point, and the indicated position is not within the display frame image displayed on the display unit, the generating unit will include in the display content a movement request that requests the shooting terminal to move backward from the movement indicated by the motion information of the movement determination. Camera / shooting device.

2. A movement determination unit determines whether or not the shooting terminal has moved from the second time point onward, based on the motion information of the movement determination; The generation unit determines whether the indicated position exists within the display frame image based on the indicated coordinate information and the motion information for the movement determination, and generates the display content according to whether the shooting terminal has moved since the second time point and whether the indicated position exists within the display frame image. The imaging terminal according to claim 1.

3. If the generation unit includes the movement request in the display content and then the indicated position is located within a new display frame image displayed on the display unit, the generation unit will include the indicated content in the display content. The imaging terminal according to claim 2.

4. The system further includes a command position calculation unit that corrects the coordinates of the command position based on the motion information of the movement determination and calculates the corrected coordinates. If the shooting terminal moves after the second time point and the indicated position is within the displayed frame image, the generation unit includes the indicated content in the displayed content and superimposes the indicated content onto the corrected coordinates within the displayed frame image. The imaging terminal according to claim 2 or 3.

5. If the shooting terminal does not move after the second time point, the generation unit includes the instruction content in the display content and superimposes the instruction content onto the coordinates in the display frame image. The imaging terminal according to any one of claims 2 to 4.

6. The recording unit further records each of the frame images, If the shooting terminal moves after the second time point, the generation unit includes the target frame video read from the recording unit and played back in the display content, and superimposes the instruction content onto the coordinates within the target frame video. The imaging terminal according to claim 2 or 3.

7. A communication system comprising: a shooting unit that takes pictures in a time series and acquires multiple frame images; a recording unit that records motion information indicating the movement of the shooting terminal at a first time, associated with a first timestamp information indicating a first time when each of the multiple frame images was acquired; an instruction information receiving unit that receives instruction information associated with instruction content information indicating instruction content for a target frame image among the multiple frame images and a second timestamp information indicating a second time when the target frame image was acquired; and a generation unit that generates display content to be displayed on a display unit based on the instruction content information and motion information for movement determination associated with a first timestamp information for movement determination indicating a time after the second time, and an instruction terminal that can communicate with the shooting terminal, wherein The aforementioned shooting terminal is A video transmission unit that associates each frame video with the first timestamp information and transmits it to the instruction terminal. Furthermore, The aforementioned instruction terminal is Instruction information transmission unit that transmits the instruction information to the camera terminal. Equipped with, The instruction content information includes instruction coordinate information indicating the coordinates of the instruction position within the target frame video, If the generating unit has moved since the second time point and the indicated position is not within the display frame image displayed on the display unit, it will include a still request in the display content that requests the shooting terminal to stop moving. Communication system.

8. A communication system comprising: a shooting unit that takes pictures in a time series and acquires multiple frame images; a recording unit that records motion information indicating the movement of the shooting terminal at a first time, associated with a first timestamp information indicating a first time when each of the multiple frame images was acquired; an instruction information receiving unit that receives instruction information associated with instruction content information indicating instruction content for a target frame image among the multiple frame images and a second timestamp information indicating a second time when the target frame image was acquired; and a generation unit that generates display content to be displayed on a display unit based on the instruction content information and motion information for movement determination associated with a first timestamp information for movement determination indicating a time after the second time, and an instruction terminal that can communicate with the shooting terminal, wherein The instruction content information includes instruction coordinate information indicating the coordinates of the instruction position within the target frame video, The aforementioned shooting terminal is A video transmission unit that associates each frame video with the first timestamp information and transmits it to the instruction terminal, If the shooting terminal moves after the second time point and the indicated position is not within the display frame image displayed on the display unit, the instruction response transmission unit transmits instruction response information indicating that instruction is not possible to the instruction terminal. Furthermore, The aforementioned instruction terminal is An instruction information transmission unit that transmits the instruction information to the camera terminal, The instruction result output unit outputs a message indicating that the instruction is not possible when it receives the instruction response information from the aforementioned shooting terminal, A communication system equipped with [the following features].

9. If the shooting terminal moves after the second time point and the indicated position is not within the displayed frame image, the generation unit includes a still request in the display content requesting the shooting terminal to stop moving. If the camera terminal comes to a standstill after the standstill request is displayed on the display unit, the instruction response transmission unit transmits a re-instruction request to the instruction terminal requesting new instruction information. The communication system according to claim 8.

10. The process involves taking photos in chronological order to acquire multiple frames of video, A step of recording motion information indicating the movement of the shooting terminal at the first time, associated with a first timestamp information indicating the first time when each frame image among the plurality of frame images was acquired, A step of receiving instruction information which is associated with instruction content information indicating the content of an instruction for a target frame video among the plurality of frame videos, and a second timestamp information indicating a second time point in time when the target frame video was acquired. A step of generating display content to be displayed on the display unit based on the instruction content information and motion information of a movement determination associated with the first timestamp information of the movement determination indicating a time after the second time point, Includes, The instruction content information includes instruction coordinate information indicating the coordinates of the instruction position within the target frame video, In the process of generating the display content, if the shooting terminal moves after the second time point, and the indicated position is not within the display frame image displayed on the display unit, the display content should include a movement request that requests the shooting terminal to move backward from the movement indicated by the motion information of the movement determination. Work instruction method.

11. On the computer, A function that captures multiple frames of video by shooting in chronological order, A function to record motion information indicating the movement of the shooting terminal at the first time, associated with a first timestamp information indicating the first time when each frame video among the plurality of frame videos was acquired, A function for receiving instruction information that associates instruction content information indicating the content of an instruction for a target frame video among the plurality of frame videos, and a second timestamp information indicating a second time point in time when the target frame video was acquired. A function to generate display content to be displayed on the display unit based on the instruction content information and motion information of the movement determination associated with the first timestamp information of the movement determination indicating a time after the second time point, Make it run, The instruction content information includes instruction coordinate information indicating the coordinates of the instruction position within the target frame video, In the function that generates the display content, if the shooting terminal moves after the second time point, and the indicated position is not within the display frame image displayed on the display unit, the display content includes a movement request that requests the shooting terminal to move backward from the movement indicated by the motion information of the movement determination. program.

Images