Video sharing system and video sharing method

Abstract

According to the present invention, a photographing terminal sets the position of the photographing terminal in a photographing reference state, which is set in a real space, as a photographing reference point and a direction from the photographing reference point toward the front of the photographing terminal as a photographing reference direction, acquires a change amount of the position and orientation of the photographing terminal from the photographing reference state, and transmits the change amount to a display terminal together with photographed video information as first correction information. The display terminal sets the position of the display terminal in a display reference state, which is set in the real space, as a display reference point and a direction from the display reference point toward the front of the display terminal as a display reference direction. The display terminal acquires a change amount of the position and orientation of the display terminal from the display reference state and uses this change amount and the first correction information to correct the photographed video information such that a positional relationship with respect to an object to be photographed with the photographing reference point and the photographing reference direction used as references and an apparent positional relationship with respect to a video of the object to be photographed with the display reference point and the display reference direction used as references coincide with each other, and displays the corrected photographed video information on the display terminal.

Description

[Technical Field]

[0001] The present invention relates to a video sharing system and a video sharing method. [Background technology]

[0002] Patent Document 1 states that "a mounted device includes a camera that photographs the work site over a wider area than the worker's field of view, a sensor that acquires the worker's gaze as sensor information, and a gaze movement detection unit that uses the sensor information to detect the worker's gaze movement as gaze movement information, and transmits the camera's captured image data and gaze movement information to an operation instruction device via a network. The operation instruction device includes a display setting unit that extracts a first image from the camera's captured image covering an area wider than the worker's field of view, corrects the first image using the gaze movement information, and displays the corrected first image on a display, and an instruction image generation unit that generates an instruction image for the worker, and transmits the instruction image to the mounted device via a network. (Abstract Excerpt)" [Prior art documents] [Patent Documents]

[0003] [Patent Document 1] International Publication No. 2020 / 178960 [Overview of the project] [Problems that the invention aims to solve]

[0004] In conventional technology, a large, fixed display device is used by the worker to view a wide area of ​​the work site without being affected by the movement of the camera terminal that photographs the work site, which presents the challenge of requiring a large and cumbersome device.

[0005] The present invention has been made in view of the above circumstances, and aims to provide a video sharing system and video sharing method that enables display unaffected by the movement of the camera, by linking a camera operating with a camera operator that captures work conditions, etc., with a display terminal that receives and displays captured video transmitted from the camera. [Means for solving the problem]

[0006] To solve the above problems, the present invention has the configuration described in the claims. To give an example, the present invention provides a video sharing system configured by communicating a shooting terminal and a display terminal, wherein the shooting terminal comprises a camera, a first position and orientation sensor for detecting the position and orientation of the shooting terminal, a first processor, and a first communication device for sending and receiving data with the display terminal, the display terminal comprises a display, a second position and orientation sensor for detecting the position and orientation of the display terminal, a second processor, and a second communication device for sending and receiving data with the shooting terminal, the first processor of the shooting terminal sets the position of the shooting terminal as a shooting reference point in a shooting reference state set in the real space in which the shooting terminal exists, sets the direction in which the front of the shooting terminal faces from the shooting reference point as the shooting reference direction, transmits the shooting reference point and the shooting reference direction to the display terminal, and based on the sensor information of the first position and orientation sensor, records the change in the position of the shooting terminal relative to the shooting reference point when the camera photographs an object, and the shooting relative to the shooting reference state The amount of change in the attitude of the shadow terminal is acquired, and the change in the position of the shooting terminal and the amount of change in the attitude of the shooting terminal are transmitted to the display terminal as first correction information along with the captured video information generated when the camera photographs the object to be photographed. The second processor of the display terminal sets the position of the display terminal as the display reference point and the direction in which the front of the display terminal faces from the display reference point as the display reference direction, based on the sensor information of the second position and attitude sensor, the change in the position of the display terminal relative to the display reference point and the amount of change in the attitude of the display terminal relative to the display reference state are acquired as second correction information, and the captured video information is corrected using the first correction information and the second correction information so that the positional relationship of the object to be photographed relative to the shooting reference point and the shooting reference direction matches the apparent positional relationship of the image of the object to be photographed relative to the display reference point and the display reference direction, and the corrected captured video information is displayed on the display. [Effects of the Invention]

[0007] According to the present invention, it is possible to provide a video sharing system and method that enables display unaffected by the movement of the camera, by linking a shooting terminal operated by a photographer of work conditions, etc., with a display terminal that receives and displays the captured images transmitted from the shooting terminal, without using a large, fixed display device. Other objectives, configurations, and effects will be clarified in the following embodiments. [Brief explanation of the drawing]

[0008] [Figure 1] This is a schematic diagram of the video sharing system according to the first embodiment. [Figure 2] This is a hardware configuration diagram of the camera and display terminals. [Figure 3] This is a hardware configuration diagram of a smartphone. [Figure 4A] This is a functional block diagram of the video sharing system, specifically the shooting terminal 1A portion. [Figure 4B] This is a functional block diagram of the display terminal 1B portion of the video sharing system. [Figure 5A] This diagram shows an example of the relationship between the shooting reference point, shooting reference direction, and the position of the object being photographed in the shooting space where the shooting terminal is located. [Figure 5B] This figure shows the captured video in the positional relationship shown in Figure 5A. [Figure 6A] This figure shows an example of the positional relationship between the display reference point, the display reference direction, and the image of the object being captured in the display space where the display terminal is located. [Figure 6B] This figure shows the displayed image in the positional relationship shown in Figure 6A. [Figure 7A] This figure shows an example of the positional relationship when the position and orientation of the camera terminal change, based on the positional relationship of the camera terminal shown in Figure 5A. [Figure 7B] This figure shows the captured video in the positional relationship shown in Figure 7A. [Figure 7C] This diagram shows the image displayed on the display terminal when the shooting terminal is in the state shown in Figure 7A and the display terminal is in the position shown in Figure 6A (reference position). [Figure 8A]This figure shows an example of the positional relationship when the position and orientation of the display terminal change, based on the positional relationship of the display terminal shown in Figure 6A. [Figure 8B] This figure shows the displayed image in the positional relationship shown in Figure 8A. [Figure 9] This diagram shows the parameters used in the explanation of the image correction method. [Figure 10] This diagram shows the relative relationship between the captured image and the displayed image in relation to the reference image. [Figure 11] This is an explanatory diagram of the parameters used in the real space where the shooting terminal exists. [Figure 12] This is an explanatory diagram of each parameter used in the real space where the display terminal exists. [Figure 13] This is a flowchart showing the processing flow of the video sharing system according to the first embodiment. [Figure 14] This figure shows an example of a displayed image in the second embodiment. [Figure 15] This is a flowchart showing the processing flow according to the second embodiment. [Modes for carrying out the invention]

[0009] This invention is expected to improve the visibility of images of workers performing tasks on-site when viewed by support personnel providing rear-echelon assistance. Therefore, this invention is expected to contribute to improving technology for labor-intensive industries that require work support and rear-echelon assistance, and thus to achieving Sustainable Development Goal 8.2 of the United Nations (Increase economic productivity through diversification, technological advancement and innovation, particularly in industries that enhance the value of goods and services and in labor-intensive industries).

[0010] Hereinafter, embodiments of the present invention will be described with reference to the drawings. In all the drawings used to describe the embodiments, the same reference numerals are used for identical components, and repeated descriptions of them will be omitted.

[0011] [First Embodiment] Figure 1 is a schematic diagram of the video sharing system according to the first embodiment. The shooting terminal 1A, which is a portable information terminal worn by user 2A, transmits video information of the object to be photographed 7 to the display terminal 1B, which is a portable information terminal worn by user 2B. At the same time, information that shows the relationship between the video at the time of shooting, the shooting reference point, and the shooting reference direction is also transmitted. On the display terminal 1B, the image of the object to be photographed 7 is displayed, with the positional relationship corrected so that the object to be photographed 7 is located in the space in which the display terminal 1B is located. The means of communication between the shooting terminal 1A and the display terminal 1B may be via the communication network 4 through access points 3A and 3B, or it may be direct communication between the shooting terminal 1A and the display terminal 1B.

[0012] In this embodiment, HMDs (Head Mounted Displays) are used as the shooting terminal 1A and the display terminal 1B.

[0013] Users 2A and 2B may each operate smartphones 5A and 5B, and smartphones 5A and 5B may also be connected to access points 3A and 3B and the communication network 4.

[0014] Figure 2 is a hardware configuration diagram of the imaging terminal and the display terminal. Since imaging terminal 1A and display terminal 1B have the same configuration, in the following description, components common to imaging terminal 1A and display terminal 1B will be distinguished by the subscripts A and B in their reference numerals.

[0015] The camera terminal 1A and the display terminal 1B are connected to each other via buses 140A and 140B, which connect the respective components. The camera terminal 1A and the display terminal 1B consist of rear cameras 111A and 111B, distance sensors 113A and 113B, position and orientation sensors 135A and 135B, geomagnetic sensors 117A and 117B, positioning sensors 118A and 118B, RTCs (real-time clocks) 114A and 114B, displays 119A and 119B, microphones 121A and 121B, speakers 122A and 122B, processors 125A and 125B, memory 128A and 128B, and network communication devices 120A and 120B. The network communication devices 120A and 120B are connected to antennas 123A and 123B, which transmit and receive network communication signals. The displays 119A and 119B and the speakers 122A and 122B are output devices, while the microphones 121A and 121B are input devices.

[0016] Position and orientation sensors 135A and 135B are used to measure the position and orientation of the shooting terminal 1A and the display terminal 1B. Here, orientation refers to the three-dimensional rotational position of the shooting terminal 1A and the display terminal 1B in the real space in which they exist. More details will be described later. Specific configuration examples of position and orientation sensors 135A and 135B include position and orientation cameras 136A and 136B (Position AND Posture cameras: hereafter abbreviated as "PP cameras"), position and orientation distance sensors 137A and 137B (Position AND Posture distance sensors: hereafter abbreviated as PP distance sensors), acceleration sensors 115A and 115B, and gyro sensors 116A and 116B. PP cameras 136A and 136B may be the same as the rear cameras 111A and 111B. Also, PP distance sensors 137A and 137B may be the same as the distance sensors 113A and 113B.

[0017] To measure position and orientation using PP cameras 136A and 136B and PP distance measuring sensors 137A and 137B, feature points existing in the external environment are utilized. Multiple feature points can be used; for example, they may be feature points of the object being photographed 7, feature points indoors, or, outdoors, feature points of buildings. As preparation, the relative positional relationship of the feature points is measured by measuring their direction and distance. This preparation can be omitted if the coordinate values ​​of the feature points in the external coordinate system are known. Then, by measuring the change in the direction and distance of the feature points as seen from the shooting terminal 1A and the display terminal 1B, the change in position and orientation of the shooting terminal 1A and the display terminal 1B is measured. When measuring distance using the stereo camera method, only PP cameras 136A and 136B may be used. Furthermore, if the external coordinate system values ​​of the feature points are known, and the relationship between the external coordinate system and the vertical direction is known, the relative relationship between the housing orientation of the shooting terminal 1A and the display terminal 1B and the vertical direction can be determined by attitude measurement using PP cameras 136A and 136B and PP distance measuring sensors 137A and 137B.

[0018] Position and orientation are measured using acceleration sensors 115A and 115B and gyro sensors 116A and 116B by integrating acceleration and angular acceleration. This allows for the determination of displacement from a reference position and reference orientation. Furthermore, the relative vertical relationship between the housing orientation of the camera terminal 1A and display terminal 1B can also be determined by measuring the gravitational acceleration vector using acceleration sensors 115A and 115B.

[0019] The position and orientation of the shooting terminal 1A and the display terminal 1B may be measured by referring to both the measurements obtained using PP cameras 136A and 136B and PP distance sensors 137A and 137B, and the measurements obtained using acceleration sensors 115A and 115B and gyro sensors 116A and 116B, or by using the measurements obtained from other types of sensors.

[0020] Positioning sensors 218A and 218B are used to measure the indoor and outdoor positions of the camera terminal 1A and display terminal 1B. They are used when searching for library images related to the object being photographed 7. Specific examples of indoor positioning include using beacon signals or location markers. Outdoor positioning includes, for example, GPS (Global Positioning System) receivers that utilize signals from satellites.

[0021] Network communication devices 120A and 120B are communication interfaces that perform wireless communication between at least the camera terminal 1A and display terminal 1B and access points 3A and 3B via short-range wireless communication, wireless LAN, or base station communication. They include communication processing circuits corresponding to various predetermined communication interfaces and are connected to antennas 123A and 123B. Network communication devices 120A and 120B transmit and receive biometric information, image data, control signals, etc. Short-range wireless communication can be performed using Bluetooth®, IrDA (Infrared Data Association®), Zigbee®, HomeRF (Home Radio Frequency®), or wireless LAN such as Wi-Fi®. For base station communication, long-range wireless communication such as LTE (LongTerm Evolution®), LTE-Advanced, Mobile WiMAX (Worldwide Interoperability for Microwave Access®), or WiMAX2 can be used.

[0022] Displays 119A and 119B are positioned in front of both eyes of users 2A and 2B, who are wearing the shooting terminal 1A and display terminal 1B. Processors 125A and 125B control the operation of the shooting terminal 1A and display terminal 1B. In particular, they correct the video of real-world information captured by the rear cameras 111A and 111B and the captured video received from access points 3A and 3B and display them on displays 119A and 119B.

[0023] Processors 125A and 125B are composed of, for example, a CPU.

[0024] Memory 128A and 128B consist of flash memory and non-volatile memory. Memory 128A and 128B also store programs 126A and 126B such as the OS (Operating System) and operation control applications, as well as data 127A and 127B used by processors 125A and 125B.

[0025] Processors 125A and 125B load programs 126A and 126B into memory 128A and 128B and execute them, and read data 127A and 127B as needed and use them in the execution of programs 126A and 126B.

[0026] Figure 3 is a hardware configuration diagram of a smartphone. Components with the same names as those in an HMD (Head-Mounted Display) have essentially the same function, so detailed explanations are omitted unless there are significant differences.

[0027] Smartphones 5A and 5B consist of rear cameras 211A and 211B, front cameras 212A and 212B, distance sensors 213A and 213B, position and orientation sensors 235A and 235B, geomagnetic sensors 217A and 217B, positioning sensors 218A and 218B, RTC 214A and 214B, displays 219A and 219B, microphones 221A and 221B, speakers 222A and 222B, processors 225A and 225B, memory 228A and 228B, telephone network communication devices 231A and 231B, and network communication devices 220A and 220B, all connected to each other via buses 240A and 240B that connect each component. Network communication devices 220A and 220B are connected to antennas 223A and 223B that transmit and receive network communication signals.

[0028] Specific configuration examples for position and orientation sensors 235A and 235B include PP cameras 236A and 236B, PP distance sensors 237A and 237B, acceleration sensors 215A and 215B, and gyro sensors 216A and 216B, similar to the case of HMDs. PP cameras 236A and 236B may be the same as the rear cameras 211A and 211B. Also, PP distance sensors 237A and 237B may be the same as the distance sensors 213A and 213B.

[0029] Memory 228A and 228B store programs 226A and 226B and data 227A and 227B.

[0030] The displays 219A and 219B have touch panels 230A and 230B stacked on top of them, serving as input interfaces.

[0031] Figure 4A is a functional block diagram of the shooting terminal 1A portion of the video sharing system. Figure 4B is a functional block diagram of the display terminal 1B portion of the video sharing system.

[0032] As shown in Figure 4A, the shooting terminal 1A includes a position and attitude measurement unit 301 that acquires sensor information from a position and attitude sensor and a distance measurement sensor and generates first correction information to be transmitted to the display terminal 1B, an image processing unit 302 that acquires and processes captured image information from a camera, a transmission information generation unit 303 that transmits the first correction information and the captured image information, and a communication control unit 304 that transmits and receives data with the display terminal 1B. The first correction information will be described later.

[0033] As shown in Figure 4B, the display terminal 1B includes a communication control unit 311 that transmits and receives data with the shooting terminal 1A, a position and attitude measurement unit 312 that acquires sensor information from a position and attitude sensor mounted on the display terminal 1B and generates second correction information indicating changes in the position and attitude of the display terminal 1B, a video information storage unit 313 that stores the received captured video information, a display video generation unit 314 that performs correction processing of the captured video using the first correction information and the second correction information, and a display control unit 315 that controls the display on the display. The second correction information will be described later.

[0034] The video sharing system 100 according to this embodiment is characterized by the display method when the video information generated by the shooting terminal 1A shooting the object 7 is transmitted to and displayed on the display terminal 1B. This characteristic will be explained with reference to Figures 5A, 5B, 6A, and 6B. In the following, the combination of HMDs of the shooting terminal 1A and the display terminal 1B will be described, but each HMD may be replaced with smartphones 5A and 5B. A combination of an HMD and a smartphone is also acceptable. Furthermore, other types of portable information terminals with equivalent functionality may also be used.

[0035] First, Figure 5A shows an example of the positional relationship between the shooting reference point 410, the shooting reference direction 420, and the object to be photographed 7 in the shooting space where the shooting terminal 1A is located. This positional relationship will be called the shooting reference state 400. Also, the point where the straight line extended from the shooting reference point 410 in the shooting reference direction 420 intersects with the object to be photographed 7 will be called the object center point 470.

[0036] The shooting reference state 400 is set, for example, as the shooting reference point 410, which is the position of the shooting terminal 411 when user 2A is in a position that is convenient for them to work in. Furthermore, the shooting reference direction 420 is set as the direction in which the front direction 430, which is the optical axis direction of the shooting terminal 1A's rear camera 111A, is mainly pointed during work. In addition, as the work progresses, the position that is convenient for working and the direction in which the front direction 430 of the shooting terminal 1A is mainly pointed may change. In that case, the shooting reference state 400 is updated based on instructions from user 2A.

[0037] Figure 5B shows the captured image in the shooting reference state 400 of Figure 5A. The image shows the captured image of shooting terminal 1A when the shooting terminal 1A and the object to be shot 7 are in the positional relationship shown in Figure 5A, and furthermore, shooting terminal 1A is at the shooting reference point 410, and the front direction 430 of shooting terminal 1A is facing the shooting reference direction 420. In this positional relationship, the object center point 470 on the object to be shot 7 is located at the center of the captured image 450, which is the center of the captured image 440.

[0038] The distance between the shooting reference point 410 and the object center point 470 in the shooting reference state 400 is defined as the object reference distance 415. The object reference distance 415 is measured by the distance measuring sensor 113A or the position and orientation sensor 135A. The shooting reference point 410 and the shooting reference direction 420 are measured by the position and orientation sensor 135A. The shooting terminal 1A transmits information on the shooting reference point 410, the shooting reference direction 420, and the object reference distance 415 to the display terminal 1B at least each time the shooting reference state 400 is set or updated.

[0039] Figure 6A is a diagram illustrating an example of the positional relationship between the display reference point 510, the display reference direction 520, and the captured object image 8 in the display space where the display terminal 1B exists. This state will be referred to as the display reference state 500. When the captured object 7 is in the positional relationship shown in Figure 5A (shooting reference state 400) in the shooting space, the positional relationship of the captured object image 8 in the display space where the display terminal 1B exists will be as shown in Figure 6A. Here, the position of the captured object image 8 is the position where user 2B perceives the captured object image 8 as being there in the display space, and is the display position as 3D in 3D video. In other words, the captured image 440 is displayed on the display terminal 1B so that the position of the captured object 7 relative to the shooting reference point 410 and the shooting reference direction 420 is the same as the apparent position of the captured object image 8 relative to the display reference point 510 and the display reference direction 520. In other words, the display terminal 1B displays the image of the object being photographed 8 in such a way that it appears to user 2B as if they were viewing the object 7 from the shooting reference point 410. Furthermore, representing the occlusion relationships with other objects in the display space would result in a more realistic image. In this specification, unless otherwise specified, the expression "image of the object being photographed 8" means an image that appears to be located in the display space to user 2B, as described above.

[0040] The display reference state 500 is set, for example, to the display terminal position 511 when user 2B is in a position where the displayed image is easily viewable, and this is set as the display reference point 510. Furthermore, the display reference direction 520 is set to the direction in which the display terminal 1B is primarily directed, the front direction 530, while viewing the image. However, as a standard setting method, the elevation angle of the display reference direction 520 with respect to the horizontal plane is matched to the elevation angle of the shooting reference direction 420. This is so that the user can view the state of the shooting site with the same sense as user 2A. Here, the front direction 530 of the display terminal 1B refers to the direction extending perpendicularly from the center of the display surface 119B of the display terminal 1B or the surface on which user 2B perceives that a display is being performed. If the display area is not a shape where the center can be obviously defined, such as a rectangle, the center of the display surface is defined as appropriate. Also, during image sharing, it may be desirable to change the viewing position or the direction in which the front direction 530 of the display terminal 1B is primarily directed. In that case, the shooting reference state 400 is updated based on instructions from user 2B.

[0041] Figure 6B is a diagram showing the display image in the display reference state 500 of Figure 6A. As shown in Figure 6B, if the image of the object being photographed 8 is in the positional relationship of Figure 6A in the display space, and the display terminal 1B is at the display reference point 510, and the front direction 530 of the display terminal 1B is facing the display reference direction 520, then the display image 540 of the display terminal 1B will be as shown in Figure 6B. In this case, the object center point 570 on the image of the object being photographed 8 is located at the display image center 550, which is the center of the display image 540.

[0042] The above is an example of captured video 440 and displayed video 540 when the shooting terminal 1A and display terminal 1B are in a standard relative position. The following explains the case when the general relative positions are assumed.

[0043] First, we will explain the case where the position and orientation of the imaging terminal 1A change, referring to Figures 7A and 7B. Figure 7A is a diagram showing an example of the positional relationship when the position and orientation of the imaging terminal 1A change, based on the relationship between the position and orientation of the imaging terminal 1A shown in Figure 5A.

[0044] First, the orientation of the terminal will be explained using the shooting terminal 1A as an example. In this invention, the three-dimensional rotational position of a portable information terminal in real space is called the orientation. In the case of the shooting terminal 1A, the reference for the rotational position is the shooting reference state 400. The amount of three-dimensional rotation from the shooting reference state 400 when the shooting terminal 1A is in the general state shown in Figure 7A is the amount of orientation change of the shooting terminal 1A. This amount of three-dimensional rotation is measured by the position orientation sensor 135A. In the following explanation, this amount of three-dimensional rotation is decomposed into the amount of change from the shooting reference direction 420 in the front direction 430 of the shooting terminal and the shooting rotation angle 435, which is the rotation angle from the shooting reference state 400 around the optical axis of the rear camera 111A. Here, the optical axis direction of the rear camera 111A and the front direction 430 of the shooting terminal 1A are aligned in the same direction.

[0045] The same applies to the display terminal 1B, in which case the reference for the rotational position is the display reference state 500. The amount of three-dimensional rotation from the display reference state 500 when the display terminal 1B is in the general state shown in Figure 8A is the amount of attitude change of the display terminal 1B. This amount of three-dimensional rotation is measured by the position and attitude sensor 135B. This amount of attitude change is then decomposed into the amount of change from the display reference direction 520 in the front direction 530 of the display terminal 1B and the display rotation angle 535, which is the rotation angle from the display reference state 500 around the vertical axis of the display surface of the display 119B. Here, the direction of the vertical axis of the display surface of the display 119B and the front direction 530 of the display terminal 1B are aligned in the same direction.

[0046] Now, the position 411 and the front direction 430 of the imaging terminal 1A shown in Figure 7A are both deviated from the imaging reference point 410 and imaging reference direction 420 set in the imaging reference state 400 shown in Figure 5A. The point where the line extending the front direction 430 of the imaging terminal 1A intersects with the object to be photographed 7 will be called the front imaging point 480.

[0047] Figure 7B shows the captured image 441 when the device is in the position shown in Figure 7A. In this case, user 2A shifts their gaze and inspects the meter 600 in the lower left. In the captured image 441, the shooting front point 480 is located at the center of the captured image 441, and the object center point 470 is located in the upper right. Here, in order to correct the displayed image 541 (see Figure 8B), the position and orientation sensor 135A measures the change in the position of the shooting terminal position 411 from the shooting reference point 410, the change in the front direction 430 of the shooting terminal 1A from the shooting reference direction 420, and the shooting rotation angle 435, which is the rotation angle of the shooting terminal 1A's out-camera 111A around the optical axis from the shooting reference state, as correction information.

[0048] The change in terminal position from the shooting reference state 400, the change in the terminal's front direction, and the rotation angle around the camera's optical axis, measured by the shooting terminal 1A, are one of the correction pieces of information used to correct the displayed image 541. Hereinafter, the change in terminal position from the shooting reference state 400, the change in the terminal's front direction, and the rotation angle around the camera's optical axis, measured by the shooting terminal 1A, will be referred to as the first correction information. Here, the change in the terminal's front direction and the rotation angle around the camera's optical axis are the specific details of the change in attitude. On the other hand, as will be described later, the change in terminal position from the display reference state 500, the change in the terminal's front direction, and the rotation angle around the vertical axis of the display surface, measured by the display terminal 1B, will be referred to as the second correction information. In the display terminal, the change in the terminal's front direction and the rotation angle around the vertical axis of the display surface are the specific details of the change in attitude.

[0049] This first correction information is then transmitted to the display terminal 1B along with the captured video information. Furthermore, the terminal's front direction 430 includes vertical information. This also makes it possible to adjust the elevation angle.

[0050] Figure 7C shows the image on display terminal 1B when the shooting terminal 1A is in the state shown in Figure 7A and the display terminal 1B is in the state shown in Figure 6A (display reference state 500). When the position of shooting terminal 1A changes from Figure 5A to Figure 7A, if the position of display terminal 1B does not change, the object center point 570 on display terminal 1B will be located at the center 550 of the displayed image. In other words, the captured image information is corrected to offset the change in terminal position, the change in the terminal's front direction, and the rotation angle around the camera's optical axis, as indicated in the first correction information, and displayed on display terminal 1B.

[0051] However, in Figure 7C, the image of the object being photographed 8 transmitted from the shooting terminal 1A is displayed only within the shooting image range 490, and there is a missing area outside the shooting image range 490 where no video data is displayed. The missing area is, for example, a screen area that is solid black. In Figure 7C, the missing area is illustrated with diagonal lines.

[0052] In Figure 7C, the screen including the missing area is displayed on the display 119B, whereas in Figure 6B, the entire area of ​​the display terminal 1B's display 119B is the same as the captured video range 490, so no missing area occurs on the display 119B, which is the difference between the two. Also, in Figure 7C, as in Figure 6B, the object's center point 570 is located at the center 550 of the displayed video.

[0053] Figure 8A shows an example of the positional relationship when the position and orientation of the display terminal 1B change from the position and orientation relationship of the display terminal 1B shown in Figure 6A (display reference state 500). Assume that the position and orientation of the display terminal 1B are as shown in Figure 8A.

[0054] Figure 8B is a diagram showing the displayed image in the positional relationship shown in Figure 8A. Here, the center of the displayed image 550 is the point located at the center of the displayed image 541 on the image of the object being captured 8. Furthermore, the center of the displayed image 550 is the point that would coincide with the image of the object being captured 8 if the display terminal 1B were extended in the forward direction, assuming that the user 2B was in a position in the space where the display terminal 1B exists and perceived the image of the object being captured 8 as if it were there. The second correction information, which is the change in the display terminal position 511 of the display terminal 1B from the display reference point 510, the change in the forward direction 530 from the display reference direction 520, and the display rotation angle 535, which is the rotation angle from the horizontal state around the vertical axis of the display surface, is measured by the position and orientation sensor 135B inside the display terminal 1B.

[0055] Based on the second correction information and the first correction information sent from the shooting terminal 1A, the display terminal 1B corrects the position and magnification of the captured image information and displays it, as shown in Figure 8B, so that the display image 541 appears as if the display terminal 1B is viewing the object 7 from a position corresponding to the display terminal position 511 in the shooting space.

[0056] If the shooting front point 480 and the display front point 580 are misaligned, or depending on the magnification correction, it may not be possible to cover the entire display area of ​​the display terminal 1B's display 119B with the captured image range 490. Therefore, as a modification, a configuration may be used in which the shooting terminal 1A is equipped with a wide-angle camera to narrow the missing area.

[0057] Furthermore, the display terminal 1B may enlarge or reduce the image of the corrected video as appropriate.

[0058] Note that while the marks indicating the object's center point, the shooting center point, and the display center point are drawn in the diagram for illustrative purposes, you may actually display such marks. Displaying these marks makes it easier to understand the status of the image processing.

[0059] Below, an example of an image correction method will be explained with reference to Figures 9 to 12. Figure 9 is a diagram showing the parameters used in the explanation of the image correction method. Figure 10 is a diagram showing the relative relationship between the captured image and the displayed image with respect to the reference image. Figure 11 is an explanatory diagram of the parameters used in the real space where the shooting terminal exists. Figure 12 is an explanatory diagram of each parameter used in the real space where the display terminal exists.

[0060] In the video sharing system and method of this embodiment, video correction is performed in the following two steps. First correction: Restores the position of the object being filmed in the image to its original state, which was caused by changes in the position and orientation of the filming terminal 1A. Second correction: This reflects the change in the position of the object being filmed on the image due to changes in the position and orientation of the display terminal 1B.

[0061] First, in this invention, conceptually, a reference image is considered to be an intermediate image between the image captured by the camera, i.e., the shooting terminal 1A, and the display image shown on the display 119B, i.e., the display terminal 1B. The reference image is an image of an external object when the shooting terminal 1A is placed in the shooting reference state 400. That is, it is the image shown in Figure 5B.

[0062] Here, let me provide some supplementary explanation regarding the terminal's reference state. In the shooting reference state 400 for shooting terminal 1A and the display reference state 500 for display terminal 1B, it is assumed that each terminal is in a horizontal state. When setting the reference state to a state where the terminal is in a certain position and facing a certain direction, the elevation angle remains the same, and the state in which the casing is in a horizontal state is used as the reference state. A horizontal state of the casing means, for example, in the case of an HMD, that both eyes of the user wearing the HMD are aligned horizontally, and in the case of a smartphone, that the short side or long side of the casing is horizontal. Normally, the imaging range and video display range are rectangular, so a horizontal state means that one side of that rectangle is horizontal.

[0063] Assuming the reference state is set as described above, a terminal coordinate system is established for each image, with the center of the image as the origin, the image direction that is horizontal in the reference state as the X-axis, and the direction perpendicular to the X-axis as the Y-axis (Figure 9). Furthermore, extending the definition to terminals where the image range is not rectangular, the horizontal direction of the image is defined as the direction on the image that corresponds to the horizontal direction perpendicular to the front direction of the terminal in the outside world, when the terminal is in a horizontal position at the time of use.

[0064] Now, the position of point P on the image is given by coordinate values ​​(X P , Y P This is expressed as L0 in Figure 11. The field of view is the angle that includes the entire width of the image from the terminal position of the shooting terminal 1A. Figure 9 shows the horizontal field of view and the vertical field of view. The field of view may differ between the captured image and the displayed image. Here, L is the distance between the object to be shot 7 or the image of the object to be shot 8 and the terminal position. In the case of the reference image, L is the distance between the shooting reference point 410 and the object center point 470, and is also the distance between the display reference point 510 and the object center point 570, which is the object reference distance 415 (Figures 5A and 6A). This is expressed as L0 in Figure 11. In the case of the captured image, L is the distance between the shooting terminal position 411 and the shooting front point 480 (Figure 7A). This is expressed as L0 in Figure 11. C This is denoted as follows. In the case of a displayed image, L is the distance between the display terminal position 511 and the display front point 580 (Figure 8A). In Figure 12, this is represented as L. D This is how it is expressed. If this L differs between images, the image range and the apparent size of the object 7 will differ at the position of the object 7 or the image 8 of the object, according to the ratio of L. Here, we assume that the apparent size changes inversely proportional to L. The position of the image 8 of the object refers to the position at which the display terminal 1B displays the object 7 captured by the shooting terminal 1A as if it were there.

[0065] In FIG. 10, the position of the imaging object 7 is common to each video. Depending on the position and orientation of the imaging terminal 1A, the captured video will be different from the reference video. Here, the case where the position of the imaging terminal 1A is at the imaging reference point 410, the front direction 430 of the imaging terminal 1A faces the imaging reference direction 420, and the horizontal direction of the captured video is in the horizontal direction of the real space is called the imaging reference state 400 (the state in FIG. 5A). When the position and orientation of the imaging terminal 1A deviate from the imaging reference state 400, the point that should be at the center in the reference video will deviate from the center in the captured video. Further, when the orientation of the imaging terminal 1A rotates around the optical axis of the camera and deviates from the direction of the imaging reference state 400, the captured image will be tilted with respect to the reference video.

[0066] Similarly, depending on the position and orientation of the display terminal 1B, the displayed video will be different from the reference video. Here, the case where the position of the display terminal 1B is at the display reference point 510, the front direction 530 of the display terminal 1B faces the display reference direction 520, and the horizontal direction of the displayed video is in the horizontal direction of the real space is called the display reference state 500 (the state in FIG. 6A). When the position and orientation of the display terminal 1B deviate from the display reference state 500, the point that should be at the center in the reference video will deviate from the center in the displayed video. Further, when the orientation of the display terminal 1B rotates around the vertical axis of the display surface and the horizontal direction of the video deviates, the displayed image will be tilted with respect to the reference video.

[0067] To explain the relationship of position representation on each video, a coordinate system for each video is defined. For the reference video, the reference coordinate system {X (0) , Y (0)}, for the captured video, the captured coordinate system {X (C) , Y (C)}, and for the displayed video, the display coordinate system {X (D) , Y (D)} are used. The superscript (0), (C), (D) represents each coordinate system. The captured coordinate system and the display coordinate system fix the terminal coordinate system at the imaging terminal 1A and the display terminal 1B in their respective reference states, and change following the position and orientation changes of the imaging terminal 1A and the display terminal 1B. Hereafter, which coordinate system the position representation is based on is represented by these superscripts.

[0068] Now, in the following, the relationship of position representation in each coordinate system will be expressed by the coordinate system origin, the rotation angle of the coordinate system from the horizontal direction, and the distance L between the object being photographed or the image of the object being photographed and the terminal position. The origin of the shooting coordinate system is (X (0) C ,Y (0) C )=x (0) C , the rotation angle of the shooting coordinate system is ω C Represented by ω. C This refers to a rotation angle of 435 degrees (Figure 7A). Here, x (0) C This is a vector representation of the position coordinates, and the same applies below. Then, the origin position of the display coordinate system is (X (0) D ,Y (0) D )=x (0) D , the rotation angle of the display coordinate system is ω D Let's assume that ω D This refers to a display rotation angle of 535. Here, the rotation angle of the coordinate system is defined as positive counterclockwise with respect to the reference coordinate system. Let R(ω) be the matrix representing the rotation with rotation angle ω (positive counterclockwise). Hereafter, position will be represented using vector notation.

[0069] First, point x on the displayed image (D) P The corresponding point x on the reference image (0) P When we find this, we get equation (1). x (0) P =(L D / L0)R(ω D )x (D) P +x (0) D ...(1)

[0070] Here, (L D / L0) represents the apparent size multiplier due to the difference in distance L.

[0071] Next, point x on the reference image (0) P The corresponding point x on the captured video. (C)P When we find this, we get equation (2). x (C) P =(L0 / L C )R(-ω C )(x (0) P - x (0) C )···(2)

[0072] As a result, point x on the displayed image (D) P The corresponding point x on the captured video. (C) P When we find this, we get equation (3). x (C) P =(L D / L C )R(-ω C )x (D) P +(L0 / L C )R(-ω C )(x (0) D -x (0) C )···(3)

[0073] Depending on the relative positions of the devices, the corresponding point x (C) P It is possible that the subject may not be within the field of view of the camera of the shooting terminal 1A. The shooting terminal 1A contains the first correction information: shooting reference direction, L0, L C , ω C , x (0) C The captured video information is acquired and transmitted to the display terminal 1B along with it. The display terminal 1B receives the L, which is the second correction information. D , ω D , x (0) D The data is acquired and combined with the information transmitted from the shooting terminal 1A to create the display image.

[0074] Next, the method of measuring data for obtaining the first correction information will be summarized. L0 is measured as the distance between the imaging reference point 410 and the object center point 470 by the distance measurement sensor 113A when the imaging terminal 1A is in the imaging reference state 400 (FIG. 5A). ω C is measured as the imaging rotation angle 435, which is the rotation angle from the imaging reference state 400 around the lens optical axis of the camera of the imaging terminal 1A, by the position and orientation sensor 135A when the imaging terminal 1A is imaging. L C , x (0) C The measurement of will be described below.

[0075] First, L C is obtained from the front direction 430 of the imaging terminal 1A. Although it may be measured as the distance between the imaging terminal position 411 and the imaging front point 480, there is a possibility of unexpected errors such as the imaging front point 480 hitting the protrusion of the imaging object 7. Therefore, the origin of the imaging coordinate system is set at the point where the plane including the reference object center point 470 and perpendicular to the imaging reference direction 420 intersects the front direction 430 of the imaging terminal. Then, the distance between the determined origin of the imaging coordinate system and the imaging terminal position 411 is set as L C . x (0) C is also obtained as the coordinate value in the reference coordinate system of the origin of this imaging coordinate system.

[0076] Next, the specific determination procedure of L C , x (0) C will be described (FIG. 7A, FIG. 11). First, the displacement r C from the imaging reference point 410 of the imaging terminal position 411 at the imaging time is measured by the position and orientation sensor 135A. At the same time, the rotation of the front direction 430 of the imaging terminal 1A from the imaging reference direction 420 is measured, and the unit vector u CFind it. Here, let the unit vector of the shooting reference direction 420 be u0. L0 and u0 are measured when the shooting terminal 1A is in the shooting reference state 400. In FIG. 11, a three-dimensional coordinate system with a two-dimensional reference coordinate system as the XY plane is considered. The reference coordinate system plane is perpendicular to the shooting reference direction 420, and the X-axis direction of the reference coordinate system is set in the horizontal direction of the real space. x (0) C When interpreted as a vector in the three-dimensional coordinate system defined here, it is expressed as the following formula (4). x (0) C =r C +L C u C -L0u0 ···(4)

[0077] x (0) C Since x is orthogonal to u0, the following formula (5) holds. (r C +L C u C -L0u0)·u0 = 0 ···(5) Here, "·" represents the inner product.

[0078] Solve this formula (5), and L C is obtained as the following formula (6). L C =(L0 - r C ·u0) / (u C ·u0) ···(6)

[0079] Substitute formula (6) into formula (5) to obtain formula (7). x (0) C =r C +{(L0 - r C ·u0) / (u C ·u0)}u C -L0u0 ···(7)

[0080] From the above, L C and x (0) C are L0, u0, u C , r CSince it can be calculated from, the information that the shooting terminal 1A measures and transmits to the display terminal 1B as first correction information is L0, u0, u C , r C , ω C That's fine.

[0081] Next, the second correction information L obtained from the display terminal 1B. D , x (0) D This will be explained. The same discussion can be held on the display terminal 1B side (Figures 8A and 12). However, the direction vectors u0 and L0 from the display reference point 510 to the object center point 570 will be those transmitted from the shooting terminal 1A. In the real space where the display terminal 1B is located, the terminal position and orientation when the display terminal 1B is set to the display reference state 500 is set to the display reference point 510, and the coordinate system is set so that the vector from this display reference point 510 to the object center point 570 is L0u0. On the display terminal 1B, the object center point 570 is the position that is displayed as if the object center point 470 of the shooting object 7 were located there.

[0082] In the display terminal 1B, first, the position and orientation sensor 135B measures the displacement r of the display terminal position from the display reference point 510 at the time of display. D Measure the rotation of the display terminal 1B in the front direction 530 from the display reference direction 520, and measure the unit vector u of the display terminal 1B in the front direction 530. D We will find this. Using the same reasoning as for the shooting terminal 1A, we can obtain the following equations (8) and (9). L D =(L0-r D ·u0) / (u D ·u0)···(8) x (0) D =r D +{(L0-r D ·u0) / (u D ·u0)}u D -L0u0···(9)

[0083] Based on the above, the number measured by display terminal 1B 2 The correction information can be uD, rD, and ωD.

[0084] If the front direction 430 of the shooting terminal 1A differs from the shooting reference direction 420, or if the front direction 530 of the display terminal 1B differs from the display reference direction 520, trapezoidal distortion will appear in the image. However, this trapezoidal distortion may be corrected before display. Alternatively, other image distortion correction methods may be used in combination.

[0085] Furthermore, in order to synchronize the video information and the first correction information, timestamps may be added to both the video information and the first correction information.

[0086] Figure 13 is a flowchart showing the processing flow of the video sharing system according to the first embodiment.

[0087] First, user 2A puts on and starts up the shooting terminal 1A. If the shooting reference state 400 is updated during startup or while video sharing is active, the branching decision in S01 branches to YES and proceeds to the process in S02. Otherwise, the branch branches to NO and proceeds to the process in S03.

[0088] The processor 125A of the shooting terminal 1A sets the shooting reference point 410 and the shooting reference direction 420(u0) according to the input operation from user 2A. After this setting, while keeping the shooting terminal 1A in the shooting reference state 400, the distance sensor 113A measures the reference distance 415(L0) of the object to be photographed and the position and orientation sensor 135A measures the shooting reference direction 420(u0) as first correction information (S02).

[0089] In the shooting terminal 1A, the position and orientation sensor 135A provides correction information for the displacement (r) of the shooting terminal position 411 from the shooting reference point 410. C ), 430 (u C ), shooting rotation angle 435 (ω C ) is measured (S03).

[0090] The recording terminal 1A captures video including the user 2A's field of view using the rear camera 111A (S04).

[0091] The shooting terminal 1A transmits video information and first correction information to the display terminal 1B (S05). Until the termination condition is met (S06: No), the shooting terminal 1A returns to S01 and repeats the process. When the termination condition is met (S06: Yes), the shooting terminal 1A terminates its processing. The "termination condition" here refers to, for example, an operation to shut down the shooting terminal 1A, or the termination operation of the video sharing application.

[0092] Meanwhile, user 2B attaches and activates display terminal 1B. Display terminal 1B receives video information and correction information from shooting terminal 1A (S11).

[0093] If the display reference state 500 is updated during startup or while video sharing is active, the branching decision in S12 is to select YES and proceed to the process in S13. Otherwise, the branching decision is to select NO and proceed to the process in S14.

[0094] The processor 125B of the display terminal 1B sets the display reference direction 520 according to the input operation from user 2B (S13).

[0095] In the display terminal 1B, the position and orientation sensor 135B provides the displacement (r) of the display terminal position 511 from the display reference point 510 as second correction information. D ), 530 (u D ), display rotation angle 535(ω D ) is measured (S14).

[0096] The processor 125B of the display terminal 1B corrects the received video information using the first correction information measured by the shooting terminal 1A and the terminal position and orientation information of the display terminal 1B measured in step S13, and displays it on the display 119B (S15).

[0097] Display terminal 1B returns to S11 and repeats processing until the termination condition is met (S16: No). When the termination condition is met (S16: Yes), processing on display terminal 1B is terminated.

[0098] In this embodiment, the shooting terminal 1A determines the shooting reference point 410 and the shooting reference direction 420 based on the setting of the shooting reference state 400 in the shooting space in which the shooting terminal 1A is located. The display terminal 1B then determines the display reference point 510 and the display reference direction 520 based on the setting of the display reference state 500 in the display space in which the display terminal 1B is located. The display terminal 1B then displays the captured image 440 such that the position of the object to be photographed 7 relative to the shooting reference point 410 and the shooting reference direction 420 is the same as the apparent position of the image of the object to be photographed 8 relative to the display reference point 510 and the display reference direction 520.

[0099] As a result, even when using a small, portable display terminal 1B, the user 2B of the display terminal 1B (e.g., a work support worker) can view the image in the direction they want to see, in the vicinity of the direction that user 2A (e.g., the worker) is looking, without being affected by the movement of the camera terminal 1A, simply by orienting the display terminal 1B in that direction.

[0100] The following are possible modifications of the first embodiment.

[0101] (Adjustment of display standard state 500) The basic idea behind setting the display reference state 500 was that when user 2B is in a position and posture that is comfortable for working, they should be able to see the same image as in the shooting reference state 400. From that state, by changing the orientation of the display terminal 1B, user 2B can freely see the part they want to see. However, in some cases, there may be a desire to see the image of the object being photographed 8 from a location far away from the same image as in the shooting reference state 400, or from a different angle. In that case, the display terminal position 511 and the front direction 530 of the display terminal when user 2B is in a position and posture that is comfortable for working may be shifted from the display reference state 500. That is, during image sharing, the relative relationship between the display terminal position 511, the front direction 530 of the display terminal, and the display reference point 510 and display reference direction 520 of the display reference state 500 may be changed as appropriate by user 2B's instructions, or returned to the original state. The same may be done to shift the reference of the display rotation angle 535 from the horizontal direction.

[0102] This expands the degree of freedom in how user 2B views the video on display terminal 1B.

[0103] (Shooting standard conditions) In the first embodiment, for example, the shooting reference state 400 was updated in response to instructions from user 2A as the work progressed. In this case, in order to smooth the changes on the displayed image due to changes in the shooting reference point 410 and the shooting reference direction 420, the changes to the shooting reference point 410 and the shooting reference direction 420 may be made continuously or in small increments.

[0104] Furthermore, as a modification, the average position and orientation of the imaging terminal 1A may be used as the imaging reference state 400. That is, the average position of the imaging terminal position 411 is set as the imaging reference point 410, and the front direction of the imaging terminal 4 The average direction of 30 may be used as the reference direction for shooting, 420.

[0105] This improves the visibility of the display image shown to user 2B on display terminal 1B.

[0106] (Load balancing) Throughout the process, in order to distribute the processing load between the shooting terminal 1A and the display terminal 1B, a portion of the processing may be performed by a terminal paired with each of the shooting terminal 1A and the display terminal 1B, such as smartphones 5A and 5B, or a server.

[0107] (Selfie mode) If the shooting device 1A is a shooting device capable of taking pictures of user 2A, such as a smartphone, the object being photographed 7 may be user 2A. In this case as well, it is possible to view images that are not affected by the movement of the device. In selfie mode, where the rear camera 111A is described above, it should be read as the front camera 212A.

[0108] (Interactive mode) A single mobile device can handle both the processing of the camera device 1A and the display device 1B, allowing them to mutually display the video footage captured by the other device. In this mutual mode, users can converse while seeing the surrounding environment that the other person is viewing, thus deepening the level of communication. It can also be used in combination with selfie mode for a video call-like experience.

[0109] [Second Embodiment] This embodiment relates to alternative video for areas that cannot be captured by the camera terminal.

[0110] The captured video is recorded on display terminal 1B or a server (not shown). If there are missing areas in the displayed video, the missing areas are extracted from past footage and the past footage is superimposed for display. This is useful when past footage contains information that can be used as a reference. For example, information that does not change, such as descriptions of the subject being filmed, can be used as a reference even in past footage.

[0111] Figure 14 shows an example of a displayed image in the second embodiment.

[0112] As shown in Figure 14, the processor 125B of the display terminal 1B extrapolates the past captured video range 590 into the captured video range 490 and displays it if the captured video range 490 does not cover the entire display area of ​​the display terminal 1B's display 119B. The video information to be extrapolated may be video information that the shooting terminal 1A has previously captured of the entire shooting space, or it may be data of the shooting space on a server (not shown).

[0113] Figure 15 is a flowchart showing the processing flow according to the second embodiment. Steps identical to those in the flowchart of Figure 13 are given the same step numbers, and redundant explanations are omitted.

[0114] The processing of the imaging terminal 1A is the same as in the first embodiment.

[0115] When the display terminal 1B receives video information and first correction information (S11), it records the video (S21).

[0116] After measuring the second correction information of the display terminal 1B (S14), only the newly acquired captured video range from the video information acquired in S11 is switched to real-time video (S22), and the video is displayed (S15).

[0117] In other words, when video information and correction information are received for the first time in step S11, the captured video range 490 included in the received video information is displayed as the displayed video 542. When new video information is received in step S12 from the second time onward through loop processing, if the newly acquired captured video range overlaps with the previously extrapolated captured video range 590, the previously captured video range 590 is replaced with the captured video range 490 included in the new video information.

[0118] According to this embodiment, when there is a large discrepancy between the area viewed by user 2A and the area viewed by user 2B, and a non-display area is noticeable on the display terminal 1B, the image displayed to user 2B can be shown over a wider area than the newly received image range 490 by extrapolating and displaying the previously captured image range. This makes it possible to link a shooting terminal operated by a photographer of work conditions, etc., with a display terminal that receives and displays the captured image transmitted from that shooting terminal, without using a large fixed display device, and to display an image with a wider area than the captured image on the display terminal.

[0119] As another variation, if there are multiple shooting terminals, multiple captured images may be combined and displayed on the display terminal.

[0120] According to this embodiment, first, the shooting terminal 1A determines the shooting reference point 410 and the shooting reference direction 420 based on the setting of the shooting reference state 400 in the shooting space where the shooting terminal 1A is located. Then, the display terminal 1B determines the display reference point 510 and the display reference direction 520 in the display space where the display terminal 1B is located in the display space where the display terminal 1B is located in the display reference state 500. The display terminal 1B then displays the captured image 440 so that the position of the object to be photographed 7 relative to the shooting reference point 410 and the shooting reference direction 420 is the same as the apparent position of the image of the object to be photographed 8 relative to the display reference point 510 and the display reference direction 520. Recorded past images are also composited and displayed. As a result, even when using a small portable display terminal 1B, the user 2B of the display terminal 1B can view images in a wide range of directions that user 2B wants to see, in the vicinity of the direction that user 2A of the shooting terminal 1A is looking, by naturally pointing the display terminal 1B in that direction, without being affected by the movement of the shooting terminal 1A.

[0121] While embodiments of the present invention have been described above, it goes without saying that the configurations for realizing the technology of the present invention are not limited to the above embodiments, and various modifications are conceivable. For example, the embodiments described above are described in detail for the purpose of explaining the present invention in an easy-to-understand manner, and are not necessarily limited to those comprising all the described configurations. Furthermore, it is possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. All of these fall within the scope of the present invention. In addition, the numbers and messages that appear in the text and figures are merely examples, and using different ones will not impair the effects of the present invention.

[0122] Furthermore, the programs described in each processing example may be independent programs, or multiple programs may constitute a single application program. The order in which each processing step is executed may also be changed.

[0123] The functions of the present invention described above may be implemented in hardware, in whole or in part, for example, by designing them as an integrated circuit. Alternatively, they may be implemented in software by a microprocessor unit, CPU, etc., interpreting and executing an operating program that realizes each function. Furthermore, the scope of software implementation is not limited, and hardware and software may be used in combination. In addition, some or all of each function may be implemented on a server. The server only needs to be able to execute functions in cooperation with other components via communication, and its form is not limited, for example, a local server, cloud server, edge server, network service, etc. Information such as programs, tables, files that realize each function may be stored in memory, a recording device such as a hard disk or SSD (Solid State Drive), or a recording medium such as an IC card, SD card, or DVD, or it may be stored on a device on a communication network.

[0124] Furthermore, the control lines and information lines shown in the diagram are those deemed necessary for explanation and do not necessarily represent all control lines and information lines on the product. In reality, it is reasonable to assume that almost all components are interconnected.

[0125] The above-mentioned embodiment includes the following forms.

[0126] (Note 1) A video sharing system configured by communicating between a shooting terminal and a display terminal, The aforementioned shooting terminal is Camera and, A first position and orientation sensor for detecting the position and orientation of the aforementioned shooting terminal, The first processor and The system includes a first communication device that transmits and receives data with the aforementioned display terminal, The aforementioned display terminal is The display and A second position and orientation sensor for detecting the position and orientation of the display terminal, The second processor, It includes a second communication device that transmits and receives data with the aforementioned shooting terminal, The first processor of the aforementioned shooting terminal is In the real space where the aforementioned shooting terminal exists, in the shooting reference state set, the position of the shooting terminal is set as the shooting reference point, and the direction in which the front of the shooting terminal faces from the shooting reference point is set as the shooting reference direction, and the shooting reference point and the shooting reference direction are transmitted to the display terminal. Based on the sensor information of the first position and orientation sensor, the camera acquires the change in the position of the shooting terminal relative to the shooting reference point and the amount of change in the orientation of the shooting terminal relative to the shooting reference state when the camera photographs the object to be photographed, and transmits the change in the position of the shooting terminal and the amount of change in the orientation of the shooting terminal as first correction information to the display terminal together with the captured video information generated when the camera photographs the object to be photographed. The second processor of the aforementioned display terminal is In the display reference state set in the physical space where the display terminal exists, the position of the display terminal is set as the display reference point, and the direction in which the front of the display terminal faces from the display reference point is set as the display reference direction. Based on the sensor information of the second position and orientation sensor, the change in the position of the display terminal relative to the display reference point and the amount of change in the orientation of the display terminal relative to the display reference state are acquired as second correction information. The first and second correction information are used to correct the captured video information so that the positional relationship of the object to be photographed relative to the shooting reference point and the shooting reference direction matches the apparent positional relationship of the image of the object to be photographed relative to the display reference point and the display reference direction, and the corrected captured video information is displayed on the display. Video sharing system.

[0127] (Note 2) A video sharing method that is performed by communicating between a shooting terminal and a display terminal, The aforementioned shooting terminal includes the steps of setting the position of the shooting terminal as the shooting reference point and the direction in which the front of the shooting terminal faces from the shooting reference point as the shooting reference direction, in a shooting reference state set in the real space in which the shooting terminal exists, The steps include obtaining the change in the position of the shooting terminal with respect to the shooting reference point when the object to be photographed is photographed, and the amount of change in the orientation of the shooting terminal with respect to the shooting reference state, The shooting terminal transmits to the display terminal, as first correction information, the change in the position of the shooting terminal and the amount of change in the attitude of the shooting terminal, along with the captured video information generated by shooting the object to be photographed. The steps include setting the position of the display terminal as the display reference point and the direction in which the front of the display terminal faces from the display reference point as the display reference direction, in the display reference state set in the physical space in which the display terminal exists, The steps include acquiring, in the display terminal, the change in the position of the display terminal with respect to the display reference point, and the amount of change in the attitude of the display terminal with respect to the display reference state, as second correction information, The steps include correcting the captured video information using the first and second correction information so that the positional relationship of the object to be photographed relative to the shooting reference point and the shooting reference direction matches the apparent positional relationship of the image of the object to be photographed relative to the display reference point and the display reference direction, and displaying the corrected captured video information on a display, Video sharing methods including [specific methods]. [Explanation of symbols]

[0128] 1A: Camera terminal 1B: Display terminal 2A: User 2B: User 3A: Access Point 3B: Access Point 4: Communication Network 5A: Smartphone 5B: Smartphone 7: Subject of photography 8: Footage of the subject being filmed 100: Video sharing system 111A: Rear camera 111B: Rear Camera 113A: Distance measuring sensor 113B: Distance measuring sensor 114A: RTC 114B: RTC 115A: Accelerometer 115B: Accelerometer 116A: Gyroscope sensor 116B: Gyroscope sensor 117A: Geomagnetic sensor 117B: Geomagnetic sensor 118A: Positioning sensor 118B: Positioning sensor 119A: Display 119B: Display 120A: Network communication device 120B: Network communication device 121A: Microphone 121B: Mike 122A: Speaker 122B: Speaker 123A: Antenna 123B: Antenna 125A: Processor 125B: Processor 126A: Program 126B: Program 127A: Data 127B: Data 128A: Memory 128B: Memory 135A: Position and orientation sensor 135B: Position and orientation sensor 136A: PP Camera 136B: PP Camera 137A: PP distance measuring sensor 137B: PP distance measuring sensor 140A: Bus 140B: Bus 211A: Rear camera 211B: Rear Camera 212A: Front camera 212B: Front camera 213A: Distance measuring sensor 213B: Distance measuring sensor 214A: RTC 214B: RTC 215A: Accelerometer 215B: Accelerometer 216A: Gyroscope sensor 216B: Gyroscope sensor 217A: Geomagnetic sensor 217B: Geomagnetic sensor 218A: Positioning sensor 218B: Positioning sensor 219A: Display 219B: Display 220A: Network communication device 220B: Network communication device 221A: Microphone 221B: Mike 222A: Speaker 222B: Speaker 223A: Antenna 223B: Antenna 225A: Processor 225B: Processor 226A: Program 226B: Program 227A: Data 227B: Data 228A: Memory 228B: Memory 230A: Touch panel 230B: Touch panel 231A:Telephone network communication equipment 231B:Telephone network communication equipment 235A: Position and orientation sensor 235B: Position and orientation sensor 236A: PP Camera 236B: PP Camera 237A :PP distance sensor 237B :PP ranging sensor 240A: Bus 240B: Bus 301:Position and orientation measurement unit 302: Image Processing Unit 303: Transmission Information Generation Unit 304: Communication Control Unit 311: Communication Control Unit 312:Position and orientation measurement unit 313: Video Information Storage Unit 314: Display image generation unit 315: Display Control Unit 400: Standard shooting conditions 410: Shooting reference point 411: Location of the shooting device 415: Reference distance to the subject being photographed 420: Reference direction for shooting 430: Front direction 435: Shooting rotation angle 440: Filmed footage 441: Filmed footage 450: Primarily filmed footage 470: Center point of the object 480: Front view of the shot 490: Shooting range 500: Display standard status 510:Display reference point 511: Display terminal location 520:Display reference direction 530: Front direction 535: Display rotation angle 540: Displayed video 541: Displayed video 542: Displayed video 550: Display image center 570: Center point of the object 580:Display front point 590: Shooting range 600: Meter

Claims

1. A video sharing system configured by communicating between a shooting terminal and a display terminal, The aforementioned shooting terminal is Camera and, A first position and orientation sensor for detecting the position and orientation of the aforementioned shooting terminal, The first processor and The system includes a first communication device that transmits and receives data with the aforementioned display terminal, The aforementioned display terminal is The display and A second position and orientation sensor for detecting the position and orientation of the display terminal, The second processor, It includes a second communication device that transmits and receives data with the aforementioned shooting terminal, The first processor of the aforementioned shooting terminal is In the real space where the aforementioned shooting terminal exists, in the shooting reference state set, the position of the shooting terminal is set as the shooting reference point, and the direction in which the front of the shooting terminal faces from the shooting reference point is set as the shooting reference direction, and the shooting reference point and the shooting reference direction are transmitted to the display terminal. Based on the sensor information of the first position and orientation sensor, the change in the position of the shooting terminal relative to the shooting reference point and the amount of change in the orientation of the shooting terminal relative to the shooting reference state are acquired when the camera photographs the object to be photographed, and the change in the position of the shooting terminal and the amount of change in the orientation of the shooting terminal are transmitted to the display terminal as first correction information together with the captured video information generated when the camera photographs the object to be photographed. The second processor of the aforementioned display terminal is In the display reference state set in the physical space where the display terminal exists, the position of the display terminal is set as the display reference point, and the direction in which the front of the display terminal faces from the display reference point is set as the display reference direction. Based on the sensor information of the second position and orientation sensor, the change in the position of the display terminal relative to the display reference point and the amount of change in the orientation of the display terminal relative to the display reference state are acquired as second correction information. The first and second correction information are used to correct the captured video information so that the positional relationship of the object to be photographed relative to the shooting reference point and the shooting reference direction matches the apparent positional relationship of the image of the object to be photographed relative to the display reference point and the display reference direction, and the corrected captured video information is displayed on the display. Video sharing system.

2. A video sharing system according to claim 1, The aforementioned shooting reference state and the aforementioned display reference state are set by user instruction. Video sharing system.

3. A video sharing system according to claim 1, The average position and orientation of the aforementioned shooting terminal are set as the shooting reference state. Video sharing system.

4. In the video sharing system described in claim 1, It is further equipped with a video recording memory unit that stores information on previously recorded video footage, If the captured image range of the corrected captured image information does not cover the entire display area of ​​the display, the second processor extrapolates and displays past captured images around the captured image range. Video sharing system.

5. In the video sharing system described in claim 1, The aforementioned shooting terminal is further equipped with a wide-angle camera, The aforementioned shooting terminal transmits the video captured by the wide-angle camera to the aforementioned display terminal. If the image range of the corrected image information does not cover the entire display area of ​​the display, the second processor extrapolates the image captured by the wide-angle camera to the periphery of the image range and displays it. Video sharing system.

6. A video sharing method that is performed by communicating between a shooting terminal and a display terminal, The aforementioned shooting terminal includes the steps of setting the position of the shooting terminal as the shooting reference point and the direction in which the front of the shooting terminal faces from the shooting reference point as the shooting reference direction, in a shooting reference state set in the real space in which the shooting terminal exists, The steps include obtaining the change in the position of the shooting terminal with respect to the shooting reference point when the object to be photographed is photographed, and the amount of change in the orientation of the shooting terminal with respect to the shooting reference state, The steps include: transmitting to the display terminal, as first correction information, the change in the position of the shooting terminal and the amount of change in the attitude of the shooting terminal, together with the captured video information generated by shooting the object to be photographed; The steps include setting the position of the display terminal as the display reference point and the direction in which the front of the display terminal faces from the display reference point as the display reference direction, in the display reference state set in the physical space in which the display terminal exists, The steps include acquiring, in the display terminal, the change in the position of the display terminal with respect to the display reference point, and the amount of change in the attitude of the display terminal with respect to the display reference state, as second correction information, The steps include correcting the captured video information using the first and second correction information so that the positional relationship of the object to be photographed relative to the shooting reference point and the shooting reference direction matches the apparent positional relationship of the image of the object to be photographed relative to the display reference point and the display reference direction, and displaying the corrected captured video information on a display, Video sharing methods including [specific methods].

Images