Image sharing system and image sharing method

Abstract

The image pickup terminal sets the position of the image pickup terminal in an image pickup reference state set in an actual space as an image pickup reference point, and sets a direction oriented from the image pickup reference point from the front surface of the image pickup terminal as an image pickup reference direction. The amount of change in the position and orientation of the imaging terminal from the imaging reference state is acquired as first correction information and transmitted to the display terminal together with the captured image information. The display terminal sets, as a display reference point, the position of the display terminal in a display reference state set in an actual space, and sets, as a display reference direction, the direction oriented from the display reference point from the front surface of the display terminal. In the display terminal, the amount of change in the position and orientation of the display terminal from the display reference state is acquired, and the amount of change and the first correction information are used. The captured image information is corrected so that the positional relationship of the captured object when the image capture reference point and the image capture reference direction are taken as references matches the positional relationship on the appearance of the captured object image when the display reference point and the display reference direction are taken as references, and the captured image information is displayed on the display terminal.

Description

Technical Field

[0001] This invention relates to an image sharing system and an image sharing method. Background Technology

[0002] Patent Document 1 describes a wearable device comprising: a camera for capturing images of the work site in a range wider than the worker's field of view; a sensor for acquiring the worker's gaze as sensor information; and a gaze movement detection unit for detecting gaze movement as gaze movement information using the sensor information, wherein the camera's image data and the gaze movement information are transmitted to an operation instruction device via a network. The operation instruction device comprises: a display setting unit for cutting out a first image from the camera's image, a first image with a range wider than the worker's field of view, correcting the first image using the gaze movement information, and displaying the corrected first image on a display; and an instruction image generation unit for generating an instruction image for the worker, wherein the instruction image is transmitted to the wearable device via a network. (Abstract excerpt)

[0003] Existing technical documents

[0004] Patent Document 1: International Publication No. 2020 / 178960 Summary of the Invention

[0005] In the prior art, in order to avoid being affected by the movement of the camera terminal that is recording the work site and to observe a wide range of the site, operation support personnel use fixed, large display devices, which presents the challenge of increasing the size of the devices.

[0006] The present invention was made in view of the above circumstances, and its object is to provide an image sharing system and image sharing method that does not require the use of a fixed large display device, but enables the camera terminal operated by the camera operator to cooperate with the display terminal that receives and displays the video images sent from the camera terminal, so as to achieve a display that is not affected by the movement of the camera terminal.

[0007] To address the aforementioned issues, the present invention comprises the structure described in the claims. As an example, this invention provides an image sharing system that connects a camera terminal and a display terminal via communication. The camera terminal includes: a camera; a first position and posture sensor for detecting the position and posture of the camera terminal; a first processor; and a first communication device for transmitting and receiving data with the display terminal. The display terminal includes: a display; a second position and posture sensor for detecting the position and posture of the display terminal; a second processor; and a second communication device for transmitting and receiving data with the camera terminal. The first processor of the camera terminal: in a camera reference state set in the actual space where the camera terminal exists, sets the position of the camera terminal as a camera reference point, and sets the direction from the camera reference point toward the front of the camera terminal as a camera reference direction, and sends the camera reference point and the camera reference direction to the display terminal; based on sensor information from the first position and posture sensor, it obtains the change in the position of the camera terminal relative to the camera reference point when the camera is filming an object, and the posture of the camera terminal relative to the camera reference state. The change in position and the change in posture of the camera terminal are used as first correction information and sent together with the video image information generated by the camera capturing the object to the display terminal. The second processor of the display terminal: in a display reference state set in the actual space where the display terminal exists, the position of the display terminal is set as a display reference point, and the direction from the front of the display terminal to the display reference point is set as the display reference direction; according to the sensor information of the second position and posture sensor, the change in position of the display terminal relative to the display reference point and the change in posture of the display terminal relative to the display reference state are obtained as second correction information; the video image information is corrected using the first correction information and the second correction information in a manner that makes the positional relationship of the object captured when the camera reference point and the camera reference direction are used consistent with the positional relationship of the image of the object captured when the display reference point and the display reference direction are used as references, and the corrected video image information is displayed on the display.

[0008] According to the present invention, an image sharing system and method can be provided that eliminates the need for a fixed, large display device. Instead, a camera terminal operated by a camera operator, which monitors the work situation, collaborates with a display terminal that receives and displays video images transmitted from the camera terminal, achieving a display unaffected by the directional movement of the camera terminal. Other objectives, structures, and effects described above will become clear in the following embodiments. Attached Figure Description

[0009] Figure 1 This is a schematic structural diagram of the image sharing system according to the first embodiment.

[0010] Figure 2 This is a hardware structure diagram of the camera terminal and the display terminal.

[0011] Figure 3 This is a hardware structure diagram of a smartphone.

[0012] Figure 4A This is a functional block diagram of the camera terminal 1A part in the image sharing system.

[0013] Figure 4B This is a functional block diagram of display terminal 1B in the image sharing system.

[0014] Figure 5A This is a diagram illustrating an example of the positional relationship between the camera reference point, camera reference direction, and the camera object in the camera space where the camera terminal exists.

[0015] Figure 5B It is shown Figure 5A The image is a photograph of the camera in terms of its positional relationship.

[0016] Figure 6A This is a diagram illustrating an example of the positional relationship between the display reference point, the display reference direction, and the image of the object being photographed in the display space where the display terminal exists.

[0017] Figure 6B It is shown Figure 6A A diagram showing the image in terms of positional relationship.

[0018] Figure 7A It shows from Figure 5A The diagram shows an example of the positional relationship between the camera terminal's location and orientation, and the positional relationship when the camera terminal's location and orientation have changed.

[0019] Figure 7B It is shown Figure 7A The image is a photograph of the camera in terms of its positional relationship.

[0020] Figure 7C This shows that the camera terminal is Figure 7A The status and the display terminal are Figure 6A A diagram showing the image of the display terminal in the (reference position) situation.

[0021] Figure 8A It shows from Figure 6A The diagram shows an example of the positional relationship between the display terminal's position and orientation, and the positional relationship when the display terminal's position and orientation change.

[0022] Figure 8B It is shown Figure 8A A diagram showing the image in terms of positional relationship.

[0023] Figure 9 This is a diagram showing the parameters used in the description of the image correction method.

[0024] Figure 10 It is a diagram showing the relative relationship between the photographic image and the displayed image relative to the reference image.

[0025] Figure 11 This is an explanatory diagram of the parameters used in the actual space where the camera terminal exists.

[0026] Figure 12 This is an explanatory diagram of the parameters used in the actual space where the display terminal exists.

[0027] Figure 13 This is a flowchart illustrating the processing flow of the image sharing system according to the first embodiment.

[0028] Figure 14 This is a diagram illustrating an example of a display image in the second embodiment.

[0029] Figure 15 This is a flowchart illustrating the process involved in the second embodiment. Detailed Implementation

[0030] This invention is expected to improve the visibility of video images of workers performing operations on-site by support personnel providing back-office support. Therefore, this invention is anticipated to improve technology for labor-intensive industries requiring operational and back-office support, and thus is expected to contribute to UN Sustainable Development Goals (SDGs) 8.2 (enhancing economic productivity through diversification, technological advancement and innovation, with a focus on industries that further increase the value of goods and services, particularly labor-intensive industries).

[0031] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In all the drawings used to describe the embodiments, the same reference numerals are generally used for the same parts, and repeated descriptions are omitted.

[0032] [First Implementation]

[0033] Figure 1This is a schematic structural diagram of the image sharing system according to the first embodiment. The image information of the object 7 captured by the camera terminal 1A, which is a portable information terminal worn by user 2A, is sent to the display terminal 1B, which is a portable information terminal worn by user 2B. At the same time, information informing the relationship between the image at the time of capture, the camera reference point, and the camera reference direction is also sent. In the display terminal 1B, for the image of the object 7 captured, an image 8 of the object, whose positional relationship has been corrected, is displayed within the space where the display terminal 1B exists, as if the object 7 were present at the set position. The means of communication between the camera terminal 1A and the display terminal 1B can be either communication via the communication network 4 through access points 3A and 3B, or direct communication between the camera terminal 1A and the display terminal 1B.

[0034] In this embodiment, an HMD (Head Mounted Display) is used as both the camera terminal 1A and the display terminal 1B.

[0035] Users 2A and 2B can also operate smartphones 5A and 5B respectively, and connect smartphones 5A and 5B to access points 3A and 3B and communication network 4.

[0036] Figure 2 This is a hardware structure diagram of the camera terminal and the display terminal. The camera terminal 1A and the display terminal 1B have the same structure, so in the following description, for the common components in the camera terminal 1A and the display terminal 1B, the additional characters of the reference symbols will be set as A and B to distinguish them.

[0037] In camera terminal 1A and display terminal 1B, external cameras 111A and 111B, ranging sensors 113A and 113B, position and attitude sensors 135A and 135B, geomagnetic sensors 117A and 117B, positioning sensors 118A and 118B, RTC (real-time clock) 114A and 114B, displays 119A and 119B, microphones 121A and 121B, speakers 122A and 122B, processors 125A and 125B, memory 128A and 128B, and network communication devices 120A and 120B are interconnected via buses 140A and 140B connecting the various components. Network communication devices 120A and 120B are connected to antennas 123A and 123B for transmitting and receiving network communication signals. The displays 119A and 119B, speakers 122A and 122B are equivalent to output devices, and the microphones 121A and 121B are equivalent to input devices.

[0038] Position and posture sensors 135A and 135B are used to determine the position and posture of camera terminal 1A and display terminal 1B. Here, posture refers to the three-dimensional rotational position of camera terminal 1A and display terminal 1B in the actual space in which they exist. Details will be described later. Specific structural examples of position and posture sensors 135A and 135B include position and posture cameras 136A and 136B (Position and Posture cameras, hereinafter referred to as "PP cameras"), position and posture ranging sensors 137A and 137B (Position and Posture ranging sensors, hereinafter referred to as PP ranging sensors), accelerometers 115A and 115B, and gyroscope sensors 116A and 116B. PP cameras 136A and 136B can also be the same as external cameras 111A and 111B. Furthermore, PP ranging sensors 137A and 137B can also be the same as ranging sensors 113A and 113B.

[0039] In determining the position and orientation of the PP cameras 136A and 136B and the PP range sensors 137A and 137B, feature points existing in the external environment are utilized. Multiple feature points are used; for example, these can be feature points of the object being filmed 7, indoor feature points, or, if outdoors, feature points of buildings. As preparation, the relative positional relationship of the feature points is determined based on their direction and distance. If the coordinate values ​​of the feature points in the external coordinate system are known, this preparation can be omitted. Furthermore, by measuring the changes in the direction and distance of the feature points seen from the camera terminal 1A and the display terminal 1B, the changes in the position and orientation of the camera terminal 1A and the display terminal 1B are determined. When measuring distance using a stereo camera method, only the PP cameras 136A and 136B are required. Furthermore, if the coordinates of the feature points in the external coordinate system are known, and the relationship between the external coordinate system and the vertical direction is known, then the relative relationship between the frame posture and the vertical direction of the camera terminal 1A and the display terminal 1B can be determined by using the posture measurement of PP cameras 136A and 136B and PP range sensors 137A and 137B.

[0040] The position and orientation of the camera terminal 1A and display terminal 1B are determined by integrating the acceleration and angular acceleration using accelerometers 115A and 115B and gyroscopes 116A and 116B. This allows the determination of displacement from the reference position and reference orientation. Furthermore, by measuring the gravitational acceleration vector using accelerometers 115A and 115B, the relative orientation of the camera terminal 1A and display terminal 1B in the vertical direction can also be determined.

[0041] In determining the position and posture of the camera terminal 1A and the display terminal 1B, the measurements can be taken from both the PP camera 136A, 136B and the PP range sensor 137A, 137B and the accelerometer sensor 115A, 115B and the gyroscope sensor 116A, 116B, or the measurements from other types of sensors.

[0042] Positioning sensors 218A and 218B are used to determine the indoor and outdoor positions of camera terminal 1A and display terminal 1B. They are used when retrieving library images associated with the photographed object 7. Specific examples include, indoors, the use of beacon signals or location markers. Outdoors, a GPS (Global Positioning System) receiver, for example, uses signals from satellites.

[0043] Network communication units 120A and 120B are communication interfaces that enable wireless communication between at least camera terminal 1A and display terminal 1B and access points 3A and 3B via near-field wireless communication, wireless LAN, or base station communication. They include communication processing circuits corresponding to various predetermined communication interfaces and are connected to antennas 123A and 123B. Furthermore, network communication units 120A and 120B transmit and receive biological information, image data, control signals, etc. In addition, as near-field wireless communication, wireless LANs such as Bluetooth, IrDA (Infrared Data Association), Zigbee, HomeRF, or Wi-Fi are used. In addition, for base station communication, long-distance wireless communication methods such as LTE (Long Term Evolution, registered trademark in Japan), LTE-Advanced, mobile WiMAX (Worldwide Interoperability for Microwave Access, registered trademark in Japan), and WiMAX2 can be used.

[0044] Displays 119A and 119B are positioned in front of the eyes of users 2A and 2B wearing camera terminal 1A and display terminal 1B. Processors 125A and 125B control the operation of camera terminal 1A and display terminal 1B. Specifically, they correct and display images of real-world spatial information captured by external cameras 111A and 111B, as well as images received from access points 3A and 3B.

[0045] Processors 125A and 125B, for example, consist of CPUs.

[0046] Memory 128A and 128B consist of flash memory and non-volatile memory. Memory 128A and 128B also store programs 126A and 126B such as OS (Operating System) and motion control applications, as well as data 127A and 127B used by processors 125A and 125B.

[0047] Processors 125A and 125B load programs 126A and 126B into memories 128A and 128B for execution, and read data 127A and 127B as needed for the execution of programs 126A and 126B.

[0048] Figure 3 This is a hardware structure diagram of a smartphone. The parts with the same names as the components of an HMD (Hardware Module) essentially have the same function, so detailed explanations are omitted unless there are specific differences.

[0049] In smartphones 5A and 5B, external cameras (rear cameras) 211A and 211B, internal cameras (front cameras) 212A and 212B, ranging sensors 213A and 213B, position and gesture sensors 235A and 235B, geomagnetic sensors 217A and 217B, positioning sensors 218A and 218B, RTC 214A and 214B, displays 219A and 219B, microphones 221A and 221B, speakers 222A and 222B, processors 225A and 225B, memory 228A and 228B, telephone network communication devices 231A and 231B, and network communication devices 220A and 220B are interconnected via buses 240A and 240B connecting the various components. Network communication devices 220A and 220B are connected to antennas 223A and 223B for transmitting and receiving network communication signals.

[0050] As specific structural examples of position and attitude sensors 235A and 235B, similar to the HMD case, they include PP cameras 236A and 236B, PP ranging sensors 237A and 237B, accelerometers 215A and 215B, and gyroscope sensors 216A and 216B. PP cameras 236A and 236B can also be the same as external cameras 211A and 211B. Furthermore, PP ranging sensors 237A and 237B can also be the same as ranging sensors 213A and 213B.

[0051] The memories 228A and 228B store program 226A and 226B and data 227A and 227B.

[0052] Touch panels 230A and 230B, which serve as input interfaces, are stacked on top of displays 219A and 219B.

[0053] Figure 4A This is a functional block diagram of the camera terminal 1A part in the image sharing system. Figure 4B This is a functional block diagram of display terminal 1B in the image sharing system.

[0054] like Figure 4A As shown, the camera terminal 1A includes: a position and posture determination unit 301, which acquires sensor information from a position and posture sensor and a ranging sensor, and generates first calibration information to be sent to a display terminal 1B; a camera processing unit 302, which acquires and processes camera image information from a camera; a transmission information generation unit 303, which transmits the first calibration information and the camera image information; and a communication control unit 304, which transmits and receives data with the display terminal 1B. The first calibration information will be described later.

[0055] In addition, such as Figure 4B As shown, the display terminal 1B includes: a communication control unit 311 for transmitting and receiving data with the camera terminal 1A; a position and posture measurement unit 312 for acquiring sensor information from a position and posture sensor mounted on the display terminal 1B and generating second correction information indicating changes in the position and posture of the display terminal 1B; an image information storage unit 313 for storing received camera image information; a display image generation unit 314 for performing correction processing on the camera image using the first and second correction information; and a display control unit 315 for performing display control on the display. The second correction information will be described later.

[0056] The image sharing system 100 involved in this embodiment is characterized by its display format when the image information generated by the camera terminal 1A capturing the image of the object 7 is sent to the display terminal 1B for display. According to Figure 5A , Figure 5B , Figure 6A as well as Figure 6B To illustrate its features. Furthermore, the following describes the combination of the HMDs of camera terminal 1A and display terminal 1B, but each HMD can also be replaced with a smartphone 5A or 5B. It can also be a combination of an HMD and a smartphone. Additionally, different types of portable information terminals can also be used as long as they have equivalent functions.

[0057] first, Figure 5AThis diagram illustrates an example of the positional relationship between the camera reference point 410, the camera reference direction 420, and the camera object 7 in the camera space where the camera terminal 1A exists. This positional relationship is referred to as the camera reference state 400. Furthermore, the point on the camera object 7 reached by a straight line extending from the camera reference point 410 along the camera reference direction 420 is referred to as the object center point 470.

[0058] Regarding the camera reference state 400, for example, user 2A sets the camera terminal position 411 when they are in a position conducive to operation as the camera reference point 410. Furthermore, the direction of the optical axis of the external camera 111A, i.e., the frontal direction 430, which is primarily facing the camera terminal 1A during operation, is set as the camera reference direction 420. Additionally, as the operation progresses, the position conducive to operation and the frontal direction 430 primarily facing the camera terminal 1A may change. In this case, the camera reference state 400 is updated upon instruction from user 2A.

[0059] Figure 5B It is shown Figure 5A The image is a diagram of the camera image under the camera reference state 400. It shows the camera terminal 1A and the camera object 7 in the following states: Figure 5A The image is a diagram of the camera image of camera terminal 1A in the case where the camera terminal 1A is positioned at camera reference point 410 and the front direction 430 of camera terminal 1A faces the camera reference direction 420. If this is the positional relationship, the center point 470 of the object on the object 7 is located at the center of the image 440, i.e., the center 450 of the image.

[0060] The distance between the camera reference point 410 and the center point 470 of the object under the camera reference state 400 is defined as the camera object reference distance 415. The camera object reference distance 415 is measured using a range sensor 113A or a position and posture sensor 135A. Furthermore, the camera reference point 410 and the camera reference direction 420 are measured using the position and posture sensor 135A. The camera terminal 1A sends information about the camera reference point 410, the camera reference direction 420, and the camera object reference distance 415 to the display terminal 1B at least whenever the camera reference state 400 is set or updated.

[0061] Figure 6A This diagram illustrates an example of the positional relationship between the display reference point 510, the display reference direction 520, and the image of the photographed object 7 in the display space where the display terminal 1B exists. The state at this position is referred to as the display reference state 500. When the photographed object 7 is in the camera space... Figure 5A Given the positional relationship shown (camera reference state 400), the positional relationship of the image 8 of the camera object in the display space where the display terminal 1B exists becomes as follows: Figure 6AIn this context, the position of the image 8 is the position where the user 2B perceives the image 8 as being present in the display space; it is the 3D display position within a 3D image. That is, the display terminal 1B displays the image 440 in such a way that the position based on the camera reference point 410 and camera reference direction 420 of the image 7 is the same as the position of the appearance based on the display reference point 510 and display reference direction 520 of the image 8. In other words, the display terminal 1B displays the image 8 in a position that appears to the user 2B as if it were viewed from the camera reference point 410, similar to the state of viewing the image 7 from the camera reference point 410. Furthermore, if the occlusion relationship with other objects in the display space is also represented, the image becomes more realistic. In this specification, unless otherwise specified, when referred to as "image 8," as described above, it means the image that appears to be located in the display space for the user 2B.

[0062] In display reference state 500, for example, the display terminal position 511 when user 2B is in a position where it is easy to view the displayed image is set as display reference point 510. Furthermore, the direction primarily facing the front of display terminal 1B 530 during image visual recognition is set as display reference direction 520. However, as a standard setting method, the tilt angle of display reference direction 520 relative to the horizontal plane is the same as the tilt angle of camera reference direction 420. This is to ensure that the viewing of the camera scene is similar to that of user 2A. Here, the front direction 530 of display terminal 1B refers to the direction extending vertically from the center of the display surface of display terminal 1B's monitor 119B, or the surface that user 2B perceives as being displayed on its surface. When the display area is not a shape with a clearly defined center, such as a rectangle, the center of the display surface is appropriately defined. Additionally, during image sharing, there may be situations where it is desirable to change the position for visual recognition, the direction primarily facing the front of display terminal 1B 530. In this case, the camera reference state 400 is updated according to user 2B's instruction.

[0063] Figure 6B It is shown Figure 6A The image shows the display image under the reference state 500. (See diagram.) Figure 6B As shown, the image 8 of the photographed object is located in the display space. Figure 6A Given the positional relationship, and with display terminal 1B positioned at display reference point 510 and its front direction 530 facing display reference direction 520, the display image 540 of display terminal 1B becomes as follows: Figure 6B In that case, the center point 570 of the object on the captured object image 8 is located at the center of the displayed image 540, i.e., the center of the displayed image 550.

[0064] The above are examples of camera image 440 and display image 540 when the camera terminal 1A and display terminal 1B are in a reference positional relationship. The following will describe a situation where the positional relationship is normal.

[0065] First, refer to Figure 7A , Figure 7B This describes the changes in the position and posture of the camera terminal 1A. Figure 7A It shows from Figure 5A The diagram shows an example of the relationship between the position and orientation of the camera terminal 1A, and the positional relationship when the position and orientation of the camera terminal 1A change.

[0066] First, the posture of the terminal will be explained using camera terminal 1A as an example. In this invention, the three-dimensional rotational position of a portable information terminal in actual space is referred to as posture. In the case of camera terminal 1A, the reference for the rotational position is camera reference state 400. Camera terminal 1A becomes... Figure 7A The three-dimensional rotation amount from the camera reference state 400 in the normal state shown is the change in posture of the camera terminal 1A. This three-dimensional rotation amount is measured using a position and posture sensor 135A. Furthermore, in the following description, this three-dimensional rotation amount is decomposed into the change in the frontal direction 430 of the camera terminal from the camera reference direction 420, and the rotation angle around the optical axis of the external camera 111A from the camera reference state 400, namely the camera rotation angle 435. Here, the optical axis direction of the external camera 111A and the frontal direction 430 of the camera terminal 1A are aligned in the same direction.

[0067] The same applies to display terminal 1B; in this case, the reference for the rotation position is display reference state 500. Display terminal 1B becomes... Figure 8A The three-dimensional rotation amount from the display reference state 500 in the normal state shown is the posture change amount of the display terminal 1B. This three-dimensional rotation amount is measured using a position and posture sensor 135B. Furthermore, this posture change amount is decomposed into the change in the front direction 530 of the display terminal 1B from the display reference direction 520, and the rotation angle from the display reference state 500 about the vertical axis of the display surface of the display 119B, namely the display rotation angle 535. Here, the direction of the vertical axis of the display surface of the display 119B and the front direction 530 of the display terminal 1B are aligned in the same direction.

[0068] in addition, Figure 7A The position 411 and the front orientation 430 of the camera terminal 1A shown are both from the position 411 and the front orientation 430. Figure 5AThe camera reference point 410 and camera reference direction 420 are set under the camera reference state 400 shown. The point where the line extending in the front direction 430 of the camera terminal 1A reaches the camera object 7 is called the camera front point 480.

[0069] Figure 7B Showing the position Figure 7A The camera image 441 shows the position of the object. In this situation, user 2A moves their gaze to check the lower left gauge 600. On camera image 441, the front viewpoint 480 is located in the center of camera image 441, and the object centerpoint 470 is located in the upper right. Here, in order to correct the display image 541 (reference...), Figure 8B As information for correction, the position and posture sensor 135A is used to measure the position change of the camera terminal 411 from the camera reference point 410, the change of the front direction 430 of the camera terminal 1A from the camera reference direction 420, and the rotation angle of the optical axis of the external camera 111A of the camera terminal 1A from the camera reference state, namely the camera rotation angle 435.

[0070] The changes in terminal position, the change in the front orientation of the terminal, and the rotation angle around the optical axis of the camera, measured by the camera terminal 1A from the camera reference state 400, are among the correction information used to correct the displayed image 541. Hereinafter, the changes in terminal position, the change in the front orientation of the terminal, and the rotation angle around the optical axis of the camera, measured by the camera terminal 1A, will be referred to as the first correction information. Here, the change in the front orientation of the terminal and the rotation angle around the optical axis of the camera are specific aspects of the change in posture. On the other hand, as described later, the changes in terminal position, the change in the front orientation of the terminal, and the rotation angle around the vertical axis of the display surface, measured by the display terminal 1B from the display reference state 500, will be referred to as the second correction information. In the display terminal, the change in the front orientation of the terminal and the rotation angle around the vertical axis of the display surface are specific aspects of the change in posture.

[0071] Furthermore, the first correction information is sent to the display terminal 1B along with the camera image information. Also, vertical direction information is included in the front view 430 of the terminal. This allows for adjustment of the tilt angle.

[0072] Figure 7C The camera terminal 1A is shown as Figure 7A The status and display terminal 1B is Figure 6A The image displayed on the display terminal 1B shows the situation as shown in reference state 500. When the position of the camera terminal 1A changes from... Figure 5A Change to Figure 7AWhen the position of display terminal 1B remains unchanged, the center point 570 of the object is located at the center 550 of the displayed image in display terminal 1B. That is, the camera image information is corrected and displayed on display terminal 1B in a manner that compensates for changes in the terminal position, changes in the front orientation of the terminal, and rotation angle around the optical axis of the camera as shown in the first correction information.

[0073] However, in Figure 6B and Figure 7C In Figure 7C The image of the object 8 transmitted from the camera terminal 1A is displayed only within the camera image range 490; the area outside the camera image range 490 becomes a missing area where no image data is displayed. This missing area may, for example, be a completely black area of ​​the screen. Figure 7C In the diagram, a diagonal line is used to indicate the missing area.

[0074] exist Figure 7C The image displayed on monitor 119B includes the missing area, but... Figure 6B The entire area of ​​the display 119B of the display terminal 1B is the same as the camera image range 490, so there is no missing area in the display 119B, which is a difference between the two. Furthermore, in Figure 7C In, also with Figure 6B Similarly, the object center point 570 is located at the display image center 550.

[0075] Figure 8A It shows from Figure 6A This diagram illustrates an example of the positional relationship between the position and orientation of the display terminal 1B (display reference state 500) and the positional relationship when the position and orientation of the display terminal 1B change. Let the position and orientation of the display terminal 1B be as follows... Figure 8A That way.

[0076] Figure 8B It is shown Figure 8A The diagram shows the displayed image in terms of positional relationship. Here, the display image center 550 is the point located at the center of the display image 541 on the image of the photographed object 8. Furthermore, the display image center 550 is the point in the space where the display terminal 1B exists, located at a position where the user 2B feels as if the image of the photographed object 8 is there, and extending to the front direction of the display terminal 1B. The position and posture sensor 135B within the display terminal 1B measures the change of the display terminal position 511 of the display terminal 1B from the display reference point 510, the change of the front direction 530 from the display reference direction 520, and the rotation angle around the vertical axis of the display surface from the horizontal state, i.e., the display rotation angle 535, which serves as the second correction information.

[0077] Display terminal 1B, based on the second calibration information and the first calibration information sent from camera terminal 1A, becomes as follows: Figure 8B The display terminal 1B displays the image 541 in a manner that appears as if it were viewing the object 7 from the corresponding position of the display terminal position 511 in the camera space, and corrects the position and magnification of the camera image information to display it.

[0078] There are cases where the camera front point 480 and the display front point 580 are offset, making it impossible to cover the entire display area of ​​the display 119B of the display terminal 1B with the camera image range 490 through magnification correction. Therefore, as a variation, a structure can also be used that mounts a wide-angle camera in the camera terminal 1A to reduce the missing area.

[0079] In addition, the above-corrected image can also be displayed in a magnified or reduced manner in the display terminal 1B.

[0080] Furthermore, the markings indicating the center point of the object, the center point of the camera, and the center point of the display are depicted in the accompanying drawings for ease of explanation, but such markings may also be displayed in practice. By displaying these markings, it becomes easier to grasp the status of image processing.

[0081] The following is for reference. Figures 9 to 12 This is an example illustrating an image correction method. Figure 9 This is a diagram showing the parameters used in the description of the image correction method. Figure 10 It is a diagram showing the relative relationship between the photographic image and the displayed image relative to the reference image. Figure 11 This is an explanatory diagram of the parameters used in the actual space where the camera terminal exists. Figure 12 This is an explanatory diagram of the parameters used in the actual space where the display terminal exists.

[0082] In the image sharing system and method of this embodiment, image correction is performed in the following two stages.

[0083] First correction: restores the positional changes of the photographed object in the image caused by changes in the position and posture of the camera terminal 1A to their original state.

[0084] Second correction: Reflects the positional changes of the photographed object in the image caused by changes in the position and posture of the display terminal 1B.

[0085] First, in this invention, as a concept, a reference image is considered between the video image obtained by the camera (i.e., the video recording terminal 1A) and the display image displayed on the display 119B (i.e., the display terminal 1B). The reference image refers to the image captured when the camera terminal 1A is set to video recording reference state 400 and a video is taken of an external object. That is, it is... Figure 5B The images.

[0086] Here, we provide further explanation of the terminal's reference state. We assume that each terminal is in a horizontal state under the camera reference state 400 of camera terminal 1A and the display reference state 500 of display terminal 1B. When setting the reference state as the terminal facing a certain direction from a certain position, the state in which the frame is horizontal while maintaining a constant pitch angle is taken as the reference state. Making the frame horizontal means, for example, the state where the eyes of a user wearing an HMD are horizontally aligned; for a smartphone, it means that either the short or long side of the frame is horizontal. Typically, the camera area and image display area are rectangular, so a horizontal state means that one side of this rectangle is horizontal.

[0087] Assuming the baseline state is set as described above, the following terminal coordinate system is set for each image: Figure 9 ): The image center is set as the origin, the horizontal image direction in the reference state is set as the X-axis, and the direction orthogonal to the X-axis is set as the Y-axis. In addition, when the definition is extended for terminals whose image range is not rectangular, at the point of use of the terminal, when the terminal is made horizontal, the direction on the image corresponding to the horizontal direction orthogonal to the external frontal direction of the terminal is set as the horizontal image direction.

[0088] Additionally, using coordinate values ​​(X) P Y P The symbol () represents the position of point P on the image. The angle at which the entire image is viewed from the terminal position of camera terminal 1A is the field of view. Figure 9 The horizontal and vertical field of view are shown in the image. The field of view can also be different in the captured image and the displayed image. Here, the distance between the captured object 7 or the captured object image 8 and the terminal position is set to L. In the case of the reference image, L is the reference distance 415 ( ) of the captured object. Figure 5A , Figure 6A This distance is both the distance between the camera reference point 410 and the center point 470 of the object, and the distance between the display reference point 510 and the center point 570 of the object. It is placed in... Figure 11 The value is recorded as L0. In the case of the video image, L represents the distance between the camera terminal position 411 and the camera's front point 480. Figure 7A ). Put it in Figure 11 The text records it as L. C The L in the case of the displayed image is the distance between the display terminal position 511 and the display front point 580. Figure 8A ). Put it in Figure 12 The text records it as L. DBetween images, if L is different, the image range and the size of the object 7 will differ depending on the ratio of L at the position of the object 7 or the image 8. Here, it is assumed that the size of the appearance changes inversely proportional to L. Furthermore, although it refers to the position of the image 8, in the display terminal 1B it is displayed as if the object 7 captured by the camera terminal 1A is actually there.

[0089] exist Figure 10 In this context, the position of the object 7 being photographed is common to all images. However, due to the position and orientation of the camera terminal 1A, the photographed image differs from the reference image. Here, the situation where the camera terminal 1A is positioned at the camera reference point 410, the frontal direction 430 of the camera terminal 1A faces the camera reference direction 420, and the horizontal direction of the photographed image is in the actual horizontal direction of space is referred to as the camera reference state 400 (is). Figure 5A (State). When the position and orientation of the camera terminal 1A deviate from the camera reference state 400, the point that should be at the center in the reference image deviates from the center in the camera image. In addition, when the posture of the camera terminal 1A rotates around the optical axis of the camera and deviates from the direction of the camera reference state 400, the camera image is tilted relative to the reference image.

[0090] Similarly, due to the position and orientation of the display terminal 1B, the displayed image differs from the reference image. Here, the situation where the display terminal 1B is positioned at the display reference point 510, the front orientation 530 of the display terminal 1B faces the display reference direction 520, and the horizontal direction of the displayed image is in the horizontal direction of actual space is referred to as the display reference state 500 (is). Figure 6A (State). When the position and orientation of the display terminal 1B shift from the display reference state 500, the point that should be centered in the reference image shifts away from the center in the displayed image. Additionally, when the orientation of the display terminal 1B rotates around the vertical axis of the display surface and the image shifts horizontally, the displayed image tilts relative to the reference image.

[0091] To illustrate the positional relationships between the various images, a coordinate system is defined for each image. For the reference image, the reference coordinate system is {X...}. (0) Y (0)}, in relation to camera images, is the camera coordinate system {X}. (C) Y (C)}, in terms of displaying images, refers to the display coordinate system {X}. (D) Y (D)The superscript characters (0), (C), and (D) represent different coordinate systems. The camera coordinate system and the display coordinate system are coordinate systems that fix the terminal coordinate system within camera terminal 1A and display terminal 1B under their respective reference states, and change according to the position and orientation of camera terminal 1A and display terminal 1B. The following superscript characters will be used to indicate which coordinate system the position representation is based on.

[0092] Furthermore, the following uses the position of the origin of the coordinate system, the rotation angle of the coordinate system from the horizontal direction, and the distance L between the object being photographed or its image and the terminal position to represent the positional relationship in each coordinate system. (X...) (0) C Y (0) C )=x (0) C To represent the position of the origin of the camera coordinate system, use ω. C ω represents the rotation angle of the camera coordinate system. C The camera rotation angle is 435 degrees. Figure 7A Here, x (0) C This is a vector representation of the position coordinates, and the same applies below. Furthermore, the position of the origin of the display coordinate system will be set to (X... (0) D Y (0) D )=x (0) D The rotation angle of the display coordinate system will be set to ω. D ω D This displays a rotation angle of 535 degrees. Here, the rotation angle with respect to the coordinate system is defined with counterclockwise as positive, based on the reference coordinate system. R(ω) is defined as a matrix representing the rotation angle ω (with counterclockwise as positive). The position representation will be shown as a vector representation below.

[0093] First, find the point x on the displayed image. (D) P Point x on the corresponding reference image (0) P When, it becomes the following formula (1).

[0094] x (0) P =(L D / L0)R(ω D )x (D) P +x (0) D …(1)

[0095] Here, (L) D / L0) represents the size ratio of the appearance caused by the difference in distance L.

[0096] Next, we calculate the point x on the reference image. (0) P The corresponding point x on the camera image (C) P When, it becomes the following formula (2).

[0097] x (C) P =(L0 / L C )R(-ω C (x) (0) P -x (0) C (2)

[0098] Therefore, the point x on the displayed image is calculated. (D) P The corresponding point x on the camera image (C) When P, it becomes the following equation (3).

[0099] x (C) P =(L D / L C )R(-ω C )x (D) P +(L0 / L C )R(-ω C (x) (0) D -x (0) C (3)

[0100] Depending on the location of the terminals, there may also be a corresponding point x. (C) P The situation where the camera is not within the field of view of the camera at camera terminal 1A. In camera terminal 1A, the camera reference direction, L0, and L2 are acquired as the first correction information. C ω C x (0) C They are then sent to display terminal 1B along with the camera image information. In display terminal 1B, L, which serves as the second calibration information, is obtained. D ω D x (0) D It is then combined with information sent from camera terminal 1A to create a display image.

[0101] Next, the method for measuring the data used to obtain the first calibration information is summarized. When the camera terminal 1A is in the camera reference state 400, the distance between the camera reference point 410 and the center point 470 of the object is measured by the distance sensor 113A as L0( Figure 5A When the camera terminal 1A is recording, the position and posture sensor 135A measures the rotation angle of the camera of the camera terminal 1A around the lens optical axis from the recording reference state 400, i.e., the recording rotation angle 435, as ω. C The following explains L. C x (0) C The determination.

[0102] First, regarding L C The distance is determined based on the frontal direction 430 of the camera terminal 1A. While it could also be measured as the distance between the camera terminal position 411 and the frontal point 480, there is a possibility of unexpected errors due to the frontal point 480 being located on a protrusion of the object 7 being photographed. Therefore, the origin of the camera coordinate system is defined as the point where the frontal direction 430 of the camera terminal intersects with a plane including the center point 470 of the object (which serves as a reference) and perpendicular to the camera reference direction 420. Furthermore, the distance between the origin of this determined camera coordinate system and the camera terminal position 411 is defined as L. C Regarding x (0) C This is also used as the reference coordinate value in the origin of the camera coordinate system to calculate the coordinates.

[0103] Next, let's explain L. C x (0) C The specific decision-making process ( Figure 7A , Figure 11 First, the position and posture sensor 135A measures the displacement r of the camera terminal position 411 from the camera reference point 410 at the time of the recording. C Simultaneously, the rotation of the front direction 430 of the camera terminal 1A from the camera reference direction 420 is measured, and the unit vector u of the front direction 430 of the camera terminal is calculated. C Here, the unit vector of the camera reference direction 420 is set to u0. When the camera terminal 1A is in camera reference state 400, L0 and u0 are measured. Figure 11 In this study, we consider setting the two-dimensional reference coordinate system as a three-dimensional coordinate system on the XY plane. The reference coordinate system plane is perpendicular to the camera reference direction by 42°, and the X-axis direction of the reference coordinate system is set to the horizontal direction in actual space. Then, we set the x... (0) C When interpreted as a vector in the three-dimensional coordinate system defined here, it is as shown in equation (4) below.

[0104] x (0) C =r C +L C u C -L0u0…(4)x (0) C Since it is orthogonal to u0, the following equation (5) holds true.

[0105] (r C +L C u C -L0u0)·u0=0…(5)

[0106] Here, "·" represents the inner product.

[0107] Solve equation (5) as shown in equation (6) to obtain L. C .

[0108] L C =(L0-r C ·u0) / (u C ·u0)…(6)

[0109] Substituting equation (6) into equation (5) yields equation (7).

[0110] x (0) C =r C +{(L0-r C ·u0) / (u C ·u0)}u C -L0u0…(7)

[0111] Based on the above, we can determine the relationship between L0, u0, and u. C r C Calculate L C and x (0) C Therefore, the information measured by the camera terminal 1A and sent to the display terminal 1B as the first calibration information can be L0, u0, u C r C ω C .

[0112] Next, the second calibration information L obtained from the display terminal 1B will be explained. D x (0) D The same inference can be made on the display terminal 1B side. Figure 8A , Figure 12However, the direction vectors u0 and L0 from the display reference point 510 to the object center point 570 are used, based on information sent from the camera terminal 1A. In the actual space where the display terminal 1B exists, the coordinate system is set as follows: the display reference point 510 is set as the terminal position that makes the display terminal 1B the display reference state 500, and the terminal position when in a terminal posture; and the vector from this display reference point 510 toward the object center point 570 is set as L0u0. In the display terminal 1B, the object center point 570 is the position where the object center point 470 of the camera object 7 appears to be present.

[0113] In display terminal 1B, the position and posture sensor 135B first measures the displacement r of the display terminal position from the display reference point 510 at the display time point. D Simultaneously, the rotation of the front direction 530 of the display terminal 1B from the display reference direction 520 is measured, and the unit vector u of the front direction 530 of the display terminal 1B is calculated. D The following equations (8) and (9) are solved by the same reasoning as that used for camera terminal 1A.

[0114] L D =(L0-r D ·u0) / (u D ·u0)…(8)

[0115] x (0) D =r D +{(L0-r D ·u0) / (u D ·u0)}u D -L0u0…(9)

[0116] Based on the above, the first calibration information measured by display terminal 1B can be u D r D ω D .

[0117] Furthermore, if the front orientation 430 of the camera terminal 1A differs from the camera reference orientation 420, or if the front orientation 530 of the display terminal 1B differs from the display reference orientation 520, trapezoidal distortion may occur in the image. However, this trapezoidal distortion can be corrected for display. Alternatively, other image distortion correction techniques can also be used.

[0118] Furthermore, in order to synchronize the image information and the first calibration information, timestamps can be added to the image information and the first calibration information.

[0119] Figure 13This is a flowchart illustrating the processing flow of the image sharing system according to the first embodiment.

[0120] First, user 2A puts on camera terminal 1A and starts it up. During startup and when updating the camera reference state 400 in the image sharing state, the branch decision in S01 branches to "Yes" and proceeds to processing in S02. Otherwise, the branch determines "No" and proceeds to processing in S03.

[0121] The processor 125A of the camera terminal 1A sets the camera reference point 410 and the camera reference direction 420 (u0) according to the input operation from the user 2A. After this setting, while the camera terminal 1A is kept in the camera reference state 400, the distance 415 (L0) of the camera object is measured by the range sensor 113A as the first correction information, and the camera reference direction 420 (u0) is measured by the position and posture sensor 135A (S02).

[0122] In the camera terminal 1A, the position and posture sensor 135A is used as calibration information to measure the displacement (r) of the camera terminal position 411 from the camera reference point 410. C ), the front view of the camera terminal 430 (u C ), camera rotation angle 435 (ω) C (S03).

[0123] The camera terminal 1A uses the external camera 111A to capture images of the field of view including the user 2A (S04).

[0124] Camera terminal 1A sends image information and first correction information to display terminal 1B (S05). Until the end condition is met in camera terminal 1A (S06: "No"), it returns to S01 and repeats the process. When the end condition is met (S06: "Yes"), the processing of camera terminal 1A ends. The "end condition" referred to here includes, for example, the operation of turning off camera terminal 1A or the termination action of the image sharing application.

[0125] On the other hand, user 2B wears display terminal 1B and starts it up. Display terminal 1B receives image information and correction information from camera terminal 1A (S11).

[0126] When the display reference state 500 is updated during startup and in the image sharing state, the branch in S12 branches to "Yes" and proceeds to the processing in S13. Otherwise, the branch branches to "No" and proceeds to the processing in S14.

[0127] The processor 125B of the display terminal 1B sets the display reference direction 520 according to the input operation from the user 2B (S13).

[0128] In display terminal 1B, the displacement (r) of display terminal position 511 from display reference point 510 is measured by position and posture sensor 135B as second correction information. D ), display terminal front direction 530 (u D ), display rotation angle 535 (ω) D (S14).

[0129] The processor 125B of the display terminal 1B performs correction on the received image information using the first correction information measured by the camera terminal 1A and the terminal position and posture information of the display terminal 1B measured in step S13, and displays it on the display 119B (S15).

[0130] The process continues until the termination condition is met in display terminal 1B (S16: "No"), then returns to S11 and repeats the process. When the termination condition is met (S16: "Yes"), the processing of display terminal 1B ends.

[0131] In this embodiment, the camera terminal 1A determines the camera reference point 410 and the camera reference direction 420 in the camera space where the camera terminal 1A exists, based on the setting of the camera reference state 400. Furthermore, the display terminal 1B determines the display reference point 510 and the display reference direction 520 in the display space where the display terminal 1B exists, based on the setting of the display reference state 500. The display terminal 1B displays the camera image 440 in such a manner that the position of the camera object 7, which uses the camera reference point 410 and the camera reference direction 420 as references, is the same as the position of the appearance of the camera object image 8, which uses the display reference point 510 and the display reference direction 520 as references.

[0132] Therefore, even when using a small, portable display terminal 1B, the user 2B (e.g., a work support staff member) of the display terminal 1B can view the image in the direction that the user 2B wants to view, near the direction that the user 2A (e.g., a worker) is looking, by naturally turning the display terminal 1B toward that direction, without being affected by the movement of the camera terminal 1A.

[0133] The following are also possible variations of the first embodiment.

[0134] (Adjustment of baseline state 500)

[0135] The basic idea behind setting the display reference state 500 is that when user 2B is in a position and posture conducive to operation, they can see the same image as in the camera reference state 400. If the orientation of display terminal 1B is changed from this state, user 2B can freely view the portion they wish to see. However, depending on the situation, there may be a desire to view the image 8 of the camera object from a location significantly different from the image in the camera reference state 400, or from a different angle. In such cases, the display terminal position 511 and the front orientation 530 of the display terminal when user 2B is in a position and posture conducive to operation can be offset from the display reference state 500. That is, in image sharing, the relative relationships of the display terminal position 511, the front orientation 530 of the display terminal, the display reference point 510 of the display reference state 500, and the display reference direction 520 can be appropriately changed according to user 2B's instructions, or these relative relationships can be returned to their original states. The same applies to the case where the reference for the display rotation angle 535 is offset from the horizontal direction.

[0136] This increases the freedom of how users 2B view images on display terminal 1B.

[0137] (Camera reference state)

[0138] In the first embodiment, for example, the camera reference state 400 is updated upon receiving an instruction from user 2A as the operation progresses. In this case, in order to smooth out the changes in the displayed image that accompany the changes in the camera reference point 410 and the camera reference direction 420, the changes in the camera reference point 410 and the camera reference direction 420 may be made continuously or little by little.

[0139] Furthermore, as a variation, the camera reference state 400 can be set according to the average position and posture of the camera terminal 1A. That is, the average position of the camera terminal position 411 can be set as the camera reference point 410, and the average direction of the front direction 530 of the camera terminal can be set as the camera reference direction 420.

[0140] As a result, the visibility of the displayed image for user 2B on display terminal 1B is improved.

[0141] (Handling distributed loads)

[0142] In order to distribute the processing load of camera terminal 1A and display terminal 1B globally, camera terminal 1A and display terminal 1B can also be paired with terminals such as smartphones 5A and 5B, or servers for processing.

[0143] (Selfie Mode)

[0144] In the case where the camera terminal 1A is a camera terminal capable of recording the user 2A's own image, such as a smartphone, the object being recorded 7 can also be the user 2A. In this case, it is also possible to view images that are not affected by the movement of the terminal. In selfie mode, the part that was referred to as external camera 111A in the above description is renamed internal camera 212A.

[0145] (Mutual Mode)

[0146] Alternatively, a portable information terminal can process the video feed from camera terminal 1A and display terminal 1B, allowing them to mutually display the video images from each other's portable information terminals. In this interactive mode, they can simultaneously observe each other's surroundings and engage in conversation, thus deepening their communication. Furthermore, when combined with the selfie mode, it can also achieve usage similar to a TV phone.

[0147] [Second Implementation]

[0148] This embodiment relates to an implementation of alternative images for areas that cannot be captured by a camera terminal.

[0149] The camera footage is recorded in the display terminal 1B or a server (not shown). If a missing area occurs in the displayed footage, the missing area is extracted from a past image and embedded into that past image for composite display. Even past images are convenient if there is relevant information. For example, even in past images, unchanging information such as descriptions of the photographed object can be used as a reference.

[0150] Figure 14 This is a diagram illustrating an example of a display image in the second embodiment.

[0151] like Figure 14 As shown, when the camera image range 490 does not cover the entire display area of ​​the display 119B of the display terminal 1B, the processor 125B of the display terminal 1B displays an interpolated camera image range 590 outside the camera image range 490. Regarding the interpolated image information, either image information obtained by the camera terminal 1A from pre-capturing the entire camera space can be used, or data of the camera space stored on a server (not shown) can be used.

[0152] Figure 15 This is a flowchart illustrating the process involved in the second embodiment. Furthermore, regarding... Figure 13 For flowcharts, add the same step number for the same steps and omit repeated descriptions.

[0153] The processing of the camera terminal 1A is the same as in the first embodiment.

[0154] When the display terminal 1B receives image information and first correction information (S11), it records the image (S21).

[0155] After measuring the second calibration information of the display terminal 1B (S14), only the newly acquired camera image range in the image information obtained in S11 is switched to the real-time image (S22), and the image is displayed (S15).

[0156] That is, when image information and correction information are received for the first time in step S11, the camera image range 490 contained in the received image information is displayed as display image 542. When new image information is received in step S12 after the second time through loop processing, if the newly acquired camera image range is the same as the interpolated past camera image range 590, the past camera image range 590 is replaced with the image range 490 contained in the new image information.

[0157] According to this embodiment, when the offset between the viewing area of ​​user 2A and the viewing area of ​​user 2B is large, and the non-display area is significant in the display terminal 1B, the image is displayed by interpolating the range of the overlay camera image. This allows the image displayed to user 2B to cover a wider area than the newly received camera image range 490. Therefore, without using a fixed, large display device, a camera terminal operated by a camera operator (e.g., for work-related situations) and a display terminal that receives and displays camera images transmitted from that camera terminal can cooperate to display an image covering a wider area than the camera image.

[0158] As another variation, when there are multiple camera terminals, multiple camera images can also be synthesized and displayed on a display terminal.

[0159] According to this embodiment, firstly, the camera terminal 1A determines the camera reference point 410 and camera reference direction 420 in the camera space where the camera terminal 1A exists, based on the setting of the camera reference state 400. Then, the display terminal 1B determines the display reference point 510 and display reference direction 520 under the display reference state 500 in the display space where the display terminal 1B exists. Furthermore, the display terminal 1B displays the camera image 440 in such a manner that the position using the camera reference point 410 and camera reference direction 420 of the object 7 as a reference is the same as the position of the appearance using the display reference point 510 and display reference direction 520 of the image 8 of the object 8 as a reference. Additionally, recorded past images are also synthesized and displayed. Therefore, even when using a small, portable display terminal 1B, the user 2B of the display terminal 1B can view images in a wide range of directions that the user 2B wants to view, near the viewing direction of the user 2A of the camera terminal 1A, by naturally turning the display terminal 1B toward that direction, without being affected by the movement of the camera terminal 1A.

[0160] The embodiments of the present invention have been described above, but the structure for implementing the technology of the present invention is obviously not limited to the above embodiments, and various modifications can be considered. For example, the above embodiments are detailed in detail for the purpose of easily understanding the present invention, and are not limited to having all the structures described. In addition, a part of the structure of a certain embodiment can be replaced by the structure of other embodiments, and it is also possible to add the structure of another embodiment to the structure of a certain embodiment. All of these fall within the scope of the present invention. Furthermore, the numerical values, messages, etc. appearing in the text and figures are merely examples, and using different parameters will not impair the effect of the present invention.

[0161] Furthermore, the programs described in each processing example can be either separate, independent programs, or multiple programs forming an application. Additionally, the order in which the processes are performed can be reversed.

[0162] Regarding the functions of the present invention described above, some or all of them can be implemented in hardware using, for example, integrated circuits. Alternatively, they can be implemented in software by having a microprocessor unit, CPU, or similar device interpret and execute operation programs to perform each function. Furthermore, the scope of software installation is not limited, and both hardware and software can be used. Additionally, some or all of the functions can be implemented using a server. Moreover, the server is not limited in form as long as it can cooperate with other structural components via communication to perform functions; for example, it can be a local server, cloud server, edge server, network service, etc. The programs, tables, files, and other information implementing the functions can be stored in recording devices such as memory, hard disks, SSDs (Solid State Drives), or recording media such as IC cards, SD cards, and DVDs, or stored in devices on a communication network.

[0163] Additionally, regarding the control lines and information lines shown in the diagram, only lines deemed necessary for explanation are shown, and not all control lines and information lines on the product are necessarily shown. It can also be assumed that almost all structures are actually interconnected.

[0164] The implementation methods include the following approaches.

[0165] (Postscript 1)

[0166] An image sharing system is an image sharing system that connects a camera terminal and a display terminal through communication.

[0167] The camera terminal has the following features:

[0168] Camera;

[0169] The first position and posture sensor detects the position and posture of the camera terminal;

[0170] The first processor; and

[0171] The first communication unit transmits and receives data with the display terminal.

[0172] The display terminal includes:

[0173] monitor;

[0174] The second position and posture sensor detects the position and posture of the display terminal;

[0175] Second processor; and

[0176] The second communication unit transmits and receives data with the camera terminal.

[0177] The first processor of the camera terminal:

[0178] In the actual space where the camera terminal exists, under the camera reference state, the position of the camera terminal is set as the camera reference point, and the direction from the front of the camera terminal to the camera reference point is set as the camera reference direction. The camera reference point and the camera reference direction are then sent to the display terminal.

[0179] Based on the sensor information from the first position and posture sensor, the change in the position of the camera terminal relative to the camera reference point and the amount of posture change of the camera terminal relative to the camera reference state are obtained when the camera is capturing the object. The change in the position and the amount of posture change of the camera terminal are used as first correction information and sent together with the video image information generated by the camera capturing the object to the display terminal.

[0180] The second processor of the display terminal:

[0181] In the display reference state set in the actual space where the display terminal exists, the position of the display terminal is set as the display reference point, and the direction from the display reference point to the front of the display terminal is set as the display reference direction;

[0182] Based on the sensor information of the second position posture sensor, the change in the position of the display terminal relative to the display reference point and the amount of posture change of the display terminal relative to the display reference state are obtained as second correction information;

[0183] The camera image information is corrected using the first correction information and the second correction information in a manner that makes the positional relationship of the camera object when the camera reference point and the camera reference direction are used consistent with the positional relationship of the image of the camera object when the display reference point and the display reference direction are used as references. The corrected camera image information is then displayed on the display.

[0184] (Postscript 2)

[0185] An image sharing method, which is an image sharing method executed by establishing a communication connection between a camera terminal and a display terminal, includes:

[0186] In the camera terminal, under the camera reference state set in the actual space where the camera terminal exists, the step of setting the position of the camera terminal as the camera reference point, and setting the direction from the front of the camera terminal towards the camera reference point as the camera reference direction;

[0187] The camera terminal includes the steps of obtaining the change in position of the camera terminal relative to the camera reference point when the camera object is being photographed, and the amount of change in posture of the camera terminal relative to the camera reference state;

[0188] In the camera terminal, the step of sending the change in the position of the camera terminal and the amount of change in the posture of the camera terminal as first correction information, together with the camera image information generated by photographing the object being photographed, to the display terminal;

[0189] In the display terminal, under a display reference state set in the actual space where the display terminal exists, the step of setting the position of the display terminal as a display reference point and setting the direction from the display reference point toward the front of the display terminal as the display reference direction;

[0190] In the display terminal, the steps of obtaining the change in the position of the display terminal relative to the display reference point and the amount of change in the posture of the display terminal relative to the display reference state as second correction information; and

[0191] The step involves using the first correction information and the second correction information to correct the camera image information in a manner that makes the positional relationship of the camera object when the camera reference point and the camera reference direction are used consistent with the positional relationship of the image of the camera object when the display reference point and the display reference direction are used as references, and then displaying the corrected camera image information on a display.

[0192] Explanation of symbols

[0193] 1A: Camera terminal; 1B: Display terminal; 2A: User; 2B: User; 3A: Access point; 3B: Access point; 4: Communication network; 5A: Smartphone; 5B: Smartphone; 7: Camera object; 8: Camera object image; 100: Image sharing system; 111A: External camera; 111B: External camera; 113A: Distance sensor; 113B: Distance sensor; 114A: RTC; 114B: RTC; 115A: Accelerometer; 115B: Accelerometer; 116A: Gyroscope sensor; 116B: Gyroscope sensor; 117A: Geomagnetic sensor; 117B: Geomagnetic sensor; 118A: Positioning sensor; 118B: Positioning sensor; 119 A: Display; 119B: Display; 120A: Network communication unit; 120B: Network communication unit; 121A: Microphone; 121B: Microphone; 122A: Speaker; 122B: Speaker; 123A: Antenna; 123B: Antenna; 125A: Processor; 125B: Processor; 126A: Program; 126B: Program; 127A: Data; 127B: Data; 128A: Memory; 128B: Memory; 135A: Position and posture sensor; 135B: Position and posture sensor; 136A: PP camera; 136B: PP camera; 137A: PP ranging sensor; 137B: PP ranging sensor; 140A: Bus; 140B: Bus; 21 1A: External camera; 211B: External camera; 212A: Internal camera; 212B: Internal camera; 213A: Distance sensor; 213B: Distance sensor; 214A: RTC; 214B: RTC; 215A: Accelerometer; 215B: Accelerometer; 216A: Gyroscope sensor; 216B: Gyroscope sensor; 217A: Geomagnetic sensor; 217B: Geomagnetic sensor; 218A: Positioning sensor; 218B: Positioning sensor; 219A: Display; 219B: Display; 220A: Network communication device; 220B: Network communication device; 221A: Microphone; 221B: Microphone; 222A: Speaker; 222B: Speaker; 223 A: Antenna; 223B: Antenna; 225A: Processor; 225B: Processor; 226A: Program; 226B: Program; 227A: Data; 227B: Data; 228A: Memory; 228B: Memory; 230A: Touch panel; 230B: Touch panel; 231A: Telephone network communication device; 231B: Telephone network communication device; 235A: Position and posture sensor; 235B: Position and posture sensor; 236A: PP camera; 236B: PP camera; 237A: PP ranging sensor; 237B: PP ranging sensor; 240A: Bus; 240B: Bus; 301: Position and posture measurement unit; 302: Image processing unit; 303: Transmission information generation unit;304: Communication control unit; 311: Communication control unit; 312: Position and posture measurement unit; 313: Image information storage unit; 314: Display image generation unit; 315: Display control unit; 400: Camera reference state; 410: Camera reference point; 411: Camera terminal position; 415: Camera object reference distance; 420: Camera reference direction; 430: Frontal direction; 435: Camera rotation angle; 440: Camera image; 441: Camera image; 450: Camera image center; 470: Object center point; 480: Camera front view point; 490: Camera image range; 500: Display reference status; 510: Display reference point; 511: Display terminal position; 520: Display reference direction; 530: Frontal direction; 535: Display rotation angle; 540: Display image; 541: Display image; 542: Display image; 550: Display image center; 570: Object center point; 580: Display frontal view point; 590: Camera image range; 600: Meter.

Claims

1. An image sharing system, comprising a camera terminal and a display terminal connected by communication, wherein, The camera terminal has the following features: Camera; The first position and posture sensor detects the position and posture of the camera terminal; The first processor; and The first communication unit transmits and receives data with the display terminal. The display terminal includes: monitor; The second position and posture sensor detects the position and posture of the display terminal; Second processor; and The second communication unit transmits and receives data with the camera terminal. The first processor of the camera terminal: In the actual space where the camera terminal exists, under the camera reference state, the position of the camera terminal is set as the camera reference point, and the direction from the front of the camera terminal to the camera reference point is set as the camera reference direction. The camera reference point and the camera reference direction are then sent to the display terminal. Based on the sensor information from the first position and posture sensor, the change in the position of the camera terminal relative to the camera reference point and the amount of posture change of the camera terminal relative to the camera reference state are obtained when the camera is capturing the object. The change in the position and the amount of posture change of the camera terminal are used as first correction information and sent together with the video image information generated by the camera capturing the object to the display terminal. The second processor of the display terminal: In the display reference state set in the actual space where the display terminal exists, the position of the display terminal is set as the display reference point, and the direction from the display reference point to the front of the display terminal is set as the display reference direction; Based on the sensor information of the second position posture sensor, the change in the position of the display terminal relative to the display reference point and the amount of posture change of the display terminal relative to the display reference state are obtained as second correction information; The camera image information is corrected using the first correction information and the second correction information in a manner that makes the positional relationship of the camera object when the camera reference point and the camera reference direction are used consistent with the positional relationship of the image of the camera object when the display reference point and the display reference direction are used as references. The corrected camera image information is then displayed on the display.

2. The image sharing system according to claim 1, wherein, The camera reference state and the display reference state are set according to user instructions.

3. The image sharing system according to claim 1, wherein, The camera reference state is set based on the average position and posture of the camera terminal.

4. The image sharing system according to claim 1, wherein, The image sharing system also includes an image storage unit for storing previously captured image information. If the corrected image range does not cover the entire display area of ​​the monitor, the second processor will interpolate the past image to the periphery of the image range for display.

5. The image sharing system according to claim 1, wherein, The camera terminal also features a wide-angle camera. The camera terminal sends the images captured by the wide-angle camera to the display terminal. If the corrected image range does not cover the entire display area of ​​the monitor, the second processor will interpolate the image captured by the wide-angle camera to the periphery of the image range for display.

6. An image sharing method, which is an image sharing method executed by establishing a communication connection between a camera terminal and a display terminal, comprising: In the camera terminal, under the camera reference state set in the actual space where the camera terminal exists, the step of setting the position of the camera terminal as the camera reference point, and setting the direction from the front of the camera terminal towards the camera reference point as the camera reference direction; The camera terminal includes the steps of obtaining the change in position of the camera terminal relative to the camera reference point when the camera object is being photographed, and the amount of change in posture of the camera terminal relative to the camera reference state; In the camera terminal, the step of sending the change in the position of the camera terminal and the amount of change in the posture of the camera terminal as first correction information, together with the camera image information generated by photographing the object being photographed, to the display terminal; In the display terminal, under a display reference state set in the actual space where the display terminal exists, the step of setting the position of the display terminal as a display reference point and setting the direction from the display reference point toward the front of the display terminal as the display reference direction; In the display terminal, the step of obtaining the change in the position of the display terminal relative to the display reference point and the amount of change in the posture of the display terminal relative to the display reference state as second correction information; as well as The step involves using the first correction information and the second correction information to correct the camera image information in a manner that makes the positional relationship of the camera object when the camera reference point and the camera reference direction are used consistent with the positional relationship of the image of the camera object when the display reference point and the display reference direction are used as references, and then displaying the corrected camera image information on a display.

Images