Electronic device and method for controlling electronic device

Abstract

To facilitate determination of a desirable angle of view and support imaging in a three-dimensional space.SOLUTION: An electronic apparatus for supporting imaging of an imaging device includes: generation means for generating a first object in a virtual space corresponding to a current imaging range of the imaging device and a second object in a virtual space corresponding to an imaging range when the imaging device is in a predetermined position and faces a predetermined direction; and display control means for displaying the first object and the second object.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to an electronic device and a method for controlling an electronic device. [Background technology]

[0002] Traditionally, cameras and cameras built into smartphones have been equipped with functions to assist in photography. Examples of functions to assist in photography include the display of grid lines and horizon lines on the live view (LV) screen, a peaking function that shows the user the in-focus part of the subject displayed on the LV screen, and a function that instructs the user on where to move the camera when taking panoramic shots. Furthermore, in recent years, technologies such as AR (Augmented Reality) and MR (Mixed Reality) have been used in devices such as head-mounted displays (HMDs), camera-equipped smartphones, and tablet terminals.

[0003] For example, Patent Document 1 discloses a device for supporting fixed-point photography as a technology for supporting angle-of-view determination. Specifically, the device disclosed in Patent Document 1 generates an AR object based on the camera position and shooting range when a photo was previously taken, and displays the generated AR object superimposed on the LV screen of the camera of the real object. This allows the user to move the camera of the real object so that it is superimposed on the camera of the AR object displayed on the LV screen, and further adjust the shooting range, thereby enabling the user to take a photo with the same angle of view as the previously taken photo.

[0004] Patent Document 2 discloses a device that supports panoramic photography. Specifically, the device according to Patent Document 2 displays a two-dimensional object on the LV screen that guides the optimal movement speed of the device for panoramic photography. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Publication No. 2020-167499 [Patent Document 2] Japanese Patent Application Laid-Open No. 2011-193286 Summary of the Invention [Problem to be solved by the invention]

[0006] However, even if the camera of the AR object is displayed on the LV screen of the camera of the real object, it may be difficult to accurately align the camera of the real object with the camera of the AR object. Also, if the shape of the camera of the AR object differs from the shape of the camera used for shooting, it is difficult to correctly align the position and orientation of the camera.

[0007] Furthermore, if a two-dimensional object that guides the movement speed is displayed on the LV screen, it can assist in moving the camera on a two-dimensional plane parallel to the imaging plane, but it is difficult to assist in shooting in three-dimensional space.

[0008] Therefore, an object of the present invention is to provide an electronic device that makes it easy to determine a suitable angle of view and supports shooting in a three-dimensional space. [Means for solving the problem]

[0009] In order to achieve the above object, the electronic device of the present invention comprises: An electronic device for supporting photography by an imaging device, a generation means for generating, in a virtual space, a first object indicating a current imaging range of the imaging device and a second object presenting the imaging range when the imaging device is at a predetermined position and in a predetermined orientation as a destination of the first object; a display control means for displaying the first object and the second object; With death, the generation means determines the predetermined position and the predetermined orientation based on changes between a past position and orientation of the imaging device and a current position and orientation of the imaging device, and generates the second object. It is characterized by: [Effects of the Invention]

[0010] According to the present invention, it is possible to easily determine a suitable angle of view and to support shooting in a three-dimensional space. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is a diagram illustrating an example of an external view of an imaging device. [Figure 2] FIG. 2 is a diagram illustrating an example of the appearance of an HMD. [Figure 3] FIG. 2 is a diagram illustrating an example of the hardware configuration of the imaging apparatus according to the first embodiment. [Figure 4] 4 is a flowchart illustrating a shooting support process according to the first embodiment. [Figure 5] FIG. 10 is a diagram illustrating trajectory information. [Figure 6] FIG. 10 is a diagram illustrating trajectory information. [Figure 7] 10 is a diagram illustrating an example of generation of an imaging object and a support object in the first embodiment. [Figure 8] 10 is a display example of an imaging object and a support object on an LV screen. [Figure 9] 10 is a flowchart illustrating a shooting support process according to a second embodiment. [Figure 10] 10 is a diagram illustrating an example of generation of an imaging object and a support object in the second embodiment. [Figure 11] FIG. 10 is a diagram illustrating an example of the hardware configuration of an imaging device and an HMD according to a third embodiment. [Figure 12] 10 is a flowchart illustrating a shooting support process according to a third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0012] The configurations of an imaging device and an HMD as electronic devices according to the present invention will be described with reference to Figures 1 and 2. The configurations described in Figures 1 and 2 are common to the following embodiments.

[0013] FIG. 1 is a diagram illustrating an example of the appearance of an imaging device. The imaging device 1 may be incorporated into an electronic device or configured as a separate device from the electronic device. FIGS. 1(A) and 1(B) show the front and back surfaces of the imaging device 1, respectively. The imaging device 1 is a device that has an imaging element, such as a camera, a smartphone, or a tablet terminal. The imaging device 1 has an imaging unit 11, a display unit 12, and an operation unit 13.

[0014] One or more imaging units 11 are mounted on at least one of the front and back surfaces of the imaging device 1. The display unit 12 is a display equipped with a touch panel, and displays various images. The touch panel equipped on the display unit 12 accepts operations by the user's finger or a stylus pen. The operation unit 13 is a member for inputting user operations, such as a button or dial. One or more operation units 13 may be mounted on the imaging device 1, or may be realized by a touch panel equipped on the display unit 12. The display unit 12 is an example of a "display control means."

[0015] FIG. 2 is a diagram illustrating an example of the appearance of the HMD 2. The HMD 2 illustrated in FIG. 2 is an optical see-through HMD. The HMD 2 is a head-mounted display device, also referred to as AR glasses. The HMD 2 has a frame and two display units 21a and 21b. The frame has a rim 25 and temples 26a and 26b joined to both sides of the rim 25. The display units 21a and 21b are joined to one surface of the rim 25.

[0016] Light from a display element (not shown) of display unit 21a is guided by image projection unit 22a and light guide unit 23a to the right eye of the wearer wearing HMD 2. Similarly, light from a display element (not shown) of display unit 21b is guided by image projection unit 22b and light guide unit 23b to the left eye of the wearer wearing HMD 2.

[0017] The wearer can simultaneously perceive the images displayed on the display units 21a and 21b and the light incident from in front of the HMD 2. The HMD 2 may also have two imaging units, 24a and 24b. The imaging units 24a and 24b capture the surroundings including the area in front of the HMD. Note that although the HMD 2 will be described as an optical see-through HMD using a transparent display, it may also be a video see-through HMD using an opaque display.

[0018] The imaging device 1 or HMD 2 according to each of the following embodiments can recognize the surrounding space using an on-board imaging unit or the like, and present various virtual objects that fit the surrounding space to the user as three-dimensional information. Information presentation using virtual space or virtual objects can present depth information that has been difficult for users to perceive until now. Virtual objects mapped onto real space can be used for various photography support means.

[0019] The first and second embodiments are embodiments for supporting shooting with the imaging device 1 or the HMD 2 when shooting using either the imaging device 1 or the HMD 2. The third embodiment is an embodiment for supporting shooting with the imaging device 1 by linking the imaging device 1 and the HMD 2.

[0020] First Embodiment The first embodiment is an embodiment for supporting movement of the imaging device 1 or the HMD 2 by presenting the movement destination in order to realize shooting with an optimal angle of view using either the imaging device 1 or the HMD 2. Note that the following description will be given for a case in which shooting with the imaging device 1 is supported.

[0021] 3 is a diagram showing an example of the hardware configuration of the imaging device 1 according to the first embodiment. The imaging device 1 has a calculation unit 101, an image processing unit 102, a primary storage unit 104, a secondary storage unit 105, a display unit 106, an imaging unit 107, an operation unit 108, and a three-dimensional space recognition unit 109. The components of the imaging device 1 are mutually connected via a bus.

[0022] The calculation unit 101 is, for example, a CPU (Central Processing Unit), and controls the entire imaging device 1. The image processing unit 102 generates objects (hereinafter, imaging objects) that indicate the imaging angle of view in the virtual space recognized by the three-dimensional space recognition unit 109, and support objects that support the user in taking pictures with the imaging device 1.

[0023] The imaging object is an AR object in a virtual space corresponding to the current imaging range of the imaging device. The support object is an AR object in a virtual space corresponding to the imaging range when the imaging device 1 is at a predetermined position and facing a predetermined direction. The support object is an AR object that presents an ideal destination for the imaging object so that the imaging device 1 can move with ideal camerawork. The imaging object corresponds to the "first object." The support object corresponds to the "second object."

[0024] The image processing unit 102 generates a support object so that the movement of the imaging device 1 is controlled by the user aligning the imaging object with the support object. The user can take pictures at a constant speed or along a regular trajectory by moving the imaging device 1 so that the imaging object overlaps the support object. The image processing unit 102 is an example of a "generation means."

[0025] The primary storage unit 104 is, for example, a dynamic random access memory (DRAM). The secondary storage unit 105 is a memory that temporarily stores image data. It is a flash memory that records captured images.

[0026] The display unit 106 is a display that displays captured images and other visual objects. The display unit 106 displays captured objects and support objects superimposed on an image captured of real space. The imaging unit 107 converts light into an electrical signal. The operation unit 108 accepts operations from the user. The display unit 106, the imaging unit 107, and the operation unit 108 correspond to the display unit 12, the imaging unit 11, and the operation unit 13 shown in FIG. 1(B), respectively.

[0027] The three-dimensional space recognition unit 109 detects the location of objects in real space and the distance to the objects, and acquires information about the virtual space. The three-dimensional space recognition unit 109 can recognize the real three-dimensional space using, for example, multiple cameras. The three-dimensional space recognition unit 109 is an example of an "acquisition means." Note that the processing of the three-dimensional space recognition unit 109 may be performed by the imaging unit 107.

[0028] 4 is a flowchart illustrating a shooting support process 400 according to the first embodiment. The shooting support process 400 is a process for supporting a user in shooting by generating an imaging object and a support object and displaying them on the LV screen. The imaging object and the support object are displayed superimposed on an image captured in real space. In the first embodiment, the support object is generated based on trajectory information indicating future movements of the imaging device 1, such as a moving speed or trajectory.

[0029] The photographing support process 400 is started, for example, when a user operates the operation unit 108 of the imaging device 1 and the imaging device 1 enters a photographing mode. The calculation unit 101 of the imaging device 1 executes a program for causing each component to perform the processing of each step of the photographing support process 400.

[0030] In S401, the image processing unit 102 acquires trajectory information to be used for generating a support object. The trajectory information is information indicating the future movement (camera work) of the shooting range of the captured image. The trajectory information includes information for determining the position, orientation (directional component from the position of the support object), movement speed, and trajectory of the support object in the virtual space.

[0031] The support object is an AR object in a virtual space that corresponds to the imaging range of the imaging device when the imaging device is at a predetermined position and a predetermined orientation. The image processing unit 102 can determine the predetermined position and a predetermined orientation based on the acquired trajectory information. The image processing unit 102 may acquire trajectory information stored in advance in the secondary storage unit 105, or may acquire trajectory information input by the user via the operation unit 108.

[0032] Here, the trajectory information will be described with reference to Figures 5(A) to 5(C), 6(A), and 6(B). The trajectory information is information indicating the movement of the imaging device 1, and is, for example, information indicating the movement of the imaging device 1 moving straight at a constant speed, or the movement of rotating the imaging device 1 or the user in a horizontal direction around the imaging device 1 or the user. The trajectory information may also be information indicating the movement of rotating the imaging device at a constant speed around the optical axis incident on the imaging unit 107 (the imaging axis of the imaging device 1), or the movement of rotating the imaging device around the subject.

[0033] 5(A) shows an example in which a person (user taking a photo) moves straight ahead at a constant speed. When the user aligns the imaging object 510 with the support object 511, the display unit 106 displays the support object 512 ahead of the support object 511. Furthermore, when the user aligns the imaging object 510 with the support object 512, the display unit 106 displays the support object 513 ahead of the support object 512.

[0034] 5(B) shows an example in which a person moves laterally at a constant speed. When the user aligns the imaging object 520 with the support object 521, the display unit 106 displays the support object 522 laterally of the support object 521.

[0035] 5(C) shows an example in which the imaging device 1 is rotated horizontally at a constant speed around the imaging device 1. When the user aligns the imaging object 530 with the support object 531, the display unit 106 displays the support object 532 in the direction in which the imaging device 1 is rotating. Furthermore, when the user aligns the imaging object 530 with the support object 532, the display unit 106 displays the support object 533 in the direction in which the imaging device 1 is rotating.

[0036] 6(A) shows an example in which the imaging device 1 is rotated at a constant speed around the imaging axis of the imaging device 1. When the user aligns the imaging object 610 with the support object 611, the display unit 106 displays a support object 612 in the direction in which the imaging device 1 rotates around the imaging axis. Furthermore, when the user aligns the imaging object 610 with the support object 612, the display unit 106 displays a support object 613 in the direction in which the imaging device 1 rotates around the imaging axis.

[0037] 6(B) shows an example in which the imaging device 1 is rotated at a constant speed around the subject. When the user aligns the imaging object 620 with the support object 621, the display unit 106 displays the support object 622 in the direction in which the imaging device 1 rotates around the subject.

[0038] The support objects may be displayed one by one in accordance with the movement of the imaging device 1 based on the trajectory information. Also, multiple support objects may be displayed with predetermined positions and predetermined orientations that are the positions and orientations of the imaging device 1 at multiple future points in time based on the trajectory information. Also, a support object that was previously displayed may be hidden or may remain displayed when the difference in position or orientation between the imaging object and the previously displayed support object becomes equal to or greater than a threshold.

[0039] The trajectory information is not limited to the speed or trajectory of the imaging device 1, but may also be information indicating a movement to place the subject at a predetermined position such as the center or edge of the imaging range, or a movement to fit the front of the subject into the imaging range.

[0040] In S402 of FIG. 4, the three-dimensional space recognition unit 109 acquires the arrangement of objects in real space and the distance from the image capture device 1 to the objects as virtual space information, and stores it in the primary storage unit 104.

[0041] In S403, the image processing unit 102 generates an imaging object indicating the imaging range captured by the imaging unit 107 in the virtual space recognized by the three-dimensional space recognition unit 109. The image processing unit 102 adds information about the generated imaging object to the virtual space information held in the primary storage unit 104. The imaging object is an AR object indicating a position and orientation linked to the imaging range of the imaging device 1.

[0042] The image processing unit 102 uses, for example, template matching, to search for a portion in the virtual space information stored in the primary storage unit 104 that matches the captured image captured by the imaging device 1. The image processing unit 102 can generate an imaging object based on the position and orientation of the captured image found in the virtual space.

[0043] The shape of the imaging object can be a shape whose position and orientation can be uniquely determined, such as a frame, a rectangular parallelepiped, a cursor, points indicating the four corners of the imaging range, etc. The position of the imaging object in the depth direction can be, for example, the position of the focal length at which the image is best in focus.

[0044] The position of the imaging object in the depth direction may be set to a position at a predetermined distance from the imaging device 1, and the size of the imaging object may be set to be variable. For example, if the imaging object is shaped like a frame, the distance from the imaging device 1 to the frame may be fixed to a predetermined distance of 1 meter, and the size of the frame may be variable. Note that the distance from the imaging device 1 to the frame may be set to, for example, the distance from the imaging device 1 to the center of the frame.

[0045] Furthermore, the position of the imaging object in the depth direction may be set so that the size of the imaging object is a predetermined size and the distance from the imaging device 1 to the imaging object is variable. For example, if the shape of the imaging object is a frame, the size of the frame can be fixed to a predetermined size of 30 cm as the diagonal length in real space, and the distance from the imaging device 1 to the frame can be variable.

[0046] In the case of a narrow space where an obstacle is present in front of the imaging device 1, the image processing unit 102 may generate an imaging object so that the imaging object is displayed at a position closer to the imaging device 1 than the obstacle. The image processing unit 102 can change the display position of the imaging object depending on the surrounding space.

[0047] In the above example, the position and orientation of the imaging object are acquired based on the imaging range captured by the imaging unit 107. The position and orientation of the imaging object can also be acquired from the position and orientation of the imaging device 1 in real space.

[0048] In S404, the image processing unit 102 generates a support object, which is an AR object for supporting photography, based on the trajectory information acquired in S401.

[0049] The trajectory information is information that indicates the movement (camera work) of the imaging range of the captured image. For example, when the trajectory information is information that indicates the movement of the imaging device 1 moving straight at a constant speed, the image processing unit 102 generates a support object so that the current imaging object is displayed at a position where the imaging object moves straight in the direction of the imaging object. Specifically, the image processing unit 102 sets the orientation of the imaging device 1 to a predetermined direction, and generates the support object by setting the position where the imaging device 1 moves straight in the predetermined direction at a constant speed for one second as the predetermined position.

[0050] The image processing unit 102 can generate a support object at a position to which the current imaging object has been moved in accordance with the movement indicated by the trajectory information. The image processing unit 102 may generate the support object so that the relative relationship between a predetermined position of the imaging device 1 and the position of the support object is equal to the relative relationship between the current position of the imaging device 1 and the position of the imaging object. The image processing unit 102 adds information about the generated support object to the virtual space information held in the primary storage unit 104.

[0051] 7 is an example of generating an imaging object and a support object, and schematically shows how information about an imaging object 702 and information about a support object 703 are added to information about a virtual space 700.

[0052] 4, the display unit 106 superimposes the captured image of the real space with the captured object and the support object stored in the primary storage unit 104 and displays the superimposed image on the LV screen. The superimposition process may be performed by the image processing unit 102.

[0053] In the superimposition process, the display unit 106 first acquires which part of the virtual space 700 corresponds to the image of the real space captured by the imaging unit 107. Next, the display unit 106 displays the imaging object 702 and the support object 703 present in the virtual space 700 on the real space. The captured image is superimposed at the corresponding position.

[0054] In S406, the calculation unit 101 determines whether or not the user has instructed the operation unit 108 to end the shooting mode. If the user has instructed the operation unit 108 to end the shooting mode (S406: YES), the shooting support process 400 ends. If the user has not instructed the operation unit 108 to end the shooting mode (S406: NO), the process returns to S402, and the processes from S402 to S406 are repeated.

[0055] The processes from S402 to S406 are executed for each frame of the moving image captured by the imaging unit 107. Note that the processes from S402 to S406 may be executed for each multiple frames (for example, five frames) as long as the support object is displayed to support the user in taking pictures.

[0056] Fig. 8 shows an example of displaying an imaging object and a support object on an LV screen. Fig. 8 shows an example of displaying an imaging object and a support object superimposed on an image captured in real space on an LV screen 800 (display unit 106).

[0057] 8(A) shows an example in which a frame-shaped imaging object 801 and a frame-shaped support object 802 are present at the same depth and in the same direction on an LV screen 800. The user can achieve an appropriate imaging angle of view or camerawork by moving the imaging device 1 so that the imaging object 801, which moves in conjunction with the imaging device 1, is aligned with the support object 802.

[0058] 8(B) shows an example in which a frame-shaped imaging object 803 and a frame-shaped support object 804 are present at different depths and in different directions on an LV screen 800. The user can achieve an appropriate imaging angle of view or camerawork by moving the imaging device 1 so that the imaging object 803, which moves in conjunction with the imaging device 1, is aligned with the support object 804.

[0059] 8(C) shows an example in which a rectangular parallelepiped imaging object 805 and a rectangular parallelepiped support object 806 are present at different depths and in different orientations on an LV screen 800. The user can achieve an appropriate imaging angle of view or camerawork by moving the imaging device 1 so that the imaging object 805, which moves in conjunction with the imaging device 1, is aligned with the support object 806.

[0060] In the above example, the case where the imaging device 1 assists the user in taking pictures has been described, but this embodiment may also be realized by an HMD 2 having a configuration similar to that of the imaging device 1 shown in FIG.

[0061] The support object may be displayed constantly while shooting in the imaging mode, or may be displayed when a predetermined condition is met. The predetermined condition may be, for example, that either the difference in position or the difference in orientation between the imaging object and the support object is equal to or greater than a threshold. The predetermined condition may also be that the moving speed of the imaging device 1 is not constant, or that the moving trajectory of the imaging device 1 is irregular.

[0062] The display unit 106 can notify the user that the imaging device 1 is not moving according to the trajectory information by temporarily displaying the support object when a predetermined condition is met. When the predetermined condition is met and the imaging device 1 is moving according to the trajectory information, the display unit 106 may stop displaying the support object.

[0063] The imaging object and the support object are input to the image processing unit 102 of the imaging device 1. However, the generation process may be requested from an external device on the cloud. In this case, the imaging device 1 has a communication unit capable of communicating with the cloud and transmits information used to generate the imaging object and the support object to the cloud. The information transmitted to the cloud includes, for example, images captured by the imaging device 1, trajectory information indicating the movement of the imaging device 1, and virtual space information.

[0064] The imaging device 1 receives (acquires) imaging objects and support objects generated on the cloud, and can display the real space on the LV screen by superimposing them on the captured image. By having an external device generate AR content such as imaging objects and support objects, the imaging device 1 does not need to be equipped with a high-performance, large-scale SoC, and can be made smaller.

[0065] According to the above-mentioned imaging device 1 or HMD 2, the user can easily determine a suitable angle of view or achieve suitable camera work by moving the imaging device 1 or HMD 2 so as to align the imaging object displayed on the LV screen with the support object.

[0066] Second Embodiment In the first embodiment, the image processing unit 102 generates a support object based on the trajectory information and the current imaging range, independently of the movement of the imaging object. In contrast, in the second embodiment, a support object is generated by reflecting the movement of the imaging object in real time. Therefore, the imaging device 1 can generate an optimal support object taking the movement of the user into consideration.

[0067] The hardware configuration of the imaging device 1 according to the second embodiment is similar to the hardware configuration of the imaging device 1 according to the first embodiment shown in Fig. 3, and therefore description thereof will be omitted. Note that, similar to the first embodiment, the second embodiment may be realized by an HMD 2 having a configuration similar to the configuration of the imaging device 1 shown in Fig. 3.

[0068] 9 is a flowchart illustrating a photographing support process 900 according to the second embodiment. The photographing support process 900 is a process for generating a support object based on the movement of an imaging object.

[0069] Of the processing steps in the shooting support processing 900, the same processing as in the shooting support processing 400 of the first embodiment shown in Fig. 4 is denoted by the same reference numerals, and detailed description thereof will be omitted. The processing of S901, S902, and S903, which differs from the shooting support processing 400 of the first embodiment, will be described.

[0070] The photographing support process 900 is started, for example, when a user operates the operation unit 108 of the imaging device 1 and the imaging device 1 enters a photographing mode. The calculation unit 101 of the imaging device 1 executes a program for causing each component to perform the processing of each step of the photographing support process 400.

[0071] As in the first embodiment, in S402, the three-dimensional space recognition unit 109 acquires virtual space information. In S403, the image processing unit 102 generates an imaging object indicating the imaging range captured by the imaging unit 107 in the virtual space.

[0072] In S901, it is determined whether or not it is the first loop. In the second embodiment, the image processing unit 102 generates a support object based on the movements of a plurality of imaging objects. Since only one imaging object is generated in the first loop, the image processing unit 102 generates a further imaging object.

[0073] If it is the first loop (S901: YES), the process returns to S402. The processes of S402 and S403 are repeated, and the image processing unit 102 can generate multiple imaging objects with a time difference. If it is the second loop or later (S901: NO), the process proceeds to S902.

[0074] In S902, the image processing unit 102 generates a support object, which is an AR object for supporting photography, from the two imaging objects generated for the current frame and the previous frame. Information about the generated support object is added to the virtual space information held in the primary storage unit 104.

[0075] Fig. 10 is an example of generation of an imaging object and a support object. Fig. 10 schematically shows a state in which information on an imaging object 1001 of a previous frame, an imaging object 1002 of a current frame, and a support object 1003 is added to information on a virtual space 1000. Note that the previous frame is not limited to the frame immediately before the current frame, and may be a frame several frames before the current frame.

[0076] Specifically, the support object can be generated by the following method: The image processing unit 102 acquires the amount of change in the position, orientation, moving speed, etc. when the imaging object 1001 of the previous frame moves to the imaging object 1002 of the current frame in the virtual space 1000. The amount of change is the amount of movement, rotation, acceleration, etc. from the imaging object 1001 to the imaging object 1002.

[0077] Based on the acquired amount of change, the image processing unit 102 estimates the position and orientation of the imaging object in the next frame as the position and orientation of the support object 1003. The image processing unit 102 adds information about the estimated support object 1003 to information about the virtual space 1000.

[0078] The image processing unit 102 can determine a predetermined position and a predetermined orientation of the imaging device 1 based on changes between the past position and orientation of the imaging object 1001 and the current position and orientation of the imaging object 1002, and generate a support object.

[0079] In S903, the display unit 106 superimposes the captured object of the current frame and the estimated support object on the captured image of the real space and displays the superimposed image on the LV screen. The superimposition process may be performed by the image processing unit 102. The process of S406 is similar to the process in the first embodiment shown in FIG.

[0080] The image processing unit 102 can generate imaging objects in a virtual space corresponding to the imaging range when the imaging device 1 has a past position and orientation. The display unit 106 may display the past imaging objects on the LV screen together with the support object and the current imaging object. The past imaging objects correspond to the "third object."

[0081] In the above example, the case where the imaging device 1 assists the user in taking pictures has been described, but this embodiment may also be realized by an HMD 2 having a configuration similar to that of the imaging device 1 shown in FIG.

[0082] The support object may be displayed so as to move in real time while shooting in the imaging mode, or may be displayed when a predetermined condition is met. The predetermined condition may be, for example, that the moving speed of the imaging device 1 is not constant, or that the trajectory of the imaging device 1 is not regular. By temporarily displaying the support object when the predetermined condition is met, the display unit 106 can make it appear as if the imaging device 1 is moving at a constant speed. It is possible to notify the user that the device has not been properly installed.

[0083] Furthermore, the imaging objects and the support objects are generated by the image processing unit 102 of the imaging device 1, but may also be generated on the cloud. The imaging device 1 can receive (acquire) the imaging objects and the support objects generated on the cloud and display them superimposed on the LV screen.

[0084] Furthermore, although an example has been described in which the support object is generated based on the amount of change in the imaging object, the support object may also be generated based on the amount of change in the position, orientation, movement speed, etc. of the imaging device 1 when capturing the previous frame and the current frame. The image processing unit 102 can determine a predetermined position and orientation of the imaging device 1 based on changes between the past position and orientation of the imaging device 1 and the current position and orientation of the imaging device 1, and generate a support object.

[0085] The imaging device 1 or HMD 2 can generate a support object for determining a suitable angle of view in consideration of the user's movement. By moving the imaging device 1 or HMD 2 so that the imaging object matches the support object, the user can move at a constant speed in accordance with the user's own movement or rotate at a constant angular velocity.

[0086] <Third embodiment> The first and second embodiments are embodiments that assist the user in taking pictures using either the imaging device 1 or the HMD 2. In contrast, the third embodiment is an embodiment that uses the imaging device 1 and the HMD 2 in cooperation with each other. In the third embodiment, it is possible to utilize the advantages of both the imaging device 1, which is specialized in imaging functions, and the HMD 2, which can display AR objects over a wide range.

[0087] 11 is a diagram showing an example of the hardware configuration of an imaging device 11 and an HMD 2 according to the third embodiment. Similar to the imaging device 1 according to the first embodiment, the imaging device 11 according to the third embodiment has a calculation unit 101, an image processing unit 102, a primary storage unit 104, a secondary storage unit 105, an imaging unit 107, and an operation unit 108. The imaging device 11 further has a communication unit 1103. The components of the imaging device 11 are connected to each other via a bus. Components that are the same as those of the imaging device 1 are given the same reference numerals, and descriptions thereof will be omitted.

[0088] The HMD2 according to the third embodiment has, like the imaging device 1 according to the first embodiment, a calculation unit 201, an image processing unit 202, a primary storage unit 204, a secondary storage unit 205, a display unit 206, an operation unit 208, and a three-dimensional space recognition unit 209. The HMD2 also has a communication unit 203. The components of the HMD2 are connected to each other via a bus. Components with the same names as those of the imaging device 1 function in the same way as the corresponding components, so detailed explanations will be omitted. The display unit 206 corresponds to the display unit 21a and the display unit 21b of the HMD2. The display unit 206 is an example of a "display control means."

[0089] The imaging device 11 and the HMD 2 communicate with each other via the communication units 1103 and 203. The imaging device 11 transmits a captured image captured by the imaging unit 107 to the HMD 2 via the communication units 1103 and 203.

[0090] 12 is a flowchart illustrating the shooting support processing according to the third embodiment. The shooting support processing according to the third embodiment is processing in which the imaging device 11 and the HMD 2 are used in cooperation with each other. The shooting support processing according to the third embodiment includes shooting processing 1200a by the imaging device 11 and shooting support processing 1200b by the HMD 2.

[0091] The photographing process 1200a of the imaging device 11 is started when the user operates the operation unit 108 to set the imaging mode. The photographing support process 1200b of the HMD 2 is started when the user operates the operation unit 208 to set the imaging mode. Note that the photographing support process 1200b may be started when the imaging device 11 is set to the photographing mode or when a photographed image is received from the imaging device 11.

[0092] The calculation unit 101 of the imaging device 11 and the calculation unit 201 of the HMD 2 execute programs for causing each component to perform the processing of each step of the photographing process 1200a and the photographing support process 1200b, respectively.

[0093] In S1201b, the three-dimensional space recognition unit 209 of the HMD 2 acquires the location of an object in real space and the distance to the object as virtual space information, and stores the information in the primary storage unit 204. The three-dimensional space recognition unit 209 is an example of an "acquisition unit." Note that the virtual space information may be acquired after receiving the captured image from the imaging device 1 in S1202b.

[0094] In S1202a, the communication unit 103 of the imaging device 1 transmits the captured image captured by the imaging unit 107 to the HMD 2 via the communication unit 203. In S1202b, the communication unit 203 of the HMD 2 determines whether the captured image has been received from the communication unit 103 of the imaging device 1.

[0095] If a captured image has been received (S1202b: YES), the process proceeds to S1203b. If a captured image has not been received (S1202b: NO), the process of S1203b is repeated until a captured image is received.

[0096] In S1203b, the image processing unit 202 generates an imaging object in the virtual space, which indicates the imaging range captured by the imaging unit 107, using the captured image received from the imaging device 1. The image processing unit 202 adds information about the generated imaging object to the virtual space information held in the primary storage unit 204. The image processing unit 202 is an example of a "generation means."

[0097] In S1204b, the image processing unit 202 generates a support object in the same manner as S403 of the photographing support processing 400 according to the first embodiment or S902 of the photographing support processing 900 according to the second embodiment. The image processing unit 202 adds information about the generated support object to the virtual space information held in the primary storage unit 104.

[0098] In S1205b, the display unit 206 of the HMD 2 displays the captured image of the real space by superimposing the captured object and the supporting object on the captured image.

[0099] In S1206b, the calculation unit 201 determines whether or not the user has instructed the operation unit 208 to end the shooting mode. If the user has instructed the operation unit 208 to end the shooting mode (S1206b: YES), the shooting support process 1200b ends. If the user has not instructed the operation unit 208 to end the shooting mode (S1206b: NO), the process returns to S1201b, and the shooting support process 1200b is repeated.

[0100] As in S1206b, in S1206a, the calculation unit 101 determines whether or not the user has instructed the operation unit 108 to end the shooting mode. If an instruction to end the shooting mode has been issued (S1206a: YES), the shooting process 1200a ends. If an instruction to end the shooting mode has not been issued (S1206a: NO), the process returns to S1202a, and the shooting process 1200a is repeated.

[0101] Note that the photographing support process 1200b does not perform the termination determination in S1206b, and the photographing process 12 When the photographing support process 1200a is completed, the photographing support process 1200b may be terminated upon receiving a notification from the photographing device 11. Furthermore, the photographing support process 1200b may be terminated when no photographed image is received from the photographing device 11 for a predetermined time.

[0102] In addition, in S1203b of the photographing support process 1200b, the image processing unit 202 receives a photographed image from the imaging device 1 and acquires the position and orientation of the imaging object from the received photographed image, but this is not limiting. The image processing unit 202 may acquire the position and orientation of the imaging object by acquiring the position and orientation of the imaging device 11 from a photographed image photographed by an imaging unit included in the HMD 2.

[0103] Furthermore, the imaging objects and the support objects are generated by the image processing unit 202 of the HMD 2, but may also be generated on the cloud. In this case, the HMD 2 communicates with the cloud, receives the imaging objects and the support objects generated on the cloud, and can display them superimposed on the display unit 206 of the HMD 2. Furthermore, whether or not to link the imaging device 11 and the HMD 2 may be set in advance, or may be set when the user switches the imaging device 11 to the imaging mode.

[0104] The imaging device 11 and HMD 2 described above enable the user to take photographs with assistance that takes advantage of the advantages of the imaging device 11, which is specialized in imaging functions, and the HMD 2, which can display AR objects over a wide range.

[0105] <Other embodiments> The imaging object is not limited to an AR object that indicates the actual imaging range. The imaging object may be made part of the imaging range instead of being made to coincide with the imaging range. In this case, the support object may be made part of the imaging range of the ideal movement destination. The range of the support object may be determined so that the ideal camera work can be realized by the user aligning the imaging object with the support object. By making the imaging object part of the imaging range, the user can make the imaging object follow the support object with greater accuracy than when the entire imaging range is aligned with the support object.

[0106] Furthermore, the number of support objects displayed is not limited to one, and multiple support objects may be displayed. For example, the image processing unit 102 (image processing unit 202) can generate a second support object based on the amount of change in at least one of the position, orientation, and movement speed between the imaging object and the generated first support object. Similarly, the image processing unit 102 (image processing unit 202) can also generate three or more support objects. Displaying multiple support objects makes it easier for the user to predict the destination of the imaging device.

[0107] Furthermore, the predetermined position and predetermined orientation of the imaging device 1 for generating the support object may be externally provided. For example, the position and orientation of the imaging device 1 when an image was taken in the past (for example, several days ago) using the imaging device 1 may be provided as the predetermined position and orientation for generating the support object. This allows the shooting range of the previously captured image to be superimposed and displayed as the support object, allowing the user to capture an image at the same position and orientation as the previously captured image.

[0108] The above-described embodiments merely exemplify the configuration of the present invention, and the present invention is not limited to the specific embodiments described above, and various combinations and modifications are possible within the scope of the technical concept thereof.

[0109] The present invention provides a program that realizes one or more functions of the above-described embodiments, over a network or The program may be provided to a system or device via a storage medium, and one or more processors in the computer of the system or device may read and execute the program. The program may also be implemented by a circuit (e.g., ASIC) that implements one or more functions. [Explanation of symbols]

[0110] 1: imaging device, 101: calculation unit (CPU), 102: image processing unit, 106: display unit, 107: imaging unit, 109: three-dimensional space recognition means

Claims

1. An electronic device for supporting photography by an imaging device, a generation means for generating, in a virtual space, a first object indicating a current imaging range of the imaging device and a second object presenting the imaging range when the imaging device is at a predetermined position and in a predetermined orientation as a destination of the first object; a display control means for displaying the first object and the second object; and the generating means determines the predetermined position and the predetermined orientation based on changes between a past position and orientation of the imaging device and a current position and orientation of the imaging device, and generates the second object. An electronic device characterized by:

2. the display control means displays the first object and the second object by superimposing the first object and the second object on an image captured of a real space.

2. The electronic device according to claim 1, wherein the electronic device is a semiconductor device.

3. the generation means further generates a third object in the virtual space corresponding to an imaging range when the imaging device has the past position and orientation; the display control means further displays the third object.

3. The electronic device according to claim 1, wherein the first and second electrodes are electrically connected to the first and second electrodes.

4. The present invention further includes an acquisition means for acquiring information about the virtual space by recognizing the location of an object in the real space and the distance to the object.

4. The electronic device according to claim 1, wherein the first and second electrodes are electrically connected to the first and second electrodes.

5. the generating means generates the first object so that the position of the first object is at a position at a focal length of the imaging device, a position at a predetermined distance from the imaging device, or a position where the size of the first object is a predetermined size.

5. The electronic device according to claim 1, wherein the first and second electrodes are electrically connected to the first and second electrodes.

6. the generating means generates the second object so that a relative relationship between the predetermined position and a position of the second object is equal to a relative relationship between a current position of the imaging device and a position of the first object.

6. The electronic device according to claim 1, wherein the first and second electrodes are electrically connected to the first and second electrodes.

7. An electronic device for supporting photography with an imaging device, a generation means for generating, in a virtual space, a first object indicating a current imaging range of the imaging device and a second object presenting the imaging range when the imaging device is at a predetermined position and in a predetermined orientation as a destination of the first object; a display control means for displaying the first object and the second object; and the display control means displays the second object when either a difference in position or a difference in orientation between the first object and the second object is equal to or greater than a threshold, when the moving speed of the electronic device is not constant, or when the trajectory of the moving electronic device is irregular. An electronic device characterized by:

8. The imaging device is incorporated into the electronic device.

8. The electronic device according to claim 1, wherein the first and second electrodes are electrically connected to the first and second electrodes.

9. the imaging device is separate from the electronic device; 8. The electronic device according to claim 1, wherein the first and second electrodes are electrically connected to the first and second electrodes.

10. The electronic device is a head-mounted display device.

10. The electronic device according to claim 1.

11. the generating means requests an external device to perform a process of generating the first object and the second object, and acquires the first object and the second object generated by the external device; 11. The electronic device according to claim 1.

12. the generating means generates, when a user aligns the first object with the second object, the second object corresponding to a shooting range in a case where the second object has a newly determined predetermined position and a newly determined predetermined orientation.

12. The electronic device according to claim 1.

13. A control method for an electronic device for supporting photography by an imaging device, comprising: a generation step of generating, in a virtual space, a first object indicating a current imaging range of the imaging device and a second object presenting the imaging range when the imaging device is at a predetermined position and in a predetermined orientation as a movement destination of the first object; a display control step of displaying the first object and the second object; and In the generating step, the predetermined position and the predetermined orientation are determined based on changes between a past position and orientation of the imaging device and a current position and orientation of the imaging device, and the second object is generated. A method for controlling an electronic device.

14. A control method for electronic equipment for supporting photography by an imaging device, comprising: In a virtual space, a first object indicating a current imaging range of the imaging device and a first object indicating an imaging range when the imaging device is at a predetermined position and a predetermined orientation are provided. a generating step of generating a second object to be presented as a destination; a display control step of displaying the first object and the second object; and In the display control step, the second object is displayed when either a difference in position or a difference in orientation between the first object and the second object is equal to or greater than a threshold, when a moving speed of the electronic device is not constant, or when a moving trajectory of the electronic device is irregular. A method for controlling an electronic device.

15. A program for causing a computer to function as each of the means of the electronic device according to any one of claims 1 to 12.

Images