Non-transitory computer readable medium, image processing method and image processing system

Abstract

A non-transitory computer-readable medium, an image processing method and an image processing system containing a program, wherein the program enables a computer connected to a camera device to implement the following functions: specifying a virtual space; generating a virtual space image obtained by photographing a virtual space including at least one virtual object; outputting a composite image of the photographed image and the virtual space image to a display device having a touch screen; determining the content of a user operation on the touch screen; updating the position or posture of each of a virtual camera and a virtual object when the user operation is a first operation in which a touch position on the touch screen does not move after the user operation is touched on the touch screen; and rotating the virtual object in a predetermined direction when the user operation is a second operation in which a touch position slides on the touch screen.

Description

Technical Field

[0001] At least one of the embodiments of the present invention relates to a non-transitory computer-readable medium (Non-transitory Computer-readable Medium) including an information processing program, an information processing method, and an image processing system, which are used to share virtual processing of actual objects fixedly arranged in the real world with multiple users through augmented reality technology. Background Art

[0002] In recent years, augmented reality (AR) technology has been provided. The augmented reality technology is a technology for displaying a virtual object in a real space by superimposing and displaying a plane or three-dimensional object (hereinafter referred to as an “AR object”) on an image in a real space.

[0003] In addition, a technology for allowing a user to operate an object displayed by augmented reality technology has been proposed. As such a technology, for example, there is a technology for moving an object in an augmented reality space when the user operation is a first operation (refer to JP2014-191718A, hereinafter referred to as "Patent Document 1").

[0004] However, in this conventional technology, there are cases where the user cannot intuitively operate the AR object. Summary of the invention

[0005] The object of the present invention is to provide a non-transitory computer-readable medium, an information processing method and an image processing system, which can improve the operability of AR objects while solving the above-mentioned problems.

[0006] According to one non-limiting aspect of the present invention, a non-transitory computer-readable medium is provided, which contains a program for causing a computer to implement the following functions, wherein the computer is connected to an image capturing apparatus.

[0007] The functions include: a designation function for designating a virtual space corresponding to a captured image, wherein the captured image is captured by the camera device.

[0008] The functions further include: a generating function of generating a virtual space image, wherein the virtual space image is obtained by photographing the virtual space including at least one virtual object using a virtual camera corresponding to the imaging device.

[0009] The functions also include: an output function of outputting a composite image to a display device, wherein the composite image is obtained by combining the captured image and the virtual space image, and the display device includes a touch screen.

[0010] The functions also include: a determination function to determine the content of the user operation on the touch screen.

[0011] The functions also include: an update function, in the case where it is determined that the user operation is a first operation, while the first operation continues, updating the position or posture of each of the virtual camera and the virtual object in a state where the relationship between the virtual camera and the virtual object is fixed according to the movement or posture change of the camera device, and the first operation is an operation in which the touch position of the touch screen does not move after the touch screen is touched.

[0012] The function also includes: a rotation function, rotating the virtual object along a predetermined direction when it is determined that the user operation is a second operation, the second operation is an operation in which the touch position slides on the touch screen, and the predetermined direction is defined by the second operation.

[0013] According to another non-limiting aspect of the present invention, there is provided an image processing method for controlling the performance of information processing in response to user operations.

[0014] The image processing method includes: specifying processing, specifying a virtual space corresponding to a captured image, wherein the captured image is captured by a camera device connected to a computer.

[0015] The image processing method further includes: generating a virtual space image, wherein the virtual space image is obtained by photographing the virtual space including at least one virtual object using a virtual camera corresponding to the imaging device.

[0016] The image processing method further includes: output processing, outputting a synthesized image to a display device, wherein the synthesized image is obtained by synthesizing the captured image and the virtual space image, and the display device includes a touch screen.

[0017] The image processing method further includes: determining processing to determine the content of the user operation on the touch screen;

[0018] The image processing method also includes: update processing, in a case where it is determined that the user operation is a first operation, while the first operation continues, according to the movement or posture change of the camera device, while the relationship between the virtual camera and the virtual object is fixed, updating the position or posture of each of the virtual camera and the virtual object, wherein the first operation is an operation in which the touch position of the touch screen does not move after the touch screen is touched.

[0019] The image processing method further includes: rotation processing, rotating the virtual object along a predetermined direction when it is determined that the user operation is a second operation, the second operation is an operation in which the touch position slides on the touch screen, and the predetermined direction is defined by the second operation.

[0020] According to another non-limiting aspect of the present invention, there is provided an image processing system for controlling information processing in response to user operations. The image processing system comprises a communication network, a server, and a user terminal connected to a camera device.

[0021] The image processing system includes a designation unit that designates a virtual space corresponding to a captured image captured by the imaging device.

[0022] The image processing system includes a generation unit that generates a virtual space image obtained by photographing the virtual space including at least one virtual object using a virtual camera corresponding to the imaging device.

[0023] The image processing system further includes: an output unit, which outputs a composite image to a display device, wherein the composite image is obtained by combining the captured image and the virtual space image, and the display device includes a touch screen.

[0024] The image processing system further includes a determination unit configured to determine content of a user operation on the touch screen.

[0025] The image processing system also includes: an updating unit, which, when determining that the user operation is a first operation, updates the position or posture of each of the virtual camera and the virtual object according to the movement or posture change of the camera device while the first operation continues, while the relationship between the virtual camera and the virtual object is fixed. The first operation is an operation in which the touch position of the touch screen does not move after the touch screen is touched.

[0026] The image processing system further includes: a rotating unit which rotates the virtual object in a predetermined direction if it is determined that the user operation is a second operation, the second operation being an operation in which a touch position slides on the touch screen, and the predetermined direction is defined by the second operation.

[0027] According to various embodiments of the present application, one or more than two deficiencies are solved.

[0028] The foregoing and other objects, features and advantages of the present invention will become more apparent from the following detailed description of preferred embodiments of the present invention with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 is a block diagram showing an example of a configuration of an image processing system corresponding to at least one of the embodiments of the present invention;

[0030] Figure 2 is a block diagram showing a configuration of a server corresponding to at least one of the embodiments of the present invention;

[0031] Figure 3 is a flow chart illustrating an example of information processing corresponding to at least one of the embodiments of the present invention;

[0032] Figure 4 is a flowchart showing an example of actions on the server side in information processing corresponding to at least one of the embodiments of the present invention;

[0033] Figure 5 is a flowchart showing an example of actions on the terminal side in information processing corresponding to at least one of the embodiments of the present invention;

[0034] Figure 6 is a block diagram showing a configuration of a server corresponding to at least one of the embodiments of the present invention;

[0035] Figure 7 is a flow chart illustrating an example of information processing corresponding to at least one of the embodiments of the present invention;

[0036] Figure 8 is a block diagram showing a configuration of a server corresponding to at least one of the embodiments of the present invention;

[0037] Fig. 9 is a flow chart illustrating an example of information processing corresponding to at least one of the embodiments of the present invention;

[0038] Fig.10 is a block diagram showing a configuration of a server corresponding to at least one of the embodiments of the present invention;

[0039] Fig.11 is a flow chart illustrating an example of information processing corresponding to at least one of the embodiments of the present invention;

[0040] Fig.12 is a block diagram showing a configuration of a server corresponding to at least one of the embodiments of the present invention;

[0041] Fig.13 is a flow chart illustrating an example of information processing corresponding to at least one of the embodiments of the present invention;

[0042] Fig.14 is a block diagram showing a configuration of a server corresponding to at least one of the embodiments of the present invention;

[0043] Fig.15 is an explanatory diagram for explaining an example of a storage state of information corresponding to at least one of the embodiments of the present invention;

[0044] Fig.16 is a flowchart illustrating an example of information processing corresponding to at least one of the embodiments of the present invention; and

[0045] Fig.17 It is an explanatory diagram showing an example of a display screen corresponding to at least one of the embodiments of the present invention. DETAILED DESCRIPTION

[0046] The following will describe examples of embodiments according to the present invention with reference to the accompanying drawings. It is noted that the various constituent elements in the embodiments described below can be appropriately combined without any contradiction, etc. In addition, the content described in a certain embodiment can be omitted in another embodiment. In addition, the content of actions or processing that are not related to the characteristic parts of each embodiment can be omitted. In addition, the order of various processing that constitutes the various processes described below can be changed without any contradiction, etc. in the processing content.

[0047] [First embodiment]

[0048] Figure 1 1 is a block diagram showing an example of the configuration of an image processing system 100 according to an embodiment of the present invention. Figure 1 As shown, the image processing system 100 includes an image processing server 10 and user terminals 20 to 20n ("n" is an arbitrary integer) used by users of the image processing system 100. In this regard, the configuration of the image processing system 100 is not limited to this configuration. The image processing system 100 may be configured so that multiple users use a single user terminal, or may be configured to include multiple image processing servers.

[0049] The image processing server 10 and each of the plurality of user terminals 20, 201 to 20n are connected to a communication network 30 such as the Internet. In this regard, although not shown in the drawings, the plurality of user terminals 20, 201 to 20n are connected to the communication network 30 by performing data communication with a base station managed by a communication operator using, for example, a wireless communication line.

[0050] The image processing system 100 includes an image processing server 10 and a plurality of user terminals 20, 201 to 20n, thereby realizing various functions for executing various processes in response to user operations.

[0051] The image processing server 10 is managed by the manager of the image processing system 100, and has various functions of providing information about various processes to each of the plurality of user terminals 20, 201 to 20n. In the present embodiment, the image processing server 10 is constituted by an information processing device such as a WWW server, and the image processing server 10 includes a storage medium for storing various information. In this regard, the image processing server 10 has a general configuration for performing various processes like a computer, such as a control unit or a communication unit, and its description is omitted here. In addition, in the image processing system 100, from the viewpoint of reducing the processing load of each of the plurality of user terminals 20, 201 to 20n, it is preferred that the image processing server 10 manages various information. However, the storage unit configured to store various information may be provided with a storage area and be in a state where the image processing server 10 can access the storage area. For example, the image processing server 10 may be configured to include a dedicated storage area outside the image processing server 10.

[0052] Each of the plurality of user terminals 20, 201 to 20n is managed by a user and has various functions for performing various processes by using information received from the image processing server 10. In the present embodiment, for example, each of the plurality of user terminals 20, 201 to 20n may be a communication terminal such as a cellular phone terminal or a PDA (Personal Digital Assistant). Each of the plurality of user terminals 20, 201 to 20n is equipped with a shooting device for shooting at least one of a moving image or a still image by the user, and each of the plurality of user terminals 20, 201 to 20n includes a display for displaying the shot image data, etc. In addition, the shooting device may include an optical camera, or may be a camera used together with a three-dimensional shooting device. In addition, each of the plurality of user terminals 20, 201 to 20n includes an input device such as a mouse, a touch screen, or a stylus pen for inputting drawing information. In this regard, as other examples of the configuration of the user terminal that can be included in the image processing system 100, there are so-called wearable devices (Wearable Device) such as smart watches and combinations of wearable devices and communication terminals.

[0053] Furthermore, each of the plurality of user terminals 20, 201 to 20n is connected to the communication network 30, and includes hardware and software for performing various processes by communicating with the image processing server 10. In this regard, each of the user terminals 20, 201 to 20n may be configured to be able to directly communicate with each other without using the image processing server 10.

[0054] In the process of performing information processing, the image processing server 10 appropriately sends the processing stage (Process Stage) to each of the plurality of user terminals 20, 201-20n. Each of the user terminals 20, 201-20n causes the display device to display the screen content (Screen Content) on the display screen of the display device based on the content of the received processing stage. The image processing server 10 may generate the screen content, or each of the user terminals 20, 201-20n may generate the screen content.

[0055] Figure 2 1 is a block diagram showing an image processing server 10A (hereinafter referred to as “server 10A”) as an example of the configuration of the image processing server 10. Figure 2 As shown, the server 10A at least includes a specifying unit 11 , a generating unit 12 , an output unit 13 , a determining unit 14 , an updating unit 15 , and a rotating unit 16 .

[0056] The designation unit 11 has a function of designating a virtual space corresponding to a captured image (Captured Image) captured by an imaging device.

[0057] Here, as an example of a camera device, there is a camera connected to the user terminal 20, which can communicate with the server 10A. In addition, the virtual space corresponding to the captured image refers to a virtual space used to capture the AR object to be superimposed and displayed on the captured image. Since the configuration of the specified virtual space is implemented by using a well-known technology, its detailed description is omitted. However, preferably, the real space specified from the captured image is associated with the specified virtual space by a predetermined rule. As an example of such a configuration, there is a configuration in which a virtual space including virtual objects is specified, and the arrangement of the virtual objects is determined based on the position and direction of the camera device.

[0058] The generation unit 12 has a function of generating a virtual space image obtained by photographing a virtual space including at least one virtual object using a virtual camera corresponding to an imaging device.

[0059] Here, a virtual object refers to an object arranged in a virtual space. There is no particular limitation on the configuration of the virtual object as long as the virtual object can move in response to a user operation. As examples of virtual objects, there are three-dimensional shapes that are stationary in a virtual space and characters that act in a virtual space according to predetermined rules.

[0060] In addition, the virtual camera corresponding to the camera device refers to a virtual camera that the user can operate via the camera device. The configuration of generating the virtual space image is not particularly limited. However, preferably, the generation unit 12 is configured to be able to change the reflection (Reflection) of the shooting in the virtual space specified by the specification unit 11 in response to the user operation. As an example of such a configuration, there is a configuration in which, when at least one of the position or direction of the camera device is changed, the virtual image to be generated is also changed. In this regard, the configuration of the virtual space image is not particularly limited. However, preferably, the virtual space image is configured to draw at least one virtual object thereon.

[0061] The output unit 13 has a function of outputting a synthesized image obtained by synthesizing a captured image and a virtual space image to a display device provided with a touch panel (Touch Panel).

[0062] Here, the configuration of outputting the composite image is not particularly limited. However, preferably, the output unit 13 is configured so that the user can operate the virtual space image in the composite image. As an example of such a configuration, there is a configuration in which: the output position of the virtual object on the touch screen is controlled and output; and the composite image is output in the case of receiving a touch operation for the output position to perform a predetermined process.

[0063] The determination unit 14 has a function of determining the content of a user operation on the touch screen.

[0064] Here, the configuration for determining the content of the user operation is not particularly limited. However, preferably, the determination unit 14 is configured to determine one operation content among multiple operation contents. As an example of such a configuration, there is a configuration in which it is determined whether a so-called long press operation or a sliding operation is received based on the position of the touch operation.

[0065] The updating unit 15 has the following function: when it is determined that the user operation is the first operation, while the first operation is continued, the position or posture of each of the virtual camera and the virtual object is updated according to the movement or posture change of the camera device in a state where the relationship between the virtual camera and the virtual object is fixed. At this point, it is preferred that the movement or posture change of the camera device is the same as the movement or posture change of the user terminal to which the camera device is connected (Synonymous). However, it can also be configured that the camera device can move or change its posture independently of the user terminal.

[0066] Here, the first operation refers to an operation that satisfies a predetermined first operation condition. The configuration of the first operation is not particularly limited as long as the first operation can be distinguished from the second operation described later. However, preferably, the first operation is an operation in which, after the finger or stylus touches the touch screen, for example, the touch position of the user's finger or stylus on the touch screen does not move.

[0067] In addition, the configuration for determining whether the first operation is to be continued is not particularly limited. The updating unit 15 may be configured to determine that the first operation is not to be continued when the first operation condition becomes unsatisfied. Alternatively, the updating unit 15 may be configured to determine that the first operation is to be continued until a predetermined end condition is satisfied even if the touch position of the finger or the like on the touch screen no longer exists after determining that the user operation is the first operation.

[0068] In addition, the state in which the relationship between the virtual camera and the virtual object is fixed means that the relationship between the two does not change in the virtual space. Here, the configuration for defining the relationship between the two is not particularly limited, but preferably, the updating unit 15 is configured so that the relative distance or posture relationship between the two does not change. As an example of such a configuration, there is a configuration in which the virtual camera and the virtual object perform the same behavior in the virtual space. At this point, the virtual camera and the virtual object can be configured to maintain the relationship of at least one of the position or the posture, or can be configured to maintain the relationship of both the position and the posture.

[0069] In addition, the phrase "update the position or posture of each of the virtual camera and the virtual object (i.e., the virtual camera, etc.) according to the movement or posture change of the camera device" means: in the case where the position of the camera device in the real space changes, the position of the virtual camera, etc. is updated; or in the case where the posture of the camera device in the real space changes, the posture of the virtual camera, etc. is updated. That is, when the imaging direction (Imaging Direction) of the camera device changes, the shooting direction of the virtual camera changes in the same way as the change, and the position of the virtual object also moves to fall within the shooting area of ​​the virtual camera. In addition, when the posture of the camera device changes, the posture of the virtual camera changes in the same way as the change, and the posture of the virtual object also changes in the same way as the posture of the virtual camera. At this point, the update unit 15 can be configured to simultaneously update the position and posture of the virtual camera, etc. when the movement and posture change of the camera device occur simultaneously, whereby the image processing server 10A can be configured to provide the user with a sense of linkage between the camera device and the virtual object. Alternatively, the update unit 15 may be configured to prioritize any one of the movement and posture change of the virtual object, etc., whereby the image processing server 10A may be configured to provide a feature in its operability. As an example of the latter configuration, there is a configuration in which, in a case where the movement of the camera device and the posture change are performed simultaneously, a priority condition is defined as to whether the movement of the virtual camera, etc. or the posture change is performed preferentially. In this regard, the configuration for detecting the movement or posture change of the camera device is not particularly limited, and a known technique using various sensors may be adopted. Therefore, its description is omitted here.

[0070] The rotation unit 16 has a function of rotating the virtual object in a predetermined direction if it is determined that the user operation is the second operation. The predetermined direction is defined by the second operation.

[0071] Here, the second operation refers to an operation that satisfies a predetermined second operation condition. The configuration of the second operation is not particularly limited as long as the second operation can be distinguished from the first operation. However, preferably, the second operation is an operation in which the position of a finger or the like touching the touch screen slides on the touch screen (i.e., an operation in which the touch position of the finger or the like moves after the finger or the like touches the touch screen).

[0072] In addition, the configuration of rotating the virtual object by the second operation is not particularly limited. However, preferably, the user can easily predict the rotation of the virtual object. As an example of such a configuration, there is a configuration in which, in the case where it is determined that the operation of moving the touch position of the finger or the like from the left side of the screen to the right side of the screen with respect to the touch screen is the second operation, the virtual object is rotated around its own axis in the direction from the left side of the screen to the right side of the screen. In this regard, in this case, the rotation unit 16 can be configured to pre-set a point or axis for the virtual object to rotate around its own axis, or can be configured to be defined by a user operation.

[0073] Next, the action of the image processing system 100 (hereinafter referred to as “system 100 ”) according to the present embodiment will be described.

[0074] Figure 3 1 is a flowchart showing an example of information processing performed by the system 100. The following description will be given by taking as an example a case where the image processing server 10A and the user terminal 20 perform information processing.

[0075] For example, when the user terminal 20 receives a photography request using the AR function from the user, the information processing starts. The following description will take the case where the server 10A receives the photography request from the user terminal 20 as an example.

[0076] In the information processing, the server 10A first specifies a virtual space corresponding to a captured image (step S11). In this embodiment, the server 10A specifies a virtual space corresponding to a captured image, and the captured image is captured by a camera (ie, a real camera) of the user terminal 20.

[0077] After specifying the virtual space, the server 10A generates a virtual space image (step S12). In the present embodiment, the server 10A generates a virtual space image obtained by photographing the virtual space with a virtual camera, and the user can operate the virtual camera through the camera of the user terminal 20. The virtual space includes a predetermined virtual object (a cube as an AR object).

[0078] After generating the virtual space image, the server 10A outputs the synthesized image (step S13). In this embodiment, the server 10A outputs the synthesized image obtained by synthesizing the captured image and the virtual space image to a display device provided with a touch screen. That is, augmented reality is achieved by superimposing and displaying the virtual object captured by the virtual camera on the real image captured by the real camera.

[0079] After outputting the synthesized image, the server 10A determines the content of the user operation (step S14). In the present embodiment, the server 10A determines the content of the user operation on the touch screen.

[0080] After determining the content of the user operation, the server 10A updates the position or posture of each of the virtual camera and the virtual object (step S15). In this embodiment, when the server 10A determines that the user operation is the first operation, while the first operation continues, the position or posture of the virtual camera is updated according to the movement or posture change of the user terminal 20, and the position or posture of the virtual object is also updated, so that the user can be provided with a feeling of operating the virtual object in the real space via the user terminal 20.

[0081] After updating the position or posture of each of the virtual camera and the virtual object, the server 10A rotates the virtual object (step S16). In the present embodiment, in the case where it is determined that the user operation is the second operation, the server 10A rotates the virtual object in the direction defined by the second operation.

[0082] Figure 4 1 is a flowchart showing an example of the operation on the server 10 side in the information processing. Here, the operation of the server 10A in the system 100 will be described.

[0083] The server 10A starts information processing, for example, by receiving an execution request for an AR function from the user terminal 20, the AR function including information about the captured image. The server 10A specifies a virtual space corresponding to the captured image (step S101); generates a virtual space image (step S102); outputs a composite image (step S103); and determines the content of the user operation (step S104). In the present embodiment, the server 10A determines the content of the user operation on the touch screen. Then, in the case where it is determined that the user operation is the first operation, the server 10A updates the position or posture of each of the virtual camera and the virtual object (step S15). On the other hand, in the case where it is determined that the user operation is the second operation, the server 10A rotates the virtual object (step S16).

[0084] Figure 5 1 is a flowchart showing an example of actions in the case where the user terminal 20 performs information processing. The following description will be given by taking as an example the case where the user terminal 20A (hereinafter referred to as "terminal 20A") alone performs information processing as an example of the configuration of the user terminal 20. At this point, the terminal 20A is configured to include the same functions as the configuration of the server 10A except for receiving various information from the server 10A. Therefore, the description thereof is omitted from the perspective of avoiding duplication of description.

[0085] For example, in the case where a real camera connected to the terminal 20A captures the real space, when a request for executing the AR function is received from the user, the terminal 20A starts information processing. The terminal 20A specifies the virtual space corresponding to the captured image (step S201); generates a virtual space image (step S202); outputs a composite image (step S203); and determines the content of the user operation (step S204). In this embodiment, the terminal 20A determines the content of the user operation on the touch screen. Then, in the case of determining that the user operation is the first operation, the terminal 20A updates the position or posture of each of the virtual camera and the virtual object (step S205). On the other hand, in the case of determining that the user operation is the second operation, the terminal 20A rotates the virtual object (step S206).

[0086] As described above, as one aspect of the first embodiment, the server 10A connected to the camera included in the user terminal is configured to include the specifying unit 11 , the generating unit 12 , the output unit 13 , the determining unit 14 , the updating unit 15 , and the rotating unit 16 . Therefore, the designation unit 11 designates a virtual space corresponding to a captured image captured by a camera device; the generation unit 12 generates a virtual space image obtained by capturing a virtual space using a virtual camera corresponding to the camera device, wherein the virtual space includes at least one virtual object; the output unit 13 outputs a synthesized image obtained by synthesizing the captured image and the virtual space image to a display device provided with a touch screen; the determination unit 14 determines the content of a user operation for the touch screen; the update unit 15 updates the position or posture of each of the virtual camera and the virtual object in a state where the relationship between the virtual camera and the virtual object is fixed according to the movement or posture change of the camera device while the first operation is continued, in the case where it is determined that the user operation is a first operation, the first operation is an operation in which the touch position does not move after a finger or the like touches the touch screen; and the rotation unit 16 rotates the virtual object in a predetermined direction in which the position of the finger or the like touching the touch screen slides in the second operation, and the predetermined direction is defined by the second operation. Therefore, the operability of the virtual object (i.e., AR object) can be improved.

[0087] That is, as a configuration for enabling the user to operate the AR object, the image processing server 10A is configured so that the first operation and the second operation can be received in the same state, the operation for the touch screen and the camera device is received in the first operation, and the sliding operation is received in the second operation. Therefore, the operability of the AR object can be further improved compared to the case where one type of operation method is received.

[0088] [Second embodiment]

[0089] Figure 61 is a block diagram showing a configuration of an image processing server 10B (hereinafter referred to as “server 10B”) as an example of the image processing server 10. In the present embodiment, the server 10B includes at least a specifying unit 11, a generating unit 12, an output unit 13, a determining unit 14, an updating unit 15B, and a rotating unit 16.

[0090] The updating unit 15B has a function of updating the position or posture of a virtual object in the virtual space, the virtual object being displayed at the touch position of the first operation.

[0091] Here, the configuration for specifying the virtual object displayed at the touch position is not particularly limited. The specifying unit 11 may be configured to specify the virtual object by comparing the coordinates of the touch position with the coordinates of the virtual object, and may be configured to specify a plurality of virtual objects simultaneously.

[0092] Figure 7 1 is a flowchart showing an example of information processing performed by the image processing system 100 (hereinafter referred to as "system 100"). The following description will be given by taking the actions of the server 10B and the user terminal 20 (hereinafter referred to as "terminal 20") as an example. At this point, the description of the flowchart showing the actions of each of the server 10B and the terminal 20 is omitted from the perspective of avoiding duplicate descriptions.

[0093] In the information processing, after determining the content of the user operation, the server 10B updates the position or posture of each of the virtual camera and the virtual object displayed at the touch position (step S2-11). In the present embodiment, the server 10B sets the virtual object displayed at the touch position of the first operation to the operation state; and while the user's touch operation continues, whenever information indicating at least one of the position or posture of the terminal 20 or the camera included in the terminal 20 is received from the terminal 20, the server 10B updates the information about the virtual object so that the virtual object in the virtual space changes in the same manner as the change.

[0094] As described above, as one aspect of the second embodiment, the server 10B is configured to include the designation unit 11, the generation unit 12, the output unit 13, the determination unit 14, the update unit 15B, and the rotation unit 16. Therefore, the update unit 15B updates the position or posture of the virtual object displayed at the touch position of the first operation in the virtual space. Therefore, the user can intuitively designate the operation target.

[0095] [Third embodiment]

[0096] Figure 81 is a block diagram showing the configuration of an image processing server 10C (hereinafter referred to as "server 10C") as an example of the image processing server 10. In the present embodiment, the server 10C includes at least a specifying unit 11, a generating unit 12, an output unit 13, a determining unit 14, an updating unit 15, and a rotating unit 16C.

[0097] The rotation unit 16C has a function of rotating the virtual object in a state where the position of the virtual object in the virtual space is fixed.

[0098] Here, the configuration of rotating the virtual object in a state where the position of the virtual object is fixed is not particularly limited. However, preferably, the rotation unit 16C is configured so that the reference position of the virtual object (e.g., the coordinates of the center of gravity) does not change before and after the rotation. As an example of such a configuration, there is a configuration in which, even if the position or posture of the terminal 20 or the camera of the terminal 20 changes within a predetermined erroneous operation range during the rotation of the virtual object when the second operation is received, this can be prevented from affecting the position or posture of the virtual object in the virtual space.

[0099] Fig. 9 1 is a flowchart showing an example of information processing performed by the image processing system 100. The following description will be given by taking the actions of the server 10C and the terminal 20 as an example. At this point, the description of the flowchart showing the actions of each of the server 10C and the terminal 20 is omitted from the perspective of avoiding duplicate descriptions.

[0100] After updating the position or posture of each of the virtual camera and the virtual object, the server 10C rotates the virtual object in a state where the position of the virtual object is fixed (step S3-11). In this embodiment, when it is determined that the user operation is the second operation, the server 10C fixes the position of the virtual object in the virtual space and rotates the virtual object in the direction defined by the second operation.

[0101] As described above, as one aspect of the third embodiment, the server 10C is configured to include the designation unit 11, the generation unit 12, the output unit 13, the determination unit 14, the update unit 15, and the rotation unit 16C. Therefore, the rotation unit 16C rotates the virtual object in a state where the position of the virtual object in the virtual space is fixed. Therefore, the user's erroneous operation can be prevented, and this makes it possible to further improve the operability of the AR object.

[0102] [Fourth embodiment]

[0103] Fig.101 is a block diagram showing an image processing server 10D (hereinafter referred to as "server 10D") as an example of the image processing server 10. In the present embodiment, the server 10D includes at least a specifying unit 11, a generating unit 12, an output unit 13, a determining unit 14, an updating unit 15D, and a rotating unit 16D.

[0104] The updating unit 15D has a function of rotating the virtual object around the shooting direction of the virtual camera to update the posture of the virtual object when determining that the received user operation is the first operation and the posture of the user terminal 20 (hereinafter referred to as the "terminal 20") is rotated by using the shooting direction of the camera as an axis. That is, the updating unit 15D has a function of controlling the virtual object to rotate the virtual object in the same manner as the rotation of the camera in the real space.

[0105] Here, the configuration of updating the posture of the virtual object by rotating the virtual object around the shooting direction of the virtual camera is not particularly limited. The updating unit 15D may be configured to set an axis from the touch position of the first operation toward the depth direction of the touch screen, or may be configured so that the position where the axis is set on the virtual object is predefined. As an example of such a configuration, there is a configuration in which, when the virtual object is rotated around the shooting direction of the virtual camera, an axis passing through a predetermined position (e.g., the center of gravity) of the virtual object is set.

[0106] The rotation unit 16D has a function of rotating the virtual object around a direction orthogonal to the shooting direction of the virtual camera when determining that the user operation is a second operation in which, for example, the position of a user's finger or a stylus pen touching the touch screen slides.

[0107] Here, the configuration of rotating the virtual object around a direction orthogonal to the shooting direction of the virtual camera is not particularly limited. However, preferably, the server 10D is configured to be able to rotate the virtual object three-dimensionally using the update unit 15D and the rotation unit 16D. As an example of such a configuration, there is a configuration in which, in a case where it is determined that the sliding direction as the second operation is the left-right direction of the touch screen, the rotation axis of the virtual object is set to the up-down direction of the virtual space; and in a case where it is determined that the sliding direction as the second operation is the up-down direction of the touch screen, the rotation axis of the virtual object is set to the left-right direction of the virtual space. In addition, the server 10D can be configured to set the rotation axis of the virtual object to the diagonal direction of the virtual space in a case where it is determined that the sliding direction as the second operation is the diagonal direction.

[0108] Fig.111 is a flowchart showing an example of information processing performed by the image processing system 100. The following description will be given by taking the actions of the server 10D and the terminal 20 as an example. At this point, the description of the flowchart showing the actions of each of the server 10D and the terminal 20 is omitted from the perspective of avoiding duplicate descriptions.

[0109] After determining the content of the user operation, the server 10D sets the rotation axis for each of the virtual camera and the virtual object and updates its posture (step S4-11). In the present embodiment, when it is determined that the user operation is the first operation, while the first operation is continued, the server 10D sets the shooting direction of the virtual camera as the rotation axis to rotate the virtual camera and the virtual object around the shooting direction of the camera through the terminal 20, and the server 10D updates its posture, thereby rotating the virtual camera and the virtual object according to the rotation of the terminal 20 or the camera.

[0110] After updating the postures of the virtual camera and the virtual object, the server 10D (i.e., the rotation unit 16D) sets the rotation axis to rotate the virtual object (step S4-12). In this embodiment, when it is determined that the user operation is the second operation (i.e., the sliding operation), while the second operation continues, the server 10D specifies the direction of the second operation on the touch screen of the terminal 20; sets the rotation axis in such a way that the virtual object rotates in the specified direction; and rotates the virtual object in the set direction.

[0111] As described above, as one aspect of the fourth embodiment, the server 10D is configured to include a designation unit 11, a generation unit 12, an output unit 13, a determination unit 14, an update unit 15D, and a rotation unit 16D. Therefore, in the case where it is determined that the received user operation is the first operation and the posture of the terminal 20 rotates around the shooting direction of the camera device, the update unit 15D rotates the virtual object around the shooting direction of the virtual camera to update the posture of the virtual object; and in the case where it is determined that the received user operation is the second operation of sliding the position of the finger touching the touch screen, etc., the rotation unit 16D rotates the virtual object around the direction orthogonal to the shooting direction of the virtual camera. Therefore, the posture of the AR object can be controlled three-dimensionally using two types of operations, and this can further improve the operability of the AR object.

[0112] [Fifth embodiment]

[0113] Fig.12 1 is a block diagram showing a configuration of an image processing server 10E (hereinafter referred to as “server 10E”) as an example of the server 10. In the present embodiment, the server 10E includes at least a specifying unit 11, a generating unit 12, an output unit 13, a determining unit 14, an updating unit 15E, and a rotating unit 16.

[0114] The updating unit 15E has a function of changing and updating a state in which the relationship between the virtual camera and the virtual object is fixed to a state in which the relationship between the virtual camera and the virtual object is not fixed when a predetermined condition in the virtual space is satisfied while the first operation is continued.

[0115] Here, the content of the predetermined condition is not particularly limited. However, preferably, the predetermined condition has content that the user can grasp. As examples of the predetermined condition, there are a condition that a predetermined time has passed after the first operation starts and a condition that a virtual object touches another object in a virtual space.

[0116] In addition, the configuration in which the relationship between the virtual camera and the virtual object is changed from a state in which the relationship between the virtual camera and the virtual object is fixed to a state in which the relationship between the virtual camera and the virtual object is not fixed is not particularly limited. However, preferably, the updating unit 15E is configured so that the user can grasp that it is no longer a fixed state. As examples of such a configuration, there are a configuration in which a virtual object that is no longer a fixed state stays at a position in the virtual space and a configuration in which a virtual object is moved to a predetermined position.

[0117] Fig.13 1 is a flowchart showing an example of information processing performed by the image processing system 100. In the following, the actions of the server 10E and the user terminal 20 (hereinafter referred to as "terminal 20") will be described as an example. At this point, the description of the flowchart showing the actions of each of the server 10E and the terminal 20 is omitted from the perspective of avoiding repeated description.

[0118] After the content of the user operation is determined to be the first operation, while the first operation continues, the server 10E (i.e., the updating unit 15E) sets the relationship between the virtual camera and the virtual object to a fixed state or a non-fixed state (step S5-11). In the present embodiment, when the virtual object specified by the first operation does not contact any other virtual object, the server 10E fixes the relationship between the virtual camera and the virtual object; controls the terminal 20 so that the virtual camera is moved or its posture is changed according to the movement or posture change of the terminal 20 or the camera of the terminal 20; and moves or changes the posture of the virtual camera and the virtual object set to the fixed state. In addition, when the virtual object specified by the first operation contacts any other virtual object, the server 10E sets the relationship between the virtual camera and the virtual object to a non-fixed state; controls the terminal 20 so that the virtual camera is moved or its posture is changed according to the movement or posture change of the terminal 20 or the camera of the terminal 20; and makes the virtual camera and the virtual object set to the non-fixed state stationary at a position that satisfies a predetermined condition.

[0119] As described above, as one aspect of the fifth embodiment, the server 10E is configured to include the designation unit 11, the generation unit 12, the output unit 13, the determination unit 14, the update unit 15E, and the rotation unit 16. Therefore, when the predetermined condition is satisfied in the virtual space while the first operation is continued, the update unit 15E changes the relationship between the virtual camera and the virtual object from a fixed state to a non-fixed state and updates the relationship. Therefore, the status of the virtual space can be reflected in the operation of the AR object, and this makes it possible to further improve the operability of the AR object.

[0120] [Sixth embodiment]

[0121] Fig.14 The image processing system 100 (see Figure 1 ) is a block diagram of the configuration of a user terminal 20Z (hereinafter referred to as "terminal 20Z") as an example of a user terminal 20 in FIG. In this embodiment, the terminal 20Z includes at least a specifying unit 21Z, a generating unit 22Z, an output unit 23Z, a determining unit 24Z, an updating unit 25Z, and a rotating unit 26Z.

[0122] The terminal 20Z has a function of enabling an operation of moving or rotating an object in an AR space by giving the user a feeling of grasping the object with the user's hands or lazy tongs (or a magic hand). The following will describe a case where an object existing in an AR space is moved or rotated by moving or rotating the body of the terminal 20Z, which is a smartphone, or swiping the screen of the terminal 20Z.

[0123] The specifying unit 21Z has a function of specifying a virtual space corresponding to a captured image captured by the imaging device.

[0124] Here, as an example of a camera device, there is a camera connected to the terminal 20Z. In addition, as an example of a virtual space corresponding to a captured image, there is a predefined virtual space for capturing an AR object to be superimposed and displayed on the captured image. That is, the terminal 20Z associates the real space with the virtual space, thereby specifying the so-called AR space.

[0125] The generation unit 22Z has a function of generating a virtual space image obtained by photographing the virtual space using a virtual camera corresponding to the camera device. The virtual space includes at least one virtual object.

[0126] Here, as an example of a virtual object, there is a three-dimensional object. In addition, the configuration for generating a virtual space image is not particularly limited. However, preferably, it can occur in a state where no virtual objects are arranged in the virtual space. As an example of such a configuration, there is a configuration in which information about the virtual space is specified in a state where no virtual objects are arranged in the virtual space; virtual objects are arranged in the virtual space according to predetermined rules; the virtual space after arrangement is photographed by a virtual camera; and a virtual space image is subsequently generated. That is, by photographing the virtual space, the generation unit 22Z can generate a virtual space image that does not include any virtual objects, or can generate a virtual space image that includes virtual objects. By configuring the terminal 20Z in this way, it is also possible for the user to realize a mode in which the synthetic image (or AR image) changes from a state in which no virtual objects are displayed to a state in which virtual objects are displayed, and this can make the operability of the AR object diverse.

[0127] The output unit 23Z has a function of outputting a synthesized image obtained by synthesizing the captured image and the virtual space image to a display device provided with a touch screen.

[0128] Here, as an example of a composite image, there is a so-called AR image obtained by synthesizing a real space image and a virtual space image. By specifying a virtual space corresponding to the real space, the user can operate a virtual object. Thus, augmented reality can be achieved.

[0129] The determination unit 24Z has a function of determining the content of a user operation on the touch screen.

[0130] Here, as examples of the user operation, there are movement or posture change of the terminal 20Z and a touch operation on the touch screen of the terminal 20Z.

[0131] The update unit 25Z has the following function: when it is determined that the user operation is the first operation, while the first operation continues, the position or posture of each of the virtual camera and the virtual object is updated according to the movement or posture change of the camera device while the relationship between the virtual camera and the virtual object is fixed.

[0132] Here, as an example of the first operation, there are operations such as so-called hold or long touch, in each of which, for example, the touch position of the user's finger or stylus does not move after the finger, etc. touches the touch screen. By fixing the relationship between the virtual camera and the virtual object while the first operation is continued, an operating system for moving or rotating the terminal 20Z with a feeling of holding an object in the AR space with his or her hand or an idle gripper can be provided to the user in the case where the user holding the terminal 20Z with his or her hand moves the terminal 20Z.

[0133] In addition, as an example of a state in which the relationship between the virtual camera and the virtual object is fixed, there is a state in which the position or posture of the virtual object changes in the same way when the position or posture of the virtual camera in the virtual space changes according to the movement or posture change of the terminal 20Z.

[0134] At this point, in the case where it is determined that the user operation is an operation of changing the posture of the terminal 20Z without touching the touch screen, the update unit 25Z updates only the position or posture of the virtual camera. That is, in the case where the AR space is specified after the camera is started, the update unit 25Z moves the virtual camera in response to the movement of the camera. However, the update unit 25Z may not move the virtual object in the virtual space.

[0135] The rotation unit 26Z has a function of rotating the virtual object in a predetermined direction when determining that the user operation is the second operation. The predetermined direction is defined by the second operation.

[0136] Here, as examples of the second operation, there are an operation of sliding a position where a finger or the like touches the touch screen and so-called swiping.

[0137] Fig.15 It is used to explain the storage unit (in Fig.14 1 is an explanatory diagram of an example of a storage state of information stored in the memory (not shown). Fig.15 As shown, the storage unit stores associated operation targets, coordinates, postures, and rotation axes.

[0138] Here, as an example of an operation target, there is a virtual object specified by a user operation. In addition, as an example of a coordinate, there is a position of a feature point of the operation target in a three-dimensional space. In addition, as an example of a posture, there is a position of one feature point or each of a plurality of feature points for specifying the posture of the operation target. And, as an example of a rotation axis, there is an axis selected or defined when rotating the operation target.

[0139] Fig.16 20Z. In the information processing according to the present embodiment, a process for controlling a virtual object (i.e., an AR object) superimposed and displayed on an image in a real space in response to a user operation is performed. Each process will be described below. At this point, the order of each process may be changed if there is no contradiction in the process content, etc.

[0140] For example, when the terminal 20Z receives a shooting request using the AR function from the user, the information processing starts. The following description will be given by taking the case where the terminal 20Z receives the shooting request as an example.

[0141] In the information processing, the terminal 20Z first specifies the virtual space corresponding to the captured image with the specifying unit 21Z (step S301). In this embodiment, the terminal 20Z specifies a reference position from the captured image captured by the camera of the terminal 20Z (i.e., the real camera), and specifies the virtual space where a plurality of virtual objects are arranged based on the specified reference position.

[0142] After specifying the virtual space, the terminal 20Z generates a virtual space image with the generation unit 22Z (step S302). In the present embodiment, the terminal 20Z generates a virtual space image obtained by photographing the virtual space with a virtual camera that can be operated by the camera of the terminal 20Z.

[0143] After generating the virtual space image, the terminal 20Z outputs the synthesized image with the output unit 23Z (step S303). In this embodiment, the terminal 20Z outputs the synthesized image obtained by synthesizing the real image and the virtual space image to a display device provided with a touch screen. The virtual space image is obtained by photographing a virtual space including a plurality of virtual objects.

[0144] After outputting the synthesized image, the terminal 20Z determines the content of the user operation by the determination unit 24Z (step S304). In the present embodiment, the terminal 20Z determines the content of the user operation on the touch screen.

[0145] After determining the content of the user operation, the terminal 20Z updates the position or posture of each of the virtual camera and the virtual object with the updating unit 25Z (step S305). In the present embodiment, when the terminal 20Z receives from the user an operation (an example of the first operation) of holding (i.e., long touching) any one of the multiple objects (i.e., AR objects) displayed on the screen of the terminal 20Z and moving the body of the terminal 20Z while maintaining such a state, the terminal 20Z controls the held AR object to move the same distance. In addition, when the terminal 20Z receives from the user an operation (another example of the first operation) of rotating the terminal 20Z while maintaining the above state, the terminal 20Z controls the held AR object to move rotationally around the coordinates of the terminal 20Z. At this point, when the finger or the like leaves the screen, the terminal 20Z determines that the operation is terminated.

[0146] After updating the position or posture of each of the virtual camera and the virtual object, the terminal 20Z rotates the virtual object with the rotation unit 26Z (step S306). In the present embodiment, when the terminal 20Z receives an operation of swiping the screen (an example of the second operation), the terminal 20Z controls the AR object to rotate in situ. Here, the terminal 20Z causes the AR object to perform a yaw rotation by receiving a horizontal swipe, and the terminal 20Z causes the AR object to perform a pitch rotation by using a vertical swipe. At this point, the operation of rotating the terminal 20Z includes a rolling rotation. At this point, the terminal 20Z can be configured so that the above-mentioned controls are executed simultaneously, or can be configured to execute each of the above-mentioned controls separately.

[0147] Fig.17 is an explanatory diagram for explaining an example of a composite image. Fig.17 As shown, a ceiling 1701 of a real space, a display 1702 of a real space, a plurality of AR objects 1711 , 1712 , 1713 , a touch position image 1721 , and an indication image 1722 are displayed on the display screen of the terminal 20Z that displays the synthesized image.

[0148] Here, each of the ceiling 1701 and the display 1702 of the real space is an image captured by a camera included in the terminal 20Z.

[0149] In addition, each of the multiple AR objects 1711, 1712, 1713 is an image obtained by photographing a virtual object arranged in a virtual space using a virtual camera. In the present embodiment, an indication image 1722 is displayed during the operation of the AR object 1711 that becomes the operation target. The indication image 1722 is formed by an object having a predetermined thickness, and the predetermined thickness is obtained by connecting the operation target to the terminal 20Z (i.e., a lower side of the display screen). In addition, the color of the indication image 1722 changes according to the distance from the lower side to the operation target. In addition, during the operation of the AR object 1711, a light is set at the center of the operation target. The light illuminates objects near the AR object 1711, so that the user can easily visually recognize the positional relationship. Fig.17 , a display area of ​​a portion or all of the AR object 1712 and the AR object 1713 located behind the AR object 1711 is illuminated by a light set at the center of the AR object 1711 .

[0150] In addition, the touch position image 1721 is an image indicating the touch of the user. The configuration of the touch position image 1721 is not particularly limited. The touch position image 1721 can be configured to notify the user of information other than the touch position. As an example of such a configuration, there is a configuration in which the presence or absence of an operation target, the type of the operation target, or its state is displayed in a manner that the user can recognize.

[0151] In this regard, the configuration of the display screen is not particularly limited as long as the user can operate the AR image. That is, for example, the display screen may be configured so that a virtual object that affects the AR object but cannot be operated by the user is displayed, or may be configured so that a UI indicating the operation time is displayed.

[0152] In addition, the display screen can be configured to include functions other than image display as functions related to the operation of the AR object. As an example of such a configuration, there is a configuration in which a vibration unit is set, and whenever the AR object as the operation target is moved a fixed distance, the vibration unit vibrates the terminal 20Z. By configuring the terminal 20Z in this way, the user can easily identify the distance that the AR object actually moves. That is, after the real space is associated with the virtual space, when the terminal 20Z is moved while the finger or the like touches the display screen, the operation target also moves in the virtual space. At this time, the size of the virtual object on the display screen changes according to the positional relationship with the virtual camera. Therefore, even for the same operation target, the user may feel difficult to accurately identify the moving distance between the case where its display size is large and the case where its display size is small. Therefore, by providing a function to notify the user of the effect whenever the operation target moves a fixed distance in the virtual space, the operability of the AR object can be further improved.

[0153] As described above, as one aspect of the sixth embodiment, the user terminal 20Z provided with the imaging device is configured to include the specifying unit 21Z, the generating unit 22Z, the output unit 23Z, the determining unit 24Z, the updating unit 25Z, and the rotating unit 26Z. Therefore, the designation unit 21Z designates a virtual space corresponding to a captured image captured by a camera device; the generation unit 22Z generates a virtual space image obtained by capturing a virtual space including at least one virtual object using a virtual camera corresponding to the camera device (for example, the generation unit 22Z generates an image of a virtual object including coordinate information); the output unit 23Z outputs a synthesized image obtained by synthesizing the captured image and the virtual space image to a display device provided with a touch screen; the determination unit 24Z determines the content of a user operation with respect to the touch screen; the update unit 25Z updates the position or posture of each of the virtual camera and the virtual object in a state where the relationship between the virtual camera and the virtual object is fixed according to a movement or posture change of the camera device while the first operation is continued, in a case where it is determined that the user operation is a first operation (for example, in a case where the terminal 20Z is held), the first operation is an operation in which the touch position with respect to the touch screen does not move after a finger or the like touches the touch screen; and the rotation unit 26Z rotates the virtual object in a predetermined direction in a case where it is determined that the user operation is a second operation in which the position of the finger or the like touching the touch screen slides (for example, in a case where the screen is swiped), the predetermined direction being defined by the second operation. Therefore, the operability of the virtual object (i.e., the AR object) can be improved.

[0154] That is, by making the movement of the user operating the terminal 20Z with one hand (especially the movement of his or her hand) consistent with the movement of the AR object, the user can intuitively operate the AR object. In addition, for example, compared with the method of detecting gestures from an image, it is possible to improve the tracking performance and stably operate the AR object. In addition, by configuring so that the terminal 20Z provided with a camera performs various processes, it is also useful in that no additional equipment is required.

[0155] In addition, in the sixth embodiment described above, the configuration of the user terminal 20Z displaying an object extending from a predetermined position toward an AR object that becomes an operation target is described. However, the configuration of the object (i.e., the specified object) is not particularly limited as long as the configuration is a configuration that enables the user to recognize the operation target. As an example of the configuration of the specified object, there is a configuration in which its form (color or shape) changes according to a predetermined rule based on the distance from the position of the terminal 20Z in the virtual space to the operation object.

[0156] Furthermore, in the sixth embodiment described above, a configuration in which the user terminal 20Z sets a light at the center of the operation target in the case where the user operates an AR object is described. However, the configuration in which the light is set is not particularly limited as long as the configuration is a configuration that allows the user to easily recognize the positional relationship between multiple AR objects in a virtual space or in a synthetic image. As examples of such a configuration, there is a configuration in which a light is set to emit light of a predetermined color onto an object that is an operation target and a configuration in which light of a light whose luminous color changes according to a specific rule is emitted onto an operation target or object selected according to a predetermined rule (for example, a configuration in which a colored light is provided that emits light whose color changes according to the positional relationship between the operation target and another object).

[0157] In addition, although not specifically mentioned in the sixth embodiment described above, the user terminal 20Z may be configured to adopt a character appearing in a video game as a virtual object as part of a function of controlling the progress of the video game. In this case, the user terminal 20Z may be configured so that various parameters that change according to the progress of the video game are reflected in the AR space. As an example of such a configuration, there is a configuration in which a change in the form of a character in a video game is reflected in the form of a virtual object in the AR space.

[0158] As described above, each embodiment according to the present application can solve one or more than two deficiencies. In this regard, the effects of each embodiment are non-limiting effects or an example of non-limiting effects.

[0159] In addition, in the above-mentioned embodiments, each of the plurality of user terminals 20, 201 to 20n and the server 10 performs the above-mentioned various processes according to various control programs (eg, image processing programs) stored in a storage device provided in the corresponding terminal or server.

[0160] In addition, the configuration of the system 100 is not limited to the configurations that have been described as examples of the various embodiments described above. For example, the system 100 may be configured so that the server 10 performs a part or all of the processing described as the processing performed by the user terminal 20. Alternatively, the system 100 may be configured so that any one of the multiple user terminals 20, 201 to 20n (for example, the user terminal 20) performs a part or all of the processing described as the processing performed by the server 10. In addition, the system 100 may be configured so that a part or all of the storage unit included in the server 10 is included in any one of the multiple user terminals 20, 201 to 20n. That is, the system 100 may be configured so that a part or all of the functions of any one of the user terminal 20 and the server 10 of the system 100 are included in the other.

[0161] Furthermore, the system 100 may be configured so that a single device not including a communication network can be programmed to implement a part or all of the functions as exemplified in the above-described embodiments.

[0162] [appendix]

[0163] The explanation of the above-mentioned embodiments has been described so that at least the following invention can be implemented by those having ordinary skill in the art to which the present invention pertains. (1)

[0165] A non-transitory computer-readable medium, comprising a program for causing a computer to implement the following functions, the computer being connected to a camera device, the functions comprising:

[0166] A designation function for designating a virtual space corresponding to a captured image, wherein the captured image is captured by the camera device;

[0167] a generating function for generating a virtual space image, wherein the virtual space image is obtained by photographing the virtual space including at least one virtual object using a virtual camera corresponding to the camera device;

[0168] an output function for outputting a composite image to a display device, wherein the composite image is obtained by combining the captured image and the virtual space image, and the display device includes a touch screen;

[0169] Determine a function, and determine the content of a user operation on the touch screen;

[0170] an updating function of, in a case where it is determined that the user operation is a first operation, updating the position or posture of each of the virtual camera and the virtual object in a state where the relationship between the virtual camera and the virtual object is fixed according to a movement or posture change of the camera device while the first operation is continued, the first operation being an operation in which a touched position of the touch screen does not move after the touch screen is touched; and

[0171] A rotation function is provided for rotating the virtual object along a predetermined direction when it is determined that the user operation is a second operation, the second operation is an operation in which the touch position slides on the touch screen, and the predetermined direction is defined by the second operation. (2)

[0173] The non-transitory computer-readable medium according to (1),

[0174] The updating function includes a function of updating a position or a posture of the virtual object displayed at the touch position of the first operation in the virtual space. (3)

[0176] The non-transitory computer-readable medium according to (1) or (2),

[0177] The rotation function includes a function of rotating the virtual object when the position of the virtual object in the virtual space is fixed. (4)

[0179] The non-transitory computer readable medium according to any one of (1) to (3),

[0180] The updating function includes the following function: when it is determined that the user operation is the first operation and the posture of the computer is rotated by using the shooting direction of the camera device as an axis, the virtual object is rotated around the shooting direction of the virtual camera to update the posture of the virtual object,

[0181] Furthermore, the rotation function includes a function of rotating the virtual object around a direction orthogonal to the shooting direction when it is determined that the user operation is a second operation in which a touch position slides on the touch screen. (5)

[0183] The non-transitory computer readable medium according to any one of (1) to (4),

[0184] Among them, the update function includes the following functions: while the first operation continues, when predetermined conditions in the virtual space are met, the state in which the relationship between the virtual camera and the virtual object is fixed is changed and updated to a state in which the relationship between the virtual camera and the virtual object is not fixed. (6)

[0186] The non-transitory computer readable medium according to any one of (1) to (5),

[0187] The updating function includes the following function: when it is determined that the user operation is an operation of changing the posture of the computer without touching the touch screen, only the position or posture of the virtual camera is updated. (7)

[0189] A non-transitory computer-readable medium containing a program for causing a server to implement at least one of the functions that the program causes a computer to implement as described in any one of (1) to (6), the server being capable of communicating with the computer. (8)

[0191] A computer having installed therein the program described in any one of (1) to (6).

[0192] (9) An image processing system for controlling information processing in response to a user operation, the image processing system comprising a communication network, a server, and a user terminal connected to a camera device, the image processing system further comprising:

[0193] a designation unit that designates a virtual space corresponding to a captured image captured by the camera device;

[0194] a generating unit configured to generate a virtual space image obtained by photographing the virtual space including at least one virtual object using a virtual camera corresponding to the imaging device;

[0195] an output unit, which outputs a composite image to a display device, wherein the composite image is obtained by combining the captured image and the virtual space image, and the display device includes a touch screen;

[0196] a determining unit, configured to determine content of a user operation on the touch screen;

[0197] an updating unit that, in a case where it is determined that the user operation is a first operation, updates the position or posture of each of the virtual camera and the virtual object in a state where the relationship between the virtual camera and the virtual object is fixed according to the movement or posture change of the camera while the first operation is continued, the first operation being an operation in which a touched position on the touch screen does not move after the touch screen is touched; and

[0198] A rotating unit rotates the virtual object in a predetermined direction when it is determined that the user operation is a second operation, the second operation being an operation in which a touch position slides on the touch screen, and the predetermined direction is defined by the second operation. (10)

[0200] The image processing system according to (9),

[0201] The server includes the specifying unit, the generating unit, the determining unit, the updating unit and the rotating unit.

[0202] Furthermore, the user terminal includes the output unit. (11)

[0204] A non-transitory computer-readable medium, comprising a program for causing a user terminal to implement:

[0205] A function to receive from a server information about the functions included in the server; and

[0206] Execute the input or output function corresponding to each of the above functions,

[0207] The functions of the server include:

[0208] A designation function for designating a virtual space corresponding to a captured image, wherein the captured image is captured by a camera device connected to the user terminal;

[0209] a generating function for generating a virtual space image, wherein the virtual space image is obtained by photographing the virtual space including at least one virtual object using a virtual camera corresponding to the camera device;

[0210] an output function for outputting a composite image to a display device, wherein the composite image is obtained by combining the captured image and the virtual space image, and the display device includes a touch screen;

[0211] Determine a function, and determine the content of a user operation on the touch screen;

[0212] an updating function of, in a case where it is determined that the user operation is a first operation, updating the position or posture of each of the virtual camera and the virtual object in a state where the relationship between the virtual camera and the virtual object is fixed according to a movement or posture change of the camera while the first operation is continued, the first operation being an operation in which a touched position on the touch screen does not move after the touch screen is touched; and

[0213] A rotation function is provided for rotating the virtual object along a predetermined direction when it is determined that the user operation is a second operation, the second operation is an operation of sliding a touch position on the touch screen, and the predetermined direction is defined by the second operation. (12)

[0215] A non-transitory computer-readable medium including a program for enabling a server to implement functions, the server being capable of communicating with a user terminal connected to a camera device, the functions including:

[0216] A designation function for designating a virtual space corresponding to a captured image, wherein the captured image is captured by the camera device;

[0217] a generating function for generating a virtual space image, wherein the virtual space image is obtained by photographing the virtual space including at least one virtual object using a virtual camera corresponding to the camera device;

[0218] an output function for outputting a composite image to a display device, wherein the composite image is obtained by combining the captured image and the virtual space image, and the display device includes a touch screen;

[0219] Determine a function, and determine the content of a user operation on the touch screen;

[0220] an updating function of, in a case where it is determined that the user operation is a first operation, updating the position or posture of each of the virtual camera and the virtual object in a state where the relationship between the virtual camera and the virtual object is fixed according to a movement or posture change of the camera device while the first operation is continued, the first operation being an operation in which a touched position on the touch screen does not move after the touch screen is touched; and

[0221] A rotation function is provided for rotating the virtual object along a predetermined direction when it is determined that the user operation is a second operation, the second operation is an operation of sliding a touch position on the touch screen, and the predetermined direction is defined by the second operation. (13)

[0223] A non-transitory computer-readable medium includes a program for causing a user terminal to implement at least one of the functions that the program in (12) causes the server to implement, wherein the user terminal is capable of communicating with the server. (14)

[0225] A server on which the program described in (11) or (12) is installed.

[0226] (15) An image processing method for controlling information processing in response to a user operation, the image processing method comprising:

[0227] specifying processing to specify a virtual space corresponding to a captured image, wherein the captured image is captured by a camera device connected to the computer;

[0228] generating a virtual space image, wherein the virtual space image is obtained by photographing the virtual space including at least one virtual object using a virtual camera corresponding to the camera device;

[0229] Output processing, outputting a synthesized image to a display device, wherein the synthesized image is obtained by synthesizing the captured image and the virtual space image, and the display device includes a touch screen;

[0230] Determine processing, determining the content of the user operation on the touch screen;

[0231] an updating process of, in a case where it is determined that the user operation is a first operation, updating the position or posture of each of the virtual camera and the virtual object in a state where the relationship between the virtual camera and the virtual object is fixed according to a movement or posture change of the camera device while the first operation is continued, the first operation being an operation in which a touched position on the touch screen does not move after the touch screen is touched; and

[0232] Rotation processing, rotating the virtual object along a predetermined direction when it is determined that the user operation is a second operation, the second operation is an operation of sliding the touch position on the touch screen, and the predetermined direction is defined by the second operation. (16)

[0234] An image processing method, wherein an image processing system controls information processing in response to user operations, wherein the image processing system includes a communication network, a server, and a user terminal, and the image processing method includes:

[0235] Specifying a virtual space corresponding to a captured image, wherein the captured image is captured by a camera connected to the user terminal;

[0236] generating a virtual space image, wherein the virtual space image is obtained by photographing the virtual space including at least one virtual object using a virtual camera corresponding to the camera device;

[0237] Output processing, outputting a synthesized image to a display device, wherein the synthesized image is obtained by synthesizing the captured image and the virtual space image, and the display device includes a touch screen;

[0238] Determine processing, determining the content of the user operation on the touch screen;

[0239] an updating process of, in a case where it is determined that the user operation is a first operation, updating the position or posture of each of the virtual camera and the virtual object in a state where the relationship between the virtual camera and the virtual object is fixed according to a movement or posture change of the camera device while the first operation is continued, the first operation being an operation in which a touched position on the touch screen does not move after the touch screen is touched; and

[0240] Rotation processing, rotating the virtual object along a predetermined direction when it is determined that the user operation is a second operation, the second operation is an operation of sliding the touch position on the touch screen, and the predetermined direction is defined by the second operation.

[0241] According to one of the embodiments of the present invention, it is useful in improving the operability of an AR object.

[0242] This application relates to subject matter contained in Japanese Patent Application No. 2019-125973 filed on Jul. 5, 2019, the disclosure of which is expressly incorporated by reference in its entirety.

Claims

1. A non-transitory computer-readable medium comprising a program for causing a computer to implement the following functions, the computer being connected to a camera device, the functions comprising: A designation function for designating a virtual space corresponding to a captured image, wherein the captured image is captured by the camera device; a generating function for generating a virtual space image, wherein the virtual space image is obtained by photographing the virtual space including at least one virtual object using a virtual camera corresponding to the camera device; an output function for outputting a composite image to a display device, wherein the composite image is obtained by combining the captured image and the virtual space image, and the display device includes a touch screen; Determine a function, and determine the content of a user operation on the touch screen; an updating function of, in a case where it is determined that the user operation is a first operation, updating the position or posture of each of the virtual camera and the virtual object in a state where the relationship between the virtual camera and the virtual object is fixed according to the movement or posture change of the camera device while the first operation is continued, the first operation being an operation in which the touched position of the touch screen does not move after the touch screen is touched; as well as A rotation function, in the case where it is determined that the user operation is a second operation, rotating the virtual object along a predetermined direction, the second operation being an operation of sliding the touch position on the touch screen, the predetermined direction being defined by the second operation; The determining function implements the following function: after the virtual object is rotated by the second operation, if the position after sliding on the touch screen from the touch position does not move, determining the operation as the first operation.

2. The non-transitory computer readable medium of claim 1, wherein: The update function includes a function of updating a position or a posture of the virtual object displayed at the touch position of the first operation in the virtual space.

3. The non-transitory computer readable medium of claim 1, wherein: The rotation function includes a function of rotating the virtual object in a state where the position of the virtual object in the virtual space is fixed.

4. The non-transitory computer readable medium of claim 1, wherein: The updating function includes a function of updating the posture of the virtual object by rotating the virtual object around the shooting direction of the virtual camera in a case where it is determined that the user operation is the first operation and the posture of rotating the computer is performed by using the shooting direction of the shooting device as an axis, Furthermore, the rotation function includes a function of rotating the virtual object around a direction orthogonal to the shooting direction when it is determined that the user operation is a second operation in which a touch position slides on the touch screen.

5. The non-transitory computer readable medium of claim 1, wherein: The update function includes the following function: while the first operation continues, if a predetermined condition in the virtual space is met, the state in which the relationship between the virtual camera and the virtual object is fixed is changed and updated to a state in which the relationship between the virtual camera and the virtual object is not fixed.

6. The non-transitory computer readable medium of claim 1, wherein: The update function includes a function of updating only the position or posture of the virtual camera when it is determined that the user operation is an operation of changing the posture of the computer without touching the touch screen.

7. An image processing method for controlling information processing in response to user operation, the image processing method comprising: specifying processing to specify a virtual space corresponding to a captured image, wherein the captured image is captured by a camera device connected to the computer; generating a virtual space image, wherein the virtual space image is obtained by photographing the virtual space including at least one virtual object using a virtual camera corresponding to the camera device; Output processing, outputting a synthesized image to a display device, wherein the synthesized image is obtained by synthesizing the captured image and the virtual space image, and the display device includes a touch screen; Determine processing, determining the content of the user operation on the touch screen; updating processing, in a case where it is determined that the user operation is a first operation, updating the position or posture of each of the virtual camera and the virtual object in a state where the relationship between the virtual camera and the virtual object is fixed according to the movement or posture change of the camera device while the first operation is continued, the first operation being an operation in which the touched position of the touch screen does not move after the touch screen is touched; as well as Rotation processing, rotating the virtual object along a predetermined direction when it is determined that the user operation is a second operation, the second operation is an operation in which the touch position slides on the touch screen, and the predetermined direction is defined by the second operation; In the determination process, after the virtual object is rotated by the second operation, if the position after sliding on the touch screen from the touch position does not move, the operation is determined as the first operation.

8. An image processing system for controlling information processing in response to user operations, the image processing system comprising a communication network, a server, and a user terminal connected to a camera device, the image processing system further comprising: a designation unit that designates a virtual space corresponding to a captured image captured by the camera device; a generating unit configured to generate a virtual space image obtained by photographing the virtual space including at least one virtual object using a virtual camera corresponding to the imaging device; an output unit, which outputs a composite image to a display device, wherein the composite image is obtained by combining the captured image and the virtual space image, and the display device includes a touch screen; a determining unit, configured to determine content of a user operation on the touch screen; an updating unit that, when determining that the user operation is a first operation, updates a position or a posture of each of the virtual camera and the virtual object in a state where the relationship between the virtual camera and the virtual object is fixed according to a movement or a change in posture of the camera device while the first operation is continued, the first operation being an operation in which a touched position on the touch screen does not move after the touch screen is touched; as well as a rotating unit, which rotates the virtual object along a predetermined direction when it is determined that the user operation is a second operation, the second operation being an operation in which a touch position slides on the touch screen, and the predetermined direction is defined by the second operation; The determination unit determines the operation as the first operation if the virtual object does not move from a position after sliding on the touch screen from a touch position after rotating the virtual object by the second operation.

Images