Character display control system, character display control method and program

Abstract

To provide a system, a method and a program which perform display control of a behavior that allows a user to feel to communicate with an augmented reality character.SOLUTION: There is provided a character display control system 1000 in which a character display control device, a plurality of imaging devices and an image display device such as an augmented reality (AR) device are connected capable of enabling an intercommunication through a network N, wherein each of the imaging devices moves on a rail, images a state of a room where a user exists, and transmits image information to the character display control device, the character display control device specifies the condition of the user by performing analysis processing using the image information, generates output information indicating the action of the character relating to the condition of the specified user and transmits the output information to the image display device, and the image display device outputs display information indicating a state in which the character acts, on the basis of the output information while being overlapped with an image of a real world.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a character display control system, a character display control method, and a program. [Background technology]

[0002] In recent years, games and video content using Augmented Reality (AR) and Virtual Reality (VR) have become increasingly popular. In an image representing Augmented Reality, for example, a virtual object character created by CG (Computer Graphics) or the like is synthesized and displayed on an image of a real-world landscape captured by a device such as a smartphone or a Head Mounted Display (HMD).

[0003] Patent Document 1 discloses an augmented reality technology that displays a virtual object superimposed on an image captured by an augmented reality marker. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2013-186691 A Summary of the Invention [Problem to be solved by the invention]

[0005] Generally, images representing virtual reality are displayed when, for example, a specific location on a map is photographed with a device such as a smartphone, or when certain conditions are met regardless of the location, virtual objects such as characters are composited into an image of the real world.

[0006] However, the behavior of the character, which is a virtual object, is merely a patterned movement and is not linked to the user's actions, remarks, etc., so there is a problem that the user does not easily feel as if he or she is communicating with the character. Note that, in the technology described in Patent Document 1, when a mobile phone photographs an object, the virtual object is simply synthesized with an image of the real world and displayed on the device, and communication between the user and the virtual object is not taken into consideration.

[0007] The present invention has been made in consideration of the above-mentioned problems, and has an object to perform display control of an augmented reality character so as to cause the user to feel as if he or she is communicating with the user. [Means for solving the problem]

[0008] The present application includes a number of means for solving at least a part of the above problems, examples of which are as follows: A character display control system according to one aspect of the present invention for solving the above problems is a character display control system having a character display control device, an imaging device, and an image display device, in which the imaging device moves on a rail to capture an image of a room in which a user is present, and transmits the image information to the character display control device, the character display control device identifies the user's state by performing an analysis process using the image information, generates output information representing the character's behavior related to the identified user's state, and transmits the output information to the image display device, and the image display device outputs display information representing the character's behavior, superimposed on an image of the real world, based on the output information.

[0009] In addition, in the above character display control system, the character display control device may identify at least one or more of the user's state from among the user's gaze direction, facial expression, and motion status by analysis processing using the image information.

[0010] In addition, in the above-mentioned character display control system, the character display control device may generate display information or the display information and audio information representing an object in the user's line of sight, the user's facial expression, and the character's behavior related to the user's operational status as the output information.

[0011] In addition, in the above-mentioned character display control system, the character display control device may generate as the output information both the display information in which the character operates around the user, or the display information and the audio information which is the character's speech related to at least one or more of an object in the user's line of sight, the user's facial expression, and the user's motion status.

[0012] In addition, in the above character display control system, at least one of the imaging devices may be installed on the rail, and the character display control device may perform analysis processing using the image information captured by each of the imaging devices.

[0013] Furthermore, in the above-mentioned character display control system, when there are multiple imaging devices, the character display control device may set the imaging device that is in a position where it can capture an image of the user's facial expression as a first imaging device, and follow and capture the user, and set the imaging device other than the first imaging device as a second imaging device, and capture an image of an object in the room.

[0014] In addition, in the above-mentioned character display control system, when the first imaging device is no longer able to capture the user's facial expression due to the user's movement, the character display control device may switch the imaging device that can capture the user's facial expression to the first imaging device.

[0015] Moreover, in the above-mentioned character display control system, when the character display control device generates the output information, it may transmit to the imaging device an output instruction for notification information notifying a user that new output information is available, and when the imaging device receives the instruction, it may notify the user that new output information is available by light or sound.

[0016] In addition, in the above-mentioned character display control system, the character display control device may identify changes in the room by analytical processing using the image information, generate the output information representing character behavior related to the identified indoor changes, and transmit the output information to the image display device.

[0017] In addition, in the above-mentioned character display control system, when the character display control device identifies the content of a user's utterance or a sound in the room, it may generate the output information representing the character's behavior related to the identified content of the utterance or the sound in the room and transmit the output information to the image display device.

[0018] In addition, in the above-mentioned character display control system, the character display control device may generate the output information representing the character's behavior using definition information that defines the character's behavior corresponding to at least one of the user's state, changes in the room, the user's speech, and sounds in the room.

[0019] Furthermore, a character display control method according to another embodiment of the present invention is a character display control method performed by a character display control device, in which the character display control device performs the steps of acquiring image information capturing an image of the state of a room in which a user is present from an imaging device that moves on a rail arranged in the room to capture images, identifying the user's state by performing analytical processing using the image information, generating output information representing character behavior related to the identified user's state, and transmitting the output information to an image display device.

[0020] Furthermore, a program according to another aspect of the present invention is a program that causes a computer to function as a character display control device, and causes the computer to acquire image information capturing an image of the state of a room in which a user is present from an imaging device that moves on a rail placed in the room to capture images, identify the user's state by performing analytical processing using the image information, generate output information representing the behavior of a character related to the identified user's state, and transmit the output information to an image display device. Effect of the Invention

[0021] According to the present invention, it is possible to control the display of an augmented reality character to exhibit behavior that gives the user the impression that the character is communicating with the user.

[0022] Problems, configurations and effects other than those described above will become apparent from the following description of the embodiments. [Brief description of the drawings]

[0023] [Figure 1] 1 is a diagram showing an example of an overall configuration of a character display control system. [Diagram 2] FIG. 1 illustrates an example of an imaging device. [Diagram 3] FIG. 13 is a diagram showing an example of a room visualized based on indoor map information. [Figure 4] FIG. 11 is a diagram showing an example of character action definition information. [Diagram 5] FIG. 11 is a diagram showing another example of character action definition information. [Figure 6] FIG. 11 is a diagram showing another example of character action definition information. [Figure 7] FIG. 1 is a diagram showing an example of positional relationships of users and the like in a room. [Figure 8] 13 is a diagram illustrating an example of a scene in which an imaging device blinks a light source to notify a user that there is new AR output information. FIG. [Figure 9] FIG. 11 is a flow diagram showing an example of a character display control process. [Figure 10] FIG. 11 is a diagram showing an example of character behavior definition information according to the second embodiment. [Figure 11] FIG. 11 is a diagram showing another example of character behavior definition information according to the second embodiment. [Figure 12] FIG. 2 is a diagram illustrating an example of a hardware configuration of a character display control device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0024] Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[0025] First Embodiment [Overall configuration of character display control system 1000] 1 is a diagram showing an example of the overall configuration of a character display control system 1000 according to this embodiment. As shown in the figure, the character display control system 1000 has a character display control device 100, an imaging device 200, and an AR (Augmented Reality) device. The character display control device 100, the imaging device 200, and the AR device 300 are each connected to each other so as to be able to communicate with each other via a predetermined network N such as the Internet or a LAN (Local Area Network).

[0026] In the character display control system 1000, the imaging device 200 captures images of a user in a room and objects such as furniture placed in the room, and transmits the image information to the character display control device 100. The character display control device 100 identifies the state of the user and objects from the acquired image information, and generates AR output information for displaying a character performing an action (e.g., an action, an utterance, etc.) related to the state of the user, etc., synthesized with an image of the real world. The AR device 300 uses the AR output information acquired from the character display control device 100 to output, for example, an augmented reality image in which a character is synthesized with an image of the room captured by a camera mounted on the device itself, or to output the voice uttered by the character from a speaker.

[0027] According to such a character display control system 1000, it is possible to control the display of an augmented reality character so as to cause the user to feel as if the character is communicating with the user.

[0028] Each device in the system will be described in detail below.

[0029] [Imaging Equipment 200] The imaging device 200 is a camera that captures images of a user present in a room or objects such as furniture placed in the room.

[0030] 2 is a diagram showing an example of an imaging device 200. As shown in the figure, the imaging device 200 is fixed to a dolly 201 that moves on a rail 220 installed on a wall 210 inside a room or the like, and moves on the rail 220 to capture images of people and objects. The imaging device 200 moves based on a movement control signal generated by the character display control device 100. Specifically, when the imaging device 200 acquires a movement control signal from the character display control device 100, the imaging device 200 moves on the rail 220 by controlling the dolly 201 based on the movement direction and movement amount indicated by the signal. As an example, the dolly 201 includes a motor that is driven according to the movement control signal and wheels that rotate with the power of the motor, and is configured to be movable by the rotated wheels running on the rail.

[0031] In addition, the imaging device 200 performs imaging processing periodically (for example, every second or every few seconds) and transmits image information to the character display control device 100 via the network N.

[0032] Although there may be only one imaging device 200, it is preferable to have a plurality of imaging devices 200. In the example of Fig. 1, rails 220 are installed on two walls 210 that form an angle, and an imaging device 200 is attached to each rail 220.

[0033] [AR Device 300] The AR device 300 is a device that outputs an augmented reality image in which a character, which is a virtual object, is superimposed on an image of the real world, and a voice uttered by the character. Specifically, the AR device 300 executes a rendering process (e.g., real-time rendering of 3DCG: 3 Dimensional Computer Graphics, etc.) using AR output information to superimpose a character expressing an action related to the behavior of a user or the like on an image of the real world (an image of an interior room) captured by a camera mounted on the device itself, and outputs a voice indicating the contents of the character's speech from a speaker.

[0034] Such an AR device 300 is realized by, for example, a smartphone or a tablet terminal. The smartphone or tablet terminal is equipped with a calculation device, an acceleration sensor, and a gyro sensor, and the calculation device performs rendering processing on the AR output information to superimpose the character on an image of the real world using display information of the character included in the AR output information so that the character has a shape and appearance according to the orientation and inclination of the AR device 300 detected by the acceleration sensor and the gyro sensor. In addition, the calculation device of the AR device 300 performs rendering processing on the AR output information to output the character's spoken voice to a speaker using the character's voice information included in the AR output information.

[0035] The AR device 300 may be smart glasses or an HMD equipped with a computing device, an acceleration sensor, and a gyro sensor. In this case, the smart glasses or the like generate display information for characters and the like by performing the same processing as a smartphone or the like.

[0036] In addition, when the computing device does not have the specifications to perform rendering processing, or when the smart glasses or the like only have the function of a display device, the smart glasses or the like are connected to a smartphone, tablet terminal, or personal computer capable of performing rendering processing, and together form the AR device 300.

[0037] In this case, the smart glasses etc. sequentially output the orientation and tilt detected by the acceleration sensor and gyro sensor to the smartphone etc. or personal computer. In addition, the smartphone etc. or personal computer outputs display information of the character to the smart glasses etc. by rendering processing of the AR output information so that the shape and appearance correspond to the orientation and tilt of the smart glasses etc.

[0038] In addition, it is sufficient for application software that executes processing for displaying augmented reality images to be installed on the smartphone, tablet terminal, smart glasses, or personal computer.

[0039] [Character display control device 100] The character display control device 100 is a device that generates AR output information including display information and audio information of a character displayed in an augmented reality image. The character display control device 100 also generates a movement control signal that controls the movement of the imaging device 200 on the rail 220.

[0040] As shown in the figure, the character display control device 100 has a storage unit 110, a processing unit 120, and a communication unit 130.

[0041] The storage unit 110 is a functional unit that stores various types of information used in the processes executed by the processing unit 120. Specifically, the storage unit 110 stores indoor map information 111, imaging target allocation information 112, and character behavior definition information 113.

[0042] The indoor map information 111 is information in which the positional relationships of furniture and the like in a room are registered.

[0043] 3 is a diagram showing an example of a room visualized based on indoor map information 111. The indoor map information 111 is information for identifying the positions of each object in the room (a sofa and a television in the illustrated example), and is, for example, coordinate information indicated by a coordinate system set in the indoor space (hereinafter, sometimes referred to as an indoor coordinate system). Based on the indoor map information 111, the positional relationship of each object placed in the room is identified.

[0044] Such indoor map information 111 is generated by the indoor state analysis unit 123 performing analysis by image recognition using image information acquired from the imaging device 200, and is stored in the storage unit 110.

[0045] The imaging target allocation information 112 is information that registers imaging targets and the like allocated to each imaging device 200. Specifically, the imaging target allocation information 112 associates identification information of each imaging device 200, the imaging target, and the status of the imaging device 200.

[0046] The identification information of the imaging device 200 is, for example, an identifier such as imaging device 1 or imaging device 2. The imaging target is information that identifies the imaging target such as a user, indoor area 1, or indoor area 2. The status of the imaging device 200 is assigned a status according to the imaging target. For example, the imaging device 200 that images the user is assigned a status of "main device," and the imaging device 200 that images the indoor area 1, indoor area 2, etc. is assigned a status of "sub device 1" or "sub device 2."

[0047] The status of the imaging device 200 is assigned by the user status analysis unit 122. The status is changed (updated) according to the analysis result by the user status analysis unit 122. The process of assigning the status will be described later.

[0048] The character action definition information 113 is information that defines the action of the character related to the action of the user.

[0049] 4 is a diagram showing an example of character behavior definition information 113. The illustrated character behavior definition information 113 is information that defines character behavior related to the user's line of sight, and has records that associate objects (such as a television or a book) in the user's line of sight with actions (such as pointing at the television or book) that are the character's behavior, and questions to the user about the objects.

[0050] 5 is a diagram showing another example of the character behavior definition information 113. The illustrated character behavior definition information 113 is information that defines the character's behavior related to the user's facial expression, and has records in which the user's facial expression (smiling, looking lonely, etc.) is associated with a predetermined action (smiling, looking lonely, etc.) that is the character's behavior, and a question to the user regarding an object is associated with the user's facial expression (smiling, looking lonely, etc.).

[0051] 6 is a diagram showing another example of the character behavior definition information 113. The illustrated character behavior definition information 113 is information that defines the behavior of the character related to the user's motion status, and has records in which the user's motion status (e.g., cooking, drinking, etc.) is associated with a predetermined action (e.g., mimicking the user's motion) that is the character's behavior, and a question to the user regarding an object.

[0052] Such character action definition information 113 is used to generate display information and voice information of the character.

[0053] Returning to Fig. 1, the processing unit 120 is a functional unit that executes various processes executed by the character display control device 100. The processing unit 120 has an instruction unit 121, a user state analysis unit 122, an indoor state analysis unit 123, and a character behavior generation unit 124 as functional units for executing each process.

[0054] The instruction unit 121 is a functional unit that generates a movement control signal that is a movement instruction to the imaging device 200. Specifically, the instruction unit 121 determines a movement direction and a movement amount on the rail 220 based on the analysis result of the image recognition by the user state analysis unit 122 and the indoor state analysis unit 123, and generates a movement control signal for instructing the imaging device 200 to move.

[0055] The user state analysis unit 122 is a functional unit that analyzes the user state. Specifically, the user state analysis unit 122 performs analysis by image recognition using image information acquired from the imaging device 200, and identifies the user's line of sight, facial expression, and motion status.

[0056] Furthermore, the user status analysis unit 122 uses the image information acquired from the imaging devices 200 to identify the imaging devices 200 capturing the user's facial expression, and updates the status of each imaging device 200 in the imaging target allocation information 112. Specifically, the user status analysis unit 122 assigns the status of the main machine to the imaging device 200 capturing the user's facial expression.

[0057] The user state analysis unit 122 performs this process every time image information is acquired from the imaging device 200, and each time, changes the status assigned to the imaging device 200. For example, when the user moves around the room and the image information captured by the imaging device 200 to which the status of the main device is assigned no longer shows the user's facial expression, the status of the main device is assigned to the imaging device 200 that captured the image information showing the user's facial expression.

[0058] Furthermore, the user state analysis unit 122 outputs the analysis result of the image recognition to the instruction unit 121. For example, when the user's entire face is not captured in the image information, the user state analysis unit 122 outputs the analysis result and identification information of the imaging device 200 that captured the image information to the instruction unit 121. Based on the analysis result, the instruction unit 121 determines the movement direction and movement amount of the imaging device 200 so that the user's entire face is captured, and generates a movement control signal for the imaging device 200. This allows the character display control device 100 to constantly acquire images captured by following the user based on the analysis result of the image recognition.

[0059] The indoor state analysis unit 123 is a functional unit that analyzes the indoor state. Specifically, the indoor state analysis unit 123 performs analysis by image recognition using image information acquired from the imaging device 200, and identifies changes in objects in the room (for example, changes in position and posture) based on a comparison with the indoor map information 111.

[0060] Furthermore, the indoor state analysis unit 123 generates indoor map information 111. Specifically, the indoor state analysis unit 123 identifies the positional relationship of objects in the room using image information acquired from the imaging device 200, and generates indoor map information 111 in which the coordinate position of each object indicated by the indoor coordinate system is registered.

[0061] Furthermore, the indoor state analysis unit 123 uses the image information acquired from the imaging devices 200 to identify the imaging devices 200 capturing images of the state of the room, and updates the status of each imaging device 200 in the imaging target allocation information 112. For example, the indoor state analysis unit 123 assigns the status of a sub-device to an imaging device 200 capturing an image of an indoor area not including a user.

[0062] The indoor state analysis unit 123 performs the process every time it acquires image information from the imaging device 200, and changes the status assigned to the imaging device 200 each time. Note that the indoor state analysis unit 123 may periodically (for example, every few tens of minutes) transmit a movement control signal to the imaging device 200 to which the status of a sub-machine is assigned via the instruction unit 121. This allows the indoor state analysis unit 123 to periodically acquire image information of the entire indoor area and perform analysis by image recognition.

[0063] The character behavior generation unit 124 is a functional unit that generates AR output information including display information and audio information of a character performing an action related to the user or the state of the room. Specifically, the character behavior generation unit 124 specifies the behavior of the character (action or question to the user) based on the analysis result of the image recognition by the user state analysis unit 122 and the indoor state analysis unit 123 and the character behavior definition information 113.

[0064] In addition, the character behavior generation unit 124 generates AR output information including display information and voice information of the character corresponding to the specified behavior. Note that the character display information is generated taking into consideration the position of the user in the room and the positional relationship of objects in the room based on the analysis result of the image recognition.

[0065] FIG. 7 is a diagram showing an example of the positional relationship of users and the like in a room. For example, when an action of pointing at a television is specified as the character's behavior, the display position of the character is determined around the user as shown in the figure. Specifically, the character behavior generation unit 124 determines the display position of the character within a predetermined distance from the user (for example, within 1 m from the user) and in the user's line of sight. The user's line of sight may be specified based on the analysis result by the user state analysis unit 122. The character behavior generation unit 124 also calculates the direction in which the character points based on the character's display position, the user's line of sight, and the position of the television ahead of the line of sight, and generates display information reflecting this.

[0066] As another example, the character behavior generation unit 124 generates display information for expressing the movement of the character from the initial display position of the character to the position of an object ahead in the line of sight.

[0067] Furthermore, every time the character behavior generation unit 124 generates AR output information, it transmits an instruction (hereinafter, sometimes referred to as a notification instruction) to the imaging device 200 to notify the user that there is new AR output information. When the imaging device 200 acquires the notification instruction, it blinks a light source (e.g., an LED light) mounted thereon or outputs a notification sound from a mounted speaker.

[0068] FIG. 8 is a diagram showing an example of a scene in which the imaging device 200 blinks the light source to notify the user that there is new AR output information.

[0069] This allows the user to know that there is an augmented reality image showing the character's new behavior.

[0070] The communication unit 130 is a functional unit that performs information communication with an external device. Specifically, the communication unit 130 acquires image information from the imaging device 200 via the network N. The communication unit 130 also transmits a movement control signal to the imaging device 200 via the network N. The communication unit also transmits AR output information to the AR device 300 via the network N.

[0071] [Processing Description] 9 is a flow diagram showing an example of a character display control process executed by the character display control system 1000. The process is started when an execution instruction is transmitted to the character display control device 100 via the AR device 300, for example.

[0072] When the process starts, the character display control device 100 requests the imaging device 200 to obtain image information. Upon receiving the request to obtain image information, the imaging devices 1 and 2 periodically capture images of the interior of the room and transmit the image information to the character display control device 100.

[0073] When the user state analysis unit 122 and the indoor state analysis unit 123 of the character display control device 100 acquire image information from the imaging devices 1 and 2, they each perform analysis using image recognition and assign to the imaging devices 1 and 2 an imaging subject and a status such as a main device or a sub device depending on the imaging subject.

[0074] Furthermore, the user state analysis unit 122 and the indoor state analysis unit 123 each transmit a movement control signal to the imaging devices 1 and 2 via the instruction unit 121 as necessary, and acquire image information captured from the positions after the movement.

[0075] Next, the user state analysis unit 122 performs analysis by image recognition to identify the user state.

[0076] In addition, the character behavior generation unit 124 uses the character behavior definition information 113 to identify a character behavior related to the identified user's state, and generates AR output information including display information and audio information for outputting the identified character behavior.

[0077] Furthermore, the character behavior generation unit 124 transmits a notification instruction for notifying the user that new AR output information is available to the imaging devices 1 and 2 via the communication unit 130. Upon receiving the notification instruction, the imaging devices 1 and 2 notify the user that new AR output information is available by, for example, turning on an LED light or outputting a sound.

[0078] Furthermore, the character behavior generation unit 124 transmits the AR output information to the AR device 300 via the communication unit 130. The AR device 300 renders the AR output information, and when a camera mounted on the AR device 300 is directed to the display position of the character, the AR device 300 displays the display information of the character superimposed on an image of the real world. Furthermore, the AR device 300 outputs the voice information of the character from a speaker mounted on the AR device 300.

[0079] The character display control process has been described above.

[0080] According to the character display control system 1000, it is possible to control the display of the behavior of the augmented reality character so that the user feels as if they are communicating with the character. In particular, since the behavior of the character related to the user's state is output to the AR device 300, the user feels as if they are communicating with the character.

[0081] Second Embodiment The character display control system 1000 of the second embodiment generates display information and audio information for outputting the character's actions related to changes in objects in a room and sounds in the room. Note that the same components and functional units as those of the first embodiment are denoted by the same reference numerals and will not be described repeatedly.

[0082] 10 is a diagram showing an example of character behavior definition information 113 according to the second embodiment. The illustrated character behavior definition information 113 is information defining character behavior related to changes in objects in a room, and has records in which changes in objects (such as objects falling over, or more shopping bags being added) are associated with actions (such as pointing at the changed object) that are the character's behavior, and questions to the user. As another example of an action, a character's behavior such as moving from the character's initial display position to the changed position of the object may be registered.

[0083] 11 is a diagram showing another example of the character behavior definition information 113 according to the second embodiment. The illustrated character behavior definition information 113 is information defining the behavior of the character related to indoor sounds, and has records in which the user's speech or indoor sounds (such as the sound of an intercom) are associated with the character's actions (such as a pose to listen to the speech or pointing in the direction of the sound source) and questions to the user. As another example of the action, a character's behavior such as moving from the character's initial display position toward the direction of the sound (the direction from which the sound was collected) may be registered.

[0084] The indoor state analysis unit 123 according to this embodiment identifies a change in an object in a room based on a comparison between the indoor map information 111 and an analysis result by image recognition of image information acquired from the imaging device 200. Also, the indoor state analysis unit 123 acquires indoor sounds collected via a microphone provided in the imaging device 200 together with the image information from the imaging device 200, for example, and identifies the contents of the user's utterance and indoor sounds by audio analysis.

[0085] In addition, the character behavior generation unit 124 uses the character behavior definition information 113 to identify the character's behavior corresponding to the identified changes in the object or sounds in the room, and generates AR output information including display information and audio information of the character corresponding to the identified behavior.

[0086] In addition, the generation of AR output information regarding such changes in objects may be performed at the same timing as the character display control process of the first embodiment is executed and the generation of AR output information for outputting the character's behavior related to the user state described above.

[0087] According to the character display control system 1000, a character performing an action related to changes in objects or sounds in a room can be superimposed on an image of the real world and displayed, thereby increasing the variety of actions that the character can take.

[0088] [Variation 1] In the above embodiment, a case has been described where an augmented reality image is generated in which a character is superimposed on an image of the real world captured by the AR device 300, but the present invention is not limited to this. The character display control system 1000 may output, for example, a virtual reality (VR) image in which a virtual object character is displayed (appears) on the background of a virtual image that faithfully reproduces the real world, to a device such as a smartphone (corresponding to the above-mentioned AR device 300, and referred to as a VR device here).

[0089] In this case, the VR device transmits an image captured by a camera mounted on the VR device to the character display control device 100, which generates a virtual image that reproduces the real world using the image acquired from the VR device. The character display control device 100 also generates VR output information in which the character generated in the same manner as described above is superimposed on a virtual image of the real world, and transmits this to the VR device. The VR device also renders such VR output information to display a VR image in which the character is superimposed on the virtual image that reproduces the real world.

[0090] Through such processing, the character display control system 1000 can display a character performing an action related to the user state or the indoor state in a VR image.

[0091] In addition, the character display control system 1000 may output, not limited to AR and VR, MR (Mixed Reality) images to a device such as a smartphone, in which a real-world image (including a virtual image that faithfully reproduces the real world) captured by a device such as a smartphone is stereoscopically synthesized with a 3D character image.

[0092] In other words, the character display control system 1000 can realize xR (xReality: cross reality) by combining a real-world image (including a virtual image that faithfully reproduces the real world) with a character image (including both 2D and 3D images), and can display the corresponding images on an image display device such as an AR device.

[0093] [Variation 2] The character display control system 1000 may control IoT (Internet Of Things) home appliances in a room based on the analysis result of image recognition by the character display control device 100, for example. For example, when the character display control device 100 detects that a user (person) has left the room by analyzing image information acquired from the imaging device 200 by image recognition, it controls the lighting in the room by transmitting a signal to the lighting devices in the room to instruct them to turn off the lights. Also, for example, when the character display control device 100 detects (specifies) that the user looks hot (or cold) by analyzing image information acquired from the imaging device 200 by image recognition, it controls the temperature in the room by transmitting a signal to the air conditioner in the room to instruct it to lower (or raise) the set temperature.

[0094] In this way, the character display control device 100 can provide a comfortable living environment for the user by controlling IoT home appliances in the room based on the analysis results from image recognition.

[0095] [Hardware configuration of the character display control device 100] 12 is a diagram showing an example of the hardware configuration of the character display control device 100. As shown in the figure, the character display control device 100 has a processing device 410, a main memory device 420, an auxiliary memory device 430, and a communication device 440, and each component is connected by a bus 450.

[0096] The processing device 410 is an arithmetic device such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit), and executes processing according to a program stored in the main memory device 420 or the auxiliary memory device 430. In the character display control device 100, processing is performed by the processing device 410 which operates according to a program read onto the main memory device 420. The processing unit 120 realizes each function by the processing device 410 executing the program.

[0097] The main memory device 420 is a memory device such as a RAM (Random Access Memory) or a flash memory, and functions as a storage area from which programs and data are temporarily read. The auxiliary memory device 430 is a writable and readable memory device. The functions of the memory unit 110 are realized by the main memory device 420 or the auxiliary memory device 430. The communication device 440 is an interface for connecting the character display control device 100 to an external device for communication.

[0098] The processing of each component of the character display control device 100 may be executed by one piece of hardware or by multiple pieces of hardware. Also, the processing of each component of the character display control device 100 may be realized by one program or by multiple programs.

[0099] In addition, the above-mentioned embodiment examples have been described in detail to make the present invention easier to understand, and the present invention is not limited to those having all the configurations described here. In addition, it is possible to replace a part of the configuration of one example of an embodiment with the configuration of another example. In addition, it is also possible to add the configuration of another example to the configuration of one example of an embodiment. In addition, it is also possible to add, delete, or replace other configurations with respect to a part of the configuration of each example of an embodiment. In addition, each of the above-mentioned configurations, functions, processing units, processing means, etc. may be realized in hardware by designing them as integrated circuits, for example, in part or in whole. In addition, the control lines and information lines in the figures show those that are considered necessary for explanation, and do not necessarily show all of them. It may be considered that almost all of the configurations are connected to each other. [Explanation of symbols]

[0100] 1000...character display control system, 100...character display control device, 110...memory unit, 111...indoor map information, 112...imaging target allocation information, 113...character action definition information, 120...processing unit, 121...instruction unit, 122...user state analysis unit, 123...indoor state analysis unit, 124...character action generation unit, 200...imaging device, 300...AR device, 410...processing unit, 420...main memory unit, 430...auxiliary memory unit, 440...communication device, 450...bus, N...network

Claims

1. A character display control system having a character display control device, an imaging device, and an image display device, the imaging device moves on a rail, captures an image of the room in which the user is present, and transmits the image information to the character display control device; the character display control device performs an analysis process using the image information to identify a state of the user, generates output information representing a character action related to the identified state of the user, and transmits the output information to the image display device; The image display device outputs display information showing the character's behavior on the basis of the output information, superimposed on an image of the real world. A character display control system comprising:

2. The character display control system according to claim 1, The character display control device includes: By performing an analysis process using the image information, at least one of the user's state among the user's gaze direction, facial expression, and motion state is identified. A character display control system comprising:

3. The character display control system according to claim 2, The character display control device includes: Generate, as the output information, display information or the display information and audio information that represent an object in the line of sight of the user, the facial expression of the user, and the behavior of the character that is related to the action status of the user. A character display control system comprising:

4. The character display control system according to claim 3, The character display control device includes: The display information in which the character moves around the user, or The display information and the voice information, which is an utterance of the character related to at least one of an object in the user's line of sight, the user's facial expression, and the user's motion status, Generate the output information A character display control system comprising:

5. The character display control system according to claim 1, At least one imaging device is installed on the rail, The character display control device includes: The image information captured by each of the imaging devices is used to perform analysis. A character display control system comprising:

6. The character display control system according to claim 1, The character display control device includes: When there are a plurality of imaging devices, the imaging device located at a position capable of capturing an image of the facial expression of the user is set as a first imaging device, and the imaging device follows and captures the image of the user; The imaging device other than the first imaging device is set as a second imaging device, and an object in the room is photographed. A character display control system comprising:

7. The character display control system according to claim 6, The character display control device includes: When the first imaging device is no longer able to capture an image of the user's facial expression due to the user's movement, the imaging device capable of capturing an image of the user's facial expression is switched to the first imaging device. A character display control system comprising:

8. The character display control system according to claim 1, The character display control device includes: when the output information is generated, transmitting to the imaging device an output instruction for notification information notifying a user that new output information is available; When the imaging device receives the instruction, the imaging device notifies the user by light or sound that there is new output information. A character display control system comprising:

9. The character display control system according to claim 1, The character display control device includes: Identifying changes in the room through analysis processing using the image information; generating output information representing the character's behavior associated with the identified indoor change and transmitting the output information to the image display device; A character display control system comprising:

10. The character display control system according to claim 1, The character display control device includes: When the user's speech or the sounds in the room are identified, Generate the output information representing the character's behavior related to the identified speech content or the sound in the room, and transmit the output information to the image display device. A character display control system comprising:

11. The character display control system according to any one of claims 1 to 10, The character display control device includes: The output information representing the character's behavior is generated using definition information in which the character's behavior is defined in response to at least one of the user's state, a change in the room, the user's speech, and a sound in the room. A character display control system comprising:

12. A character display control method performed by a character display control device, comprising: The character display control device acquires image information capturing an image of a room in which a user is present from an imaging device that captures images by moving on a rail arranged in the room; Identifying a state of a user by performing an analysis process using the image information; generating output information representing a character's behavior associated with the identified user state; transmitting the output information to a visual display device. A character display control method comprising:

13. A program for causing a computer to function as a character display control device, The computer includes: Acquire image information capturing an image of the room in which the user is present from an imaging device that moves on a rail installed in the room and captures images; Identifying the user's condition by performing an analysis process using the image information; generating output information representing a character's behavior related to the identified user state; causing an image display device to transmit said output information; A program characterized by:

Images