Information processing device and program

Abstract

To improve the diversity of movements of at least one of the following: human-generated avatar movements and robot movements. [Solution] The skeletal information generation system H1 generates skeletal information from the detection target image KTG represented by the target video ST1, and outputs the generated skeletal information as a data sequence ST5 to the posture evaluation system H2 as the detection target image KTG ST6. When the posture evaluation system H2 performs posture evaluation of the detection target image KTG using the example data TD, it generates text information to be transmitted to the detection target KT by performing a difference AI analysis SH2 with the example using the difference between the skeletal information contained in the example data TD and the input skeletal information. This text information is output as audio from the audio output system TO, and the detection target image KTG and the example image OG contained in the example data TD are displayed by the image display system GH.

Description

Technical Field

[0001] The present invention relates to an information processing apparatus and a program.

Background Art

[0002] Techniques for estimating the skeletal information of a person included in an image captured by a camera are known (see, for example, Patent Document 1). In addition, techniques for remotely controlling a robot based on the movement of a person and causing the robot to perform a specific operation are also known (see, for example, Patent Document 2).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0004] Robots as described above are controlled to operate in accordance with human physical movements or operations. In such motion control, the motion of the robot not only becomes what is expected in advance by the person making the physical movement including the operation, but also depends only on that person. Depending on the use of the robot, it is considered desirable to improve the diversity of the robot's operations in consideration of convenience for people or the impression given to people. This is the same when operating an image such as an avatar according to a person's movement.

[0005] An object of the present invention is to provide an information processing apparatus and a program capable of improving the diversity of operations of at least one of an avatar and a robot according to human movement.

Means for Solving the Problems

[0006] An information processing apparatus in one aspect of the present disclosure includes: an image data acquisition means for acquiring image data obtained by imaging a target to be detected; a skeletal information generation means for generating skeletal information including joint position information and relationship information indicating the relationship between joints in the target to be detected image based on the image data of the target to be detected; an analysis means for analyzing the motion content in the target to be detected image based on the skeletal information; a text information generation means for generating text information based on the analysis results of the motion content; and an motion control means for operating at least one of a predetermined avatar to be displayed and a robot which is a physical machine as the target to be operated on, based on the text information.

[0007] A program according to one aspect of the present disclosure generates skeletal information, including joint position information and relationship information indicating the relationships between joints, based on an image of a target to be detected, which is represented by image data obtained by imaging of a target to be detected; analyzes the motion content in the target to be detected based on the skeletal information; generates text information based on the results of the motion content analysis; and executes a process to operate at least one of a predetermined avatar to be displayed and a robot, which is a physical machine, as the target of the motion, based on the text information. [Effects of the Invention]

[0008] This invention can improve the diversity of movements of at least one of a human avatar and a robot. [Brief explanation of the drawing]

[0009] [Figure 1] This figure illustrates a service provision system constructed by applying the present invention, and an example of the content of the services provided by that system. [Figure 2] A diagram illustrating an example of an action that includes a posture for causing the target object to perform another action. [Figure 3] This figure illustrates an example of a network environment to which an AP server, according to one embodiment of the information processing device of the present invention, is connected. [Figure 4] This diagram illustrates an example of a space for games and similar activities. [Figure 5] This is a block diagram showing an example of the hardware configuration of an AP server according to one embodiment of the information processing device of the present invention. [Figure 6] This is a functional block diagram showing an example of a functional configuration implemented on an AP server according to one embodiment of the information processing device of the present invention. [Figure 7] This flowchart shows an example of video display processing performed by the CPU installed in the AP server, which is an information processing device according to this embodiment. [Modes for carrying out the invention]

[0010] The embodiments for carrying out the present invention will be described below with reference to the figures. The embodiments described are merely examples, and the technical scope of the present invention is not limited thereto. The technical scope of the present invention also includes various modifications.

[0011] Figure 1 illustrates an example of a service provision system constructed by applying the present invention, and the content of the services provided by that system. Here, the services provided by this service provision system SY will be referred to as "this service" in the explanation.

[0012] In the example shown in Figure 1, the camera C is assumed to be capturing images (video) of an animal, primarily a human. In this example, a human is designated as the target of detection KT, which is the object of skeletal detection. It is assumed that information to be transmitted to the target KT will be communicated in a timely manner via audio output, depending on the actions of the target KT.

[0013] To verify the operation of the detected KT, skeletal information of the detected KT is generated. This skeletal information includes positional information of human joints and relationship information indicating the relationships between joints. Relationship information, for example, represents the distance and direction between adjacent joints on the skeleton. Such skeletal information is generated by analyzing the target image ST1 obtained by imaging with camera C. Thus, the skeletal information of the detected KT is generated from the target image ST1. The skeleton image SG shown in Figure 1 represents an example of the joints from which positional and relationship information is generated. The generation of skeletal information itself is performed using well-known techniques.

[0014] Note that there may be multiple cameras C used for imaging. If multiple cameras C are installed in different positions, the target images ST1 obtained from each camera C will allow for the confirmation of parts of the target KT that would be blind spots with one or more cameras C. Therefore, the generation of skeletal information can be performed with higher accuracy. For example, this is particularly useful when there are two or more targets KT. For the sake of explanation, we will assume that there is only one camera C in this explanation.

[0015] As shown in Figure 1, the service provision system SY includes a skeletal information generation system H1, a posture evaluation system H2, and a posture management system H3. In Figure 1, each system H1 to H3 is shown separately, but these systems (and their functions) may be implemented on the same information processing device. Two of them may even be implemented on the same information processing device. Here, as shown in Figure 1, we assume that each system H1 to H3 is implemented on a different information processing device; in other words, we assume that the service provision system SY is constructed using three or more information processing devices.

[0016] The skeletal information generation system H1 is implemented on an information processing device equipped with an OS (Operating System) H11. The OS H11 is equipped with a camera device video acquisition and management switch (SW) H111. This camera device video acquisition and management switch H111 is a function that switches the destination to which various information including the target video ST1 received by the skeletal information generation system H1 is passed. As a result, the target video ST1 received by the skeletal information generation system H1 is passed as the target video ST2 to the processing engine H112 by the camera device video acquisition and management SW H111. The processing engine H112 and the skeletal detection unit H113 are functions realized by, for example, application programs (hereinafter abbreviated as "applications") that operate on the OS H11.

[0017] The processing engine H112 is a function that performs overall control for providing this service. The processing engine H112 causes the skeletal detection unit H113 to execute the skeletal detection process by the skeletal detection process call ST3 and generate skeletal information using the target video ST2. For this purpose, the processing engine H112 passes, for example, the target video ST2 to the skeletal detection unit H113 by the skeletal detection process call ST3. What is passed from the processing engine H112 to the skeletal detection unit H113 may be not the target video ST2 but only the video information of the part including the detection target image. Here, it is assumed that the target video ST2 is passed from the processing engine H112 to the skeletal detection unit H113.

[0018] The skeletal detection unit H113 executes the skeletal detection process for generating skeletal information and returns the processing result ST4 of the skeletal detection process to the processing engine H112. This processing result ST4 is the skeletal information generated in the form of a data sequence of skeletal expressions.

[0019] The processing engine H112 can output the skeletal information, which is the processing result ST4 returned from the skeletal detection unit H113, as a data sequence ST5 of skeletal expressions to the posture evaluation system H2 and the posture management system H3, respectively. The posture evaluation system H2 is also output with the detection target image KTG, which is the image of the detection target KT, as the detection target image ST6.

[0020] The posture evaluation system H2 can output, as a part of the image ST8 to be displayed, the image of the detection target ST6 input from the skeleton information generation system H1 or the image of the character generated using the data sequence ST5 of the skeleton representation to the image display system GH. The image ST8 is, for example, for one-screen display, and the image of the character is arranged and displayed within that screen. The image of the character is assumed to be a clone of the detection target KT.

[0021] Even when displaying other character images, the characters may include a clone of another detection target KT or a clone of a virtual personality. The virtual personality is specifically assumed for the provision of this service. Hereinafter, among the characters, the characters positioned as such clones are referred to as "avatars". Unless otherwise specified, "character" is used to refer to those other than avatars among the characters.

[0022] The image display system GH displays a screen on the display D based on the image ST8 input from the posture evaluation system H2. Therefore, the posture evaluation system H2 can arbitrarily change the content of the screen to be displayed on the display D through the image ST8 output to the image display system GH. FIG. 1 shows a state where the detection target image KTG represented by the image ST8 is displayed on the display D. Instead of the detection target image KTG, an avatar image generated from the skeleton information may be displayed.

[0023] The display D is, for example, a large display device or a screen or the like. When the display D is a screen, the image display system GH also includes a projector capable of projecting onto the screen. On the other hand, the posture evaluation system H2 generates text information that is the target of voice output through analysis using the data sequence ST5 of the skeleton representation. For this purpose, the posture evaluation system H2 performs differential AI (Artificial Intelligence) analysis SH2 with the model. As will be explained in more detail later, this text information also functions as control information that controls various actions of the avatar image or the robot that is the target of the action. These actions include not only actions that visually change the target of the action, but also displays that do not involve visual changes, or actions that involve speaking by emitting sound.

[0024] The SH2 AI analysis of differences from the example is based on the premise that the posture (pose) that the detected object KT should adopt at some point is known in advance. For this analysis, example data TD representing the ideal posture is prepared. Examples of movements where the posture to be adopted is known in advance include dancing, games that require actions that imitate the movements of an example or actions that require taking one or more defined postures, and exercises where an ideal movement can be considered (e.g., sports). Hereafter, such requests or a series of actions resulting from exercise will be collectively referred to as "posture-requiring actions."

[0025] This example data TD includes not only the example image OG, which is an image of the example posture, but also skeletal information. Therefore, the difference AI analysis SH2 with respect to the example performs an AI analysis using this difference, including calculating the difference between the skeletal representation data sequence ST5 and the skeletal information contained in the example data TD. This analysis evaluates the posture represented by the detected image KTG, and text information is generated according to the evaluation result. In the example shown in Figure 1, the difference correction instruction ST9 output from the posture evaluation system H2 to the audio output system TO is for instructing the output of the text information as sound. Specifically, this difference correction instruction ST9 is either an audio signal generated for output, or a command containing the information necessary to generate that audio signal.

[0026] For difference-based analysis, the AI ​​employs deep learning using training data that represents the relationship between the difference and the text information to be generated. Through this deep learning, the SH2 difference analysis AI can generate appropriate text information by generating (calculating) the difference. Since the ratio between the size of the target image KTG and the size of the example image OG represented by the example data TD is not always within an appropriate range, the difference is generated after, for example, scaling one of the two skeletal information pieces.

[0027] Figure 1 shows that the display D displays both the target image KTG and the example image OG. The most significant difference between the detected image KTG and the example image OG is the right hand. The left hand is also clearly different, but the difference is smaller compared to the right hand. Therefore, in this case, text information representing a string such as "raise the right hand slightly" is generated by the difference AI analysis SH2 with the example. As a result, this string is output as audio from the audio output system TO. This audio output is performed assuming that the detected image KTG or the example image OG is an avatar image.

[0028] The voice output by the voice output system TO is performed assuming the detection target KT. Therefore, for the detection target KT, useful information for taking the appropriate posture is obtained in a timely manner through voice output from the detection target image KTG or example image OG displayed as an avatar image on the display D. As a result, the displayed detection target image KTG or example image OG is treated as the target of action, and a speaking action is virtually performed by generating text information from the detection target image KTG, causing that target to speak.

[0029] Note that the actions of the target object do not necessarily have to be speech actions, i.e., voice output. For example, the target object may be made to perform movements necessary to bring the detected KT closer to the posture of the example image OG. Alternatively, text may be displayed as the content of the avatar image's speech. Voice output (output of messages, etc.) will only be performed if the posture of the detected image KTG is evaluated as inappropriate. If the posture is evaluated as appropriate, a sound effect or dramatic sound may be emitted so that the detected KT can recognize this. Therefore, text information may represent the emission of a sound effect or dramatic sound. Speech actions may be represented by either emission or display.

[0030] As described above, the posture evaluation system H2 uses the skeletal representation data sequence ST5 input from the skeletal information generation system H1 to evaluate the posture of the detection target KT, generate text information, and realize the movement of the avatar image according to that text information. In this way, it causes the target to perform a different action than that of the detection target KT, and provides useful information to the detection target KT through this different action. As a result, not only is the diversity of actions of the target improved, but the convenience for the detection target KT is also improved. The detection target KT will be able to perform more appropriate posture request actions more easily.

[0031] For the detected KT, recognizing the audio output is required in order to move its body appropriately. Therefore, a higher level of concentration is necessary. Because it must perform actions in such a state, the detected KT is more likely to achieve a higher level of immersion. As a result, the detected KT may be given a better impression or a higher level of satisfaction.

[0032] On the other hand, the posture management system H3 uses the skeletal representation data sequence ST5 input from the skeletal information generation system H1 to control the movement of at least one of the following: a robot, which is a physical machine, or an avatar image displayed on the screen. As a result, the object being controlled moves in accordance with the movement of the detected image KTG. Therefore, the detected object KT can control the movement of the object being controlled through its own movement. Consequently, the detected object KT is provided with an environment in which it can enjoy controlling the object being controlled as entertainment, or can check its own posture or movement. A physical machine here refers to a device equipped with one or more power sources such as motors. A robot, being a physical machine, is capable of changing its overall shape and form through the power transmitted from the power source.

[0033] Robot motion control is performed, for example, by transmitting control information to the robot as an action / display instruction ST7 from the attitude management system H3, which is used to instruct the robot to assume the next desired posture. This control information is mainly for operating the power source. Avatar image motion control is performed, for example, by transmitting image information representing the screen to be displayed as an action / display instruction ST7 from the attitude management system H3 to an image / sound generation system capable of displaying a screen containing the avatar image as a video.

[0034] The proportions of each part of the robot and avatar image being operated on usually differ from those of the detected image KTG. For example, if the detected image KTG is 8 heads tall, the robot being operated on is often less than 8 heads tall, and their proportions are usually different. Therefore, the posture management system H3 performs a scaling process SH3 to match the proportions of the input skeletal information to those of the robot being operated on, and manipulates that skeletal information. This manipulation is performed by referring to parameters P, which are provided according to the difference in proportions. By generating and transmitting an operation / display instruction ST7 using the skeletal information after such manipulation, the robot being operated will move in accordance with the movement represented by the detected image KTG, without giving an unnatural impression.

[0035] Furthermore, the posture management system H3 may also prepare example data TD and generate and output text information. This would allow for timely transmission of information to the detection target KT, such as the quality of posture at key points or information about the next posture to adopt. If information about the next posture to adopt is to be transmitted, the example data TD may also be used to determine whether the detection target image KTG is in a posture that should generate text information. The other action in the aforementioned target is performed by comparing it to a single posture represented by a single example data TD. However, the other action may also be performed by comparing it to an action that includes posture.

[0036] Figure 2 illustrates an example of a movement that includes a posture for causing an object to perform another movement. Each circle in Figure 2 represents a different joint. The example shown in Figure 2 is a hypothetical scenario of a ninja throwing a shuriken. The action of throwing the shuriken is as if it were thrown with the left hand. The left side of Figure 2 shows an example of the posture taken when throwing a shuriken. In this posture, the right elbow is bent, the right hand holding the shuriken is positioned at waist height and close to the waist, and the shuriken is grasped with the left hand. Hereafter, this posture will be referred to as the "initial assumed posture" for convenience.

[0037] On the other hand, the right-hand diagram in Figure 2 shows an example of the posture after throwing a shuriken. This posture is an example of the posture after moving the left hand to the right and throwing the shuriken from the initial assumed posture shown in the left-hand diagram of Figure 2. When viewed from above, the body has rotated counterclockwise in order to throw the shuriken.

[0038] To throw a shuriken held in the left hand, a relatively large movement of that left hand is required, as shown in Figure 2. Therefore, by checking the movement from the starting point of the hand assumed to be holding the shuriken, starting from the initial assumed posture, it is possible to determine whether or not the detected KT performed the action of throwing a shuriken. Because such determination is possible, if it is determined that the action of throwing a shuriken has been performed, it is possible to generate text information and emit sounds such as the sound of the shuriken cutting through the air or other sound effects. Emitting such sounds makes it possible to provide an environment that can be made to feel more realistic, especially in games that involve physical movement. The emission of such sounds itself can be positioned as a separate action from the action of throwing a shuriken, which was part of the detected image KTG. In addition to determining whether or not the above-mentioned action of throwing a shuriken has been performed, it is also possible to further determine the position where the shuriken was released and the direction in which it was thrown.

[0039] To enable the detection of movements as exemplified in Figure 2, the example data TD may include data such as the following: a first condition which should be considered the initial assumed posture, a second condition which indicates that the action defined from that initial assumed posture has been performed, and a time range which is the period from when the first condition is no longer met until the second condition is met.

[0040] In games and other applications, if the action of throwing a shuriken is to be detected, the target of the action could be an avatar image representing an opponent or other player. In such a case, upon confirmation of the shuriken throwing action, the target could be made to make a statement indicating that a shuriken has been thrown, and also perform an action to respond to the shuriken. In other words, the generation of one piece of text information could generate one or more pieces of text information.

[0041] For these reasons, various approaches are possible regarding the movement of the detection target image (KTG) that generates text information, the content of the text information generated by that movement, and the control of the actions after outputting the generated text information. For example, one could focus on actions that assume the use of weapons other than shuriken and consider specifications accordingly. However, regardless of the specifications, this will not only broaden the range of expression through the movement of the target, but also increase the amount of information provided to the detection target (KT). As a result, the satisfaction level of the detection target (KT) can be increased.

[0042] From here on, we will explain in detail the specific implementation method for providing the service exemplified in Figure 1, referring to Figures 3 to 7. Figure 3 illustrates an example of a network environment to which an AP (Application) server, according to one embodiment of the information processing device of the present invention, is connected.

[0043] AP Server 1 is an information processing device installed by SK, the service provider that provides this service. SK contracts with organizations such as companies that provide entertainment such as games, and through the provision of this service to these contracted organizations, it helps to increase the satisfaction of visitors who use those organizations. In Figure 3, AP Server 1 is shown as being located within the service provider SK, but it may be located elsewhere. For example, it may be provided through a cloud service.

[0044] AP Server 1 is actually connected to Network N via other information processing devices, such as a proxy. However, for the sake of explanation, we will ignore the information processing devices that interpose between AP Server 1 and Network N. In other words, AP Server 1 is assumed to be directly connected to Network N. Network N includes, for example, the Internet.

[0045] Client company KK is an organization that has contracted with service provider SK. Client company KK provides a space where users can engage in physical games and other activities. This service enables the provision of a more satisfying environment for users of this space. Hereafter, "Client company KK" will be used as a general term for the contracted organization. There are multiple entities named Client company KK.

[0046] In addition to the camera C mentioned above, client company KK has a server 2, a projector 3, a sound system 4, and a microphone (abbreviated as "microphone" in Figure 3; this abbreviation will be used hereafter) 5 installed. All except server 2 are installed in a single location where games and other activities can be conducted. Therefore, it is common for all except server 2 to exist in multiples.

[0047] Figure 4 illustrates an example of a space for games and the like. The space for games, etc. (hereinafter referred to as the "game area") is a space where a screen SC is installed or can be installed, as shown in Figure 4. The space is equipped with a projector 3 capable of projecting onto the screen SC, a sound system 4 capable of emitting various sounds, a camera C, and a microphone 5. Camera C is positioned to capture images of users from the screen SC side.

[0048] Server 2 is connected to camera C and microphone 5. Projector 3 and sound system 4 are connected to AP server 1 via network N. Sound system 4 is a system that includes a speaker, which is a sound-emitting device, and also includes an information processing device that functions as a terminal capable of communication via network N. Thus, sound system 4 emits sound from the speaker in response to an audio signal transmitted from AP server 1.

[0049] Projector 3 is an information processing device equipped with communication capabilities via network N. It generates a screen to be projected based on a video signal transmitted from AP server 1, and projects the generated screen onto screen SC. Camera C is used to record video of game field users. The recording results are transmitted as image data (video information) from Server 2 to AP Server 1. All, or at least one, of these users are detection target KTs. Camera C is installed on the Screen SC side because it is assumed that users who are detection target KTs will face Screen SC and play games that involve physical movement.

[0050] Microphone 5 is installed to pick up voices emitted by users of the game area. The audio signal output from microphone 5 is converted into audio information by server 2 and sent to AP server 1.

[0051] Figure 5 is a block diagram showing an example of the hardware configuration of AP Server 1 according to one embodiment of the information processing device of the present invention. Next, with reference to Figure 5, an example of the hardware configuration of AP Server 1 will be specifically described. Note that this configuration example is just one example, and the hardware configuration of AP Server 1 is not limited to this.

[0052] As shown in Figure 5, AP Server 1 includes a CPU (Central Processing Unit) 11, ROM (Read Only Memory) 12, RAM (Random Access Memory) 13, a bus 14, an input / output interface 15, an output unit 16, an input unit 17, a storage unit 18, a communication unit 19, and a drive 20.

[0053] The CPU 11 executes various processes according to the programs recorded in the ROM 12 and the programs loaded from the storage unit 18 into the RAM 13. The programs loaded from the storage unit 18 into the RAM 13 include, for example, the OS and various application programs that run on that OS. The various application programs include one or more that were developed for the provision of this service.

[0054] RAM13 also stores data necessary for the CPU11 to perform various processes. This data includes various programs that the CPU11 executes. The CPU 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output interface 15 is also connected to this bus 14. An output unit 16, an input unit 17, a storage unit 18, a communication unit 19, and a drive 20 are connected to the input / output interface 15.

[0055] The output unit 16 includes a display such as an LCD. The output unit 16 displays various images or screens under the control of the CPU 11. The output unit 16 may be installed on the AP server 1, or it may be connected as needed. In other words, the output unit 16 is not an essential component.

[0056] The input unit 17 includes, for example, various hardware buttons such as a keyboard. It may also include one or more pointing devices such as a mouse. The operator can input various information via the input unit 17. This input unit 17 may be installed on the AP server 1, but it may also be connected as needed. In other words, the input unit 17 is not an essential component.

[0057] The storage unit 18 is, for example, an auxiliary storage device such as a hard disk drive or an SSD (Solid State Drive). Large amounts of data are stored in this storage unit 18. The communication unit 19 enables communication with other information processing devices via the network N. The server 2, projector 3, and sound system shown in Figure 3 all correspond to other information processing devices.

[0058] The drive 20 is a device that can insert and remove removable media 25, such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory card. The drive 20 can, for example, read information from and write information to the inserted removable media 25. As a result, a program recorded on the removable media 25 can be stored in the storage unit 18 via the drive 20. Furthermore, the removable media 25 inserted in the drive 20 can be used as a copy destination or transfer destination for various data stored in the storage unit 18.

[0059] The application program developed for this service may be recorded on removable media 25 and distributed. It may also be made available for distribution via network N, etc. Therefore, the recording medium on which the application program is recorded may be one mounted or attached to an information processing device directly or indirectly connected to network N, or one mounted or attached to an externally accessible device.

[0060] The hardware resources of AP Server 1 are controlled by various programs, including application programs. As a result, AP Server 1 can provide this service to customer company KK.

[0061] Figure 6 is a functional block diagram showing an example of a functional configuration implemented on AP Server 1 according to one embodiment of the information processing device of the present invention. Next, an example of a functional configuration implemented on AP Server 1 will be described in detail with reference to Figure 6. To avoid confusion, we will assume there is only one game field. We will also assume that a game takes place in the game field. We will assume that both the avatar image and the robot, which are the targets of the actions, are operated using skeletal information generated from the detection target image (hereinafter also referred to as the detection target image KTG).

[0062] On the CPU 11 of AP Server 1, the following functional configurations are implemented, as shown in Figure 6: a setting unit 111, an image extraction unit 112, a skeleton detection unit 113, a posture evaluation unit 114, a text conversion unit 115, a video generation unit 116, a posture control unit 117, a speech recognition unit 118, a conversation processing unit 119, an image recognition unit 120, and a face recognition unit 121.

[0063] These are realized by the CPU 11 executing various programs, including application programs developed for the provision of this service. As a result, or for the purpose of providing this service, the memory unit 18 is allocated a reference image storage unit 181, a character image storage unit 182, a background image storage unit 183, a dictionary storage unit 184, and a control information storage unit 185.

[0064] The settings unit 111 has functions for setting up games played in the game field, starting or ending the set games, and various settings for playing the set games. These various requests for settings are sent from, for example, the server 2.

[0065] Figures 3 and 4 omit the device that enables this configuration. This device is connected to Server 2, which processes requests from this device and generates and sends requests to AP Server 1. Server 2, for example, starts recording video using camera C from before the game starts until it ends, and continuously transmits the image data (video information) obtained from this video recording to AP Server 1. During this time, audio information obtained by converting the audio signal from microphone 5 is also transmitted from Server 2 to AP Server 1.

[0066] The image extraction unit 112 identifies the range of the detection target image KTG in the image data received from the server 2 and extracts the identified range. This range extraction is performed, for example, by generating still image data from the image data at predetermined time intervals, and then targeting the generated still image data. The image data extracted in this way will hereafter be referred to as "detection target image data".

[0067] The skeleton detection unit 113 generates skeleton information from the detection target image KTG represented by each detection target image data (see, for example, Patent Document 1). The posture evaluation unit 114 uses the skeletal information generated by the skeletal detection unit 113 to evaluate the posture of the detected image KTG, or the movement including the posture. This evaluation is basically the same as, for example, the difference AI analysis SH2 with the example shown in Figure 1, or it is based on the initial assumed posture and subsequent movement as shown in Figure 2. To perform this evaluation, reference image information corresponding to the example data TD is stored in the reference image storage unit 181. Therefore, the corresponding reference image information is also used in the evaluation.

[0068] The text conversion unit 115 generates text information by converting the evaluation results from the posture evaluation unit 114. The generated text information depends on the reference image information actually used for evaluation, and the difference between the action content represented by that reference image information and the action content represented by the detected image KTG, as explained with reference to Figures 1 and 2. The dictionary storage unit 184 stores a dictionary for generating text information, etc., according to the assumed game. As a result, for example, a sentence or phrase registered in the dictionary, or a combination of two or more strings, is generated as text information.

[0069] The video generation unit 116 generates an image (screen) to be projected onto the projector 3, and transmits the generated image as image data to the communication unit 19. The video generation unit 116 also generates information for emitting music, sound effects, sound effects, and voices intended for the detected KT to hear, as needed, and transmits the generated information to the communication unit 19. This provides an environment in which the detected KT can play games, etc., while various sounds are emitted. Hereafter, the information for emitting various sounds will be collectively referred to as "voice information".

[0070] The character image storage unit 182 stores various image information for generating various character images, including avatar images. This image information includes, for example, 3D representation image information for each part that makes up the avatar image. The background image storage unit 183 stores image information for generating images other than various character images as background image information. As a result, the video generation unit 116 generates a background image from the various image information stored in the background image storage unit 183, and generates each avatar image using the skeletal information and the various image information stored in the character image storage unit 182. After generating these, the video generation unit 116 generates an image for one screen by placing each avatar image on the background image.

[0071] The posture control unit 117 generates posture control information for operating the robot using skeletal information. This posture control unit 117 makes it possible to operate the robot in accordance with the movement of the detected image KTG, either in place of or in conjunction with the avatar image. The generation of posture control information is performed by referring to other control information stored in the control information storage unit 185. This control information is for generating posture control information, and the parameter P shown in Figure 1 is stored in the control information storage unit 185 as one of the pieces of control information.

[0072] The speech recognition unit 118 performs speech recognition by natural language processing using, for example, the speech information received from the server 2. Through this speech recognition, the speech recognition unit 118 generates a string of text information that represents the content of what the detected KT said. The string is generated by identifying each character represented by the speech information through pattern matching. This string corresponds to the first string in this embodiment. The conversation processing unit 119 considers the generated string to be the content of the detected KT's statement and generates a response string to return to that statement. To this end, the conversation processing unit 119 sequentially performs morphological analysis, syntactic analysis, semantic analysis, contextual analysis, and anaphora analysis on the generated string to identify the content that the string represents. Using this identification result, the conversation processing unit 119 generates a response string, for example, that is associated with that identification result. The string generated as a response string corresponds to the second string in this embodiment.

[0073] The response sentences generated by the conversation processing unit 119 and the audio information for emitting the text information generated by the text conversion unit 115 are generated, for example, by the video generation unit 116 and transmitted from the communication unit 19. As a result, the detected target KT can not only see the images (videos) projected onto the screen SC by the projector 3, but also hear the audio for conversations etc. emitted by the sound system 4. The sounds emitted include not only voices but also music, sound effects, and sound effects. Unless otherwise specified, here, "speech information" is used as a general term for these sounds, as well as the information that causes these various sounds, including voices, to be emitted. If the target of the operation is a robot, the generation of audio information is performed by the attitude control unit 117. As a result, even when operating a robot, the detected target KT can hear voices such as conversations.

[0074] The voice recognition unit 118 and the conversation processing unit 119 enable conversation between the detected KT and the character whose image is projected onto the screen SC. This allows the detected KT to progress through the game while conversing with the character. Through this conversation, the detected KT can make choices such as actions or ways of progressing in the game that it had not considered before. As a result, the detected KT can not only be made more immersed in the game, but the range of ways to enjoy it is also broadened, leading to greater satisfaction. Actions that the detected KT had not considered before include the appearance (and subsequent actions) of other avatar images or character images in response to its own speech, actions or changes in its own avatar image in response to the content of its speech, changes in the background image, etc. Furthermore, both the speech recognition unit 118 and the conversation processing unit 119 employ well-known technologies.

[0075] The image recognition unit 120 recognizes objects that are included in or near the detection target image KTG. These objects should be considered separate from the detection target KT. These objects are mainly items worn or held by the detection target KT. After recognizing this object image, the image recognition unit 120 identifies the operation of the object image by focusing on changes in the position, orientation, or size of the object image. Identification of the operation can also be performed by employing well-known techniques in the image recognition unit 120. By specifying the nature of the operation, the object image can be changed depending on how the detected KT handles the object. Specifically, if the object is a sword-shaped object, for example, the representation of the object image may be changed depending on its positional relationship with the detected KT, or effects may be added to the object image depending on how fast the sword-shaped object is swung.

[0076] The face recognition unit 121 extracts the face portion from the target image KTG and recognizes the gender and age of the target KTG. Known technologies can also be employed for this face recognition unit 121. The object itself may be a dedicated one equipped with an acceleration sensor and a communication device. When the detection results of such sensors are used for image recognition by the image recognition unit 120, the motion control of the object image can be performed with higher precision. By having the communication device transmit information indicating the type of object, for example, via the server 2, the AP server 1 can recognize the type of object that the detected target KT is holding or wearing.

[0077] The object recognition result by the image recognition unit 120 can be used to generate an avatar image holding the object. For example, if the detected target KT is holding a sword-shaped object, an avatar image holding a sword may be displayed. Alternatively, if the detected target KT throws an object, the movement of the thrown object may be represented as a video. The image representing the object can be stored, for example, in the character image storage unit 182, and selected from the object recognition result or the operation details of the detected target image KTG identified by skeletal information. This selection can be performed, for example, by the image recognition unit 120. Whether or not the detected target KT is holding an object can be determined, for example, by checking whether there is an area where the detected target image KTG and the object image are considered to overlap, and whether the degree of change in their sizes exceeds a set value. For example, as shown in Figure 2, when throwing a shuriken, the direction in which the shuriken is thrown, and even the position of the target, can be estimated from the direction in which the left hand moves. These estimation results can then be reflected in the movement of the shuriken.

[0078] Such screen control can be achieved by having the video generation unit 116 perform processing according to the response sentence generated by the conversation processing unit 119. Robot posture control can be achieved by having the posture control unit 117 perform processing according to the response sentence. What is operated in response to the response sentence is basically an avatar image or a robot that is intended to speak the response sentence.

[0079] The face recognition result from the face recognition unit 121 may be used to determine the type of avatar to be displayed as an image, including its gender and facial expression. In that case, it becomes possible to select a more desirable avatar for the detected KT, and it also becomes possible to change the facial expression of the avatar according to the facial expression of the detected KT. All of the above controls work to further enhance the sense of immersion in the game for the detected KT, and to provide a higher level of satisfaction.

[0080] In the functional configuration described above, the skeleton detection unit 113 corresponds to the skeleton information generation means in this embodiment. Similarly, the posture evaluation unit 114 corresponds to the analysis means, the text conversion unit corresponds to the text information generation means, the video generation unit 116 and the posture control unit 117 correspond to the motion control means, the speech recognition unit 118 corresponds to the speech recognition means, and the conversation processing unit 119 corresponds to the conversation processing means. The CPU 11 itself corresponds to the image data acquisition means and the speech information acquisition means.

[0081] Figure 7 is a flowchart showing an example of video display processing performed by the CPU installed in AP Server 1, which is an information processing device according to this embodiment. This video display processing is performed, for example, when a request to start a game is made from Server 2 or the like, in order to have the detected target KT play the game. This processing is realized by the CPU 11 executing an application program developed for providing this service. Finally, we will refer to Figure 7 and explain this video display processing in detail.

[0082] The video display process shown in Figure 7, which is an example flowchart, assumes a single game field as shown in Figure 4, in order to avoid confusion. The explanation will be based on this assumption. Accordingly, the explanation will assume that video information (image data) captured by camera C and audio information from microphone 5 are sent from server 2 to AP server 1. The avatar image is assumed to move in accordance with the movement of the detected target KT. The CPU 11 is assumed to be the entity that executes the processing.

[0083] First, in step S1, the CPU 11 determines whether or not it has received video information from server 2. If video information has been received, the determination in step S1 is YES and the process proceeds to step S2. If video information has not been received, the determination in step S1 is NO and the process proceeds to step S6. In step S2, the CPU 11 generates a still image from the received video information, for example, and extracts the detection target image KTG from the still image. In the following step S3, the CPU 11 performs a skeleton detection process to generate skeleton information from the extracted detection target image KTG. In the next step S4, the CPU 11 evaluates the pose of the detection target image KTG from the generated skeleton information. If necessary, already generated skeleton information is also used in this pose evaluation.

[0084] In step S5, following step S4, the CPU 11 converts the evaluation results into text by generating text information according to the posture evaluation results. After this text conversion, the process proceeds to step S6. In step S6, the CPU 11 determines whether or not it has received audio information from server 2. If audio information has been received, the determination in step S6 is YES and the process proceeds to step S7. If audio information has not been received, the determination in step S6 is NO and the process proceeds to step S10.

[0085] In step S7, the CPU 11 performs speech recognition processing using the received audio information and previously received audio information. This speech recognition processing includes not only generating a string from the audio information but also identifying the content of that string. In the following step S8, the CPU 11 determines whether it was able to identify the content of one conversation spoken by the detected KT. If the content of one conversation, that is, the content of the speech from the start to the end, was identified, the determination in step S8 becomes YES and the process proceeds to step S9. If the content of one conversation was not identified, the determination in step S8 becomes NO and the process proceeds to step S10. The determination of the end of speech is made, for example, if the state in which no audio from the detected KT can be confirmed continues for a set period of time. If the determination in step S8 is YES as a result, the content last identified in step S7 will be considered the content of the detected KT's speech. In step S9, the CPU 11 (conversation processing unit 119) generates a response string to be returned to the detected KT based on the identified utterance. After generation, the process proceeds to step S10.

[0086] In step S10, the CPU 11 determines whether the game has ended or not. If an end event occurs, such as a game termination instruction from the detected KT or the occurrence of an event within the game that should terminate the game, the determination in step S10 is YES and the process proceeds to step S11. If no end event has occurred, the determination in step S10 is NO and the process proceeds to step S12.

[0087] In step S11, the CPU 11 sends an end screen to the server 2 to notify the target KT that the game has ended. After this end screen is sent, the video display process ends. In step S12, the CPU 11 performs other processes necessary for the game to progress. Some games present the user with a selection of items or other options during gameplay and require them to choose one of the presented options. This presentation of options and selection from those options is achieved by executing other processes in step S12.

[0088] In the following step S13, the CPU 11 generates video information and audio information, respectively, and sends the generated video information and audio information to the projector 3 and sound system 4, respectively. After that, the process returns to step S1. If an avatar image exists on the screen represented by the video information, that avatar image is generated using the skeletal information generated by the execution of the skeletal detection process in step S3. An image for one screen is generated by creating each avatar image, each character image, and a background image, and then placing each avatar image and each character image on the background image. The text information generated in step S5 or the response statement generated in step S9 is reflected in the generation of any of the avatar images or any of the character images as needed. As a result, the behavior of the avatar image can be changed by at least one of the generated text information and response statement. The video information generated in step S13, where such screen generation takes place, is, for example, information generated by compressing the generated screen.

[0089] If text information is generated in step S5, or if a response sentence is generated in step S9, the aforementioned audio information is generated in step S13. The audio information to be transmitted is, for example, a digital audio signal. When a digital audio signal is transmitted as an audio signal, that audio signal is generated by, for example, a text-to-speech conversion application (hereinafter referred to as "audio conversion software"). In that case, step S13 involves calling the audio conversion software by specifying the text information or response sentence to be converted, and acquiring the audio signal. The transmission of the acquired audio signal as audio information continues until there is no more audio signal to be transmitted. When sound is emitted by the sound system 4 via the server 2 equipped with this audio conversion software, the text information and response sentence can be transmitted as audio information. For this reason, the audio information is not particularly limited.

[0090] If the target of the operation is a robot, attitude control processing is performed instead of video display processing. The flow of this attitude control processing is basically the same as the video display processing described above. The difference from the video display processing is that attitude control information is generated instead of video information. For this reason, a detailed explanation of the attitude control processing will be omitted.

[0091] In this embodiment, the explanation focused on games that involve physical movement, but as described above, the present invention can be broadly applied to any movement that requires the body to perform a posture-requiring action for which one or more postures to be taken are known in advance. For example, sign language, which is performed by moving the fingers, is a means of expressing what one wants to convey through speech, using a predetermined posture and predetermined movements from that posture. Therefore, it is also possible to use skeletal information to convert the content expressed in sign language into text information and output that text information. Specifically, for example, the movements of sign language may be reproduced on an object, and the content of the sign language reproduced on that object may be displayed, or at least one of the following may be output: display or audio. Considering that it will be used by both people with hearing impairments and people without, it is desirable to output text information in both ways.

[0092] In sports, various running activities (sprints, long-distance races, hurdle races, etc.) and golf are examples of activities that require specific postures. In such sports, for example, the target of the activity may perform the movement represented by skeletal information, and by comparing it with a model image (OG), text information may be generated to communicate points for improvement or good points to the detection target (KT). This text information may then be displayed or output as audio. Since it is difficult to transmit text information to a running detection target (KT) in a timely manner, it is desirable to save video information representing the target's movement along with the text information and make it playable.

[0093] Furthermore, by using skeletal information, it becomes possible to identify the state of the detected KT in a timely manner. This can be used to estimate the future movements of the detected KT, and to generate and output text information according to the estimation results. This could be used to warn a detected KT that attempts to touch an exhibit that it should not touch, or to warn it about moving to a place it should not go. It could also be used to detect customers visiting a store as the detected KT, preventing shoplifting and other criminal acts by the detected KT, or to apprehend a detected KT that has committed a criminal act. In any case, more information will be provided, and the convenience for users will be greatly enhanced.

[0094] In motion estimation, the movement speed and direction of the target image KTG can be estimated from skeletal information, and the estimation results can be reflected in the movement of the target. By performing such estimation (prediction), it is expected that the response time will be improved. When applied to detecting dangerous behavior of the target KTG, the detection can be performed more quickly.

[0095] This movement estimation can be used to control the movements of avatars, such as opponents or characters in competitive games or RPGs (Role-Playing Games). For opponents, defensive movements can be made to correspond to the movements represented by their skeletal information. However, movement estimation can make defensive movements more natural. For characters facing the player, making them perform actions corresponding to the movements of the detected KT will give the KT a stronger impression of facing them. If the movement estimation results are used to control the character's movements, the character's movements can be made more natural, giving the KT a stronger impression of actually facing an opponent. In any case, it is desirable to generate text information based on posture evaluation using skeletal information, and to perform actions and natural conversations according to the generated text information.

[0096] Furthermore, the present invention can also be applied to the control of the movements of avatars or robots in unmanned stores or unmanned reception areas. Both the avatar and the robot are used to perform actions different from those of the detected KT in response to its movements, in order to provide the services desired by the detected KT. For example, it could be combined with image recognition technology to recognize the presence or absence of goods or packages, and then respond accordingly. More specifically, if the detected KT makes an action that suggests it is showing a product, the type of product could be identified, and that identification result could be reflected in the subsequent conversation. This is because a detected KT that makes an action that suggests it is showing a product is likely to have the intention of checking for other products in different colors, other products in the same category, or inventory.

[0097] As described above, the targets of the operation can be broadly divided into two types: those that operate in accordance with the operation of the detected KT (hereinafter referred to as "first operation target") and those that operate differently from the operation of the detected KT (hereinafter referred to as "second operation target"). Therefore, the generated text information can also be broadly divided into two types: those that assume the first operation target and those that assume the second operation. The same applies to the example data TD (reference image information).

[0098] Both the first and second action targets may be subject to motion control. When controlling both simultaneously, text information may be generated with the intention of using it for both motion controls. For example, if the detection target KT assumes the transformation pose represented by the example data TD, text information may be generated to cause the first action target to output the sound "transform," and the second action target to stop moving or perform an action to a set posture. If speech recognition technology and natural speech (language) technology are also used, the detection target KT may be evaluated to determine whether it has assumed the transformation pose based on the utterance of "transform," and text information may be generated accordingly. In this case, the text information may be used for emitting a sound effect for the first action target and for similar motion control for the second action target. For these reasons, various modifications are possible for the conditions for generating text information, the content of the generated text information, and the motion control of the action targets using the text information. [Explanation of symbols]

[0099] 1 AP Server, 2 Server, 3 Projector, 4 Sound System, 5 Microphone, 11 CPU, 18 Memory Unit, 19 Communication Unit, 111 Setting Unit, 112 Image Extraction Unit, 113 Skeleton Detection Unit, 114 Pose Evaluation Unit, 115 Text Conversion Unit, 116 Video Generation Unit, 117 Pose Control Unit, 118 Voice Recognition Unit, 119 Conversation Processing Unit, 120 Image Recognition Unit, 121 Face Recognition Unit, 181 Reference Image Storage Unit, 182 Character Image Storage Unit, 183 Background Image Storage Unit, 184 Dictionary Storage Unit, 185 Control Information Storage Unit, C Camera, H1 Skeleton Information Generation System, H2 Pose Evaluation System, H3 Pose Management System, KK Customer Company, KT Detection Target, KTG Detection Target Image, OG Example Image, SK Service Provider Company, SY Service Provider System

Claims

1. Image data acquisition means for acquiring image data obtained by imaging the target to be detected, A skeletal information generation means generates skeletal information including joint position information and relationship information indicating the relationship between joints in the detected image, based on the detected image represented by the image data. An analysis means for analyzing the motion content in the detected image based on the skeletal information, A text information generation means that generates text information based on the analysis results of the aforementioned operation content, Based on the aforementioned text information, motion control means for controlling at least one of a predetermined avatar to be displayed and a robot, which is a physical machine, as the target of the operation. An information processing device having

2. The operation of the target of the operation based on the text information by the operation control means includes a speech operation by displaying and / or emitting sound. The information processing apparatus according to claim 1.

3. The motion control means is capable of operating the target object based on the skeletal information. The information processing apparatus according to claim 1 or 2.

4. A means for acquiring audio information that acquires audio information representing the sound emitted by the detected target, A speech recognition means that performs speech recognition using the aforementioned speech information and generates a first string from the aforementioned speech information, The system further comprises conversation processing means for generating a second string to be used as a response to the first string, The operation control means causes the second string to be output as the content of the statement in the statement operation of the target of the operation. The information processing apparatus according to any one of claims 1 to 3.

5. The motion control means causes the target to perform an action different from the action represented by the skeletal information, based on the skeletal information. The information processing apparatus according to claim 1, 2, or 4.

6. The text information generation means generates text information representing the content of the sign language when, as a result of analyzing the action content, the finger movements in the detected image are identified as movements for sign language. The motion control means uses the skeletal information to cause the target to perform finger movements in accordance with the movement of the fingers, and also outputs the text information. The information processing apparatus according to any one of claims 1 to 4.

7. In an information processing device, Based on the image data obtained by imaging a target object, which represents the image of the target object, skeletal information is generated that includes the position information of the joints in the image of the target object and relationship information indicating the relationships between the joints. Based on the aforementioned skeletal information, the motion details in the detected image are analyzed, Based on the analysis results of the aforementioned operation, text information is generated. Based on the aforementioned text information, the system operates on at least one of a predetermined avatar to be displayed and a robot, which is a physical machine. A program that executes a process.

Images