Information processing device, information processing method, and program

Abstract

[Problem] To improve the protectiveness of motion data. An information processing apparatus includes: an acquisition unit for acquiring a motion feature indicating a feature amount generated from motion data and transmitted from a communication terminal; and a generation unit for generating an image corresponding to the motion data used in generating the motion feature acquired by the acquisition unit.

Description

Technical Field

[0001] The present disclosure relates to an information processing device, an information processing method, and a program. Background Art

[0002] In recent years, animation production and transmission of motion information indicating user motion obtained using motion capture has become popular. For example, the obtained motion information is used to generate motion data imitating the user's motion, and then an avatar image based on the motion data is transmitted. Against this background, the amount of motion data has increased year by year, and technology for reusing the motion data generated so far is being developed.

[0003] For example, PTL 1 below discloses a technology in which, when a plurality of avatars approach each other, a server transmits motion data for rendering such as a high five exchanged between avatars or avatar data that has been rendered to a client terminal.

[0004] [Citation List]

[0005] [Patent Document]

[0006] [PTL1]

[0007] JP 2020-91504A Summary of the invention

[0008] [Technical issues]

[0009] However, in the above-mentioned PTL 1, for example, motion data transmitted by a client terminal can be extracted, which makes it difficult to protect the motion data.

[0010] Therefore, the present disclosure proposes new and improved information processing devices, information processing methods, and programs that are capable of better protecting motion data.

[0011] [Solution to the problem]

[0012] According to the present disclosure, there is provided an information processing device including: an obtaining unit that obtains a motion feature indicating a feature generated from motion data and transmitted from a communication terminal; and a generating unit that generates an image corresponding to the motion data used to generate the motion feature obtained by the obtaining unit.

[0013] According to the present disclosure, there is provided an information processing method executed by a computer, the method comprising: obtaining a motion feature indicating a feature generated from motion data and transmitted from a communication terminal; and generating an image corresponding to the motion data used to generate the obtained motion feature.

[0014] According to the present disclosure, a program is provided that enables a computer to implement: an acquisition function for obtaining a motion feature indicating a feature generated from motion data and transmitted from a communication terminal; and a generation function for generating an image corresponding to the motion data used to generate the motion feature obtained by the acquisition function. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] [ Figure 1 ] Figure 1 is an explanatory diagram showing an information processing system according to one embodiment of the present disclosure.

[0016] [ Figure 2 ] Figure 2 is an explanatory diagram showing an example of the functional configuration of the information processing terminal 10 according to the present disclosure.

[0017] [ Figure 3 ] Figure 3 is an explanatory diagram showing an example of the functional configuration of the server 20 according to the present disclosure.

[0018] [ Figure 4 ] Figure 4 is an explanatory diagram showing an example of preparation of an estimation encoder.

[0019] [ Figure 5 ] Figure 5 is an explanatory diagram showing an example of a user interface according to the present disclosure.

[0020] [ Figure 6 ] Figure 6 is an explanatory diagram showing an example of the configuration of the storage unit 220 according to a modification.

[0021] [ Figure 7 ] Figure 7 is an explanatory diagram showing an example of the operation and processing of the information processing terminal 10 according to the present disclosure.

[0022] [ Figure 8 ] Figure 8 is a block diagram showing the hardware configuration of information processing terminal 10 . DETAILED DESCRIPTION

[0023] Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In the present specification and the accompanying drawings, components having substantially the same functional configuration will be denoted by the same reference numerals, and repeated descriptions thereof will be omitted.

[0024] The embodiments will be described according to the order of items indicated below.

[0025] 1. Overview of Information Processing Systems

[0026] 2. Example of function configuration

[0027] 2-1. Example of functional configuration of information processing terminal

[0028] 2-2. Example of functional configuration of the server

[0029] 3. Details

[0030] 3-1. Background

[0031] 3-2. Preparation stage

[0032] 3-3. Operation phase

[0033] 4. Variant

[0034] 5. Operation examples

[0035] 6. Hardware Configuration Example

[0036] 7. Additional Notes

[0037] <<1. Overview of Information Processing System>>

[0038] In motion data, for example, skeletal data represented by a skeletal structure indicating a body structure is used to visualize information about the movement of a moving body such as a person or an animal. The skeletal data includes information such as the position, posture, and the like of parts. More specifically, the skeletal data includes various information such as the global position of a root joint, the relative posture of each joint, and the like. Note that the "parts" in the skeletal structure correspond to, for example, end parts, joint parts, and the like in the body. In addition, the skeletal data may include bones, which are line segments connecting parts. Although the "bones" in the skeletal structure may be equivalent to, for example, human bones, the positions and number of bones do not necessarily have to match the positions and number of bones of an actual human skeleton.

[0039] The position and posture of each part in the skeleton data can be obtained by various motion capture technologies. For example, there are: a camera-based technology in which a marker is attached to each part of the body and the position of the marker is obtained using an external camera or the like; and a sensor-based technology in which a motion sensor is attached to a part of the body and the position information of the motion sensor is obtained based on time series data obtained by the motion sensor.

[0040] In addition, the skeleton data can be applied to different settings. For example, the time series data of the skeleton data is used to improve the form in dance and sports, and for applications such as virtual reality (VR) or augmented reality (AR). The motion data as the time series data of the skeleton data is also used to generate an avatar image that imitates the user's motion and then transmits the avatar image.

[0041] Note that the motion data may also include various information, such as skeletal information indicating the length of each bone and the connection relationship with other bones, time series data indicating information that both feet are in contact with the ground, and the like.

[0042] As one embodiment of the present disclosure, an example of a configuration of an information processing system that better protects motion data will be described below.

[0043] Figure 1 1 is an explanatory diagram showing an information processing system according to one embodiment of the present disclosure. Figure 1 As shown, the information processing system according to one embodiment of the present disclosure includes an information processing terminal 10 used by a user U and a server 20 .

[0044] The information processing terminal 10 and the server 20 are connected via a network 1. The network 1 is a wired or wireless transmission path for information sent from a device connected to the network 1. For example, the network 1 may include a public line network such as the Internet, a telephone network, a satellite communication network, etc., various types of local area networks (LANs) including Ethernet (registered trademark), wide area networks (WANs), etc. In addition, the network 1 may include a dedicated line network such as an Internet Protocol-Virtual Private Network (IP-VPN).

[0045] (Information Processing Terminal 10)

[0046] The information processing terminal 10 is an example of an information processing device. The information processing terminal 10 obtains the motion feature transmitted from the server 20.

[0047] For example, the information processing terminal 10 may transmit a designated motion data identification (ID) indicating the motion data designated by the user to the server 20. Then, the information processing terminal 10 may receive the motion characteristics of the designated motion data from the server 20.

[0048] Information processing terminal 10 also generates an image corresponding to the motion data used to generate the motion feature.

[0049] Although Figure 1 A smartphone is shown as the information processing terminal 10 , but the information processing terminal 10 may be other devices such as a laptop personal computer (PC), a desktop PC, or the like.

[0050] (Server 20)

[0051] The server 20 is an example of a communication device, and stores a plurality of pieces of sports data and a sports feature of each of the plurality of sports data. The server 20 also transmits the sports feature of the sports data to the information processing terminal 10.

[0052] For example, the server 20 may receive an ID of the designated motion data from the information processing terminal 10. Then, the server 20 may transmit the motion feature of the designated motion data corresponding to the ID to the information processing terminal 10.

[0053] The overview of the information processing system according to the present disclosure has been described above. Next, an example of the functional configuration of the information processing terminal 10 and the server 20 according to the present disclosure will be described.

[0054] <<2. Example of function configuration>>

[0055] <2-1. Example of functional configuration of information processing terminal>

[0056] Figure 2 1 is an explanatory diagram showing an example of the functional configuration of the information processing terminal 10 according to the present disclosure. Figure 2 As shown, the information processing terminal 10 includes an operation and display unit 110 , a communication unit 120 , a storage unit 130 , a renderer unit 140 , and a time management unit 143 .

[0057] (Operation and display unit 110)

[0058] The operation and display unit 110 functions as a display unit that displays the search result transmitted from the server 20. For example, the operation and display unit 110 displays an image corresponding to the motion data used to generate the motion feature transmitted from the server 20.

[0059] The operation and display unit 110 also functions as an operation unit for a user to perform operation inputs. For example, the user uses the operation and display unit 110 to specify exercise data.

[0060] The display unit function is implemented by, for example, a cathode ray tube (CRT) display device, a liquid crystal display (LCD) device, or an organic light emitting diode (OLED) device.

[0061] The operating unit function is implemented, for example, by a touch panel, a keyboard, or a mouse.

[0062] Note that Figure 1 In the embodiment, the information processing terminal 10 is configured such that the functions of the display unit and the operation unit are integrated, but the configuration may be such that the functions of the display unit and the operation unit are separated.

[0063] (Communication unit 120)

[0064] The communication unit 120 communicates various types of information with the server 20 through the network 1. For example, the communication unit 120 is an example of an acquisition unit, and acquires a motion feature transmitted from the server 20.

[0065] For example, the communication unit 120 may transmit a designated motion data ID indicating motion data designated by a user to the server 20. Then, the communication unit 120 may receive a motion feature of the designated motion data transmitted from the server 20 in response to the transmitted ID.

[0066] (Storage unit 130)

[0067] The storage unit 130 stores software and various types of data. For example, the storage unit 130 stores downloaded sports data.

[0068] (renderer unit 140)

[0069] The renderer unit 140 is an example of a generating unit, and generates an image corresponding to the motion data used to generate the motion feature received from the server 20 .

[0070] For example, the renderer unit 140 may generate an image corresponding to the motion data based on the pre-learned relationship between the motion feature and the image. A specific example of the process of learning the relationship between the motion feature and the image will be described later.

[0071] (Time Management Unit 143)

[0072] The time management unit 143 manages the time of motion data corresponding to the image generated by the renderer unit 140 .

[0073] An example of the functional configuration of the information processing terminal 10 has been described above. Note that the functional configuration of the information processing terminal 10 is not limited to this example. For example, the information processing terminal 10 may also have a configuration for performing a denoising process that cancels the modification of the motion feature applied by the modification process performed by the modification unit 235 (described later). Figure 3 An example of the functional configuration of the server 20 is described.

[0074] <2-2. Example of functional configuration of server>

[0075] Figure 3 2 is an explanatory diagram showing an example of the functional configuration of the server 20 according to the present disclosure. Figure 3 As shown, the server 20 includes a communication unit 210 , a storage unit 220 and a control unit 230 .

[0076] (Communication unit 210)

[0077] The communication unit 210 communicates various types of information with the information processing terminal 10 through the network 1. For example, the communication unit 210 may receive, from the information processing terminal 10, a designated motion data ID indicating motion data designated by a user.

[0078] The communication unit 210 also transmits the motion feature of the motion data corresponding to the ID received from the information processing terminal 10 to the information processing terminal 10 .

[0079] (Storage unit 220)

[0080] The storage unit 220 stores software and various types of data. Figure 3 As shown, the storage unit 220 includes a motion data storage unit 221 and a motion feature storage unit 225 .

[0081] The motion data storage unit 221 stores a plurality of pieces of motion data. For example, the motion data storage unit 221 may store the motion data and the motion data ID in association with each other.

[0082] Note that the motion data stored by the motion data storage unit 221 may be data obtained by motion capture, or may be data obtained by modifying data obtained by motion capture.

[0083] In addition, the motion data stored by the motion data storage unit 221 may be data manually created by a person, or may be data generated by a machine learning technique such as deep learning.

[0084] The motion feature storage unit 225 stores the motion feature of each of the plurality of pieces of motion data stored in the motion data storage unit 221. For example, the motion feature storage unit 225 may store the motion feature and the ID of the motion data that has generated the motion feature in association with each other.

[0085] (Control Unit 230)

[0086] The control unit 230 controls the overall operation of the server 20. Figure 3 As shown, the control unit 230 includes a learning unit 231 , an estimating unit 233 , and a modifying unit 235 .

[0087] The learning unit 231 learns the relationship between the motion data and the motion features. For example, the learning unit 231 may learn the relationship between the motion data and the motion features using an encoder / decoder model based on deep learning.

[0088] Then, the learning unit 231 can generate an estimated encoder by learning the relationship between the motion data and the motion feature. A specific example of the process learned by the learning unit 231 will be described later.

[0089] The estimation unit 233 estimates motion features based on the motion data. For example, the estimation encoder generated by the learning unit 231 provides the function of the estimation unit 233.

[0090] The modification unit 235 performs a modification process of adding noise information to the motion feature estimated by the estimation unit 233. The noise information includes, for example, information that a predetermined modification process is applied to the image generated by the renderer unit 140.

[0091] The predetermined modification process may include, for example, a process for adding noise or a watermark to the image. The predetermined modification process may also include, for example, a process for reducing the resolution, or any desired process that makes the image difficult to see. This may reduce the risk that the motion feature will be used incorrectly.

[0092] In addition, when the motion feature is transmitted to the information processing terminal 10 of the trusted user, the modification unit 235 may perform a denoising process to remove noise information added by the modification process that has been performed.

[0093] An example of the functional configuration of the information processing system according to the present disclosure has been described above. Next, details of the information processing system according to the present disclosure will be given in order.

[0094] <<3. Details>>

[0095] <3-1. Background>

[0096] As an example of a method of reusing motion data, it is conceivable to sell motion data to users. In this case, it is expected that the user will consider purchasing after confirming whether the motion data providing the desired motion is sold when confirming the image generated from the motion data.

[0097] In one example, it can be imagined that the server sends an image generated from the motion data to an information processing terminal used by the user, and then the information processing terminal can display the received image.

[0098] However, in this example, the user can confirm the image generated from the motion data, but it is difficult for the user to confirm the image from the viewpoint they want to see or confirm the application of the motion data to the character, which makes it difficult for the user to grasp the details of the motion data.

[0099] In another example, it can be imagined that the server transmits the motion data to an information processing terminal used by the user, and then the information processing terminal can display an image rendered from the received motion data.

[0100] It is also conceivable that encryption processing is applied to the motion data exchanged between the server and the information processing terminal. In this case, the information processing terminal uses a decoder to decrypt the encrypted motion data into motion data, and then displays an image rendered from the motion data.

[0101] However, in this other example, the image is generated using a common renderer, which may introduce a risk of extracting motion data by, for example, using a browser's debugging functionality.

[0102] Therefore, the server 20 according to the present disclosure transmits the motion feature indicating the feature generated from the motion data to the information processing terminal 10. Then, the information processing terminal 10 generates and displays an image corresponding to the motion data used to generate the received motion feature.

[0103] Here, when generating motion features from motion data, the user can better protect the motion data by using a private estimation encoder.

[0104] The renderer unit 140 can also use the obtained motion features as well as camera parameters, character information, etc. (described later) to generate images corresponding to the motion data, so that, for example, images of viewpoints, characters, etc. that the application user wants to see can be displayed, and the user can grasp the details of the motion data.

[0105] In this way, the information processing system according to the present disclosure has a preparation phase including two types of preparations: one, preparation for an estimation encoder for generating motion features from motion data; and two, preparation for rendering, in which the renderer unit 140 generates an image according to the motion features. Details of the preparation phase will be given in order below.

[0106] <3-2. Preparation stage>

[0107] (Prepare to estimate encoder)

[0108] Figure 4 is an explanatory diagram showing an example of preparation of the estimation encoder. For example, the learning unit 231 may use Figure 4 The encoder / decoder model shown is used to learn the relationship between motion data and motion features.

[0109] Note that the motion data input to the encoder may be a single frame, or may be a plurality of frames.

[0110] First, study unit 231 to prepare by motion i [0:T i ] (where i = 0 to N). Here, i represents the sequence of motion data used for learning, and i = 0 to N. N represents the number of sequences, and T i represents the frame number in sequence i.

[0111] Then, the learning unit 231 encodes each item of the motion data constituting the motion data set using the following formula (Math. 1) and estimates the motion feature.

[0112] [Mathematical formula 1]

[0113] feature i [0:T i ]=Enc(motion i [0:T i ]) (Mathematical formula 1)

[0114] Next, the learning unit 231 decodes the estimated motion feature using the following formula (Math. 2) and estimates motion data.

[0115] [Mathematical formula 2]

[0116] motion i _[0:T i ]=Dec(feature i [0:T i ])

[0117] (Mathematical formula 2)

[0118] The motion data estimated through decoding may be referred to as 'estimated motion data' to distinguish the data from the motion data included in a previously prepared motion data set.

[0119] Then, the learning unit 231 performs learning using the following formula (Math. 3) to minimize the loss function Loss between each motion data set constituting the motion data set and the estimated motion data estimated based on each of these motion data items.

[0120] [Mathematical formula 3]

[0121]

[0122] Then, the learning unit 231 outputs the encoder after learning to minimize the loss function Loss as an estimated encoder to the estimation unit 233.

[0123] (Estimation of motion features)

[0124] Then, the estimation unit 233 estimates the motion feature based on the motion data stored in the motion data storage unit 221 using the estimation encoder.

[0125] Note that the motion data input to the estimation encoder may or may not include the motion data used in the learning for the estimation encoder.

[0126] In addition, if the estimation encoder cannot process variable length input, the estimation unit 233 can use the following formula (Math. 4) to estimate, for example, motion features of any desired length.

[0127] [Formula 4]

[0128]

[0129] Here, T represents the input length of the estimated encoder.

[0130] The estimation unit 233 constructs a database of motion features by storing the motion feature of each item of motion data estimated through the above-described process in the motion feature storage unit 225 .

[0131] Thus, if the user specifies motion data, the communication unit 210 included in the server 20 may transmit the motion characteristics of the specified motion data specified by the user to the information processing terminal 10 .

[0132] After receiving the motion feature, the information processing terminal 10 renders the motion feature through the renderer unit 140 and generates an image corresponding to the motion data from which the motion feature is generated.

[0133] Next, a specific example of learning for generating an image from a motion feature will be described. Note that the learning preparation of the renderer unit 140 may be performed in the server 20, or may be performed separately in other devices.

[0134] (Rendering preparation)

[0135] The renderer unit 140 renders an image at time t according to, for example, motion characteristics, camera parameters, and character information.

[0136] The camera parameters are, for example, parameters related to viewpoint information indicating a viewpoint from which an image is viewed. More specifically, the camera parameters include information related to a relative camera position and posture viewed from a character coordinate system.

[0137] The character information is, for example, information about a character that performs motion data. For example, the character information may be data consisting of at least one of a 3D mesh, skeleton information, texture, a 2D image (or image group) viewed from multiple viewpoints, or a point cloud.

[0138] Note that if the character to be rendered is set in advance, the character information may include only the ID of the character.

[0139] Additionally, if only a single character is supported, the character information need not be included in the input used by the renderer unit 140 when rendering an image.

[0140] The learning of the relationship between: motion features, camera parameters and character information; and an image will be described in detail herein. First, an image is obtained from any desired character information, any desired camera parameters and motion data using any desired renderer.

[0141] Next, the renderer unit 140 according to the present disclosure uses the following formula (Math. 5) to obtain an image based on the character information and camera parameters input to the desired renderer and the motion features generated by the motion data input to the desired renderer. The renderer unit 140 is, for example, a deep neural network (DNN) renderer and will be referred to as a "neural renderer".

[0142] [Formula 5]

[0143]

[0144] Here, T represents the input length and may be different from the input length of the estimation encoder.

[0145] For example, the following formula (Math 6) can be used to perform learning so that the image image obtained by the desired renderer i [t] and the image image_ obtained by the renderer unit 140 i The loss function Loss between [t] is minimized.

[0146] [Mathematical formula 6]

[0147]

[0148] Then, the renderer unit 140 may generate an image according to the motion features, the camera parameters, and the character information using the neural renderer obtained through learning.

[0149] Note that the images learned by the neural renderer may be images to which predetermined modification processes have been applied, such as adding noise, watermarking, reducing resolution, etc. This may mitigate the risk of the rendered images being reused.

[0150] The details of the preparation phase have been described above. Next, the operation phase, which is a phase actually used by the user, will be described in detail. Although the present disclosure mainly describes an example of selling motion data to users, the use case of the information processing system according to the present disclosure is not limited to this example.

[0151] <3-3. Operation phase>

[0152] Figure 5 1 is an explanatory diagram showing an example of a user interface according to the present disclosure. For example, the operation and display unit 110 displays a Figure 5 The sports data sales screen W is shown.

[0153] The motion data sales screen W includes, for example, a motion data list AM saved by the server 20. For example, the user can confirm a motion gif image, a still image, or the like of each item of motion data in the motion data list AM.

[0154] Note that the motion data list AM may arrange some or all of the motion data stored in the motion data storage unit 221 in any desired order. For example, the motion data list AM may be arranged in the order of the IDs of the motion data, or may be a motion data group belonging to a given category or a recommended motion data group. The motion data list AM may also include results searched by motion capture that captures the user's motion.

[0155] For example, the user specifies one item of the motion data included in the motion data list AM using the operation and display unit 110. Then, the communication unit 120 transmits the ID of the motion data specified by the user to the server 20. Then, the server 20 transmits the motion feature corresponding to the received ID to the information processing terminal 10.

[0156] Then, based on the received motion feature, the renderer unit 140 generates an image corresponding to the motion data from which the motion feature is generated. Then, the operation and display unit 110 may display the generated image in the designated motion data display area DM.

[0157] For example, the camera parameters can be changed by performing a mouse operation (eg, drag and drop, etc.) in the designated motion data display area DM. In addition, the camera parameters can be changed after selecting the rotation button B1 and the movement button B2.

[0158] For example, the user can select the rotation button B1 and then change the posture of the camera using each of the x-axis, y-axis, and z-axis as the rotation axis. The user can also select the movement button B2 and change the position of the camera in each of the x-direction, y-direction, and z-direction.

[0159] In addition, the user can select the rotation button B1 or the movement button B2 and input each camera parameter as a numerical value.

[0160] The role information can be changed by selecting a role prepared in advance on the service provider side. For example, the user selects the change role button B3. In this case, the operation and display unit 110 can display candidates that can change the role. Then, the user selects a role from the displayed role candidates. Then, the role information can be changed to various information related to the role selected by the user.

[0161] In addition, the character information can be changed by uploading a character prepared by the user. The character information to be uploaded may be, for example, a general 3D model, or may be image data.

[0162] In addition, although the renderer unit 140 generates an image corresponding to the motion data of the input time, the time may be automatically counted or may be designated by the user through the search bar S.

[0163] For example, the communication unit 120 included in the information processing terminal 10 obtains motion features from the server 20. Then, the renderer unit 140 may use the motion features of the time specified by the user through the search bar S among the obtained motion features to generate an image corresponding to the motion data of the time.

[0164] In addition, if the user specifies a given time using the search bar S, the communication unit 120 may obtain the motion characteristics of the time from the server 20. In this case, the renderer unit 140 may generate an image corresponding to the motion data of the time.

[0165] As described above, time, camera parameters, and the like are changed in real time by user operations or automatic counting, and thus the renderer unit 140 can sequentially perform rendering processing involved in image generation.

[0166] In addition, if the time and camera parameters have not changed, the operation and display unit 110 may skip the rendering process and display the image generated in the previous step.

[0167] Then, when there is sports data that the user wants to purchase, the user can select the purchase button P and proceed to the process for downloading the specified sports data. For example, if the user selects the purchase button G, the communication unit 120 included in the information processing terminal 10 sends the ID of the specified sports data to the server 20.

[0168] Then, the server 20 may transmit the motion data corresponding to the received ID to the information processing terminal 10, and the storage unit 130 may store the received motion data. This makes it possible to generate an image from the motion data using an existing renderer.

[0169] The information processing system according to the present disclosure has been described in detail above. According to the above information processing system, the risk of motion data being extracted can be reduced, and the motion data can be better protected. Note that the information processing system according to the present disclosure is not limited to the above example. Figure 6 Variations of information handling systems according to the present disclosure are described.

[0170] <<4. Variants>>

[0171] Figure 62 is an explanatory diagram showing an example of the configuration of the storage unit 220 according to a modification. For example, it can be assumed that a plurality of users will exist on both the service provider side and the service user side.

[0172] In this case, the storage unit 220 may store the motion data and motion features for each domain on the service provider side. For example, if there are three domains on the service provider side, the storage unit 220 may include a first storage unit 220A, a second storage unit 220B, and a third storage unit 220C. Figure 6 shown.

[0173] For example, limiting the domains that each user can access makes it possible to restrict users with different trust levels based on domains, which can better protect data in motion.

[0174] In addition, the motion feature storage unit 225A provided in the first storage unit 220A, the motion feature storage unit 225B provided in the second storage unit 220B, and the motion feature storage unit 225C provided in the third storage unit 220C may each store motion features estimated using different estimation encoders by the estimation unit 233. In this case, the same number of decoders as the estimation encoders are prepared.

[0175] Likewise, the renderer unit 140 included in the information processing terminal 10 can learn separately according to the corresponding estimated encoder. For example, the relationship between motion features and images can be learned by a renderer corresponding to an estimated encoder of a given domain and a renderer corresponding to an estimated encoder of another domain, respectively.

[0176] A single renderer unit 140 may also learn to process motion features estimated by multiple estimation encoders.

[0177] The above variations can, for example, make it difficult to build a general model through reverse engineering, which can better protect motion data. More specifically, even if the estimated encoder or decoder of one domain has been reverse engineered, it is difficult to recover the motion features of other domains.

[0178] The above has described variations of the information processing system according to the present disclosure. It should be noted that the above-mentioned first storage unit 220A, second storage unit 220B and third storage unit 220C may be provided in a single server 20, or a plurality of servers 20 may include storage units 220A to 220C, respectively. Figure 7 An example of the operation of the information processing terminal 10 according to the present disclosure is described.

[0179] <<5. Operation Example>>

[0180] Figure 71 is an explanatory diagram showing an example of operation and processing of the information processing terminal 10 according to the present disclosure. First, the operation and display unit 110 displays a list of motion data using a simple image such as a motion gif image or a still image (S101).

[0181] Then, one item of exercise data is selected by a user operation performed in the operation and display unit 110 (S105).

[0182] Then, the communication unit 120 transmits the ID of the selected motion data to the server 20 ( S109 ).

[0183] Then, the communication unit 120 receives the motion feature of the motion data corresponding to the transmitted ID (S113).

[0184] Then, the operation and display unit 110 obtains character information through the user operation (S117).

[0185] Next, the operation and display unit 110 obtains camera parameters through user operation ( S121 ).

[0186] Then, the renderer unit 140 determines whether the image playback mode is automatic playback (S125). If the image playback mode is automatic playback (S125 / Yes), the sequence moves to S129, and if the image playback mode is not automatic playback (S125 / No), the sequence moves to S133.

[0187] If the image playback mode is automatic playback (S125 / Yes), the time management unit 143 counts the playback time (S129).

[0188] If the image playback mode is not automatic playback (S125 / No), the time management unit 143 obtains the playback time designated by the user (S133).

[0189] Then, the renderer unit 140 generates an image of the obtained playback time based on the motion characteristics, the character information, and the camera parameters (S137).

[0190] Next, the operation and display unit 110 displays the generated image ( S141 ).

[0191] Then, the operation and display unit 110 determines whether the display of the image has ended in response to the user operation (S145). If it is determined that the display of the image has not ended (S145 / No), the sequence returns to S117. If it is determined that the display of the image has ended (S145 / Yes), the information processing terminal 10 according to the present disclosure ends the operation and processing.

[0192] <<6. Hardware Configuration Example>>

[0193] The embodiments of the present disclosure have been described above. Information processing, such as the processing of generating an image corresponding to the motion data for generating the motion feature as described above, is implemented by the software and hardware cooperative operation of the information processing terminal 10 described below. Note that the hardware configuration described below can also be applied to the server 20.

[0194] Figure 8 100 is a block diagram showing a hardware configuration of the information processing terminal 10. The information processing terminal 10 includes a central processing unit (CPU) 1001, a read-only memory (ROM) 1002, a random access memory (RAM) 1003, and a host bus 1004. The information processing terminal 10 also includes a bridge 1005, an external bus 1006, an interface 1007, an input device 1008, an output device 1010, a storage device (HDD) 1011, a drive 1012, and a communication device 1015.

[0195] The CPU 1001 functions as a calculation processing device and a control device, and controls the overall operation in the information processing terminal 10 according to various programs. The CPU 1001 may be a microprocessor. The ROM 1002 stores programs, calculation parameters, etc. used by the CPU 1001. The RAM 1003 temporarily stores programs for execution by the CPU 1001, parameters that change appropriately during execution, etc. These are connected to each other via a host bus 1004 composed of a CPU bus or the like. For example, referring to Figure 2 The described functions of the renderer unit 140 and the like can be realized by the CPU 1001 , the ROM 1002 , and the RAM 1003 operating in cooperation with software.

[0196] The host bus 1004 is connected to an external bus 1006 such as a peripheral component interconnect / interface (PCI) bus through a bridge 1005. It should be noted that the host bus 1004, the bridge 1005, and the external bus 1006 do not necessarily need to be separately configured, and their functions may be implemented by one bus.

[0197] The input device 1008 is composed of an input device such as a mouse, a keyboard, a touch panel, a button, a microphone, a switch, a joystick, etc., for a user to input information, and an input control circuit that generates an input signal based on the user's input and outputs the input signal to the CPU 1001. The user of the information processing terminal 10 can input various types of data to the information processing terminal 10, instruct processing operations, etc. by operating the input device 1008.

[0198] The output device 1010 includes, for example, a display device such as a liquid crystal display device, an OLED device, a lamp, etc. The output device 1010 also includes an audio output device such as a speaker, an earphone, etc. The output device 1010 outputs, for example, playback content. Specifically, the display device uses text or images to display various types of information, such as playback image data. On the other hand, the audio output device converts the playback audio data, etc. into sound and outputs the sound.

[0199] The storage device 1011 is a device for storing data. The storage device 1011 may include a storage medium, a recording device for recording data in the storage medium, a reading device for reading data from the storage medium, a deleting device for deleting data recorded in the storage medium, etc. The storage device 1011 is composed of, for example, a hard disk drive (HDD). The storage device 1011 drives the hard disk and stores programs executed by the CPU 1001, various types of data, etc.

[0200] The drive 1012 is a reader / writer of a storage medium, and is built in the information processing terminal 10 or is external to the information processing terminal 10. The drive 1012 reads out information recorded in a removable storage medium 30 such as a mounted magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like, and outputs the information to the RAM 1003. The drive 1012 can also write information to the removable storage medium 30.

[0201] The communication device 1015 is, for example, a communication interface constituted by a communication device or the like for connecting with the network 1. The communication device 1015 may be a wireless LAN compatible communication device, an LTE (Long Term Evolution) compatible communication device, or a wired communication device that communicates by wire.

[0202] <<7. Additional Explanation>>

[0203] Although the preferred embodiments of the present disclosure have been described in detail with reference to the accompanying drawings, the present disclosure is not limited to these examples. Obviously, without departing from the technical concepts disclosed in the scope of the claims, a person of ordinary skill in the technical field to which the present disclosure belongs can implement various changes and modifications, and such changes and modifications should certainly be understood as falling within the technical spirit of the present disclosure.

[0204] For example, the server 20 may include some functional configurations of the information processing terminal 10. For example, the server 20 may also have a configuration including the functions of the above-mentioned renderer unit 140. Then, the server 20 may generate an image according to the motion feature and send the generated image to the information processing terminal 10.

[0205] The server 20 need not include Figure 3For example, the server 20 need not include the modification unit 235.

[0206] In addition, the steps in the process of the information processing terminal 10 described in this specification do not have to be performed in chronological order in the order shown in the flowchart. For example, the steps in the process of the information processing terminal 10 may be performed in an order different from the order shown in the flowchart.

[0207] In addition, a computer program may be created for causing hardware such as a CPU, ROM, RAM provided in the information processing terminal 10 to realize functions equivalent to the configuration of the above-described information processing terminal 10. A storage medium storing such a computer program is also provided.

[0208] In addition, the effects described in this specification are only explanatory or exemplary, and not restrictive. In other words, together with or instead of the above effects, the technology according to the present disclosure can provide other effects that will be clear to those skilled in the art from this specification.

[0209] Note that configurations such as the following also fall within the technical scope of the present disclosure.

[0210] (1) An information processing device comprising:

[0211] an obtaining unit that obtains a motion feature indicating a feature generated from the motion data and transmitted from the communication terminal; and

[0212] A generating unit that generates an image corresponding to the motion data used to generate the motion feature obtained by the obtaining unit.

[0213] (2) The information processing device according to (1),

[0214] The motion feature of the motion data is estimated using an estimation encoder obtained by learning the relationship between the motion data and the motion feature.

[0215] (3) The information processing device according to (1) or (2),

[0216] The generating unit generates an image corresponding to the motion data based on a pre-learned relationship between the motion feature and the image.

[0217] (4) The information processing device according to any one of (1) to (3),

[0218] The generation unit generates an image corresponding to the motion data based on a pre-learned relationship between the motion feature and a camera parameter and the image, the camera parameter being related to viewpoint information indicating a viewpoint from which the image is viewed.

[0219] (5) The information processing device according to any one of (1) to (4),

[0220] wherein the motion data includes designated motion data indicating motion data designated by a user, and

[0221] The obtaining unit obtains the motion feature of the designated motion data.

[0222] (6) The information processing device according to (5),

[0223] In which, the generation unit generates an image corresponding to the specified motion data based on the pre-learned relationship between the motion features, camera device parameters and character information and the image, the camera device parameters are related to viewpoint information indicating the viewpoint from which the image is viewed, and the character information indicates the character that performs the motion data.

[0224] (7) The information processing device according to (6),

[0225] The generating unit generates an image corresponding to the designated motion data at the input time.

[0226] (8) The information processing device according to any one of (1) to (7),

[0227] The motion feature includes noise information of adding a predetermined modification process to the image generated by the generating unit.

[0228] (9) The information processing device according to any one of (1) to (7),

[0229] The generating unit generates an image by adding a predetermined modification process to the motion data used to generate the motion feature obtained by the obtaining unit.

[0230] (10) The information processing device according to (5),

[0231] Wherein, an encoder corresponding to the specified motion data is used to estimate the motion characteristics of the specified motion data.

[0232] (11) The information processing device according to (10),

[0233] Wherein, a plurality of encoders corresponding to the designated motion data are used to estimate motion characteristics of the designated motion data.

[0234] (12) The information processing device according to any one of (1) to (11), further comprising:

[0235] A display unit displays the image generated by the generation unit.

[0236] (13) An information processing method executed by a computer, the method comprising:

[0237] obtaining a motion feature indicating a feature generated from the motion data and transmitted from the communication terminal; and

[0238] An image is generated corresponding to the motion data used to generate the obtained motion feature.

[0239] (14) A program that causes a computer to implement the following functions:

[0240] an obtaining function for obtaining a motion feature indicating a feature generated from the motion data and transmitted from the communication terminal; and

[0241] A generating function is used to generate an image corresponding to the motion data used to generate the motion feature obtained by the obtaining function.

[0242] [reference numerals list]

[0243] 10 Information processing terminal

[0244] 20 Servers

[0245] 110 Operating and display unit

[0246] 120 Communication Unit

[0247] 130 storage units

[0248] 140 renderer units

[0249] 143 Time Management Unit

[0250] 210 Communication Unit

[0251] 220 storage units

[0252] 221 Motion Data Storage Unit

[0253] 225 Motion feature storage unit

[0254] 230 Control Unit

[0255] 231 study units

[0256] 233 Estimation Unit

[0257] 235 Modification Unit

Claims

1. An information processing device, include: an obtaining unit, the obtaining unit obtaining a motion feature indicating a feature generated from the motion data and transmitted from the communication terminal; as well as A generating unit that generates an image corresponding to the motion data used to generate the motion feature obtained by the obtaining unit.

2. The information processing device according to claim 1, in, The motion feature of the motion data is estimated using an estimation encoder obtained by learning the relationship between the motion data and the motion feature.

3. The information processing device according to claim 1, in, The generation unit generates an image corresponding to the motion data based on a pre-learned relationship between the motion feature and the image.

4. The information processing device according to claim 1, in, The generation unit generates an image corresponding to the motion data based on a relationship between the motion feature and a camera parameter related to viewpoint information indicating a viewpoint from which the image is viewed, and an image, which is learned in advance.

5. The information processing device according to claim 1, in, The motion data includes designated motion data indicating motion data designated by a user, and The obtaining unit obtains the motion feature of the designated motion data.

6. The information processing device according to claim 5, in, The generation unit generates an image corresponding to the specified motion data based on a pre-learned relationship between the motion features, camera parameters and character information and the image, wherein the camera parameters are related to viewpoint information indicating a viewpoint from which the image is viewed, and the character information indicates a character performing the motion data.

7. The information processing device according to claim 6, in, The generating unit generates an image corresponding to the designated motion data at an input time.

8. The information processing device according to claim 7, in, The motion feature includes noise information of adding a predetermined modification process to the image generated by the generating unit.

9. The information processing device according to claim 7, in, The generating unit generates an image in which a predetermined modification process is added to the motion data used to generate the motion feature obtained by the obtaining unit.

10. The information processing device according to claim 5, in, The motion characteristics of the specified motion data are estimated using an encoder corresponding to the specified motion data.

11. The information processing device according to claim 10, in, A plurality of encoders corresponding to the specified motion data are used to estimate motion characteristics of the specified motion data.

12. The information processing device according to claim 11, further comprising: include: A display unit displays the image generated by the generation unit.

13. An information processing method executed by a computer, the method include: obtaining a motion feature indicating a feature generated from the motion data and transmitted from the communication terminal; as well as An image is generated corresponding to the motion data used to generate the obtained motion feature.

14. A program that enables a computer to implement the following functions: an obtaining function for obtaining a motion feature indicating a feature generated from the motion data and transmitted from the communication terminal; and A generating function is used to generate an image corresponding to the motion data used to generate the motion feature obtained by the obtaining function.

Images