Information processing system and information processing method

Abstract

To provide an information processing system and an information processing method that can reduce a communication load when transmitting data for reproducing a three-dimensional image.SOLUTION: An information processing system includes a transmission information processing device and a reception information processing device that receives information from the transmission information processing device through a network. The transmission information processing device generates motion information indicating the motion of an object and texture information indicating the color of the object from shooting data of the object, which is a dynamic body, and transmits motion data corresponding to the motion information and texture data corresponding to the texture information to the reception information processing device. The reception information processing device reflects motion based on the motion data and texture based on the texture data in a 3D model corresponding to the object.SELECTED DRAWING: Figure 5

Description

Technical Field

[0001] The present disclosure relates to an information processing system and an information processing method used for three-dimensional video transmission.

Background Art

[0002] In recent years, a photographing device equipped with a depth sensor together with an image sensor has been developed. By photographing an object, this photographing device acquires depth information indicating the distance to the object in addition to color information indicating the color of the object (see, for example, Patent Document 1). By reproducing the photographing data including the color information and the depth information, it is possible to reproduce an image that reproduces the color and three-dimensional structure of the object.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] When the photographing data is reproduced at a location away from the photographing location of the object, the photographing data is transmitted through a network. Since the capacity of the data including the color information and the depth information is larger than that of conventional still images and moving image data including only the color information, the communication load increases. Therefore, for the transmission of data for reproducing a three-dimensional image, development of a technology capable of reducing the communication load is desired.

Means for Solving the Problems

[0005] Each aspect of an information processing system and an information processing method for solving the above problems will be described. [Aspect 1] An information processing system including a transmission information processing device and a reception information processing device that receives information from the transmission information processing device through a network, wherein the transmission information processing device generates motion information indicating the motion of a target that is a moving object and texture information indicating the color of the target from the captured data of the target, and transmits motion data corresponding to the motion information and texture data corresponding to the texture information to the reception information processing device, and the reception information processing device reflects an operation based on the motion data and a texture based on the texture data on a 3D model corresponding to the target stored in the reception information processing device.

[0006] According to the above configuration, the data of the motion and the texture are used as transmission data, and the data representing the three-dimensional shape of the target is not transmitted. Therefore, compared with the case of transmitting the three-dimensional shape data, the capacity of the transmission data is reduced, so that the load on communication can be reduced.

[0007] [Aspect 2] The information processing system according to [Aspect 1], wherein the transmission information processing device transmits data of a transmission target part that is a part of a plurality of parts of the target to the reception information processing device for at least one of the motion data and the texture data. According to the above configuration, it is possible to reduce the capacity of the transmission data as compared with the case where the data of the entire target is transmitted.

[0008] [Aspect 3] The information processing system according to [Aspect 2], wherein the transmission information processing device calculates a displacement amount for each part using the latest motion information and the past motion information, and determines the transmission target part using the displacement amount. According to the above configuration, the change in the position of the part with a large movement is likely to be reflected in the 3D model. Therefore, an accurate reproduction of the motion of the target is possible.

[0009] [Aspect 4] The information processing system according to [Aspect 3], wherein the transmission information processing device determines the transmission target part using the weighting set for each part and the amount of displacement for each part. According to the above configuration, as the weighting, it is possible to set the priority reflecting the movement for each part.

[0010] [Aspect 5] Each of the transmission information processing device and the reception information processing device stores in advance information indicating the operation for a plurality of operations, and each of the plurality of operations is a specified operation. The transmission information processing device determines whether the operation indicated by the motion information corresponds to any of the specified operations. When the corresponding specified operation is specified, the identification information of the specified operation is transmitted to the reception information processing device as the motion data. The reception information processing device reflects the specified operation corresponding to the identification information on the 3D model when the motion data is the identification information. The information processing system according to any one of [Aspect 1] to [Aspect 4]. According to the above configuration, it is possible to further reduce the capacity of the transmission data by transmitting the identification information of the specified operation as the operation information.

[0011] [Aspect 6] When the specified operation corresponding to the operation indicated by the motion information is not specified, the transmission information processing device transmits the motion information to the reception information processing device as the motion data. The reception information processing device reflects the operation indicated by the motion information on the 3D model when the motion data is the motion information. The information processing system according to [Aspect 5].

[0012] According to the above configuration, by properly using the reflection of the specified operation and the reflection of the operation indicated by the motion information on the 3D model, both the reduction of the transmission data capacity and the reproduction of the target movement can be suitably achieved.

[0013] [Aspect 7] The information processing system according to [Aspect 5] or [Aspect 6], wherein the transmission information processing apparatus executes identification of the specified operation corresponding to the operation indicated by the motion information using a machine learning model. According to the above configuration, the accuracy of querying the specified operation can be improved.

[0014] [Aspect 8] The information processing system according to any one of [Aspect 1] to [Aspect 7], wherein the target is a person.

[0015] According to the above configuration, it is possible to use the information processing system for applications that were difficult in two-dimensional information transmission, such as remote transmission of actual techniques, and it is possible to accurately transmit information including operations in such applications. And, since the communication load can be reduced, the system becomes easier to use for such applications.

[0016] [Aspect 9] An information processing method executed by a plurality of computers, comprising: generating motion information indicating the motion of a target that is a moving object and texture information indicating the color of the target from the shooting data of the target; transmitting motion data corresponding to the motion information and texture data corresponding to the texture information through a network; and reflecting an operation based on the motion data and a texture based on the texture data on a 3D model corresponding to the target that has been generated in advance.

[0017] According to the above method, the data of the motion and the texture are used as transmission data, and the data representing the three-dimensional shape of the target is not transmitted. Therefore, compared with the case of transmitting the three-dimensional shape data, the capacity of the transmission data is reduced, so that the load on the communication can be reduced.

Advantages of the Invention

[0018] According to the present disclosure, the communication load can be reduced when transmitting data for reproducing a three-dimensional image.

Brief Description of the Drawings

[0019]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Mode for Carrying Out the Invention

[0020] With reference to the drawings, an embodiment of an information processing system and an information processing method will be described. The information processing system of the present embodiment is used, for example, in a system for remote teaching of practical skills such as sports, musical instrument performance, dance, and traditional arts, a monitoring system in a factory, a construction site, etc., and other systems for remote communication.

[0021] [Overall Configuration of Information Processing System] With reference to Figure 1, the overall configuration of the information processing system will be described. As shown in Figure 1, the information processing system 100 includes a transmission system 110 and a reception system 120. The transmission system 110 transmits information generated based on the photographing of the target Tg to the reception system 120. The reception system 120 reproduces a three-dimensional image of the target Tg based on the information received from the transmission system 110. In the present embodiment, the target Tg is a person.

[0022] The transmission system 110 includes a photographing device 10 and a transmission information processing device 20. The reception system 120 includes a reception information processing device 30 and a playback device 40. The photographing device 10 continuously photographs the target Tg as a still image or a moving image. The photographing device 10 may include a camera equipped with an image sensor. For example, the photographing device 10 may be a web camera or a smartphone.

[0023] The photographing device 10 outputs the photographing data of the target Tg to the transmission information processing device 20. The photographing device 10 and the transmission information processing device 20 may be connected by wire or wirelessly, or may be connected via a network. Also, the photographing device 10 and the transmission information processing device 20 may be an integrated device.

[0024] The transmission information processing device 20 generates transmission data based on the photographing data. The transmission data includes information regarding the appearance and operation of the target Tg. The transmission information processing device 20 is a computer device such as a personal computer, a tablet terminal, a smartphone, or a server device.

[0025] The transmission information processing device 20 and the reception information processing device 30 can transmit and receive data to and from each other through a network such as the Internet, and the transmission data generated by the transmission information processing device 20 is transmitted to the reception information processing device 30.

[0026] The reception information processing device 30 generates display data based on reflecting the appearance and operation of the target Tg on a 3D model using the transmission data. The reception information processing device 30 is a computer device such as a personal computer, a tablet terminal, a smartphone, or a server device.

[0027] The reception information processing device 30 outputs the display data to the playback device 40. The reception information processing device 30 and the playback device 40 may be connected by wire or wirelessly, or may be connected via a network. Also, the reception information processing device 30 and the playback device 40 may be an integrated device.

[0028] The playback device 40 displays a three-dimensional image of the target Tg using the display data. The playback device 40 is, for example, a display device including a display panel such as a liquid crystal panel, a head-mounted display for VR (Virtual Reality) or AR (Augmented Reality), a projection device, or the like.

[0029] In the above configuration, the transmission information processing device 20 may also have at least a part of the functions of the imaging device 10, or the reception information processing device 30 may also have at least a part of the functions of the playback device 40.

[0030] [Configuration of Transmission Information Processing Device] Referring to FIG. 2, the detailed configuration of the transmission information processing device 20 will be described. The transmission information processing device 20 includes a communication unit 21, a control unit 22, and a storage unit 23.

[0031] The communication unit 21 performs communication processing between the transmission information processing device 20 and other devices, such as connecting to a network and transmitting and receiving data. Also, when the transmission information processing device 20 transmits and receives data to and from the imaging device 10 by short-range wireless communication, the communication unit 21 may have a processing unit that executes communication processing through the above-mentioned network and a processing unit that executes communication processing such as connecting to a device by short-range wireless communication and transmitting and receiving data.

[0032] The control unit 22 functions as a motion processing unit 22a, a texture processing unit 22b, and a processing target management unit 22c by executing a transmission information processing program stored in the storage unit 23.

[0033] The motion processing unit 22a analyzes the region that is the target Tg among the images included in the imaging data of the imaging device 10, and generates information indicating the motion of the target Tg. This information is the motion information Im. The motion information Im is information indicating the arrangement of the skeleton by coordinates indicating the positions of the joints of the human body, and in other words, indicates the pose of the target Tg at the timing captured by the imaging device 10. One piece of motion information Im indicates the motion of the target Tg by showing the shape of the body at one point in a series of motions of the target Tg.

[0034] The motion processing unit 22a calculates the displacement amount of the arrangement of the skeleton for each part of the target Tg based on the current and past motion information Im. In addition, the motion processing unit 22a determines whether the motion indicated by the motion information Im corresponds to a specified motion that is a pre-registered motion. This determination is executed using a machine learning model.

[0035] Based on the results of these processes, the motion processing unit 22a generates motion data Dm, which is data to be transmitted to the received information processing device 30. The motion data Dm includes the motion information Im of the specified part or the identification information of the specified motion. The motion data Dm is an example of transmission data.

[0036] The texture processing unit 22b acquires color information from the region that is the target Tg among the images included in the imaging data of the imaging device 10. This information is the texture information It, and the texture information It is information corresponding to the texture of the 3D model. The texture is attached to the surface of the 3D model to express the color and texture of the model. The texture information It, in other words, indicates the appearance including the color tone and the effect of light on the surface of the target Tg in a color including lightness and chroma.

[0037] The texture processing unit 22b generates texture data Dt including the texture information It of the specified part as data to be transmitted to the received information processing device 30. The texture data Dt is an example of transmission data.

[0038] The processing target management unit 22c determines and gives instructions on the processing targets of the motion processing unit 22a and the texture processing unit 22b. Thereby, the processing target management unit 22c manages the cooperation between the processing of the motion processing unit 22a and the processing of the texture processing unit 22b.

[0039] Specifically, the processing target management unit 22c identifies the region that is the target Tg in the image included in the shooting data of the imaging device 10. Further, the processing target management unit 22c determines the part to be the data transmission target by using the displacement amount for each part calculated by the motion processing unit 22a.

[0040] The storage unit 23 stores various programs and data necessary for the execution of the processing by the control unit 22. As an example of such a program, the storage unit 23 stores a transmission information processing program. Further, as an example of such data, the storage unit 23 stores the specified operation data 23a. The specified operation data 23a includes specified motion information Ip which is information indicating a specified operation, its identification information, and information necessary for querying the specified operation by a machine learning model. The specified motion information Ip is information indicating the arrangement or displacement of the skeleton by coordinates indicating the position of the joint, etc. That is, the specified motion information Ip indicates an operation as a specific body shape or its change.

[0041] The physical configuration, that is, the hardware configuration of the transmission information processing device 20 having the above functions will be described. The transmission information processing device 20 includes an arithmetic device such as a CPU, MPU, and GPU, a memory such as a ROM, RAM, register memory, and unbuffered memory, and a storage such as an SSD and HDD. The arithmetic device loads an operating system and various programs from the storage into the memory and executes the instructions taken out from the memory. The transmission information processing device 20 may include an integrated circuit such as an ASIC and FPGA.

[0042] In addition, the transmission information processing device 20 is provided with a communication interface for transmitting and receiving data to and from a device at the connection destination via a network. The communication interface is implemented as hardware, software, or a combination thereof.

[0043] Each process executed by the transmission information processing device 20 may be executed by software provided in the transmission information processing device 20, or may be realized by a combination of an integrated circuit and software provided in the transmission information processing device 20.

[0044] [Configuration of the reception information processing device] With reference to FIG. 3, the detailed configuration of the reception information processing device 30 will be described. The reception information processing device 30 includes a communication unit 31, a control unit 32, and a storage unit 33.

[0045] The communication unit 31 performs communication processing between the reception information processing device 30 and other devices, such as connection to a network and transmission and reception of data. Further, when the reception information processing device 30 transmits and receives data to and from the playback device 40 by short-range wireless communication, the communication unit 31 may have a processing unit that executes the communication processing through the above-described network and a processing unit that executes communication processing such as connection of devices and transmission and reception of data by short-range wireless communication.

[0046] The control unit 32 functions as a motion processing unit 32a, a texture processing unit 32b, and a display data generation unit 32c by executing a reception information processing program stored in the storage unit 33.

[0047] Based on the motion data Dm received from the transmission information processing device 20, the motion processing unit 32a specifies application motion data Rm that is data to be applied to the 3D model. The application motion data Rm includes motion information Im generated from the captured data or specified motion information Ip indicating a specified operation.

[0048] The texture processing unit 32b performs processing such as resolution adjustment on the texture data Dt received from the transmission information processing device 20, and generates applied texture data Rt which is data to be applied to the 3D model.

[0049] The display data generation unit 32c reflects the motion based on the applied motion data Rm and the texture based on the applied texture data Rt on the 3D model of the target Tg that has been generated in advance and stored in the reception information processing device 30, and generates display data.

[0050] The storage unit 33 stores various programs and data necessary for the execution of the processing by the control unit 32. As an example of such a program, the storage unit 33 stores a reception information processing program. Further, as an example of such data, the storage unit 33 stores specified operation data 33a and model data 33b. The specified operation data 33a includes specified motion information Ip and its identification information. The model data 33b is data of the 3D model of the target Tg.

[0051] The physical configuration, that is, the hardware configuration of the reception information processing device 30 having the above functions will be described. The reception information processing device 30 includes an arithmetic circuit which is an arithmetic device such as a CPU, MPU, GPU, memories such as a ROM, RAM, register memory, and unbuffered memory, and a storage such as an SSD and HDD. The arithmetic device loads the operating system and various programs from the storage into the memory, and executes the instructions taken out from the memory. The reception information processing device 30 may include an integrated circuit such as an ASIC and FPGA.

[0052] Also, the reception information processing device 30 includes a communication interface for transmitting and receiving data to and from a connected device via a network. The communication interface is implemented as hardware, software, or a combination thereof.

[0053] Each process executed by the reception information processing apparatus 30 may be executed by software provided in the reception information processing apparatus 30, or may be realized by a combination of an integrated circuit and software provided in the reception information processing apparatus 30.

[0054] [Operation of Information Processing System] As an information processing method executed by the information processing system 100, the processing flow of the information processing system 100 will be described. The processing of the information processing system 100 includes processing in a preliminary preparation stage and processing in a transmission stage. In the transmission stage, imaging of the target Tg, transmission of data, and reproduction of a three-dimensional image are performed.

[0055] With reference to FIG. 4, the processing flow in the preliminary preparation stage will be described. The processing in the preliminary preparation stage is performed before the processing in the transmission stage, for example, before the start of an action such as performance by the target Tg. As shown in FIG. 4, first, information on the three-dimensional shape and surface color of the target Tg is taken into the transmission information processing apparatus 20 (step S10). For the acquisition of the shape and color information, known 3D scanning technology may be used. For the 3D scan, the imaging device 10 may be used, or a device different from the imaging device 10 may be used.

[0056] Subsequently, the transmission information processing apparatus 20 generates a 3D model using the acquired shape and color information (step S11). The generated 3D model is a mesh model in which the three-dimensional shape of the target Tg is reflected, and has a texture in which the color of the target Tg is reflected as an initial texture.

[0057] The data of the generated 3D model is transmitted from the transmission information processing apparatus 20 to the reception information processing apparatus 30 (step S12). When the reception information processing apparatus 30 receives the data of the 3D model, it stores this data as model data 33b (step S13).

[0058] Note that information on the shape and color of the target Tg may be sent from the transmission information processing device 20 to the reception information processing device 30, and the reception information processing device 30 may generate a 3D model. Also, the 3D model may be generated from an image. Among the transmission information processing device 20 and the reception information processing device 30, the control unit of the device that generates the 3D model functions as a 3D model generation unit.

[0059] Also, by transmitting information on the location where the target Tg is photographed from the transmission information processing device 20 to the reception information processing device 30, background data used when reproducing the image of the target Tg may be generated by the reception information processing device 30. The information to be transmitted is, for example, information acquired by an image or a 3D scanner. Alternatively, the background data may be generated by the transmission information processing device 20 and sent to the reception information processing device 30, or may be separately generated and stored in the reception information processing device 30.

[0060] Subsequently, prior to the description of the processing in the transmission stage, machine learning related to the query of the specified operation will be described. Depending on the type of action of the target Tg such as sports or musical instrument performance performed in the transmission stage, there are actions assumed to be executed by the target Tg. Such actions are set as specified actions, and information indicating the poses or their changes constituting the specified actions is included in the specified motion information Ip in the specified motion data 23a, 33a. Identification information such as a number is assigned to the specified actions, and the specified motion information Ip is associated with the identification information and included in the specified motion data 23a, 33a.

[0061] In the information processing system 100, when the acquisition of the motion information Im based on the imaging of the target Tg is repeated, the characteristics of the skeletal arrangement and the characteristics of its temporal change are learned by machine learning using the accumulated motion information Im. As a result, when new motion information Im is obtained, it becomes possible to determine which defined motion the motion indicated by the motion information Im corresponds to, and to predict the motion of the target Tg following the motion indicated by the motion information Im. Since the shape of the human body is not exactly the same every time for the same motion, the accuracy of querying the defined motion and predicting the motion can be improved by machine learning. Further, as a result of machine learning, addition or modification of the defined motion may be performed with the defined motion data 23a, 33a.

[0062] When querying the defined motion, the latest motion information Im may be input alone, or the displacement may be input using the latest motion information Im and the motion information Im traced back a predetermined time from the latest.

[0063] Note that the fact that the motion indicated by the motion information Im corresponds to the defined motion means that the motion having the body shape represented by the motion information Im as a component and the motion having the body shape represented by the defined motion information Ip or its change as a component are determined to be the same motion.

[0064] The defined motion data 23a, 33a are shared between the transmission information processing device 20 and the reception information processing device 30 in the pre-preparation stage. That is, the data learned by the transmission information processing device 20 is sent from the transmission information processing device 20 to the reception information processing device 30. Alternatively, the defined motion data 23a, 33a may not be held in each of the transmission information processing device 20 and the reception information processing device 30, but may be stored in a database connectable from each device 20, 30.

[0065] With reference to FIGS. 5 and 6, the processing flow in the transmission stage will be described. First, with reference to FIG. 5, the processing flow of the transmission system 110 will be described. As shown in FIG. 5, based on the imaging device 10 imaging the target Tg, the transmission information processing device 20 acquires imaging data (step S20). The transmission information processing device 20 extracts, from the image indicated by the imaging data, the region that is the target Tg, that is, the region of a person (step S21). For example, when the imaging device 10 captures a moving image, an image of one frame is the processing target, and a person is extracted. A known algorithm may be used for extracting a person in the image.

[0066] The processing of step S21 is the processing by the processing target management unit 22c, and the extracted person region is set as the processing target region and passed on to the processing of each of the motion processing unit 22a and the texture processing unit 22b.

[0067] First, the processing of the motion processing unit 22a will be described. The motion processing unit 22a detects a skeleton by image analysis of the person region and generates motion information Im (step S30). When the motion information Im is generated, the motion processing unit 22a queries whether there is a specified motion corresponding to the motion indicated by the motion information Im (step S31).

[0068] Also, the motion processing unit 22a calculates the displacement amount in the three-dimensional space of the position of the skeleton of each part using the latest motion information Im and the past motion information Im (step S32). For example, the displacement amount per unit time may be calculated, or a specific motion information Im such as the motion information Im that was the transmission target in the past may be used as a comparison target, and the displacement amount may be calculated. The division of the parts may be determined based on joints. Note that the past motion information Im only needs to be temporarily stored for a certain period.

[0069] Subsequently, the motion processing unit 22a calculates an evaluation value for each part used for determining the transmission target using the calculated displacement amount and a predetermined calculation formula (step S33). Weights are set for each part, and in the calculation formula, the weight of the part and the displacement amount are used. The weight of each part is set such that the weight of the part with higher importance becomes larger according to the type of action of the target Tg. For example, the weight of a part that is assumed to have characteristic movement in the action of the target Tg is made larger.

[0070] In one example, the weight of the hand is 0.5, the weight of the arm is 0.3, and the weight of other parts is 0.2, and the value obtained by multiplying the displacement amount by the weight is used as the evaluation value. In this case, with the hand being the largest, the weights are set such that the weights are larger in the order of the hand, the arm, and other parts.

[0071] Then, based on the evaluation value, the transmission target part is determined (step S22). For example, a part whose evaluation value is equal to or greater than a predetermined threshold value is set as the transmission target. A part with a larger weight and a part with a larger displacement amount are more likely to be set as the transmission target. Step S22 is the process of the processing target management unit 22c, and the processing target management unit 22c instructs the texture processing unit 22b and the motion processing unit 22a about the transmission target part.

[0072] Upon receiving the instruction for the transmission target part, the motion processing unit 22a determines the motion data Dm (step S34). Specifically, when the specified motion corresponding to the motion indicated by the motion information Im is identified in the process of step S31, the motion processing unit 22a uses the identification information of the specified motion as the motion data Dm. In this case, the motion data Dm indirectly corresponds to the motion information Im obtained from the shooting data by indicating the specified motion information Ip showing the same motion as the motion information Im obtained from the shooting data through the identification information.

[0073] On the other hand, when no specified operation is identified in the process of step S31, that is, when there is no specified operation corresponding to the operation indicated by the motion information Im, the motion processing unit 22a sets the information corresponding to the transmission target part among the latest motion information Im as the motion data Dm. In this case, the motion data Dm directly corresponds to the motion information Im obtained from the shooting data.

[0074] When the motion data Dm is determined, the motion processing unit 22a compresses it by encoding the motion data Dm (step S35). The motion data Dm is identification information of a specified operation or text data indicating motion information Im such as coordinates, and a known suitable compression method may be used. The motion processing unit 22a transmits the compressed motion data Dm to the reception information processing apparatus 30 through the communication unit 21 (step S36).

[0075] Next, the process of the texture processing unit 22b will be described. After the person's area is extracted from the image indicated by the shooting data by the process of step S21, the texture processing unit 22b generates texture information It by performing image analysis on the person's area (step S40). Furthermore, the texture processing unit 22b divides the texture information It for each part of the target Tg (step S41).

[0076] When receiving an instruction of the transmission target part by the process of step S22, the texture processing unit 22b sets the information corresponding to the transmission target part among the texture information It as the texture data Dt (step S42).

[0077] Then, the texture processing unit 22b compresses the texture data Dt by encoding it (step S43). The texture data Dt is image data, and a known suitable compression method may be used.

[0078] The texture processing unit 22b transmits the compressed texture data Dt to the reception information processing apparatus 30 through the communication unit 21 (step S44). The motion data Dm and the texture data Dt are preferably transmitted by packet communication using TCP as a communication protocol. For the texture, since the sense of discomfort of the user who views the reproduced image is small even if information loss or modification occurs, UDP may be used as the communication protocol.

[0079] Subsequently, with reference to FIG. 6, the processing flow of the reception system 120 will be described. First, the processing of the motion processing unit 32a will be described. When the motion processing unit 32a receives the motion data Dm through the communication unit 31 (step S37), the motion processing unit 32a decrypts the motion data Dm (step S38).

[0080] Then, the motion processing unit 32a specifies the applied motion data Rm (step S39). Specifically, the motion processing unit 32a determines whether the decrypted motion data Dm is identification information of a specified operation. Then, when the motion data Dm is the identification information of the specified operation, the motion processing unit 32a acquires the specified motion information Ip of the specified operation corresponding to the identification information from the specified operation data 33a and sets it as the applied motion data Rm.

[0081] On the other hand, when the motion data Dm is not the identification information of the specified operation, that is, when the motion data Dm is the motion information Im generated from the photographed data, the motion processing unit 32a sets the motion information Im as the applied motion data Rm.

[0082] Next, the processing of the texture processing unit 32b will be described. When the texture processing unit 32b receives the texture data Dt through the communication unit 31 (step S45), the texture processing unit 32b decrypts the texture data Dt (step S46).

[0083] Then, the texture processing unit 32b performs conversion for application to the 3D model and display on the playback device 40, such as resolution adjustment like upscaling processing, on the decoded texture data Dt, and generates applied texture data Rt (step S47).

[0084] When the applied motion data Rm and the applied texture data Rt are obtained, the display data generation unit 32c reflects these data on the 3D model of the model data 33b to generate display data (step S50). That is, as shown by the applied motion data Rm, the arrangement of the skeleton of the 3D model is changed, and the texture indicated by the applied texture data Rt is pasted on the corresponding part of the 3D model. Further, the display data generation unit 32c synthesizes a background with the 3D model to generate display data. Then, an image is reproduced by the playback device 40 based on the generated display data (step S51).

[0085] By repeating the processes of FIGS. 5 and 6, the movement of the target Tg is reproduced together with the appearance in the image reproduced by the playback device 40. Therefore, it is possible to accurately transmit three-dimensional information including the movement.

[0086] When the target Tg moves, the shadowed part in the body of the target Tg and the position of the wrinkles on the clothes of the target Tg change. Such changes in light and shade in the appearance of the target Tg are also reflected on the 3D model as changes in the texture, so an image closer to the real object of the target Tg is reproduced, and it is difficult for the user viewing the image to feel a sense of discomfort. Also, fine shape changes such as expressions can be easily reproduced by changes in the texture.

[0087] In addition, when applying the applied motion data Rm and the applied texture data Rt, the motion data Dm and the texture data Dt corresponding to the same captured image may be synchronized using a time stamp or the like so that the corresponding data is reflected simultaneously. However, since fine texture misalignment is difficult for the user viewing the image to recognize, synchronization of the motion data Dm and the texture data Dt is not essential. By omitting the synchronization process, an improvement in the frame rate can be expected.

[0088] In the reception information processing apparatus 30, the motion information Im received as the motion data Dm and the identification number of the specified operation are accumulated together with the reception order. By using this to machine-learn the reception order of the identification numbers of the specified operations, subsequent specified operations can be estimated, and the specified operations may be reflected in the 3D model according to the estimation result. Also, such estimation may be performed in the transmission information processing apparatus 20.

[0089] Furthermore, by using the information accumulated in the reception information processing apparatus 30 as learning data, the accuracy of querying the specified operation in the transmission information processing apparatus 20 can be improved. Therefore, after the end of the transmission stage, it is preferable that the information accumulated in the reception information processing apparatus 30 be transmitted to the transmission information processing apparatus 20. That is, data obtained by adding a series of motion data Dm received by the reception information processing apparatus 30 in the transmission stage to the specified operation data 33a received by the reception information processing apparatus 30 from the transmission information processing apparatus 20 before the transmission stage is sent to the transmission information processing apparatus 20. By further learning this data in the transmission information processing apparatus 20, the accuracy of querying the specified operation can be improved. Also, for data accumulation, when the motion data Dm is identification information of a specified operation, the motion information Im may also be transmitted to the reception information processing apparatus 30. Furthermore, whether or not to transmit such motion information Im may be determined according to the network situation such as the line speed.

[0090] [Function] In the information processing system 100 of the present embodiment, operation and texture data are used as transmission data, and data representing the three-dimensional shape of the target Tg such as depth information is not transmitted. Then, by synthesizing the operation and texture onto the 3D model held in the reception information processing device 30, a three-dimensional image is reproduced. Therefore, compared with the case of transmitting three-dimensional shape data, the capacity of the transmission data is reduced, so the communication load is reduced. By reducing the communication load, even when the imaging of the target Tg and the reproduction of the image are performed in real time, the processing delay can be suppressed and smooth image reproduction is possible. Furthermore, large-scale equipment for detecting the three-dimensional shape of the target Tg and devices having specific functions such as depth cameras are not required.

[0091] By acquiring information from the target Tg in advance and generating a 3D model, it is also less likely to cause failures or a sense of discomfort to the user when reflecting the operation and texture. Also, when sequentially transmitting three-dimensional shape data, there is a limit to the accuracy of the reproduced three-dimensional shape due to constraints such as the data capacity, calculation load, and available devices. On the other hand, when generating a 3D model in advance, such constraints are reduced, so even fine parts such as fingers can be reproduced with high accuracy. Therefore, it is also possible to handle the case of reproducing an image with high resolution, and it is also possible to enhance the reproducibility of the texture of a person by increasing the resolution.

[0092] Also, since data is transmitted for a part of a plurality of parts of the target Tg, it is possible to reduce the capacity of the transmission data compared to the case where full-body data is transmitted. Also, since the texture data is divided for each part and applied to the 3D model, it is possible to suppress a large deviation in the arrangement of the texture.

[0093] And since the target transmission site is determined using the displacement amount for each site, changes in the position of sites with large movements are reflected in the 3D model. Therefore, the actions of the target Tg are accurately reproduced. Furthermore, since weighting for each site is taken into account in determining the target transmission site, when there is a site that makes characteristic movements even if the change in position is not large depending on the type of action of the target Tg, it is possible to preferentially reflect the movement of this site in the 3D model. Therefore, it is possible to accurately reproduce the actions of the target Tg without missing characteristic movements.

[0094] Also, regarding the texture, the target transmission site set based on the movement is applied. In sites with large movements, since the changes in appearance such as wrinkles and swaying of the clothes accompanying this movement are also large, by setting the same site as the transmission target of the motion information Im as the transmission target of the texture information It, the changes in the appearance of the target Tg are accurately reproduced. Also, compared to the case of determining the target transmission site of the texture separately from the movement, the computational load can be reduced.

[0095] As described above, according to the above embodiment, the following effects can be obtained. (1) When transmitting data for reproducing a three-dimensional image, the data of the movement and the texture are transmitted, and the data representing the three-dimensional shape is not transmitted. Therefore, since the capacity of the transmitted data is reduced, it is possible to reduce the communication load.

[0096] (2) For at least one of the movement and the texture, the data of the target transmission site, which is a part of the plurality of sites of the target Tg, is transmitted. Therefore, compared to the case where the entire data of the target Tg is transmitted, it is possible to reduce the capacity of the transmitted data.

[0097] (3) Since the target transmission site is determined using the displacement amount for each site, changes in the position of sites with large movements are more likely to be reflected in the 3D model. Therefore, it is possible to accurately reproduce the actions of the target Tg.

[0098] (4) The transmission target part is determined using the weighting set for each part and the displacement amount for each part. According to such a configuration, as the weighting, it is possible to set the priority of reflecting the movement for each part.

[0099] (5) When a specified operation corresponding to the operation indicated by the motion information Im is specified based on an inquiry of the specified operation, the identification information of the specified operation is transmitted as the operation information. Thereby, it is possible to further reduce the capacity of the transmission data. And, by properly using the reflection of the specified operation and the reflection of the operation indicated by the motion information Im for the 3D model, it is possible to suitably reduce the capacity of the transmission data and reproduce the movement of the target Tg.

[0100] (6) If the target Tg is a person, it is possible to use the information processing system 100 for applications that were difficult in two-dimensional information transmission such as remote transmission of actual techniques, and in such applications, the transmission of information including operations can be accurately realized. And, since it is possible to reduce the communication load, it becomes easier to use the information processing system 100 for such applications.

[0101] [Modification Example] The above embodiment can be implemented with the following modifications. The following modification examples may be implemented in combination.

[0102] · There may be an opportunity to transmit only one of the motion data Dm and the texture data Dt. For example, the transmission frequency of the texture data Dt may be reduced in consideration of the network situation and the like. Also, when a texture is associated with a specified operation and the specified operation is reflected in the 3D model, the pre-stored texture may be synthesized with the 3D model. In this case, when the motion data Dm is the identification information of the specified operation, the texture data Dt may not be transmitted.

[0103] · The determination of the transmission target parts of the motion information Im and the texture information It may be performed in a manner different from the above-described embodiment. For example, the weighting of the parts may not be performed, or the transmission target parts may be separately set for the motion information Im and the texture information It. For example, for the texture information It, only the face part may always be the transmission target. Also, the parts to be transmitted may not be set, and information about the entire target Tg may be transmitted. · The inquiry of the specified operation may not be performed, and the motion information Im may always be transmitted as the motion data Dm. Also, the inquiry of the specified operation may be performed in a manner different from the above-described embodiment using machine learning.

[0104] · The 3D model of the target Tg may not be generated based on the information acquired from the target Tg. For example, it may be a general-purpose 3D model of a person. In short, any model composed of parts similar to the target Tg is acceptable, as long as it has parts common to the target Tg and thus corresponds to the target Tg.

[0105] · The target Tg may be not only a person but also an animal or an object. Also, for an object such as a musical instrument where a part moves slightly, its movement may be expressed only by a change in texture. For example, a 3D model of such an object may be arranged near the 3D model of the target Tg, and together with the transmission data of the target Tg, the color data of the object near the target Tg may be sent from the transmission information processing device 20 to the reception information processing device 30 as texture data for the 3D model of the object.

[0106] · Each of the functions of the transmission information processing device 20 and the reception information processing device 30 may be realized by a plurality of information processing devices. One information processing device is a single computer device. That is, the transmission information processing device 20 and the reception information processing device 30 may be composed of one or a plurality of information processing devices.

[0107] [Supplementary Note] Means for solving the above problems include the following appendices as technical ideas derived from the above embodiments and modifications.

[0108] (Appendix 1) A transmission information processing device connected to a reception information processing device via a network, generating motion information indicating the motion of the object and texture information indicating the color of the object from the shooting data of the object which is a moving body; transmitting, to the reception information processing device, motion data corresponding to the motion information as data indicating an operation to be reflected in the 3D model corresponding to the object stored in the reception information processing device; transmitting, to the reception information processing device, texture data corresponding to the texture information as data indicating a texture to be reflected in the 3D model; A transmission information processing device that executes the above.

[0109] (Appendix 2) A reception information processing device connected to a transmission information processing device via a network, receiving, from the transmission information processing device, motion data corresponding to motion information indicating the motion of the object generated from the shooting data of the object which is a moving body and texture data corresponding to texture information indicating the color of the object generated from the shooting data; reflecting, on the 3D model corresponding to the object stored in the reception information processing device, the motion based on the motion data and the texture based on the texture data; A reception information processing device that executes the above.

Explanation of Reference Numerals

[0110] 10…Shooting device 20…Transmission information processing device 30…Reception information processing device 40…Playback device 100…Information processing system

Claims

1. An information processing system including a transmission information processing device and a reception information processing device that receives information from the transmission information processing device through a network, The transmission information processing device, generates motion information indicating the motion of the object and texture information indicating the color of the object from the shooting data of the object that is a moving body, and transmits motion data corresponding to the motion information and texture data corresponding to the texture information to the reception information processing device, The reception information processing device, executes reflecting the motion based on the motion data and the texture based on the texture data on the 3D model corresponding to the object stored in the reception information processing device Information processing system.

2. The transmission information processing device transmits data of a transmission target part that is a part of a plurality of parts of the object to the reception information processing device for at least one of the motion data and the texture data. The information processing system according to claim 1.

3. The transmission information processing device calculates a displacement amount for each part using the latest motion information and the past motion information, and determines the transmission target part using the displacement amount. The information processing system according to claim 2.

4. The transmission information processing device determines the transmission target part using the weighting set for each part and the displacement amount for each part. The information processing system according to claim 3.

5. Each of the transmission information processing device and the reception information processing device stores in advance information indicating the operation for a plurality of operations, and each of the plurality of operations is a specified operation, The transmission information processing device, Determine whether the operation indicated by the motion information corresponds to any of the specified operations. When the corresponding specified operation is identified, transmit the identification information of the specified operation as the motion data to the receiving information processing device. The receiving information processing device When the motion data is the identification information, reflect the specified operation corresponding to the identification information in the 3D model. The information processing system according to claim 1.

6. The transmitting information processing device When the specified operation corresponding to the operation indicated by the motion information is not identified, transmit the motion information as the motion data to the receiving information processing device. The receiving information processing device When the motion data is the motion information, reflect the operation indicated by the motion information in the 3D model. The information processing system according to claim 5.

7. The transmitting information processing device executes the identification of the specified operation corresponding to the operation indicated by the motion information using a machine learning model. The information processing system according to claim 5.

8. The object is a person The information processing system according to claim 1.

9. An information processing method executed by a plurality of computers, comprising: Generating motion information indicating the motion of the object, which is a moving object, and texture information indicating the color of the object from the shooting data of the object; Transmitting the motion data corresponding to the motion information and the texture data corresponding to the texture information through a network; Reflecting the motion based on the motion data and the texture based on the texture data in a pre-generated 3D model corresponding to the object. An information processing method including

Images