Information processing device

Abstract

This information processing device includes a performer performance information determination unit, a user viewing position determination unit, and a display unit. The performer performance information determination unit receives performer action data, determines execution of a performer performance from the performer action data, and determines performer performance information. The user viewing position determination unit receives performer performance information from the performer performance information determination unit, and determines, from the performer performance information, a viewing position at which a user views a performer performance. The display unit receives the performer action data, receives the viewing position information from the user viewing position determination unit, and displays a video from the viewing position on the basis of the performer action data and the viewing position information.

Description

Information processing device

[0001] The present invention relates to an information processing device.

[0002] At live music events, there are often large audiences and the venues are often large, making it difficult to enjoy the performers' performances (such as the performers' calls to the audience and the performers' reactions to the audience's cheers).

[0003] As a solution to the above situation, a technology has been proposed that conveys the performer's behavior to the audience through the light emitted by a penlight (Non-Patent Document 1), and a system that allows viewing of free viewpoint video in a VR space (Non-Patent Document 2).

[0004] Koyo Otsu, Fumiyasu Fukushima, Hidekazu Takahashi, Minoru Hirahara, Yuto Fukuda, Takanori Kobayashi, Yoshinori Kuno, Keiichi Yamazaki, "Affinity Live: An interactive live performance support system that enhances the sense of unity between performers and audiences," Transactions of the Information Processing Society of Japan, Vol. 59, No. 11, pp. 2019-2029, 2018; Manabu Tsukada, Yutaka Komohara, Hideaki Arai, Takashi Kasuya, Shigeki Kosaka, Keiko Ogawa, Hiroshi Ezaki, "Interactive playback of video and audio from any viewing point for music events," Proceedings of the 2017 Multimedia, Distributed, Cooperative and Mobile Symposium, pp. 1460-1467, 2017

[0005] The above method has the problem that it is difficult for the audience or user to enjoy the performer's performance that is aimed just at them.

[0006] The present invention has been made in light of the above circumstances, and its object is to provide an information processing device that enables a user to enjoy a performer's performance that is just for that user.

[0007] One aspect of the present invention is an information processing device. The information processing device has a performer performance information determination unit, a user viewing position determination unit, and a display unit. The performer performance information determination unit receives performer action data, determines the execution of a performer performance from the performer action data, and determines performer performance information. The user viewing position determination unit receives the performer performance information from the performer performance information determination unit, and determines a viewing position from which a user will view the performer performance from the performer performance information. The display unit receives the performer action data and viewing position information from the user viewing position determination unit, and displays video from the viewing position based on the performer action data and the viewing position information.

[0008] According to the present invention, an information processing device is provided that enables a user to enjoy a performer's performance that is just for him or her.

[0009] Fig. 1 is a block diagram showing the functional configuration of an information processing apparatus according to an embodiment, Fig. 2 is a block diagram showing the hardware configuration of the information processing apparatus according to an embodiment, and Fig. 3 is a flowchart showing the flow of processing executed by the information processing apparatus according to an embodiment.

[0010] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0011] (Functional Configuration) First, the functional configuration of an information processing device 10 according to an embodiment will be described with reference to Fig. 1. Fig. 1 is a block diagram showing the functional configuration of the information processing device 10 according to an embodiment. The information processing device 10 according to an embodiment is a device that enables a user to enjoy a performance from a performer that is directed just at the user at a music event or the like.

[0012] The information processing device 10 includes a performer performance information determination unit 20 , a user viewing position determination unit 30 , a display unit 40 , a performer action data storage unit 50 , and an audience reaction data storage unit 60 .

[0013] The performer action data storage unit 50 stores performer action data. The performer action data includes performer video, performer voice, motion information, etc. Preferably, the performer action data further includes line of sight information.

[0014] The audience response data storage unit 60 stores audience response data, which includes audience movement information, audience video, audience voice, vital data (heart rate, sweating), emotional information, and the like.

[0015] The performer performance information determination unit 20 receives performer action data from the performer action data storage unit 50, determines the execution of the performer performance based on the performer action data, and determines performer performance information. The performer performance information includes information on the direction of the performer's performance and information on the impact of the performer's performance. The performer performance includes calls from the performer to the audience and the performer's reactions to the audience's cheers. Without being limited to these, the performer performance may include any action performed by the performer, such as waving, winking, or making a heart pose.

[0016] Preferably, the performer performance information determination unit 20 receives audience reaction data from the audience reaction data storage unit 60 in addition to the performer action data, and determines the performer performance information by taking the audience reaction data into account in addition to the performer action data.

[0017] The user viewing position determining unit 30 receives the performer performance information from the performer performance information determining unit 20, and determines the viewing position from which the user will view the performer performance based on the performer performance information.

[0018] Preferably, the user viewing position determination unit 30 receives audience response data from the audience response data storage unit 60 in addition to the performer performance information, and determines the user's viewing position by taking the audience response data into account in addition to the performer performance information.

[0019] The display unit 40 displays video that the user can view. For example, in addition to displaying video, the display unit 40 also plays audio that the user can view. The display unit 40 receives performer action data from the performer action data storage unit 50 and receives viewing position information from the user viewing position determination unit 30, and displays video from the viewing position based on the performer action data and the viewing position information. For example, the display unit 40 displays video capturing the performer from the viewing position.

[0020] The information processing device 10 does not necessarily have to include an audience response data storage unit 60. That is, the performer performance information determination unit 20 may determine performer performance information from performer action data alone, and the user viewing position determination unit 30 may determine the user's viewing position from performer performance information alone. However, using audience response data as well is preferable because it improves the accuracy with which the performer performance information determination unit 20 and the user viewing position determination unit 30 determine the direction and influence of the performer's performance and the user's viewing position.

[0021] Furthermore, the performer action data does not need to be stored in advance in the performer action data storage unit 50. The performer action data may be acquired in real time. Similarly, the audience reaction data does not need to be stored in advance in the audience reaction data storage unit 60. The audience reaction data may be acquired in real time.

[0022] That is, the performer performance information determination unit 20 may acquire performer action data in real time and also acquire audience reaction data in real time to determine performer performance information. The user viewing position determination unit 30 may acquire audience reaction data in real time to determine the user viewing position. The display unit 40 may acquire performer action data in real time to generate a display image.

[0023] (Hardware Configuration) Next, a hardware configuration of the information processing device 10 will be described. Here, an example will be described in which the information processing device 10 is configured as a computer. For example, the information processing device 10 may be configured as a personal computer, a server computer, or the like.

[0024] 2 is a block diagram showing an example of the hardware configuration of the information processing device 10 according to the embodiment, in other words, the hardware configuration of the computer 100 that constitutes the information processing device 10. The information processing device 10 (computer 100) has a control device 120, an input device 140, and an output device 150.

[0025] The control device 120 controls the entire information processing device 10. The control device 120 includes a processor 121, a read only memory (ROM) 122, a random access memory (RAM) 123, and an auxiliary storage device .

[0026] The processor 121, ROM 122, RAM 123, auxiliary storage device 124, input device 140, and output device 150 are electrically connected to one another via a bus 130, and can transmit and receive data.

[0027] The processor 121 is configured by a general-purpose hardware processor including, for example, a CPU (Central Processing Unit), a GPU (Graphical Processing Unit), etc. The processor 121 executes programs deployed in the RAM 123 to perform various functions of the information processing device 10.

[0028] The ROM 122 is a non-volatile memory that constitutes part of the main storage device. The ROM 122 non-temporarily stores a startup program required to start up the information processing device 10. The processor 121 loads the startup program in the ROM 122 into the RAM 123 and executes it to start up the information processing device 10. The ROM 122 is configured, for example, by an EPROM (Erasable Programmable Read Only Memory), and can store various startup settings in addition to the startup program.

[0029] The RAM 123 is a volatile memory that constitutes part of the main storage device. The RAM 123 temporarily stores programs required for processing by the processor 121 and data required for executing the programs. In other words, the RAM 123 functions as a work area for the processor 121.

[0030] The auxiliary storage device 124 is configured with non-volatile memory such as a hard disk drive (HDD) or a solid state drive (SSD). The auxiliary storage device 124 can non-temporarily store various programs executed by the processor 121 and data required for executing the programs. The processor 121 executes various functions of the information processing device 10 by expanding the programs in the auxiliary storage device 124 into the RAM 123 and executing them.

[0031] The input device 140 is a device for a user to input information and instructions, and accepts input of information and instructions. The input device 140 includes a keyboard, a pointing device, etc. The pointing device includes a mouse, a trackpad, a touch screen, etc.

[0032] The output device 150 is a device that outputs information to provide information to a user. The output device 150 is, for example, a display device that displays characters, images, etc. on a screen. For example, the output device 150 is a liquid crystal display, an organic EL display, a plasma display, etc.

[0033] The output device 150 and the input device 140 may be configured as an input / output device having the functions of both. Such an input / output device may be configured as, for example, a touch panel.

[0034] The input device 140 may also include a device that inputs information or data from the outside. For example, the input device 140 may include a wired or wireless interface or receiving device.

[0035] The output device 150 may also include a device that outputs information or data to the outside. For example, the output device 150 may include a wired or wireless interface or transmission device.

[0036] The input device 140 may also include a device that reads data from a computer-readable recording medium 160 that non-temporarily records data such as a program. For example, the recording medium 160 includes disks such as flexible disks, optical disks (CD-ROM, CD-R, DVD-ROM, DVD-R, etc.), magneto-optical disks (MO, etc.), semiconductor memories, etc. The input device 140 includes drives, readers, etc. for these.

[0037] The program stored in the auxiliary storage device 124 is provided to the information processing device 10, for example, via the recording medium 160. Alternatively, the program may be stored in a server on a network and provided to the information processing device 10 by downloading it.

[0038] For example, when the information processing device 10 is started, the processor 121 executes a startup program in the ROM 122 to start up the operating system (OS). Under the control of the OS, the processor 121 monitors input instructions, connections to external devices, etc. Also, under the control of the OS, the processor 121 sets a program area and a data area in the RAM 123.

[0039] In response to an instruction to start a program, processor 121 reads the program from auxiliary storage device 124 into the program area of ​​RAM 123, and also reads data necessary for executing the program from auxiliary storage device 124 into the data area of ​​RAM 123. Processor 121 calculates the data in the data area in accordance with the program and writes the calculation results into the data area.

[0040] Through these operations, the processor 121, RAM 123, and auxiliary storage device 124 work together to execute at least some of the functions of the control device 120. Furthermore, the control device 120, input device 140, and output device 150 work together to execute at least some of the functions of the information processing device 10.

[0041] The programs non-temporarily stored in the auxiliary storage device 124 include an information processing program that causes the processor 121 to execute at least some of the functions of the control device 120. In other words, the processor 121 executes this information processing program to execute at least some of the functions of the control device 120.

[0042] As a result, the control device 120, in cooperation with the input device 140 and the output device 150, executes at least some of the functions of the performer performance information determination unit 20, the user viewing position determination unit 30, the display unit 40, the performer action data storage unit 50, and the audience reaction data storage unit 60. The control device 120 also executes the function of acquiring performer action data and audience reaction data in cooperation with the input device 140.

[0043] (Operation Example) Next, an operation example of the information processing device 10 according to the embodiment will be described with reference to Fig. 3. Fig. 3 is a flowchart showing the flow of processing executed by the information processing device 10 according to the embodiment.

[0044] The flowchart in Figure 3 shows an example of the operation of the information processing device 10 while an event is taking place. For example, the event is performed by playing back performer action data stored in the performer action data storage unit 50. In this case, the performer action data is data acquired in advance. Alternatively, the event is performed in real time outside the information processing device 10. In other words, the event is a live event. In this case, the performer action data is data acquired in real time.

[0045] While the event is taking place, the display unit 40 displays video of the event. For example, the display unit 40 displays video of the event by playing back performer action data stored in the performer action data storage unit 50. Alternatively, the display unit 40 displays video of the event by acquiring performer action data in real time from outside the information processing device 10.

[0046] While the display unit 40 is displaying the video of the event, in step S11 the performer performance information determination unit 20 acquires performer action data and determines from the performer action data whether a performer performance has been performed.

[0047] When the performer action data is pre-acquired data, for example, data recording the timing at which the performer performance is performed is created in advance. Based on the data, the performer performance information determination unit 20 determines whether the performer performance is being performed in the current display scene on the display unit 40.

[0048] Alternatively, whether the performer action data is data acquired in advance or data acquired in real time, the performer performance information determination unit 20 determines that a performer performance has been performed in the currently displayed scene if the choreography differs from a predetermined choreography, for example, using image processing technology utilizing AI technology.

[0049] In the determination of step S11, if the determination result is that the performer performance has not been performed (NO in step S11), the performer performance information determination unit 20 repeatedly determines whether the performer performance has been performed. In other words, the performer performance information determination unit 20 continues to determine whether the performer performance has been performed until the performance of the performer performance is detected.

[0050] If the determination result in step S11 is that a performer performance has been performed (YES in step S11), the performer performance information determination unit 20 determines the direction of the performer performance in step S12. The direction of the performer performance is, for example, the direction in which the performer's face is facing.

[0051] For example, information on the direction of the performer's face is stored in advance as information on the performer's movement, and the performer performance information determination unit 20 determines the direction of the performer's performance based on this information on the direction of the performer's face.

[0052] Alternatively, the performer performance information determination unit 20 estimates the direction of the performer's face in a video image showing the performer based on the performer action data, and determines the estimated direction as the direction of the performer's performance. Possible techniques for estimating the direction of the performer's face include a method using facial image processing technology and a method using deep learning.

[0053] An example of a method using face image processing technology is disclosed in "https: / / www.tytlabs.co.jp / assets / pdf / results / 2006_1.pdf."

[0054] An example of a method using deep learning is the method disclosed in "https: / / ipsj.ixsq.nii.ac.jp / ej / ?action=repository_uri&item_id=213707&file_id=1&file_no=1https: / / ipsj.ixsq.nii.ac.jp / ej / ?actio n=r repository_uri&item_id=213707&file_id=1&file_no=1".

[0055] Next, in step S13, the performer performance information determination unit 20 determines the degree of influence of the performer performance. The degree of influence of the performer performance is determined for each type of performance.

[0056] For example, the influence of a performer's performance is quantified in stages according to the strength of the influence for each type of performance, such as 1 point for waving one's hands, 5 points for winking or making a heart pose, and so on.

[0057] For example, the score may be predetermined, or the score may be dynamically changed depending on the number of times the event is performed or the reactions of the audience or users during the event.

[0058] As mentioned above, the direction of a performer's performance and the impact of that performer's performance can be determined based on the performer action data alone, or by adding audience reaction data to the performer action data. By adding audience reaction data to the performer action data, the accuracy of determining the direction of a performer's performance and the impact of that performer's performance can be improved, so it is preferable to use audience reaction data in addition to the performer action data.

[0059] Next, in step S14, the user viewing position determination unit 30 determines the viewing position from which the user will view the performer's performance. The user viewing position is determined taking into consideration the direction of the performer's performance and the degree of influence of the performer's performance.

[0060] The user's viewing position changes in response to changes in the display scene on the display unit 40. For example, the user's viewing position changes in response to changes in the performer's performance. The user viewing position determination unit 30 moves the user's viewing position in response to changes in the user's viewing position.

[0061] That is, if it is determined in step S11 that a performer performance has been performed, the user viewing position determination unit 30 moves the user's viewing position from the position where the performer performance has been viewed up until now to the position where the user will view the next performer performance.

[0062] The user's viewing position may be moved instantly, like teleportation, or over a period of time, like moving a viewpoint. The speed of the movement may be constant or may change along the way.

[0063] In one example, the user's viewing position is instantly moved to a position in the direction of the performer's performance, like teleportation, regardless of the degree of influence. In this case, the user's viewing position becomes a position in the direction of the performer's performance.

[0064] In another example, for a performer performance with an influence score of 5 or more, the user's viewing position is moved over a 2-second period from the previous position to a position that is closer to the performer's performance, and the user's viewing position moves along a trajectory from the previous position to a position that is closer to the performer's performance.

[0065] In another example, only the user's favorite performer actions are taken into account, and only when that performer's performance is performed, the user's viewing position is moved over two seconds from the previous position to a position that is closer to the performer's performance.

[0066] Next, in step S15, the display unit 40 updates the display image based on the performer action data and the viewing position information. The display unit 40 displays an image from the user's viewing position. For example, the display unit 40 displays an image capturing the performer from the viewing position.

[0067] For this purpose, a VR space is constructed based on the performer action data, and a virtual camera is set within this VR space. The virtual camera is placed at the user's viewing position, and its viewing direction is set toward the performer. The display unit 40 displays the image captured by the virtual camera set in this way.

[0068] While the user's viewing position is moving, the display unit 40 displays an image capturing the performer as he or she moves, so that the user watching the display unit 40 feels as if they are watching the performer as they move.

[0069] Furthermore, after the user's viewing position reaches the end point of the movement, the display unit 40 displays the image captured by a virtual camera positioned above the performer's performance and with its viewing direction set directly opposite the direction of the performer's performance. As a result, the user watching the display unit 40 will see an image capturing the performer's performance from directly in front (directly opposite the direction of the performer's performance). In other words, the user will see an image that makes it seem as if the performer is performing towards them.

[0070] In this way, the user of the information processing device 10 according to the embodiment can enjoy the performer's performance that is just for him or her.

[0071] (Example) The following describes an example of the information processing device 10. This example is an example of viewing an archive in a virtual venue.

[0072] The performer action data storage unit 50 stores not only the performer action data but also timing data that records the timing at which the performer performance is performed. The timing data is manually created in advance based on the performer action data.

[0073] The performer action data storage unit 50 stores, as one type of performer action data, face direction data that records information about the direction of the performer's face. The face direction data is obtained and recorded as information about the performer's movement during recording, for example.

[0074] The display unit 40 reads the actor action data from the actor action data storage unit 50 and displays the image captured by the virtual camera. The initial position of the virtual camera is set in advance, for example.

[0075] The performer performance information determination unit 20 determines whether or not a performer performance is being performed in the currently reproduced scene based on the performer action data and timing data.

[0076] When it is determined that a performer performance is being performed, a performer performance information determination section 20 determines the direction in which the performer's face is facing forward as the direction of the performer performance based on face direction data as one of the performer performance information.

[0077] The user viewing position determination unit 30 moves the viewpoint of the user's viewing position, i.e., the position of the virtual camera, from the current position to a position in the direction of the performer's performance at a constant speed over a predetermined time based on the direction of the performer's performance determined by the performer performance information determination unit 20.

[0078] Here, the current position is the final viewing position for the previous performer's performance, and the starting position for the current performer's performance. The forward position for the performer's performance is the final viewing position for the current performer's performance.

[0079] For example, suppose performer performance 1 is performed at time t1, and the next performer performance 2 is performed at time t2. The final viewing position for performer performance 1 is x1, and the final viewing position for performer performance 2 is x2. For example, points x1 and x2 may be position vectors.

[0080] In this case, the user viewing position determination unit 30 moves the user's viewing position, that is, the position of the virtual camera, from point x1 to point x2 at a constant speed (x2-x1) / (t2-t1).

[0081] As described above, the display unit 40 displays an image captured by a virtual camera. Here, the display unit 40 displays an image captured by a virtual camera moving at a constant speed.

[0082] Therefore, the user watches a video that gives the impression that he or she is moving at a constant speed from point x1 to point x2. Furthermore, when the user arrives at point x2, the user watches a video that makes the user feel as if the actor is reacting to the user.

[0083] (Effects) As described above, the information processing device 10 of the embodiment can display video that captures a performer's performance from directly in front. This allows the user of the information processing device 10 to view video that makes it seem as if the performer is performing for them. In other words, the information processing device 10 allows the user to enjoy a performer's performance that is directed only at them.

[0084] The embodiments of the present invention have been described above with reference to the drawings. However, the above embodiment is merely an example of a configuration that embodies the present invention. That is, it is clear that the present invention is not limited to the above embodiment. Therefore, addition, omission, substitution, and other modifications of components may be made within the scope of the technical concept of the present invention.

[0085] In short, the present invention is not limited to the above-described embodiments, and various modifications can be made in the implementation stage without departing from the spirit of the invention. Furthermore, the embodiments may be implemented in appropriate combinations, in which case the combined effects can be obtained. Furthermore, the above-described embodiments include various inventions, and various inventions can be extracted by combining selected elements from the disclosed elements. For example, if the problem can be solved and the desired effect can be obtained even if some elements are deleted from all elements shown in the embodiments, the configuration from which these elements are deleted can be extracted as an invention.

[0086] 10... Information processing device 20... Performer performance information determination unit 30... User viewing position determination unit 40... Display unit 50... Performer action data storage unit 60... Audience reaction data storage unit 100... Computer 120... Control device 121... Processor 122... ROM 123... RAM 124... Auxiliary storage device 130... Bus 140... Input device 150... Output device 160... Recording medium

Claims

1. An information processing device comprising: a performer performance information determination unit that receives performer action data, determines the execution of a performer performance from the performer action data, and determines performer performance information; a user viewing position determination unit that receives the performer performance information from the performer performance information determination unit and determines the viewing position from which a user will view the performer performance from the performer performance information; and a display unit that receives the performer action data, receives information about the viewing position from the user viewing position determination unit, and displays video from the viewing position based on the performer action data and the information about the viewing position.

2. The information processing device according to claim 1, wherein the performer performance information determination unit further receives audience reaction data and determines the performer performance information by taking the audience reaction data into account in addition to the performer action data.

3. The information processing device according to claim 1, wherein the performer performance information determination unit determines the direction of the performer performance and the degree of influence of the performer performance as the performer performance information.

4. The information processing device according to claim 1, wherein the performer performance information determination unit determines the direction in which the performer's face is facing as the direction of the performer's performance.

5. The information processing device according to claim 1, wherein the performer performance information determination unit quantifies the degree of influence in stages for each type of performer performance.

6. The information processing device according to claim 3, wherein the user viewing position determination unit determines the viewing position taking into consideration the direction of the performer's performance and the degree of influence of the performer's performance.

7. The information processing device according to claim 6, wherein the display unit displays an image captured by a virtual camera placed at the viewing position.

8. The information processing device according to claim 7, wherein the display unit is positioned above the performer's performance and displays an image captured by the virtual camera whose viewing direction is set directly opposite to the direction of the performer's performance.

Images