Image processing method and image processing device

By adjusting the viewpoint position according to audio data in a 3-dimensional space, the problem that the existing technology cannot automatically adjust the viewpoint is solved, meaningful viewpoint changes for live broadcast events are achieved, and user experience is improved.

CN118922866APending Publication Date: 2024-11-08YAMAHA CORP
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202380027140.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2022-03-24
Filing Date
2023-03-10
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

The existing information processing device cannot automatically adjust the viewpoint position according to the sound changes of the live broadcast event, resulting in the inability to realize the viewpoint position change that is meaningful to the live broadcast event.

Method used

By configuring objects in a 3-dimensional space, compute relative positions based on audio data related to live events, and automatically adjust the viewpoint position to output images that adapt to sound changes.

Benefits of technology

The viewpoint position changes that are meaningful for live broadcast events are achieved, which enhances visual and auditory experiences and brings new viewing experiences to users.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118922866A_ABST
    Figure CN118922866A_ABST
Patent Text Reader

Abstract

An image processing method arranges an object in a three-dimensional space, and outputs an image in which a live event is viewed from a viewpoint position set in the three-dimensional space. The video processing method sets a first viewpoint position on the basis of a first factor which is a factor related to the object, outputs a video in which the live event is viewed from the first viewpoint position, receives a second factor which is a factor related to the sound of the live event, calculates a relative position with respect to the object on the basis of the second factor, and outputs the video in which the live event is viewed from the first viewpoint position. And an output unit that changes the first viewpoint position to the second viewpoint position on the basis of the calculated relative position, and outputs an image in which the live event is viewed from the second viewpoint position.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] One embodiment of the present invention relates to an image processing method and an image processing device. Background Art

[0002] Patent document 1 discloses an information processing device for displaying a virtual viewpoint image, wherein the viewpoint change is performed by including, as a viewpoint change requirement, a relationship condition that specifies the positional relationship of a focus object, and two or more information of an object existing in a viewpoint dimensional space that specifies the specified viewpoint associated with the relationship condition. When the relationship condition is satisfied and the viewpoint change requirement is met, the viewpoint is changed.

[0003] Patent Document 1: Japanese Patent Application Publication No. 2020-190978 Summary of the invention

[0004] The information processing device of Patent Document 1 is not a device that changes the viewpoint position based on a factor of a change in sound such as a change in the venue of a live event.

[0005] An object of one embodiment of the present invention is to provide a video processing method for automatically changing a viewpoint position based on a factor of a change in sound of a live event, thereby realizing a change in the viewpoint position meaningful to the live event.

[0006] An image processing method according to one embodiment of the present invention is to arrange an object in a three-dimensional space and output an image of a live event viewed from a viewpoint position set in the three-dimensional space. The image processing method is to set a first viewpoint position based on a first factor related to the object, output an image of the live event viewed from the first viewpoint position, receive a second factor related to the sound of the live event, calculate a relative position relative to the object based on the second factor, change the first viewpoint position to the second viewpoint position based on the calculated relative position, and output an image of the live event viewed from the second viewpoint position.

[0007] Effects of the Invention

[0008] According to an embodiment of the present invention, a change in the viewpoint position that is meaningful to a live event can be achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 It is a block diagram showing the structure of the image processing device 1.

[0010] Figure 2 It is a diagram showing the three-dimensional space R1.

[0011] Figure 3This is a diagram showing an example of a video of a live event viewed from a viewpoint position set in the three-dimensional space R1.

[0012] Figure 4 It is a flowchart showing the operation of the processor 12 .

[0013] Figure 5 It is a perspective view showing an example of a three-dimensional space R2 of different live broadcast venues.

[0014] Figure 6 This is a diagram showing an example of a video of a live event viewed from a viewpoint position set in the three-dimensional space R2.

[0015] Figure 7 This is a diagram that explains the concept of coordinate transformation.

[0016] Figure 8 This is a diagram that explains the concept of coordinate transformation.

[0017] Fig. 9 This is a diagram for explaining a modified example of coordinate transformation.

[0018] Fig.10 This is a flowchart showing the operation of the processor 12 when performing the audio processing with the first viewpoint position 50A and the second viewpoint position 50B as listening points.

[0019] Fig.11 This is a perspective view showing an example of a three-dimensional space R1 according to Modification Example 1.

[0020] Fig.12 This is a diagram showing an example of a video of a live event viewed from a second viewpoint position 50B according to Modification 1.

[0021] Fig.13 This is a perspective view showing an example of a three-dimensional space R1 according to modification example 4.

[0022] Fig.14 This is a perspective view showing an example of a three-dimensional space R1 according to modification example 4. DETAILED DESCRIPTION

[0023] Figure 1 1 is a block diagram showing the structure of the image processing device 1. The image processing device 1 includes a communication unit 11, a processor 12, a RAM 13, a flash memory 14, a display 15, a user I / F 16, and an audio I / F 17.

[0024] The image processing device 1 is constituted by a personal computer, a smartphone, a tablet computer, or the like.

[0025] The communication unit 11 communicates with other devices such as a server. The communication unit 11 has, for example, a wireless communication function such as Bluetooth (registered trademark) or Wi-Fi (registered trademark), or a wired communication function such as USB or LAN.

[0026] The display 15 is composed of an LCD, an OLED, etc. The display 15 displays the image output by the processor 12 .

[0027] The user I / F 16 is an example of an operation unit. The user I / F 16 is composed of a mouse, a keyboard, a touch panel, etc. The user I / F 16 receives an operation by a user. In addition, the touch panel may be stacked on the display 15.

[0028] The audio I / F 17 has a virtual audio terminal or a digital audio terminal, etc., and is connected to an audio device. In the present embodiment, the audio I / F 17 is connected to a headphone 20 as an example of an audio device, and outputs a sound signal to the headphone 20.

[0029] The processor 12 is composed of a CPU, a DSP, or a SoC (System on a Chip), etc., and is equivalent to the processing unit of the present invention. The processor 12 reads the program from the flash memory 14 as a storage medium and temporarily stores it in the RAM 13, thereby performing various operations. In addition, the program does not need to be stored in the flash memory 14. The processor 12 can, for example, download it from other devices such as a server when necessary and temporarily store it in the RAM 13.

[0030] The processor 12 obtains data related to the live event via the communication unit 11. The data related to the live event includes space information, model data of an object, position information of an object, motion data of an object, and sound data of the live event.

[0031] Spatial information is information representing the shape of a three-dimensional space corresponding to a live broadcast venue such as a studio or concert hall, and is represented by three-dimensional coordinates with a certain position as the origin. Spatial information can be coordinate information based on 3D CAD data of an actual live broadcast venue such as a concert hall, or it can be logical coordinate information of a fictitious live broadcast venue (information standardized by 0 to 1).

[0032] The model data of the object is three-dimensional CG image data, and is composed of a plurality of image components.

[0033] The position information of an object is information indicating the position of an object in a three-dimensional space. The position information of an object in this embodiment is represented by three-dimensional coordinates of a time series corresponding to the elapsed time from the start of a live event. As an object, there are objects such as devices such as speakers whose positions do not change from the start of a live event to the end, and there are also objects such as actors whose positions change along the time series.

[0034] The motion data of the object is data indicating the arrangement relationship of the plurality of image components used to move the model data. The motion data of this embodiment is time series data corresponding to the elapsed time from the start of the live event. The motion data is set for an object whose position changes in a time series, such as an actor.

[0035] The audio data of the live event is, for example, an audio signal of a stereo (L, R) channel. The processor 12 plays the audio data of the live event and outputs it to the headphones 20 via the audio I / F 17. The processor 12 can perform effect processing such as reverberation processing on the audio signal. Reverberation processing is a process of simulating the sound (reflected sound) of a certain room by assigning a specified delay time to the received audio signal and generating a virtual reflected sound with an adjusted level. The processor 12 generates effect processing suitable for the live venue. For example, in the case where the live venue represented by the spatial information is a small venue, the processor 12 generates a virtual reflected sound with a short delay time and reverberation time and a low level. On the other hand, in the case where the live venue represented by the spatial information is a large venue, the processor 12 generates a virtual reflected sound with a long delay time and reverberation time and a high level.

[0036] In addition, the audio data of the live event may include an audio signal associated with each object that emits the sound. For example, the speaker objects 53 and 54 are objects that emit the sound. The processor 12 may perform a localization process such that the sound of the speaker objects 53 and 54 is localized at the positions of the speaker objects 53 and 54, and generate an audio signal of a stereo (L, R) channel for output to the headphones 20. The details of the localization process will be described later.

[0037] The processor 12 arranges the object in the following manner based on the position information of the object in the data related to the live event: Figure 2 The processor 12 sets a viewpoint position in the 3D space R1. The processor 12 draws a 3DCG image of each object based on the spatial information, the model data of the object, the position information of the object, the motion data of the object, and the information of the set viewpoint position to generate an image of the live event viewed from the set viewpoint position (refer to Figure 3 ).

[0038] Figure 2 It is a perspective view showing an example of the three-dimensional space R1. Figure 3 This is a diagram showing an example of a video of a live event viewed from a viewpoint position set in the three-dimensional space R1. Figure 4 It is a flowchart showing the operation of the processor 12 . Figure 4 The actions are executed by the application program read from the flash memory 14 by the processor 12 .

[0039] Figure 2 The three-dimensional space R1 is a rectangular parallelepiped space as an example. The processor 12 arranges an actor object 51, an actor object 52, a speaker object 53, a speaker object 54, and a stage object 55 in the three-dimensional space R1. The actor object 51 is an object of a guitar player. The actor object 52 is an object of a singer.

[0040] The processor 12 arranges the stage object 55 at the center of the left-right direction (X direction), the end in the front-back direction (Y direction), and the lowest position in the height direction (Z direction) of the three-dimensional space R1. In addition, the processor 12 arranges the speaker object 54 at the leftmost position in the left-right direction (X direction), the end in the front-back direction (Y direction), and the lowest position in the height direction (Z direction) of the three-dimensional space R1. In addition, the processor 12 arranges the speaker object 53 at the rightmost position in the left-right direction (X direction), the end in the front-back direction (Y direction), and the lowest position in the height direction (Z direction) of the three-dimensional space R1.

[0041] In addition, the processor 12 arranges the actor object 51 on the left side in the left-right direction (X direction), the last position in the front-back direction (Y direction), and the height direction (Z direction) of the three-dimensional space R1 on the stage object 55. In addition, the processor 12 arranges the actor object 52 on the right side in the left-right direction (X direction), the last position in the front-back direction (Y direction), and the height direction (Z direction) of the three-dimensional space R1 on the stage object 55.

[0042] Then, the processor 12 receives a first factor (S11) which is a factor related to the object, and sets the first viewpoint position 50A based on the first factor (S12). For example, the first factor is a factor related to the stage object 55. The processor 12 receives the designation of "a position in front of the stage object 55 at a predetermined distance" as the first factor in the initial processing of starting the application.

[0043] Alternatively, the user can designate a specific object via the user I / F 16. For example, the processor 12 displays on the display 15 Figure 2 The schematic diagram of the three-dimensional space R1 is shown, and various objects are displayed. The user specifies an arbitrary object via, for example, a touch panel. For example, if the user specifies the stage object 55, the processor 12 sets the first viewpoint position 50A to a position in front of the stage object 55 at a predetermined distance.

[0044] The first viewpoint position 50A set as described above corresponds to the position of the user viewing the content in the three-dimensional space R1. The processor 12 outputs Figure 3The video of the live event is viewed from the first viewpoint position 50A as shown in FIG. 5 (S13). The video viewed from the first viewpoint position 50A set as "a position in front of the stage object 55 at a predetermined distance" is as shown in FIG. Figure 3 As shown, the video will include all actors who filmed the live event.

[0045] Furthermore, the image can be output to the display 15 of the own device for display, and can be output to other devices via the communication unit 11 .

[0046] Next, the processor 12 determines whether the second factor related to the sound of the live event is received (S14). For example, the user performs an operation to change the current live venue to a different live venue via the user I / F 16.

[0047] The processor 12 repeatedly performs the process of outputting the image based on the first viewpoint position until it is determined in S14 that the second factor has been received (S14: NO→S13). As described above, the position information and motion data of the object are time series data corresponding to the elapsed time from the start of the live event. Therefore, the position and movement of the actor object 51 and the actor object 52 change according to the progress of the live event.

[0048] The second factor related to the sound of the live broadcast event is, for example, the change of the live broadcast venue. As described above, the processor 12 generates a virtual reflected sound corresponding to the size of the live broadcast venue represented by the spatial information. Therefore, if the live broadcast venue is changed, the sound of the live broadcast event also changes. Therefore, the change of the live broadcast venue is a factor related to the sound of the live broadcast event.

[0049] Figure 5 This is a perspective view showing an example of a three-dimensional space R2 of a live broadcast venue different from the three-dimensional space R1. Figure 6 This is a diagram showing an example of a video of a live event viewed from a viewpoint position set in the three-dimensional space R2.

[0050] Figure 5 The three-dimensional space R2 is an octagonal prism-shaped space as an example. The three-dimensional space R2 is a space larger than the three-dimensional space R1. The user changes the live broadcast venue from the three-dimensional space R1 to the three-dimensional space R2 via the user I / F 16.

[0051] When the processor 12 receives an operation to change from the three-dimensional space R1 to the three-dimensional space R2 in the judgment of S14 (S14: YES), it calculates the relative position relative to a certain object (the stage object 55 in the above example), and changes the first viewpoint position 50A to the second viewpoint position 50B based on the calculated relative position (S15).

[0052] Specifically, first, the processor 12 obtains the coordinates of the actor object 51 , the actor object 52 , the speaker object 53 , the speaker object 54 , and the stage object 55 in the three-dimensional space R2 .

[0053] Figure 7 and Figure 8 This is a diagram that explains the concept of coordinate transformation. Figure 7 and Figure 8 In FIG. 5 , the coordinates of the loudspeaker object 53 are transformed from the three-dimensional space R1 to the three-dimensional space R2 as an example. However, for ease of explanation, Figure 7 and Figure 8 , the three-dimensional space R1 and the three-dimensional space R2 are represented by a two-dimensional space when viewed from above.

[0054] The processor 12 transforms the coordinates of the speaker object 53 from the first coordinates of the three-dimensional space R1 to the second coordinates of the three-dimensional space R2. Figure 7 In the example, there are 8 reference points before transformation 70A (x1, y1), 70B (x2, y2), 70C (x3, y3), 70D (x4, y4), 70E (x5, y5), 70F (x6, y6), 70G (x7, y7), 70H (x8, y8), and 8 reference points after transformation 70A (x'1, y'1), 70B (x'2, y'2), 70C (x'3, y'3), 70D (x'4, y'4), 70E (x'5, y'5), 70F (x'6, y'6), 70G (x'7, y'7), 70H (x'8, y'8).

[0055] The processor 12 obtains the centroids G of the eight reference points before the transformation and the centroids G′ of the eight reference points after the transformation, and generates a triangular mesh with these centroids as the center.

[0056] The processor 12 transforms the internal space of the triangle before the transformation and the internal space of the triangle after the transformation through a predetermined coordinate transformation. The transformation uses, for example, an affine transformation. An affine transformation is an example of a geometric transformation. The affine transformation represents the transformed x-coordinate (x') and y-coordinate (y') as functions of the x-coordinate (x) and y-coordinate (y) before the transformation, respectively. That is, the affine transformation performs a coordinate transformation through the formulas x'=ax+by+c and y'=dx+ey+f. The coefficients a~f can be uniquely obtained through the coordinates of the three vertices of the triangle before the transformation and the coordinates of the three vertices of the triangle after the transformation. The processor 12 transforms from the first coordinate to the second coordinate by obtaining the affine transformation coefficients for all triangles in the same way. In addition, the coefficients a~f can be obtained by the least squares method.

[0057] In addition, the above-mentioned mesh may be a mesh of polygons other than triangles, or a combination thereof. For example, the processor 12 may Fig. 9 The quadrilateral mesh is generated as shown and the coordinate transformation is performed. The transformation method is not limited to the above-mentioned affine transformation. For example, the processor 12 can transform the quadrilateral mesh and perform coordinate transformation based on the formula shown below (where x0, y0, x1, y1, x2, y2, x3, y3 are the coordinates of the transformation points respectively).

[0058] x'=x0+(x1-x0)x+(x3-x0)y+(x0-x1+x2-x3)xyy'=y0+(y1-y0)x+(y3-y0)y+(y0-y1+y2-y3)xy In addition to the above-mentioned transformation methods, other geometric transformations such as isometric mapping, similarity transformation, or projective transformation can be used. For example, the projective transformation is represented by the formulas x'=(ax+by+c) / (gx+hy+1) and y'=(dx+ey+f) / (gx+hy+1). The coefficients are obtained in the same way as the above-mentioned affine transformation. For example, the 8 coefficients (a~h) of the projective transformation constituting a quadrilateral can be uniquely obtained by 8 simultaneous equations. Alternatively, the coefficients can be obtained, for example, by the least squares method.

[0059] In the above example, the coordinate transformation is performed in a two-dimensional space (X, Y), but the coordinate transformation in a three-dimensional space (X, Y, Z) can also be performed in the same way.

[0060] Thus, each object is transformed into coordinates matching the shape of the three-dimensional space R2.

[0061] In this example, the processor 12 also transforms the coordinates of the first viewpoint position 50A based on the above-mentioned transformation method. Thus, the processor 12 obtains the second viewpoint position 50B which is the relative position with respect to the stage object 55 in the three-dimensional space R2.

[0062] Furthermore, the processor 12 outputs Figure 5 As shown in FIG. 1 , the video of the live event is viewed from the second viewpoint position 50B ( S16 ). In this example, the video viewed from the second viewpoint position 50B becomes as follows: Figure 6 As shown, images of all the actors in the live event were captured.

[0063] However, the coordinates of actor object 51, actor object 52, speaker object 53, speaker object 54, and stage object 55 change according to the change of the live broadcast venue. Therefore, the objects are arranged at positions separated from each other in three-dimensional space R2 larger than three-dimensional space R1.

[0064] As described above, the video processing device 1 of the present embodiment can automatically change the viewpoint position according to the cause of the change in the sound of the live event, thereby realizing a change in the viewpoint position that is meaningful to the live event.

[0065] (About sound processing)

[0066] The processor 12 obtains the sound signal related to the live event and performs audio processing on the obtained sound signal. When the live venue changes, the processor 12 generates a reflected sound corresponding to the changed live venue. For example, the processor 12 generates a virtual reflected sound with a long delay time and reverberation time and a high level corresponding to the three-dimensional space R2 that is larger than the three-dimensional space R1.

[0067] In addition, the processor 12 may perform sound processing using the first viewpoint position 50A and the second viewpoint position 50B as listening points. The sound processing using the first viewpoint position 50A and the second viewpoint position 50B as listening points is, for example, the above-mentioned localization processing.

[0068] Fig.10 1 is a flowchart showing the operation of the processor 12 when performing the audio processing with the first viewpoint position 50A and the second viewpoint position 50B as the listening points. Figure 4 The same reference numerals are used to denote common structures, and description thereof will be omitted.

[0069] When the processor 12 outputs the video of the live event viewed from the first viewpoint position 50A, for example, it performs a positioning process (S21) such that the sound of the speaker objects 53 and 54 is localized at the positions of the speaker objects 53 and 54. In addition, the processes of S13 and S21 may be performed one by one or in parallel. In addition, the actor objects 51 and 52 are also objects that emit sound, and therefore, the processor 12 may perform a positioning process such that the sound of the actor objects 51 and 52 is localized at the positions of the actor objects 51 and 52, respectively.

[0070] The processor 12 performs positioning processing based on, for example, HRTF (Head Related Transfer Function). HRTF represents a transfer function from a certain imaginary sound source position to the right ear and left ear of the user. Figure 2 and Figure 3As shown, the position of the speaker object 53 is the front left side when viewed from the first viewpoint position 50A, and the position of the speaker object 54 is the front right side of the first viewpoint position 50A. The processor 12 performs binaural processing (Binaural processing) on ​​the sound signal corresponding to the speaker object 53 so as to be located at the HRTF of the position on the front left side of the user. In addition, the processor 12 performs binaural processing on the sound signal corresponding to the speaker object 54 so as to be located at the HRTF of the position on the front right side of the user. As a result, the user can perceive that the sound of the speaker object 53 on the front left side and the speaker object 54 on the front right side of the user are in the first viewpoint position 50A in the three-dimensional space R1.

[0071] After transforming the coordinates of each object, the processor 12 performs audio processing using the second viewpoint position 50B as the listening point (S22). The processing of S16 and S22 may be performed either first or in parallel.

[0072] The processor 12 performs binaural processing on the sound signal corresponding to the speaker object 53 by convolving the HRTF so as to locate the sound at the position on the left side in front of the user. However, the position of the speaker object 53 viewed from the second viewpoint position 50B in the three-dimensional space R2 is to the left and further in the depth direction than the position of the speaker object 53 viewed from the first viewpoint position 50A in the three-dimensional space R1. Therefore, the processor 12 performs binaural processing on the HRTF by convolving the sound signal corresponding to the speaker object 53 so as to locate the sound at the left side and further in the depth direction. Similarly, the processor 12 performs binaural processing on the HRTF by convolving the sound signal corresponding to the speaker object 54 so as to locate the sound at the right side and further in the depth direction.

[0073] As described above, the visual position of the object changes in accordance with the change of the live broadcast venue, and the localization position of the sound of the object also changes. Therefore, the image processing device 1 of this embodiment can realize the change of the viewpoint position that is meaningful to the live broadcast event both visually and auditorily.

[0074] Furthermore, the processor 12 also performs acoustic processing using the first viewpoint position 50A and the second viewpoint position 50B as listening points in the reflected sound generation process in the reverberation process.

[0075] The reflected sound in the reverberation processing is generated, for example, by convolving an impulse response pre-determined in the actual live broadcast venue with the sound signal. The processor 12 performs binaural processing of convolving the sound signal of the live event with the HRTF based on the impulse response pre-determined in the live broadcast venue and the position of each reflected sound corresponding to the impulse response. The position of the reflected sound in the actual live broadcast venue can be obtained by setting a plurality of microphones at the listening point of the actual live broadcast venue to measure the impulse response. Alternatively, the reflected sound can be generated based on a simulation. The processor 12 can calculate the delay time, level, and arrival direction of the reflected sound based on the position of the sound source, the position of the wall of the live broadcast venue based on 3DCAD data, etc., and the position of the listening point.

[0076] When the live broadcast venue changes, the processor 12 obtains the impulse response corresponding to the changed live broadcast venue and performs binaural processing, thereby implementing audio processing for generating reflected sounds corresponding to the changed live broadcast venue. If the live broadcast venue changes, the position of each reflected sound in the impulse response also changes. Based on the position of each reflected sound after the change, the processor 12 performs binaural processing of the convolution HRTF on the sound signal of the live broadcast event.

[0077] As a result, the live broadcast venue changes visually, and the reflected sound also changes to a sound corresponding to the changed live broadcast venue. Therefore, the image processing device 1 of this embodiment can achieve a change in the viewpoint position that is meaningful to the live broadcast event both visually and auditorily, and can bring a new customer experience to the user.

[0078] In addition, the reflected sound in the reverberation process includes the initial reflected sound and the rear reverberation sound. Therefore, the processor 12 can change the processing according to the initial reflected sound and the rear reverberation sound. For example, the initial reflected sound can be generated by simulation, and the rear reverberation sound can be generated based on the impulse response measured in the live broadcast venue.

[0079] (Variant 1)

[0080] In the above embodiment, the change of the live broadcast venue is shown as an example of the second factor related to the sound of the live broadcast event. In the first variation, the second factor is the factor of the music change of the live broadcast event. The factor of the music change is, for example, a transition from a state where all actors perform or sing to a solo part of a certain actor. Alternatively, the factor of the music change also includes a transition from a solo part of a certain actor to a solo part of another actor.

[0081] In the first modification, the data related to the live event includes information indicating the timing of the music change and information indicating the content of the music change. The processor 12 calculates the relative position with respect to the object based on the information, and changes the first viewpoint position 50A to the second viewpoint position 50B based on the calculated relative position. For example, when the data related to the live event includes information indicating the timing of the transition to the guitar solo, the processor 12 calculates the relative position with respect to the actor object 51, and changes the first viewpoint position 50A to the second viewpoint position 50B based on the relative position.

[0082] Fig.11 is a perspective view showing an example of a three-dimensional space R1 involved in Modification Example 1. For example, the processor 12 Fig.11 As shown, the second viewpoint position 50B is set at a front position at a predetermined distance from the actor object 51 .

[0083] Fig.12 5B is a diagram showing an example of an image of a live event viewed from the second viewpoint position 50B according to Modification 1. The processor 12 outputs Fig.12 As shown, the live event video is viewed from the second viewpoint position 50B. Since the second viewpoint position 50B is set at a front position at a predetermined distance from the actor object 51, the video viewed from the second viewpoint position 50B is a video in which the actor object 51 is enlarged.

[0084] As described above, in the image processing device 1 involved in variation 1, the second viewpoint position 50B is set at a relative position with respect to the actor object 51 of the guitar at the timing of the transition to the guitar solo. Therefore, when a factor of a musical change of the live event such as a transition to the solo part occurs, a change in the viewpoint position that is meaningful to the live event can be achieved.

[0085] In the above embodiment, the stage object 55 is specified as the first factor, but the processor 12 may specify the stage object 55 as the first factor. Figure 4 The selection of the actor object 52 of the lead singer, for example, is received from the user through the processing of S11, and the first viewpoint position 50A is set to the relative position of the selected actor object 51 of the lead singer. As described above, even if the user selects a specific object according to his / her preference, when a factor of a music change of the live event such as a transition to a solo part occurs, a change of the viewpoint position meaningful to the live event can be realized.

[0086] (Variant 2)

[0087] The image processing device 1 of the second modification receives an adjustment operation of the first viewpoint position 50A or the second viewpoint position 50B from the user. The processor 12 changes the first viewpoint position 50A or the second viewpoint position 50B according to the received adjustment operation. For example, when the processor 12 receives an operation to adjust the height of the second viewpoint position 50B to a high position, the processor 12 changes the second viewpoint position 50B to a high position.

[0088] (Variant 3)

[0089] The video processing device 1 according to the third modification records information indicating the viewpoint position as time-series data.

[0090] For example, the processor 12 passes Figure 4 When a selection of, for example, a favorite vocal actor object 52 is received from the user through the processing of S11, the first viewpoint position 50A can be set to the relative position of the selected vocal actor object 52, and the first viewpoint position 50A can be recorded as time series data.

[0091] Alternatively, when the processor 12 receives an adjustment operation of the second viewpoint position 50B from the user as in Modification 2, the processor 12 may record the adjusted second viewpoint position 50B as time-series data.

[0092] Thus, users can construct photography technology data, which is obtained by adding their own adjustment history to the viewpoint position that automatically changes according to the elements related to the sound of the live event. Users can make the constructed photography technology data public, and can also obtain the photography technology data of other users that have been made public, and enjoy the audio-visual experience of the live event through the photography technology data of other users.

[0093] (Variant 4)

[0094] In the above-described embodiment, the first viewpoint position 50A and the second viewpoint position 50B correspond to the position of the user viewing the content in the three-dimensional space. In contrast, the image processing device 1 of the fourth modification sets the first viewpoint position 50A and the second viewpoint position 50B to positions different from the position of the user viewing the content.

[0095] Fig.13 and Fig.14 1 is a perspective view showing an example of a three-dimensional space R1 according to Modification 4. The image processing device 1 according to Modification 4 receives an operation from a user to designate a user position in the three-dimensional space R1 of the user. The processor 12 controls an object in the three-dimensional space according to the received user position. Specifically, the processor 12 arranges the user object 70 at the received user position.

[0096] In this example, the processor 12 acquires information indicating the user positions of other users and arranges the other user objects 75. The user positions of other users are received by the image processing devices of the respective users. The user positions of other users are acquired via the server.

[0097] exist Fig.13 In the example of , the processor 12 receives the selection of the actor object 51 from the user, and sets the first viewpoint position 50A to the relative position of the selected vocalist actor object 51. For example, if the user performs an operation to change the position of the first viewpoint position 50A to the rear of the three-dimensional space R1, the processor 12 outputs the image of the user object 70 and other user objects 75.

[0098] Here, as shown in Modification 1, even when the processor 12 sets the second viewpoint position 50B to the relative position of the actor object 51 with respect to the guitar at the timing of transition to the guitar solo, the positions of the user object 70 and other user objects 75 do not change.

[0099] For example, in a live event, sometimes a specific event is completed by multiple participants at a certain time. For example, sometimes the participants of the live event light up their own penlights at a specified position and in a specified color to form a specific pattern in the entire live venue, which is a participant-participated event.

[0100] The image processing device 1 according to Modification 4 may also receive the above-mentioned participant's participation event as a factor related to the live event, i.e., a second factor, and change the first viewpoint position to a second viewpoint position according to the second factor. For example, when all users participating in the live event are arranged at the designated user positions, the image processing device 1 sets the second viewpoint position 50B to a position that overlooks the entire live venue.

[0101] The image processing device 1 according to Modification 4 receives a user position different from the first viewpoint position 50A and the second viewpoint position 50B and arranges the user object 70 and other user objects 75. Thus, when an event is completed in which a specific pattern is formed in the entire live broadcast venue as described above, the pattern can be viewed from a bird's-eye view.

[0102] It should be understood that the description of the present embodiment is merely illustrative in all aspects and is not restrictive. The scope of the present invention is not indicated by the above-mentioned embodiment, but by the claims. Moreover, the scope of the present invention includes the scope equivalent to the claims.

[0103] For example, in the present embodiment, a music live broadcast event in which a singer and a guitarist perform is shown as an example of a live broadcast event. However, live broadcast events also include events of dramas, musicals, lectures, recitations, or game conventions. For example, a game convention event can be set up with a hypothetical live broadcast venue configured with multiple player objects, a screen object that displays a game screen, a host object, an audience object, and a speaker object that emits game sounds. When the user specifies a first factor for specifying a first player object, the image processing device outputs an image of the live broadcast venue in which the first player is captured. Moreover, as a second factor related to the live broadcast event, for example, when the second player wins, the image processing device outputs an image of the second player. As described above, the image processing device can also achieve a change in the viewpoint position that is meaningful to the live broadcast event for other types of events.

[0104] Description of the label

[0105] 1: Image processing device, 11: Communication unit, 12: Processor, 13: RAM, 14: Flash memory, 15: Display, 16: User I / F, 17: Audio I / F, 20: Headphones, 50A: First viewpoint position, 50B: Second viewpoint position, 51: Actor object, 52: Actor object, 53: Speaker object, 54: Speaker object, 55: Stage object, 70: User object, 70A: Reference point, 75: Other user objects, R1: 3D space, R2: 3D space

Claims

1. A method for image processing, which is to arrange objects in a three-dimensional space and output an image of a live event viewed from a viewpoint position set in the three-dimensional space, In this image processing method, The first viewpoint position is set based on a first factor which is a factor related to the object. outputting a video of the live event viewed from the first viewpoint, receiving a second factor related to the sound of the live event, calculating a relative position with respect to the object based on the second factor, and changing the first viewpoint position to a second viewpoint position based on the calculated relative position, A video of the live event viewed from the second viewpoint is output.

2. The image processing method according to claim 1, wherein: Obtaining the audio signal involved in the live event, The acquired sound signal is subjected to acoustic processing using the second viewpoint position as a listening point.

3. The image processing method according to claim 2, wherein: The sound processing includes localization processing using the object as a sound source.

4. The image processing method according to claim 2 or 3, wherein: The acoustic processing includes a process of generating reflected sound.

5. The image processing method according to any one of claims 1 to 4, wherein: The second factor includes a change in a live broadcast venue for the live broadcast event.

6. The image processing method according to claim 5, wherein: Obtaining the audio signal involved in the live event, performing acoustic processing on the acquired audio signal with the second viewpoint position as a listening point, When the live broadcast venue is changed, acoustic processing is performed to generate reflected sound corresponding to the changed live broadcast venue.

7. The image processing method according to any one of claims 1 to 6, wherein: The second factor includes a factor of a change in music of the live event.

8. The image processing method according to any one of claims 1 to 7, wherein: receiving an adjustment operation of the second viewpoint position from a user, The second viewpoint position is changed according to the received adjustment operation.

9. The image processing method according to any one of claims 1 to 8, wherein: The first viewpoint position and the second viewpoint position are recorded as time-series data.

10. The image processing method according to any one of claims 1 to 9, wherein: receiving an operation from a user to designate a user position in the three-dimensional space of the user, The object in the three-dimensional space is controlled according to the received position of the user.

11. An image processing device comprising a processing unit that arranges objects in a three-dimensional space and outputs an image of a live event viewed from a viewpoint position set in the three-dimensional space, In the image processing device, The processing unit sets a first viewpoint position based on a first factor that is a factor related to the object. outputting a video of the live event viewed from the first viewpoint, receiving a second factor related to the sound of the live event, calculating a relative position with respect to the object based on the second factor, and changing the first viewpoint position to a second viewpoint position based on the calculated relative position, A video of the live event viewed from the second viewpoint is output.

12. The image processing device according to claim 11, wherein: The processing unit obtains the audio signal related to the live event, The acquired sound signal is subjected to acoustic processing using the second viewpoint position as a listening point.

13. The image processing device according to claim 12, wherein: The sound processing includes localization processing using the object as a sound source.

14. The image processing device according to claim 12 or 13, wherein: The acoustic processing includes a process of generating reflected sound.

15. The image processing device according to any one of claims 11 to 14, wherein: The second factor includes a change in a live broadcast venue for the live broadcast event.

16. The image processing device according to claim 15, wherein: The processing unit obtains the audio signal related to the live event, performing acoustic processing on the acquired audio signal with the second viewpoint position as a listening point, When the live broadcast venue is changed, acoustic processing is performed to generate reflected sound corresponding to the changed live broadcast venue.

17. The image processing device according to any one of claims 11 to 16, wherein: The second factor includes a factor of a change in music of the live event.

18. The image processing device according to any one of claims 11 to 17, wherein: The processing unit receives an adjustment operation of the second viewpoint position from a user, The second viewpoint position is changed according to the received adjustment operation.

19. The image processing device according to any one of claims 11 to 18, wherein: The processing unit records the first viewpoint position and the second viewpoint position as time-series data.

20. The image processing device according to any one of claims 11 to 19, wherein: The processing unit receives an operation from a user to designate a user position in the three-dimensional space of the user, The object in the three-dimensional space is controlled according to the received position of the user.