Information processing method, information processing system, and program

Abstract

To generate content with a sense of unity between a video and sound while suppressing the effect on the video represented by video data.SOLUTION: An editing system 20 processes video data X1 including a plurality of video segments and sound data Y1 including a plurality of sound segments. The editing system 20 includes a sound data processing unit 53 that processes the sound data Y1 so as to switch from a first sound segment of the plurality of sound segments to a second sound segment other than the first sound segment at a boundary point between a first video segment of the plurality of video segments and a second video segment following the first video segment.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present disclosure relates to techniques for generating content including video and audio. [Background technology]

[0002] Various techniques for creating content including moving images and sound have been proposed in the past. For example, Patent Document 1 discloses a technique for generating a slideshow moving image in such a way that the moving images are switched at the time when the melody of the music changes. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2007-188561 A Summary of the Invention [Problem to be solved by the invention]

[0004] In the technology of Patent Document 1, since the video is forcibly switched at the point where the melody of the music changes, it is actually difficult for the creator to create content that includes the video that he or she intended. In consideration of the above circumstances, one aspect of the present disclosure aims to generate content that has a sense of unity between the video and the sound while suppressing the influence on the video represented by the video data. [Means for solving the problem]

[0005] In order to solve the above problems, an information processing method according to one aspect of the present disclosure is an information processing method for processing moving image data including a plurality of moving image segments and sound data including a plurality of sound segments, the sound data being processed so as to switch from a first sound segment among the plurality of sound segments to a second sound segment other than the first sound segment at a boundary point between a first moving image segment among the plurality of moving image segments and a second moving image segment subsequent to the first moving image segment.An information processing method according to another aspect of the present disclosure is an information processing method for processing moving image data including a plurality of moving image segments and sound data representing sound, the sound data being processed so as to reduce a volume within a transition period including a boundary point between a first moving image segment among the plurality of moving image segments and a second moving image segment subsequent to the first moving image segment.

[0006] An information processing system according to one aspect of the present disclosure is an information processing system that processes moving image data including a plurality of moving image segments and sound data including a plurality of sound segments, and includes a sound data processing unit that processes the sound data so as to switch from a first sound segment among the plurality of sound segments to a second sound segment other than the first sound segment at a boundary point between a first moving image segment among the plurality of moving image segments and a second sound segment subsequent to the first sound segment.An information processing system according to another aspect of the present disclosure is an information processing system that processes moving image data including a plurality of moving image segments and sound data representing sound, and includes a sound data processing unit that processes the sound data so as to reduce a volume within a transition period that includes a boundary point between a first moving image segment among the plurality of moving image segments and a second moving image segment subsequent to the first moving image segment.

[0007] A program according to one aspect of the present disclosure is a program for processing moving image data including a plurality of moving image segments and sound data including a plurality of sound segments, the program causing a computer to function as a sound data processing unit that processes the sound data so as to switch from a first sound segment among the plurality of sound segments to a second sound segment other than the first sound segment at a boundary point between a first moving image segment among the plurality of moving image segments and a second moving image segment subsequent to the first moving image segment.A program according to another aspect of the present disclosure is a program for processing moving image data including a plurality of moving image segments and sound data representing sound, the program causing a computer to function as a sound data processing unit that processes the sound data so as to reduce a volume within a transition period that includes a boundary point between a first moving image segment among the plurality of moving image segments and a second moving image segment subsequent to the first moving image segment. [Brief description of the drawings]

[0008] [Figure 1] 1 is a block diagram illustrating a configuration of an information system according to a first embodiment. [Diagram 2] FIG. 1 is a block diagram illustrating a configuration of an editing system. [Diagram 3] FIG. 2 is a block diagram illustrating an example of a functional configuration of the editing system. [Figure 4] 4 is a flowchart of an operation executed by a control device of the editing system. [Diagram 5] FIG. 4 is an explanatory diagram of an editing process in the first embodiment. [Figure 6] 4 is a flowchart of an editing process in the first embodiment. [Figure 7] FIG. 11 is an explanatory diagram of an editing process in the second embodiment. [Figure 8] 13 is a flowchart of an editing process in the second embodiment. [Figure 9] FIG. 13 is an explanatory diagram of an editing process in the third embodiment. [Figure 10] 13 is a flowchart of an editing process in the third embodiment. [Figure 11] FIG. 13 is an explanatory diagram of an editing process in the fourth embodiment. [Figure 12] FIG. 13 is a block diagram illustrating a configuration of a terminal device in a fifth embodiment. [Figure 13] FIG. 13 is a block diagram illustrating a functional configuration of a terminal device in a fifth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] A: First embodiment 1 is a block diagram illustrating a configuration of an information system 100 according to the first embodiment. The information system 100 according to the first embodiment includes a terminal device 10 and an editing system 20. The terminal device 10 and the editing system 20 communicate with each other via a communication network 30 such as the Internet.

[0010] The terminal device 10 is an information terminal such as a mobile phone, a smartphone, a tablet terminal, or a personal computer. The terminal device 10 transmits material data D to the editing system 20. The material data D includes video data X1 and sound data Y1. The video data X1 is data representing a video. For example, the video data X1 is generated by an imaging device mounted on the terminal device 10. The sound data Y1 is data representing a sound to be played in parallel with the video of the video data X1. Specifically, the sound data Y1 in the first embodiment is data representing a performance sound (musical instrument sound or singing sound) of a piece of music played as background music for the video of the video data X1.

[0011] The editing system 20 is a computer system that generates content C using material data D received from the terminal device 10. The content C is a video content including video data X2 and sound data Y2. The video data X2 is generated by editing the video data X1. The sound data Y2 is generated by editing the sound data Y1. In other words, the material data D is data that becomes the material of the content C. The editing system 20 transmits the content C to the terminal device 10. The terminal device 10 plays the content C received from the editing system 20. In other words, the video represented by the video data X2 and the sound represented by the sound data Y2 (specifically, the sound of a musical piece being performed) are played in parallel.

[0012] 2 is a block diagram illustrating an example of the configuration of an editing system 20. The editing system 20 includes a control device 21, a storage device 22, and a communication device 23. The editing system 20 may be realized by a single device, or may be realized by a plurality of devices configured separately from each other.

[0013] The control device 21 is a single or multiple processors that control each element of the editing system 20. Specifically, the control device 21 is configured by one or more types of processors, such as a CPU (Central Processing Unit), an SPU (Sound Processing Unit), a DSP (Digital Signal Processor), an FPGA (Field Programmable Gate Array), or an ASIC (Application Specific Integrated Circuit).

[0014] The storage device 22 is one or more memories that store the programs executed by the control device 21 and various data used by the control device 21. The storage device 22 is configured with a known recording medium such as a magnetic recording medium or a semiconductor recording medium, or a combination of multiple types of recording media. In addition, a portable recording medium that is detachable from the editing system 20, or a recording medium (e.g., cloud storage) to which the control device 21 can write or read via the communication network 30 may be used as the storage device 22.

[0015] The communication device 23 communicates with the terminal device 10 via the communication network 30. Specifically, the communication device 23 receives material data D transmitted from the terminal device 10. In addition, the communication device 23 transmits content C generated from the material data D to the terminal device 10.

[0016] 3 is a block diagram illustrating a functional configuration of the editing system 20. The control device 21 of the editing system 20 executes a program stored in the storage device 22 to realize a plurality of functions (a material data acquisition unit 51, a video data processing unit 52, an audio data processing unit 53, and a content providing unit 54) for generating and providing content C from material data D. The material data acquisition unit 51 acquires the material data D transmitted from the terminal device 10 by the communication device 23.

[0017] The moving image data processing unit 52 generates moving image data X2 from the moving image data X1 of the material data D. Specifically, the moving image data processing unit 52 generates the moving image data X2 by executing image processing on the moving image data X1. The image processing includes various processes such as extraction of a specific section of a moving image or adjustment of image quality. The moving image data X1 and the moving image data X2 may be in any format.

[0018] The sound data processing unit 53 generates sound data Y2 from sound data Y1 of the material data D. In the first embodiment, the sound data Y1 and sound data Y2 are composed of a time series of samples representing sound waveforms, for example. The video data X2 generated by the video data processing unit 52 and the sound data Y2 generated by the sound data processing unit 53 constitute the content C. In other words, the video data processing unit 52 and the sound data processing unit 53 function as elements that generate the content C from the material data D. The content providing unit 54 transmits the content C including the video data X2 and the sound data Y2 from the communication device 23 to the terminal device 10.

[0019] FIG. 4 is a flow chart illustrating a specific procedure of the operation executed by the control device 21. The process of FIG. 4 is started in response to an instruction from the user of the terminal device 10. When the process is started, the control device 21 (material data acquisition unit 51) receives the material data D transmitted from the terminal device 10 via the communication device 23 (Sa). The video data processing unit 52 generates video data X2 from the video data X1 of the material data D (Sb). The audio data processing unit 53 generates audio data Y2 from the audio data Y1 of the material data D (Sc: editing process). The content providing unit 54 transmits the content C including the video data X2 and the audio data Y2 from the communication device 23 to the terminal device 10 (Sd).

[0020] FIG. 5 is an explanatory diagram of the editing process Sc. The video data X2 includes a plurality (M pieces) of video sections V1 to VM arranged on a time axis (M is a natural number of 2 or more). Each video section Vm (m=1 to M) is a period obtained by dividing the video represented by the video data X2 on a time axis by scene. The M video sections V1 to VM are continuous on the time axis without any interval between each other, and constitute a series of stories related to one theme. For example, a series of stories with a sense of conceptual unity, such as a video introducing a company or a video introducing products handled by the company, is constituted by the M video sections V1 to VM. Any one of the M video sections V1 to VM is, for example, a section corresponding to a scene with a semantic unity in a video, or a section captured in one imaging operation by an imaging device. However, it is also possible that one video section Vm includes multiple scenes, or that one video section Vm includes a period captured in multiple imaging operations. Note that the M video sections V1 to VM do not necessarily have to be temporally or conceptually continuous. For example, the video sections Vm may be spaced apart from one another, or the two or more video sections Vm may have different themes. Also, a single video section Vm may not necessarily have a semantic unity.

[0021] Each moving image section Vm has a variable length. Basically, the time length of each moving image section Vm is different. However, the time length may be the same across any two or more moving image sections Vm. In the moving image data X2, a time point Pm (hereinafter referred to as a "boundary point") at the boundary between adjacent moving image sections Vm and Vm+1 on the time axis is specified. The boundary point Pm can also be said to be the end point (specifically, the terminal point) of the moving image section Vm.

[0022] The sound data Y1 includes a plurality of (M) sound intervals A1-AM arranged on a time axis. Each moving image interval Vm of the moving image data X2 corresponds to each sound interval Am of the sound data Y1. Specifically, the material data acquisition unit 51 acquires material data D from the terminal device 10, in which each moving image interval Vm of the moving image data X1 corresponds to each sound interval Am of the sound data Y1. For example, the sound interval Am corresponding to each moving image interval Vm is selected in response to an instruction from the user of the terminal device 10. Note that the sound data Y1 is a single piece of data that is continuous across M sound intervals A1-AM, or a collection of a plurality of data corresponding to different sound intervals Am.

[0023] Each sound interval Am is a structural interval obtained by dividing the music piece represented by the sound data Y1 on the time axis according to its musical meaning. For example, each sound interval Am corresponds to a period such as an intro, verse, bridge, chorus, and outro. Data specifying the end points (start point or end point) of each sound interval Am is set in the sound data Y1. The data specifying the end points of each sound interval Am corresponds to a rehearsal mark indicating the start point of each structural interval. The duration of each sound interval Am exceeds the duration of the video interval Vm corresponding to that sound interval Am.

[0024] The sound data processing unit 53 processes the sound data Y1 to generate sound data Y2 so that a sound section Am corresponding to the video section Vm starts in each of the M video sections V1 to VM of the video data X2. That is, the sound data processing unit 53 generates sound data Y2 from the sound data Y1 so that a sound section Am of the M sound sections A1 to AM is switched to an immediately succeeding sound section Am+1 at a boundary point Pm between the video section Vm of the M video sections V1 to VM and the immediately succeeding video section Vm+1. For example, a switch from the sound section A1 to the sound section A2 occurs at a boundary point P1 between the video section V1 and the video section V2, and a switch from the sound section A2 to the sound section A3 occurs at a boundary point P2 between the video section V2 and the video section V3. Specifically, the sound data processing unit 53 of the first embodiment generates sound data Y2 by processing the sound data Y1 so that a switch occurs from a midpoint in the sound section Am that coincides with the boundary point Pm to an immediately succeeding sound section Am+1.

[0025] Specifically, the sound data processing unit 53 of the first embodiment extracts a portion of a period (hereinafter referred to as a "specific section") Bm including the start point of a sound section Am of sound data Y1, and generates sound data Y2 by linking M specific sections B1 to BM corresponding to different sound sections Am in chronological order to each other. The specific section Bm is a section of the sound section Am that extends from the start point of the sound section Am to the duration of the moving image section Vm. A portion of the sound section Am including the end point (sections other than the specific section Bm) is removed. As can be understood from the above explanation, the end section of the sound section Am is removed so that the duration of each sound section Am matches the duration of the moving image section Vm.

[0026] 6 is a flow chart illustrating a specific procedure of the edit process Sc. The edit process Sc is started when the video data processing unit 52 generates the video data X2.

[0027] When the editing process Sc is started, the sound data processing unit 53 selects one video section Vm from the M video sections V1-VM of the video data X2 (Sc11). Each video section Vm is selected sequentially in chronological order. The sound data processing unit 53 selects one sound section Am corresponding to the video section Vm from the M sound sections A1-AM of the sound data Y1 (Sc12). The sound data processing unit 53 extracts a specific section Bm from the selected sound section Am, which extends from the start point of the sound section Am to the time length of the video section Vm (Sc13: extraction process). The sound data processing unit 53 concatenates the specific section Bm extracted in the extraction process Sc13 to the end of the specific section Bm-1 extracted in the immediately preceding extraction process Sc13 (Sc14: concatenation process). Note that the specific section B1 extracted from the first sound section A1 is placed at the beginning of the sound data Y2.

[0028] The sound data processing unit 53 judges whether the above processes (Sc11-Sc14) have been performed for all of the M sound sections A1-AM (Sc15). If an unprocessed sound section Am remains (Sc15: NO), the sound data processing unit 53 selects the moving image section Vm+1 immediately following the currently selected moving image section Vm among the M moving image sections V1-VM of the moving image data X2 as the moving image section Vm to be processed (Sc11), and performs selection of a sound section Am (Sc12), extraction process Sc13, and linking process Sc14 for the updated moving image section Vm. On the other hand, if all of the M sound sections A1-AM have been processed (Sc15: YES), the sound data processing unit 53 ends the editing process Sc. As can be understood from the above explanation, the moving image data X2 is not edited in the editing process Sc.

[0029] As illustrated above, the sound data processor 53 processes the sound data Y1 so as to switch from the sound interval Am to the sound interval Am+1 at the boundary point Pm between the moving image interval Vm and the subsequent moving image interval Vm+1. Thus, the content C is generated in which each moving image interval Vm and each sound interval Am are parallel to each other, and the sound interval Am starts at the start point of the moving image interval Vm. That is, at the boundary point Pm where the moving image played by the terminal device 10 transitions from the moving image interval Vm to the moving image interval Vm+1, the sound played by the terminal device 10 switches to the sound interval Am+1 from the middle of the sound interval Am (the end point of the specific interval Bm). On the other hand, the duration of each moving image interval Vm specified by the moving image data X2 is not changed. As can be understood from the above explanation, according to the first embodiment, it is possible to generate the content C having a sense of unity between the change in the moving image and the change in the sound while suppressing the influence on the moving image represented by the moving image data X2.

[0030] Furthermore, in the first embodiment, by a simple process of starting a sound interval Am+1 at a boundary point Pm of each moving image interval Vm, it is possible to cause a switch from a sound interval Am+1 to occur at a boundary point Pm between the moving image intervals Vm and Vm+1. Note that the moving image interval Vm is an example of a "first moving image interval," and the moving image interval Vm+1 is an example of a "second moving image interval." Also, the sound interval Am is an example of a "first sound interval," and the immediately following sound interval Am+1 is an example of a "second sound interval."

[0031] B: Second embodiment A second embodiment will be described. Note that, in each of the following exemplary embodiments, for elements whose functions are similar to those of the first embodiment, the reference numerals used in the description of the first embodiment will be used and detailed descriptions of each will be omitted as appropriate.

[0032] 7 is an explanatory diagram of the editing process Sc in the second embodiment. In the first embodiment, a part including the end point of each sound interval Am is deleted to make the end points of each sound interval Am coincide with the end points of each video interval Vm on the time axis. The sound data processor 53 in the second embodiment shortens or extends each sound interval Am on the time axis to make the end points of each sound interval Am coincide with the end points of each video interval Vm on the time axis. Specifically, the sound data processor 53 adjusts the progress speed (e.g., tempo) of each sound interval Am to shorten or extend the sound interval Am to the time length of the video interval Vm.

[0033] 8 is a flowchart illustrating a specific procedure of the editing process Sc in the second embodiment. When the editing process Sc is started, the sound data processing unit 53 selects a moving image section Vm (Sc21) and a sound section Am (Sc22) in the same manner as in the first embodiment.

[0034] The sound data processing unit 53 of the second embodiment executes a resizing process Sc23 instead of the extraction process Sc13 of the first embodiment. The resizing process Sc23 is a process for shortening or extending the sound section Am so that the time length of the sound section Am coincides with the time length of the moving image section Vm. Any known resizing technique is adopted for the resizing process Sc23. The sound data processing unit 53 connects the sound section Am after resizing by the resizing process Sc23 to the end of the sound section Am-1 after resizing by the previous resizing process Sc23 (Sc24). Note that the first sound section Am is placed at the beginning of the sound data Y2. The above process is repeated for all M sound sections A1 to AM (Sc25). As in the first embodiment, the moving image data X2 is not edited in the editing process Sc of the second embodiment.

[0035] The editing process Sc illustrated above generates content C in which each video section Vm and each sound section Am are parallel to each other and the sound section Am starts at the starting point of the video section Vm. Therefore, similar to the first embodiment, it is possible to generate content C that has a sense of unity between changes in the video and changes in sound while suppressing the influence on the video represented by the video data X2. Furthermore, in the second embodiment, each sound section Am is expanded or contracted, thereby reducing the possibility that each sound section Am will be interrupted discontinuously at a point in time. Furthermore, when the duration of the sound section Am is shorter than the duration of the video section Vm, extending the sound section Am reduces the possibility that the sound section Am will be insufficient for the video section Vm.

[0036] The sound data processing unit 53 in the first and second embodiments is collectively represented as an element that processes sound data Y1 so as to switch from sound interval Am to sound interval Am+1 at boundary point Pm between moving image interval Vm and moving image interval Vm+1.

[0037] C: Third embodiment 9 is an explanatory diagram of the editing process Sc executed by the sound data processing unit 53 in the third embodiment. In the third embodiment, it is assumed that the duration of each sound section Am of the sound represented by the sound data Y1 is set to the same duration as each moving image section Vm of the moving image represented by the moving image data X2. Note that in the third and fourth embodiments, it is not essential to distinguish between each sound section Am.

[0038] The transition period Q in Fig. 9 is a period corresponding to a boundary point Pm between adjacent moving image sections Vm and Vm+1 among M moving image sections V1 to VM of the moving image represented by the moving image data X2. Specifically, the transition period Q is a period including the boundary point Pm. In the first embodiment, a period ending at the boundary point Pm is exemplified as the transition period Q. The transition period Q is set to a predetermined time length. However, the time length of the transition period Q may be variable according to, for example, an instruction from a user of the terminal device 10.

[0039] The sound data processing unit 53 of the third embodiment processes the sound data Y1 so that the volume is lower within the transition period Q than outside the transition period Q, thereby generating sound data Y2. Specifically, the sound data processing unit 53 processes the sound data Y1 so that the volume is lower from the start point q1 to the end point q2 of the transition period Q, and the volume is increased at the end point q2 of the transition period Q. For example, the sound data processing unit 53 generates sound data Y2 by multiplying each sample constituting the sound data Y1 by an adjustment value G. The sound data processing unit 53 changes the adjustment value G over time within a range between a reference value gH and a minimum value gL. The reference value gH is a numerical value that exceeds the minimum value gL. For example, the reference value gH is set to 1, and the minimum value gL is set to 0.

[0040] First, the sound data processing unit 53 of the third embodiment maintains the adjustment value G at the reference value gH outside the transition period Q. Second, the sound data processing unit 53 decreases the adjustment value G over time from the reference value gH to the minimum value gL from the start point q1 to the end point q2 of the transition period Q. Within the transition period Q, the adjustment value G decreases linearly, for example, at a predetermined rate of change. However, the adjustment value G may change curvilinearly, for example, at a variable rate of change. Third, the sound data processing unit 53 increases the adjustment value G from the minimum value gL to the reference value gH at the end point q2 of the transition period Q.

[0041] 10 is a flowchart illustrating a specific procedure of the editing process Sc in the third embodiment. When the editing process Sc is started, the sound data processing unit 53 sets multiple transition periods Q corresponding to different boundary points Pm on the time axis by referring to the video data X2 (Sc31). The sound data processing unit 53 selects one of the multiple transition periods Q (Sc32).

[0042] The sound data processing unit 53 adjusts the volume of the sound data Y1 so as to decrease the volume during the selected transition period Q (Sc33: adjustment process). Specifically, the sound data processing unit 53 decreases the adjustment value G over time from the reference value gH to the minimum value gL from the start point q1 to the end point q2 of the transition period Q, and increases the adjustment value G from the minimum value gL to the reference value gH at the end point q2.

[0043] The sound data processing unit 53 determines whether or not the adjustment process Sc33 has been performed for all of the multiple transition periods Q (Sc34). If an unprocessed transition period Q remains (Sc34: NO), the sound data processing unit 53 selects the transition period immediately following the currently selected transition period from among the multiple transition periods Q (Sc32), and performs the adjustment process Sc33 for the updated transition period Q. On the other hand, if the adjustment process Sc33 has been performed for all of the multiple transition periods Q (Sc34: YES), the sound data processing unit 53 ends the editing process Sc. As can be understood from the above explanation, the video data X2 is not edited in the editing process Sc.

[0044] As described above, in the third embodiment, the sound data Y1 is processed so that the volume decreases within the transition period Q corresponding to the boundary point Pm between the moving image section Vm and the moving image section Vm+1. Therefore, similar to the first embodiment, it is possible to generate content C that has a sense of unity between changes in the moving image and changes in the sound while suppressing the influence on the moving image represented by the moving image data X2.

[0045] In particular, in the third embodiment, the volume decreases from the start point q1 to the end point q2 (boundary point Pm) of the transition period Q, and increases at the end point q2 (the start point of the moving image section Vm) of the transition period Q. Therefore, it is possible to generate content C in which the volume decreases over time toward the end point of the moving image section Vm, and sound is played back at a sufficient volume at the start of the moving image section Vm+1.

[0046] D: Fourth embodiment 11 is an explanatory diagram of the editing process Sc in the fourth embodiment. In the fourth embodiment, similar to the third embodiment, it is assumed that the duration of each sound section Am of the sound represented by the sound data Y1 is set to the same duration as each moving image section Vm of the moving image represented by the moving image data X2. In the fourth embodiment, similar to the third embodiment, a plurality of transition periods Q including different boundary points Pm of the moving image represented by the moving image data X2 are set. Each transition period Q is a period having a boundary point Pm as an end point.

[0047] The sound data processing unit 53 of the fourth embodiment generates sound data Y2 by processing the sound data Y1 so that the volume is lower within the transition period Q than outside the transition period Q. Specifically, the sound data processing unit 53 processes the sound data Y1 so that the volume is lower at the start point q1 of the transition period Q and is increased from the start point q1 to the end point q2 of the transition period Q. For example, similar to the third embodiment, the sound data processing unit 53 changes the adjustment value G, which is multiplied by each sample of the sound data Y1, over time within a range between a reference value gH and a minimum value gL.

[0048] First, the sound data processing unit 53 of the fourth embodiment maintains the adjustment value G at the reference value gH outside the transition period Q. Second, the sound data processing unit 53 decreases the adjustment value G from the reference value gH to the minimum value gL at the start point q1 of the transition period Q. Third, the sound data processing unit 53 increases the adjustment value G over time from the minimum value gL to the reference value gH from the start point q1 to the end point q2 of the transition period Q. Within the transition period Q, the adjustment value G increases linearly, for example, at a predetermined rate of change. However, the adjustment value G may change curvilinearly, for example, at a variable rate of change.

[0049] The operations of the editing process Sc in the fourth embodiment, other than the adjustment process Sc33, are the same as those in the third embodiment. In the adjustment process Sc33 in the fourth embodiment, the sound data processing unit 53 adjusts the volume of the sound data Y1 so that the volume decreases within the transition period Q. Specifically, the sound data processing unit 53 in the fourth embodiment decreases the adjustment value G from the reference value gH to the minimum value gL at the start point q1 of the transition period Q, and increases the adjustment value G over time from the minimum value gL to the reference value gH from the start point q1 to the end point q2 of the transition period Q.

[0050] As described above, in the fourth embodiment, the sound data Y1 is processed so that the volume is reduced within the transition period Q including the boundary point Pm between the moving image section Vm and the moving image section Vm+1. Therefore, similar to the third embodiment, it is possible to generate content C that has a sense of unity between changes in the moving image and changes in the sound while suppressing the influence on the moving image represented by the moving image data X2.

[0051] In particular, in the fourth embodiment, the volume decreases at the start point q1 of the transition period Q, and increases from the start point q1 to the end point q2 (boundary point Pm) of the transition period Q. Therefore, it is possible to generate content C in which the volume increases over time as the video section Vm switches to the video section Vm+1.

[0052] E: Fifth embodiment 12 is a block diagram illustrating a configuration of a terminal device 10 in the fifth embodiment. In the first to fourth embodiments, the editing system 20 generates the content C from the material data D. In the fifth embodiment, the terminal device 10 generates the content C from the material data D. In the fifth embodiment, the editing system 20 is omitted.

[0053] The terminal device 10 includes a control device 11, a storage device 12, and a playback device 13. The terminal device 10 may be realized as a single device, or may be realized as a plurality of devices configured separately from each other. For example, the playback device 13 may be configured separately from the terminal device 10, and may be connected to the terminal device 10 by wire or wirelessly.

[0054] The control device 11 is a single or multiple processors that control each element of the terminal device 10. Specifically, the control device 11 is configured by one or more types of processors, such as a CPU, an SPU, a DSP, an FPGA, or an ASIC.

[0055] The storage device 12 is a single or multiple memories that store the programs executed by the control device 11 and various data used by the control device 11. The storage device 12 is configured with a known recording medium such as a magnetic recording medium or a semiconductor recording medium, or a combination of multiple types of recording media. In addition, a portable recording medium that is detachable from the terminal device 10, or a recording medium (e.g., cloud storage) to which the control device 11 can write or read via the communication network 30 may be used as the storage device 12.

[0056] The playback device 13 plays back the content C. Specifically, the playback device 13 includes a display device 131 that displays a moving image represented by the moving image data X2 of the content C, and a sound emitting device 132 (e.g., a speaker or a headphone) that emits a sound represented by the sound data Y2 of the content C.

[0057] 13 is a block diagram illustrating a functional configuration of the terminal device 10. The control device 11 of the terminal device 10 executes a program stored in the storage device 12 to realize a plurality of functions (a material data acquisition unit 51, a video data processing unit 52, an audio data processing unit 53, and a playback control unit 55) for generating and providing content C from material data D. That is, the functions of the terminal device 10 are such that the content providing unit 54 among the functions of the editing system 20 in the first to fourth embodiments is replaced with the playback control unit 55.

[0058] The material data acquisition unit 51 acquires material data D including video data X1 and audio data Y1. Specifically, the material data acquisition unit 51 generates or edits the material data D in response to an instruction from a user of the terminal device 10. Note that the material data acquisition unit 51 may receive the material data D from an external device with which the terminal device 10 can communicate via the communication network 30.

[0059] As in the first embodiment, the video data processing unit 52 generates video data X2 from video data X1 of the material data D. The audio data processing unit 53 generates audio data Y2 from audio data Y1 of the material data D. Specifically, the audio data processing unit 53 generates audio data Y2 by executing the editing process Sc exemplified in any of the first to fourth embodiments on the audio data Y1. As in the first embodiment, the video data X2 generated by the video data processing unit 52 and the audio data Y2 generated by the audio data processing unit 53 constitute the content C.

[0060] The playback control unit 55 causes the playback device 13 to play the content C. Specifically, the playback control unit 55 causes the display device 131 to display a moving image in response to the supply of the moving image data X2, and causes the sound emitting device 132 to emit a sound in response to the supply of the sound data Y2. Thus, the user of the terminal device 10 can view the content C. The fifth embodiment also achieves the same effects as the first embodiment.

[0061] The editing system 20 illustrated in the first to fourth embodiments and the terminal device 10 illustrated in the fifth embodiment are collectively expressed as an information processing system that processes video data X1 and audio data Y1.

[0062] F: Variation Specific modified embodiments added to each of the above-mentioned embodiments are exemplified below. Two or more embodiments selected from the following examples may be appropriately combined as long as they are not mutually contradictory.

[0063] (1) The configurations according to the above-mentioned embodiments can be appropriately combined. For example, each sound section Am may be adjusted to have the same duration as each moving image section Vm by the extraction process (Sc13, Sc23) illustrated in the first or second embodiment, and then the volume within each transition period Q may be adjusted by the adjustment process Sc33 illustrated in the third or fourth embodiment.

[0064] (2) In the third and fourth embodiments, the transition period Q having the boundary point Pm as its end point is illustrated, but the relationship between the transition period Q and the boundary point Pm is not limited to the above example. For example, it is also possible to set the transition period Q with the boundary point Pm as its start point, or to set the transition period Q with the boundary point Pm as its midpoint.

[0065] (3) In each of the above-mentioned embodiments, the sound data Y (Y1, Y2) represents the performance sound of a piece of music (musical instrument sound or singing sound), but the sound represented by the sound data Y is not limited to musical sound. For example, a form in which the sound data Y represents a speech sound (language sound) that does not include musical elements is also conceivable. For example, the sound data Y may represent a speech sound (e.g., a speech sound by a character in the video or a commentary sound of the video) to be played in parallel with the video represented by the video data X2.

[0066] (4) In each of the above embodiments, the material data D includes sound data Y1, but the material data D may include character string data instead of the sound data Y1. The character string data is data representing a character string corresponding to sound to be played in parallel with the moving image of the moving image data X2. The sound data processing unit 53 generates sound data Y1 by voice synthesis using the character string data of the material data D, and generates sound data Y2 by editing the sound data Y1 through an editing process Sc. Any known method may be used for the voice synthesis.

[0067] (5) In the first and second embodiments, the duration of each sound section Am in the sound data Y1 is adjusted (deleted or expanded) to the duration of the moving image section Vm. However, the sound data processing unit 53 may generate sound data Y2 in which the duration of each sound section Am is set according to the duration of the moving image section Vm by a synthesis process. The synthesis process is musical sound synthesis that synthesizes a performance sound from control data representing a time series of musical notes, or voice synthesis that synthesizes a voice such as a speech sound or a singing sound from control data representing a character string. The sound data processing unit 53 generates, for example, sound data Y2 in which each sound section Am is set to the same duration as the moving image section Vm by a synthesis process that applies control data. As can be understood from the above description, in the first or second embodiment, the adjustment to the sound data Y1 may be omitted. Also, the sound data Y1 used in the third or fourth embodiment may be generated by the synthesis process exemplified above.

[0068] (6) In the above embodiments, the sound data Y (Y1, Y2) is configured as a time series of samples, but the sound data Y may be in any format. For example, sound data Y in a format conforming to the MIDI (Musical Instrument Digital Interface) standard may be used.

[0069] (7) In the above-mentioned embodiments, the video data X (X1, X2) represents a video, but the video data X may represent both a video and sound that are played in parallel. When content C is played, the sound represented by the video data X2 and the sound represented by the sound data Y2 are played in parallel.

[0070] (8) As described above, the functions of the editing system 20 in the first to fourth embodiments are realized by the cooperation of one or more processors constituting the control device 21 and the program stored in the storage device 22. Similarly, the functions of the terminal device 10 in the fifth embodiment are realized by the cooperation of one or more processors constituting the control device 11 and the program stored in the storage device 12.

[0071] The program for realizing the above functions can be provided in a form stored in a computer-readable recording medium and installed in the computer. The recording medium is, for example, a non-transitory recording medium, and is exemplified by an optical recording medium (optical disk) such as a CD-ROM, but also includes any known type of recording medium such as a semiconductor recording medium or a magnetic recording medium. Note that the non-transitory recording medium includes any recording medium except for a transient, propagating signal, and does not exclude volatile recording media. In addition, in a configuration in which a distribution device distributes a program via a communication network, a storage device that stores the program in the distribution device corresponds to the above-mentioned non-transitory recording medium.

[0072] G: Notes From the above-described exemplary embodiments, the following configurations can be understood, for example.

[0073] An information processing method according to one aspect (aspect 1) of the present disclosure is an information processing method for processing moving image data including a plurality of moving image segments and sound data including a plurality of sound segments, the sound data being processed so as to switch from a first sound segment among the plurality of sound segments to a second sound segment other than the first sound segment at a boundary point between a first moving image segment among the plurality of moving image segments and a second moving image segment subsequent to the first moving image segment. In the above aspect, the sound data is processed so as to transition from the first sound segment to the second sound segment at the boundary point between the first moving image segment and the second moving image segment. Therefore, it is possible to generate content that has a sense of unity between changes in the moving image and changes in sound while suppressing the influence on the moving image represented by the moving image data.

[0074] Note that "the first sound section switches to the second sound section at the boundary point between the first and second video sections" means that the switching point from the first sound section to the second sound section substantially coincides with the boundary point between the first and second video sections. "Substantially coincident" includes cases where the switching point completely coincides with the boundary point, as well as cases where the switching point and the boundary point do not strictly coincide but can be regarded as coinciding. For example, even if the switching point and the boundary point are actually different, a state in which the switching point and the boundary point are close enough that a viewer of the content can perceive them as coinciding can be interpreted as "substantially coinciding."

[0075] In a specific example (Aspect 2) of Aspect 1, the sound data is processed so as to switch to the second sound section at an intermediate point in the first sound section that coincides with the boundary point. According to the above aspect, it is possible to switch from the first sound section to the second sound section at the boundary point between the first and second moving image sections by a simple process of starting the second sound section at an intermediate point in the first sound section that coincides with the boundary point.

[0076] In a specific example (Aspect 3) of Aspect 1, in processing the sound data, at least one of the first sound interval and the second sound interval is shortened or extended on the time axis, so that the time point at which the first sound interval switches to the second sound interval coincides with the boundary point. According to the above aspect, it is possible to reduce the possibility that the first sound interval will be cut off midway, or that the first sound interval will be insufficient for the first video interval. Note that the sound interval is expanded or contracted by, for example, adjusting the playback speed. That is, the sound interval is shortened by increasing the playback speed, and the sound interval is extended by decreasing the playback speed.

[0077] An information processing method according to another aspect (aspect 4) of the present disclosure is an information processing method for processing moving image data including a plurality of moving image segments and audio data representing sound, the audio data being processed so that the volume is reduced within a transition period including a boundary point between a first moving image segment and a second moving image segment subsequent to the first moving image segment among the plurality of moving image segments. In the above aspect, the audio data is processed so that the volume is reduced within a transition period including a boundary point between the first moving image segment and the second moving image segment. Therefore, it is possible to generate content that has a sense of unity between changes in the moving image and changes in the sound while suppressing the influence on the moving image represented by the moving image data.

[0078] In a specific example (Aspect 5) of Aspect 4, the transition period is a period ending at the boundary point, and the sound data is processed such that the volume decreases from the start point to the end point of the transition period and increases at the end point of the transition period. In the above aspect, the volume decreases from the start point to the end point (boundary point) of the transition period and increases at the end point of the transition period (the start point of the second moving image section). Therefore, it is possible to generate content in which the volume decreases over time towards the end point of the first moving image section and sound is played at a sufficient volume as the second moving image section begins.

[0079] In a specific example (aspect 6) of aspect 4, the transition period is a period ending at the boundary point, and the sound data is processed such that the volume decreases at the start of the transition period and increases from the start to the end of the transition period. In the above aspect, the volume decreases at the start of the transition period and increases from the start to the end (boundary) of the transition period. Therefore, it is possible to generate content in which a first moving image segment switches to a second moving image segment while the volume increases over time.

[0080] An information processing system according to one aspect of the present disclosure is an information processing system that processes moving image data including a plurality of moving image segments and sound data including a plurality of sound segments, and includes a sound data processing unit that processes the sound data so as to switch from a first sound segment among the plurality of sound segments to a second sound segment other than the first sound segment at a boundary point between a first moving image segment among the plurality of moving image segments and a second sound segment subsequent to the first sound segment.An information processing system according to another aspect of the present disclosure is an information processing system that processes moving image data including a plurality of moving image segments and sound data representing sound, and includes a sound data processing unit that processes the sound data so as to reduce a volume within a transition period that includes a boundary point between a first moving image segment among the plurality of moving image segments and a second moving image segment subsequent to the first moving image segment.

[0081] A program according to one aspect of the present disclosure is a program for processing video data including a plurality of video segments and sound data including a plurality of sound segments, the program causing a computer to function as a sound data processing unit that processes the sound data so as to switch from a first sound segment among the plurality of sound segments to a second sound segment other than the first sound segment at a boundary point between a first video segment among the plurality of video segments and a second video segment subsequent to the first video segment.A program according to another aspect of the present disclosure is a program for processing video data including a plurality of video segments and sound data representing sound, the program causing a computer to function as a sound data processing unit that processes the sound data so as to reduce a volume within a transition period that includes a boundary point between a first video segment among the plurality of video segments and a second video segment subsequent to the first video segment. [Explanation of symbols]

[0082] 100...information system, 10...terminal device, 11, 21...control device, 12, 22...storage device, 13...playback device, 131...display device, 132...sound emission device, 20...editing system, 23...communication device, 51...material data acquisition unit, 52...video data processing unit, 53...sound data processing unit, 54...content providing unit, 55...playback control unit.

Claims

1. 1. An information processing method for processing moving image data including a plurality of moving image segments and sound data representing sound, comprising the steps of: The sound data is processed so that the volume is reduced within a transition period including a boundary point between a first moving image section and a second moving image section subsequent to the first moving image section among the plurality of moving image sections.

1. A computer-implemented information processing method, comprising: the transition period is a period ending at the boundary point, The sound data is processed such that the volume decreases from the start point to the end point of the transition period and increases at the end point of the transition period. Information processing methods.

2. 1. An information processing method for processing moving image data including a plurality of moving image segments and sound data representing sound, comprising the steps of: The sound data is processed so that the volume is reduced within a transition period including a boundary point between a first moving image section and a second moving image section subsequent to the first moving image section among the plurality of moving image sections.

1. A computer-implemented information processing method, comprising: the transition period is a period ending at the boundary point, The sound data is processed such that the volume decreases at the start point of the transition period and increases from the start point to the end point of the transition period. Information processing methods.

3. 1. An information processing system for processing video data including a plurality of video segments and sound data representing sound, comprising: a sound data processing unit that processes the sound data so that a volume of the sound is reduced within a transition period including a boundary point between a first moving image section and a second moving image section subsequent to the first moving image section among the plurality of moving image sections; the transition period is a period ending at the boundary point, The sound data processing unit processes the sound data such that the volume decreases from the start point to the end point of the transition period and the volume increases at the end point of the transition period. Information processing system.

4. 1. An information processing system for processing video data including a plurality of video segments and sound data representing sound, comprising: a sound data processing unit that processes the sound data so that a volume of the sound is reduced within a transition period including a boundary point between a first moving image section and a second moving image section subsequent to the first moving image section among the plurality of moving image sections; the transition period is a period ending at the boundary point, The sound data processing unit processes the sound data such that the volume decreases at the start point of the transition period and increases from the start point to the end point of the transition period. Information processing system.

5. A program for processing video data including a plurality of video segments and sound data representing sounds, the program comprising: functioning as a sound data processing unit that processes the sound data so as to reduce a volume within a transition period including a boundary point between a first moving image section and a second moving image section subsequent to the first moving image section among the plurality of moving image sections; the transition period is a period ending at the boundary point, The sound data processing unit processes the sound data such that the volume decreases from the start point to the end point of the transition period and the volume increases at the end point of the transition period. program.

6. A program for processing video data including a plurality of video segments and sound data representing sounds, the program comprising: functioning as a sound data processing unit that processes the sound data so as to reduce a volume within a transition period including a boundary point between a first moving image section and a second moving image section subsequent to the first moving image section among the plurality of moving image sections; the transition period is a period ending at the boundary point, The sound data processing unit processes the sound data such that the volume decreases at the start point of the transition period and increases from the start point to the end point of the transition period. program.

Images