Processing system and program
The processing system and program utilize marker information to control song transitions and character movements, addressing the lack of smooth playback transitions by integrating volume and equalizer adjustments, ensuring a seamless and engaging musical experience.
Patent Information
- Application Number
- JP2020047764
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2020-03-18
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2040-03-18
AI Technical Summary
Existing methods lack effective utilization of connection information for smoothly transitioning playback from one song to another, leading to potential discomfort during the transition.
A processing system and program that utilize marker information to control the mixing of preceding and succeeding songs, adjusting volume and equalizer settings, and character movement to ensure a smooth transition, taking into account song structure, genre, and external feedback.
Enables an appropriate and smooth playback transition between songs by effectively using connection information, allowing for varied and engaging character movements that reflect song characteristics and audience feedback.
Smart Images

Figure 0007588467000001 
Figure 0007588467000002 
Figure 0007588467000003
Abstract
Description
[Technical field]
[0001] The present invention relates to a processing system, a program, and the like. [Background technology]
[0002] Conventionally, a method for playing two songs consecutively, such as playing a preceding song and then playing a subsequent song from the middle of the preceding song, has been known. For example, Patent Document 1 discloses a playback device that plays two songs consecutively according to a command comment of the preceding song and a command comment of the subsequent song. Patent Document 2 discloses a playback device that detects the listener's feelings and mood and automatically selects a song, smoothly transitioning the playback from the middle of the preceding song to the subsequent song without any awkwardness. [Prior art documents] [Patent documents]
[0003] [Patent Document 1] JP 2005-331905 A [Patent Document 2] JP 2006-106103 A Summary of the Invention [Problem to be solved by the invention]
[0004] When a playback transition from a preceding song to a following song occurs, it is desirable to mix the preceding song and the following song. For example, during the mixing period, a smooth playback transition without a sense of incongruity can be realized by gradually decreasing the volume of the preceding song while gradually increasing the volume of the following song. In order to transition from a preceding song to a following song while mixing them, connection information for connecting the preceding song and the following song is required. However, until now, no method has been proposed for effectively utilizing such connection information or for realizing an appropriate and smooth playback transition using the connection information.
[0005] According to the present disclosure, it is possible to provide a processing system, a program, etc. that can effectively utilize connection information between a preceding song and a succeeding song, or that can realize an appropriate and smooth transition of musical sound playback using the connection information. [Means for solving the problem]
[0006] One aspect of the present disclosure relates to a processing system including a music information storage unit that stores music information for a plurality of music pieces, a marker information storage unit that stores marker information, a sound processing unit that performs a playback process of music pieces based on the music information, a character processing unit that controls the movement of a character, and a display processing unit that performs a display process of an image of the character, the sound processing unit performing a mixing process of the preceding music piece and the subsequent music piece according to the marker information when switching from the preceding music piece to the subsequent music piece, and the character processing unit controlling the movement of the character during a mixing period of the preceding music piece and the subsequent music piece according to the marker information when switching from the preceding music piece to the subsequent music piece. Another aspect of the present disclosure relates to a program that causes a computer to function as each of the above units, or a computer-readable information storage medium having the program stored therein.
[0007] According to one aspect of the present disclosure, a music piece is played back based on music information stored in a music information storage unit, and a character's movement is controlled to display an image of the character. When switching from a previous music piece to a subsequent music piece, a mixing process of the previous music piece and the subsequent music piece according to the marker information is performed, and the character's movement is controlled during the mixing period according to the marker information. In this way, it is possible to effectively use the marker information that serves as connection information between the previous music piece and the subsequent music piece, or to realize an appropriate and smooth transition of the playback of musical tones using the marker information that serves as connection information between the previous music piece and the subsequent music piece.
[0008] In another aspect of the present disclosure, the character processing unit may change the behavior of the character depending on at least one of the mixing pattern of the preceding song and the subsequent song, the mix mode for playing multiple songs, the analysis results of external information, and the song composition of the song.
[0009] In this way, the character's movements will vary depending on the mixing pattern, mix mode, the results of analysis of external information, or the musical composition of the song, making it possible to realize character movement control that reflects the mixing pattern, mix mode, the results of analysis of external information, or the musical composition.
[0010] In another aspect of the present disclosure, the sound processing unit may change at least one of the mixing pattern of the preceding song and the subsequent song, the mix mode for playing multiple songs, and the marker information used for mixing based on the character information.
[0011] In this way, it becomes possible to perform mixing processing of the preceding music piece and the succeeding music piece by using a mixing pattern, mix mode or marker information according to the character information, thereby realizing music mixing in various forms.
[0012] In one aspect of the present disclosure, the mixing pattern is at least one of a volume adjustment pattern and an equalizer adjustment pattern in mixing the preceding song and the subsequent song, and the mix mode may be a basic mix mode, a short mix mode, or a long mix mode.
[0013] In this way, it is possible to realize character movement control that reflects the mixing volume adjustment pattern or equalizer adjustment pattern, basic mix mode, short mix mode or long mix mode, or to change the volume adjustment pattern or equalizer adjustment pattern, or these mix modes, depending on the character information.
[0014] In another aspect of the present disclosure, the marker information storage unit may store information of a marker corresponding to a start timing of the mixing period and information of a marker corresponding to an end timing of the mixing period.
[0015] In this way, it becomes possible to set the start and end timings of the mixing period using the marker information, perform mixing processing of the preceding music piece and the succeeding music piece, and control the movement of the character during the mixing period.
[0016] In one aspect of the present disclosure, the markers in the marker information may correspond to the song structure of the song, and the sound processing unit may identify a period during which the preceding song is the jth song structure (j is an integer greater than or equal to 1) and the subsequent song is the kth song structure (k is an integer greater than or equal to 1) based on the marker information, and perform the mixing process with that period as the mixing period.
[0017] In this way, when the preceding song has a jth song composition and the succeeding song has a kth song composition, a mixing process of the jth song composition and the kth song composition can be performed to switch the song being played from the preceding song to the succeeding song.
[0018] In another aspect of the present disclosure, the marker information storage unit may store the marker information such that, if the song is of a first genre, markers are set for the song's structure in a first setting pattern, and, if the song is of a second genre, markers are set for the song's structure in a second setting pattern different from the first setting pattern.
[0019] In this way, it becomes possible to mix the preceding and succeeding songs and control the movement of characters during the mixing period according to the information of markers that are set in the composition of the songs in different setting patterns depending on the genre of the songs.
[0020] Another aspect of the present disclosure relates to a processing system including a music information storage unit that stores music information for a plurality of music pieces, a marker information storage unit that stores marker information, and a sound processing unit that performs playback processing of music pieces based on the music information, wherein the marker information storage unit stores the marker information in which markers are set in a music structure of a music piece in a first setting pattern when the music piece is in a first genre, and in which markers are set in a music structure of a music piece in a second setting pattern different from the first setting pattern when the music piece is in a second genre, and the sound processing unit performs mixing processing of the preceding music piece and the following music piece in a mixing period set according to the marker information when switching from the preceding music piece to the following music piece. Another aspect of the present disclosure relates to a program that causes a computer to function as each of the above-mentioned units, or a computer-readable information storage medium that stores the program.
[0021] According to one aspect of the present disclosure, a music piece is played back based on music information stored in a music information storage unit. Marker information is set in different setting patterns according to the genre of the music piece, and a mixing process is performed between the preceding music piece and the following music piece during a mixing period set according to the marker information. Therefore, it is possible to realize an appropriate and smooth transition of the playback of musical tones by using the marker information, which is connection information between the preceding music piece and the following music piece.
[0022] In addition, one aspect of the present disclosure may include a display processing unit that performs display processing to allow a user to visually recognize the sound waveform of the preceding song, the sound waveform of the subsequent song, and the timing of the markers in the marker information.
[0023] By performing such display processing, the user can easily switch from the preceding song to the following song, or mix the preceding song and the following song, by looking at the sound waveforms of the preceding song and the following song and the marker images.
[0024] In another aspect of the present disclosure, when the amount of deviation between the timing of a user's mixing operation and the timing of a marker in the marker information is within a predetermined range, the sound processing unit may perform a process of correcting the timing of the mixing operation to match the timing of the marker.
[0025] In this way, even if the user lacks the skill to perform operations at the appropriate timing, it is possible to mix or switch music at the timing set by the markers.
[0026] In another aspect of the present disclosure, the sound processor may perform an evaluation process of the user's mixing operation based on an amount of deviation between the timing of the user's mixing operation and the timing of the marker.
[0027] In this way, when a user performs a mixing operation while looking at the markers, it becomes possible to evaluate whether or not the timing of the mixing operation was appropriate, making it possible to provide the user with the enjoyment of performing a mixing operation.
[0028] In addition, in one aspect of the present disclosure, the marker information storage unit may store information of a first marker that sets a permitted timing for starting mixing when the song is the preceding song, information of a second marker that sets a permitted timing for ending mixing when the song is the preceding song, information of a third marker that sets a permitted timing for starting mixing when the song is the following song, and information of a fourth marker that sets a permitted timing for ending mixing when the song is the following song.
[0029] By using such information on the first marker, second marker, third marker, and fourth marker, it is possible to appropriately set the mixing period of the preceding song and the following song, and realize a playback transition from the preceding song to the following song while mixing the preceding song and the following song.
[0030] In another aspect of the present disclosure, the sound processing unit may determine a timing for starting playback of the subsequent song based on the second marker of the preceding song and the fourth marker of the subsequent song, and may determine a timing for starting the mixing process based on the first marker of the preceding song and the third marker of the subsequent song.
[0031] In this way, the start and end timings of the mixing period can be determined using the information of the first to fourth markers, and a playback transition from the preceding song to the subsequent song can be realized while performing a mixing process of the preceding song and the subsequent song during the mixing period.
[0032] In one aspect of the present disclosure, the marker information storage unit stores information of a first marker set of the first to fourth markers used in a first mix mode and information of a second marker set of the first to fourth markers used in a second mix mode, and the sound processing unit may perform the mixing process in the first mix mode during the mixing period set in accordance with the information of the first marker set, and perform the mixing process in the second mix mode during the mixing period set in accordance with the information of the second marker set.
[0033] In this way, when a mix mode such as basic mix mode, short mix mode or long mix mode is set, by using a marker set corresponding to the set mix mode from among multiple marker sets set for the song, it becomes possible to perform mixing processing using markers appropriate for that mix mode. [Brief description of the drawings]
[0034] [Figure 1] FIG. 1 is a block diagram showing an example of the configuration of a processing system according to an embodiment of the present invention. [Diagram 2] An explanatory diagram of a virtual DJ character. [Diagram 3] FIG. 2 is a diagram showing an example of DJ equipment operated by a character. [Figure 4] 4 is a flowchart illustrating a processing example of the present embodiment. [Diagram 5] 11A to 11C are diagrams illustrating character movement control according to a mixing pattern. [Figure 6] 11 is a flowchart for explaining processing for controlling music playback and character movements according to a mix mode. [Figure 7] FIG. 13 is an explanatory diagram of a method of using external input information. [Figure 8] 11A to 11C are explanatory diagrams of control of character movements in accordance with the musical composition of a piece of music. [Figure 9] 11A to 11C are explanatory diagrams of control of character movements in accordance with the musical composition of a piece of music. [Figure 10] 11 is a flowchart for explaining a process for changing a mixing pattern, a mix mode, and marker information to be used based on character information. [Figure 11] FIG. 11 is an explanatory diagram of a mixing process using markers corresponding to the start and end timings of a mixing period. [Figure 12] 4 is a flowchart illustrating a processing example of the present embodiment. [Figure 13] FIG. 13 is a diagram showing an example of a DJ screen. [Figure 14] FIG. 4 is an explanatory diagram of a specific example of a marker used in the present embodiment. [Figure 15] 11 is a flowchart illustrating a mixing process using a marker. [Figure 16] Figures 16(A), 16(B), and 16(C) are explanatory diagrams about song structures in the genres of house music, EDM, and anime songs. [Figure 17] An illustration of the marker setting patterns and mixing process for house music. [Figure 18] An explanation of marker setting patterns and mixing processes in EDM. [Figure 19] An explanatory diagram of the marker setting patterns and mixing process for anime songs. [Figure 20] FIG. 13 is an explanatory diagram of a technique using a permission marker to start mixing. [Figure 21] An explanatory diagram of the marker sets for each mix mode. [Figure 22] 11 is a flowchart of a process based on the amount of deviation between a mixing operation and a marker timing. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0035] The present embodiment will be described below. Note that the present embodiment described below does not unduly limit the contents of the claims. Furthermore, not all of the configurations described in the present embodiment are necessarily essential configurations.
[0036] 1. Processing System Fig. 1 is a block diagram showing an example of the configuration of the processing system (information processing device, attraction device, game device) of this embodiment. Note that the processing system of this embodiment is not limited to the configuration shown in Fig. 1, and various modifications are possible, such as omitting some of the components or adding other components.
[0037] The operation unit 160 is for a user (operator, player) to input various operation information (input information). The operation unit 160 can be realized by, for example, a keyboard, operation buttons, directional indicator keys, a joystick, a lever, or a touch panel display.
[0038] The storage unit 170 stores various information. The storage unit 170 functions as a work area for the processing unit 100, the communication unit 196, and the like. Programs and data required for executing the programs are held in the storage unit 170. The functions of the storage unit 170 can be realized by a semiconductor memory (DRAM, VRAM), a hard disk drive (HDD), a solid state drive (SSD), an optical disk device, or the like. The storage unit 170 includes a music information storage unit 172, a marker information storage unit 174, a motion data storage unit 175, a character information storage unit 176, an object information storage unit 177, and a drawing buffer 178. The music information storage unit 172 stores information on music to be played. For example, the music information storage unit 172 stores additional data such as sound data of the music, genre, artist name, or album name. The marker information storage unit 174 stores various information on markers. For example, the setting information such as a setting pattern of markers for music is stored in association with the music. The motion data storage unit 175 stores motion data for performing motion reproduction processing of model objects such as characters. Motion reproduction processing is also called animation processing. The character information storage unit 176 stores various information about characters. For example, it stores parameter information such as a character's level or ability, and information representing the character's attributes such as personality and status. The object information storage unit 177 stores various information (shape information, position information, etc.) about objects that constitute model objects such as characters and objects that constitute a virtual space. The drawing buffer 178 is a buffer that can store image information in pixel units, such as a frame buffer or work buffer.
[0039] The information storage medium 180 is a computer-readable medium for storing programs, data, and the like. The information storage medium 180 can be realized by an optical disk (DVD, BD, CD), a HDD, or a semiconductor memory (ROM). The processing unit 100 performs various processes of this embodiment based on the programs (data) stored in the information storage medium 180. That is, the information storage medium 180 stores programs (programs for causing the computer to execute the processes of each unit) for causing a computer (a device equipped with an input device, a processing unit, a storage unit, and an output unit) to function as each unit of this embodiment.
[0040] The display unit 190 outputs the image generated by this embodiment, and the function thereof can be realized by an LCD, an organic EL display, a CRT, a touch panel display, an HMD (Head Mounted Display), etc. The sound output unit 192 outputs the sound generated by this embodiment, and the function thereof can be realized by a speaker, a headphone, etc.
[0041] The I / F unit 194 (interface unit) performs interface processing with the portable information storage medium 195, and this function can be realized by an ASIC for I / F processing or the like. The portable information storage medium 195 is a storage device for the user to store various information, and retains the storage of this information even when the power supply is cut off. The portable information storage medium 195 can be realized by an IC card (memory card), a USB memory, a magnetic card, or the like.
[0042] The communication unit 196 communicates with the outside (other devices) via a wired or wireless network, and the function can be realized by hardware such as a communication ASIC or a communication processor, or by communication firmware.
[0043] The programs (data) for causing a computer to function as each unit of this embodiment may be distributed from an information storage medium held by a server (host device) to the information storage medium 180 (or the storage unit 170) via the network and the communication unit 196. Such use of the information storage medium by the server can also be included within the scope of this embodiment.
[0044] The processing unit 100 (processor) performs sound processing, character processing, display processing, and the like, based on operation input information from the operation unit 160, programs, and the like.
[0045] Each process of the present embodiment performed by each unit of the processing unit 100 can be realized by a processor (a processor including hardware). For example, each process of the present embodiment can be realized by a processor that operates based on information such as a program, and a memory that stores information such as a program. The functions of each unit of the processor may be realized by individual hardware, or the functions of each unit may be realized by integrated hardware. For example, the processor includes hardware, and the hardware may include at least one of a circuit that processes a digital signal and a circuit that processes an analog signal. For example, the processor may be configured with one or more circuit devices (e.g., ICs, etc.) mounted on a circuit board, or one or more circuit elements (e.g., resistors, capacitors, etc.). The processor may be, for example, a CPU (Central Processing Unit). However, the processor is not limited to a CPU, and various processors such as a GPU (Graphics Processing Unit) or a DSP (Digital Signal Processor) can be used. The processor may also be a hardware circuit using an ASIC. The processor may also include an amplifier circuit, a filter circuit, etc. that process an analog signal. The memory (storage unit 170) may be a semiconductor memory such as an SRAM or DRAM, or a register. Alternatively, the memory may be a magnetic storage device such as a hard disk drive (HDD), or an optical storage device such as an optical disk drive. For example, the memory stores computer-readable instructions, and the instructions are executed by the processor to realize the processing of each part of the processing unit 100. The instructions here may be a set of instructions constituting a program, or may be instructions that instruct the hardware circuitry of the processor to operate.
[0046] The processing unit 100 includes a sound processing unit 102, a character processing unit 110, and a display processing unit 120. As described above, each process of the present embodiment executed by each of these units can be realized by a processor (or a processor and a memory). Note that various modifications are possible, such as omitting some of these components (each unit) or adding other components.
[0047] The sound processing unit 102 performs various kinds of sound processing. For example, the sound processing unit 102 performs a process of playing back music. Alternatively, the sound processing unit 102 may perform digital signal processing such as a filter process on sound data, sound analysis processing such as a voice analysis process, or various kinds of determination processing related to sound.
[0048] The character processing unit 110 performs various processes for the character. For example, the character processing unit 110 performs a process of moving a character, which is a model object, in a virtual space, which is an object space. For example, the character processing unit 110 performs a process of moving a character based on motion data stored in the motion data storage unit 175. Motion reproduction processing based on motion data is also called animation processing. For example, the character processing unit 110 performs a control process of moving or moving a character in a virtual space (motion, animation) based on a program (movement / movement algorithm) and various data (motion data, character information). Specifically, a simulation process is performed to sequentially obtain the character's motion information (position or rotation angle of a part object) and movement information (position, rotation angle, speed, or acceleration) for each frame (for example, 1 / 60 seconds). Note that a frame is a unit of time for performing character motion processing and image generation processing.
[0049] The display processing unit 120 performs processing for displaying an image on the display unit 190. For example, a drawing process is performed based on the results of various processes performed by the processing unit 100, thereby generating an image and displaying it on the display unit 190. Specifically, geometry processes such as coordinate transformation (world coordinate transformation, camera coordinate transformation), clipping processing, perspective transformation, or light source processing are performed, and drawing data (position coordinates of vertices of primitive surfaces, texture coordinates, color data, normal vectors, or alpha values, etc.) are created based on the processing results. Then, based on this drawing data (primitive surface data), an object (one or more primitive surfaces) after perspective transformation (after geometry processing) is drawn in a drawing buffer 178 (a buffer that can store image information in pixel units, such as a frame buffer or a work buffer). As a result, an image seen from a virtual camera in a virtual space is generated. The drawing process performed by the display processing unit 120 can be realized by vertex shader processing, pixel shader processing, etc.
[0050] For example, in this embodiment, the processing unit 100 can perform game processing, virtual space setting processing, virtual camera setting processing, moving body processing, etc. Game processing is, for example, processing to start a game when a game start condition is satisfied, processing to progress the started game, processing to end a game when a game end condition is satisfied, or processing to calculate game results. Virtual space setting processing is, for example, processing to arrange and set various objects (objects composed of primitive surfaces such as polygons, free-form surfaces, or subdivision surfaces) representing display objects such as moving bodies (cars, people, robots, trains, airplanes, ships, monsters, or animals), maps (terrain), buildings, spectator seats, courses (roads), items, trees, walls, and water surfaces in a virtual space. For example, in the virtual space setting processing, a process is performed in which the position and rotation angle (synonymous with orientation and direction) of an object in a world coordinate system are determined, and the object is arranged at that position (X, Y, Z) at that rotation angle (rotation angle around the X, Y, and Z axes). Specifically, the object information storage unit 177 of the storage unit 170 stores object information, which is information such as the position, rotation angle, moving speed, or moving direction of an object (part object) in a virtual space, in association with an object number. In the virtual space setting process, for example, an update process of the object information, which is virtual space information, is performed for each frame. In addition, the virtual camera setting process is a process of setting the position and direction of the virtual camera in the virtual space. For example, in the virtual camera setting process, a process of setting the position and direction of the virtual camera in the virtual space is performed based on information on the position and direction of a moving object such as a character, or setting information on the position and direction of the virtual camera. The moving object process is a process of calculating the movement information of the moving object in the virtual space. For example, in the moving object process, a process for moving the moving object in the virtual space is performed. In addition, the processing unit 100 performs a sound generation process. For example, the processing unit 100 performs a sound generation process based on the results of various processes in the processing unit 100. Specifically, music, sound effects, or voices are generated and output to the sound output unit 192.
[0051] The processing system of this embodiment includes a music information storage unit 172, a marker information storage unit 174, a sound processing unit 102, a character processing unit 110, and a display processing unit 120, as shown in FIG.
[0052] The music information storage unit 172 stores music information. For example, the music information storage unit 172 stores music information for a plurality of music pieces. For example, the music information storage unit 172 stores music information for a plurality of music pieces to be played. The music information is, for example, sound data of the music pieces. Alternatively, the music information may include additional data such as the genre, artist name, or album name. Based on this music information, the music is played back.
[0053] The marker information storage unit 174 stores marker information. The marker information is, for example, information that sets the timing of switching from a preceding song to a following song. For example, a marker in the marker information serves as a marker for the timing of switching from the preceding song to the following song. Specifically, the marker information is information that sets a mixing period of the preceding song and the following song. For example, a marker in the marker information serves as a marker for the timing of the mixing period. The preceding song is a song that precedes the current song in the playback transition when switching from the current song to the next song, and the following song is the next song that follows the preceding song.
[0054] The character processing unit 110 controls the movement of the character. For example, the character processing unit 110 performs a motion replay process for replaying the motion of the character as the movement control of the character. The motion replay process is a process for changing the posture of the skeleton of the model object to move the model object, and is also called animation processing. Specifically, the character processing unit 110 performs the motion process of the character based on the motion data stored in the motion data storage unit 175. The motion data can be obtained, for example, by performing motion capture of the movement of the actor who performs the character's movement. There are various types of motion capture methods, such as optical, inertial sensor, mechanical, and magnetic. The motion data specifies the posture of the skeleton corresponding to the character (actor), and the skeleton is composed of a plurality of joints and bones specified by the joints, and has a parent-child structure that is a hierarchical structure. Specifically, the motion data can include information such as the rotation angles α, β, and γ of the child bone relative to the parent bone around the X-axis, Y-axis, and Z-axis. Alternatively, the motion data may include information such as the scaling rate of the bone in the X-axis, Y-axis, and Z-axis directions. For example, the motion data is realized by matrix information in which rotation angles α, β, γ, etc. are matrix elements.
[0055] The display processing unit 120 performs display processing of the image of the character. For example, the display processing unit 120 performs rendering processing of the character, which is a three-dimensional model object, and generates a virtual space image including the image of the character, which is an image seen from a virtual camera in a virtual space, which is an object space.
[0056] Then, when switching from the preceding song to the following song, the sound processing unit 102 performs a mixing process of the preceding song and the following song according to the marker information. For example, the sound processing unit 102 sets a mixing period according to the marker information, and performs a mixing process of the preceding song and the following song in the set mixing period. For example, a marker corresponding to the start timing of the mixing period and a marker corresponding to the end timing of the mixing period are set in the preceding song and the following song, and the mixing period is specified using these markers. Then, the sound processing unit 102 performs a mixing process of the preceding song and the following song in the mixing period specified by these markers. In the mixing process, a volume adjustment is performed in the mixing period, for example, gradually decreasing the volume of the preceding song while gradually increasing the volume of the following song. For example, at the start timing of the mixing period, the volume of the following song is, for example, zero, and only the preceding song is reproduced and output. Then, at the end timing of the mixing period, the volume of the preceding song becomes, for example, zero, and only the following song is reproduced and output. During the mixing period, an equalizer adjustment is performed to attenuate or amplify a specific frequency band, such as a low frequency band. For example, the low frequency band of the succeeding song is kept attenuated for a predetermined period from the start of the mixing period, and at the end of the mixing period or a short period including the end of the mixing period, the low frequency band of the succeeding song is amplified to return it to its original state.
[0057] Then, when switching from the preceding music piece to the subsequent music piece, the character processing unit 110 controls the character's behavior during the mixing period of the preceding music piece and the subsequent music piece according to the marker information. For example, the character processing unit 110 controls the character's behavior so that the character performs a mixing behavior during the mixing period set by the marker information. For example, the character processing unit 110 sets a mixing period according to the marker information, and controls the character's behavior so that the character performs a mixing behavior during the set mixing period. Specifically, a marker corresponding to the start timing of the mixing period and a marker corresponding to the end timing of the mixing period are set in the preceding music piece and the subsequent music piece, and the character processing unit 110 specifies the mixing period using these markers to control the character's behavior. For example, the character processing unit 110 controls the character's behavior in conjunction with the mixing process of the sound processing unit 102 performed during the mixing period. For example, the character is made to move with a mixing motion. For example, the character processing unit 110 makes the character move with a mixing motion by correcting the character's motion. Specifically, the character's motion is corrected so that the character's hands perform a mixing operation. The mixing operation is, for example, a fader operation, an equalizer operation, etc. As one example, an inverse kinematics process is performed to move the character's hand (fingers) so that the character performs a mixing operation, thereby making the character perform a mixing motion.
[0058] The character processing unit 110 also changes the behavior of the character according to at least one of the mixing pattern of the preceding and following music pieces, the mix mode in the reproduction of multiple music pieces, the analysis result of the external information, and the music composition of the music pieces. For example, when the mixing pattern is the first mixing pattern, the character is made to move in a motion pattern corresponding to the first mixing pattern, and when the mixing pattern is the second mixing pattern, the character is made to move in a motion pattern corresponding to the second mixing pattern. When the mix mode in the reproduction of multiple music pieces is the first mix mode, the character is made to move in a motion pattern corresponding to the first mix mode, and when the mix mode is the second mix mode, the character is made to move in a motion pattern corresponding to the second mix mode. Alternatively, the processing system acquires external information and analyzes the external information, and the character processing unit 110 changes the behavior of the character based on the analysis result of the external information. The analysis result of the external information is, for example, the analysis result of an Internet vote, the analysis result of the sound of the external environment such as cheers, or the analysis result of the movement of the audience. Also, when the musical composition of the music piece is a first musical composition, the character is made to move in a motion pattern corresponding to the first musical composition, and when the musical composition of the music piece is a second musical composition, the character is made to move in a motion pattern corresponding to the second musical composition.
[0059] Furthermore, the sound processor 102 changes at least one of the mixing pattern of the preceding music piece and the following music piece, the mix mode in reproducing a plurality of music pieces, and the marker information used in mixing, based on the character information. The character information is, for example, information of a judgment condition set for the character. For example, when the character information is the first information, the sound processor 102 performs a mixing process of the preceding music piece and the following music piece in the first mixing pattern, and when the character information changes from the first information to the second information, the sound processor 102 performs a mixing process of the preceding music piece and the following music piece in the second mixing pattern. When the character information is the first information, the sound processor 102 performs a plurality of music pieces in the first mix mode, and when the character information changes from the first information to the second information, the sound processor 102 performs a plurality of music pieces in the second mix mode. When the character information is the first information, the sound processor 102 uses the first type of marker information as the marker information used in the mixing process and the display process, and when the character information changes from the first information to the second information, the sound processor 102 uses the second type of marker information. Character information is, for example, parameters of the character's level or ability, information on algorithms such as a movement algorithm set for the character, information on the character's personality or attributes, etc. Character ability is, for example, the character's ability in sense of rhythm, ability in performance technique, or ability to respond on the spot, for example, ability as a DJ. Character information is, for example, set in advance for the character, and is, for example, stored in the character information storage unit 176 in association with the character.
[0060] The mixing pattern is at least one of a volume adjustment pattern and an equalizer adjustment pattern in mixing the preceding and succeeding songs. The volume adjustment pattern is, for example, a volume adjustment pattern using a fader in mixing. For example, the volume adjustment pattern is a pattern of the volume adjustment degree and volume curve of the preceding and succeeding songs during the mixing period. The equalizer adjustment pattern is, for example, an equalizer adjustment pattern in mixing. For example, the equalizer adjustment pattern is a pattern of the adjustment degree and adjustment curve of components of each frequency band such as bass, midrange, or treble by the equalizer.
[0061] The mix mode is a basic mix mode, a short mix mode, or a long mix mode. The short mix mode is a mix mode in which the following music piece is mixed at an earlier timing than the basic mix mode with respect to the preceding music piece. For example, the short mix mode is a mix mode in which the number of music pieces played within a predetermined performance time is greater than that of the basic mix mode. The long mix mode is a mix mode in which the following music piece is mixed at a later timing than that of the basic mix mode with respect to the preceding music piece. For example, the long mix mode is a mix mode in which the number of music pieces played within a predetermined performance time is less than that of the basic mix mode. The mix mode may be set for multiple music pieces at once, or the mix mode may be set differently for each music piece. For example, a first music piece group for the basic mix mode, a second music piece group for the short mix mode, and a third music piece group for the long mix mode may be prepared, and the music information for these may be stored in the music information storage unit 172. The mix mode may be set automatically by the processing system, or may be set by the user using the operation unit 160 or the like.
[0062] Furthermore, the marker information storage unit 174 stores information on a marker corresponding to the start timing of a mixing period and information on a marker corresponding to the end timing of the mixing period. The marker corresponding to the start timing of a mixing period is, for example, a marker associated with the start timing, and may be, for example, a marker that sets the start timing of the mixing period, or may be a marker that sets the permitted timing for starting mixing. The marker corresponding to the end timing of a mixing period is, for example, a marker associated with the end timing, and may be a marker that sets the end timing of the mixing period, or may be a marker that sets the permitted timing for ending mixing.
[0063] Also, the markers in the marker information are associated with the song structure of the music piece. For example, if the music piece has a first song structure to an mth song structure (m is an integer of 2 or more), the marker is associated with, for example, the i-th song structure (1≦i≦m) among the first song structure to the mth song structure. For example, the marker is associated with the beginning timing, the end timing, or the middle timing of the song structure. Note that the markers are set, for example, in units of bars. Also, for the song structure of the music piece, a marker when the song is a preceding song and a marker when the song is a succeeding song may be associated. The song structure is a block of multiple bars, such as 2 bars, 4 bars, 8 bars, 16 bars, or 32 bars, and the music piece is composed of multiple song structures, which are blocks, connected together. There are various song structures, such as an intro, an intro chorus, a build-up, a drop, a break, a chorus, a verse, an A melody, a B melody, or an outro. The sound processor 102 then identifies a period during which the preceding music piece has the jth music piece structure (j is an integer equal to or greater than 1) and the succeeding music piece has the kth music piece structure (k is an integer equal to or greater than 1) based on the marker information, and performs mixing processing with the period as a mixing period. For example, when a marker for the preceding music piece is set to the jth music piece structure of the preceding music piece and a marker for the succeeding music piece is set to the kth music piece structure of the succeeding music piece, the sound processor 102 identifies a mixing period based on these markers. As the marker for the preceding music piece, for example, a marker corresponding to the start timing of the jth music piece structure of the preceding music piece or a marker corresponding to the end timing of the jth music piece structure can be assumed. As the marker for the succeeding music piece, for example, a marker corresponding to the start timing of the kth music piece structure of the succeeding music piece or a marker corresponding to the end timing of the kth music piece structure can be assumed. Then, the sound processor 102 sets a mixing period using these markers, and performs mixing processing of the preceding music piece and the succeeding music piece. For example, the jth song structure of the preceding song and the kth song structure of the succeeding song are song structures that are specified according to, for example, the genre of the songs, so that there is little sense of incongruity or unnaturalness when the two song structures are mixed together.
[0064] The processing system of this embodiment includes a music information storage unit 172 that stores music information for a plurality of music pieces, a marker information storage unit 174 that stores marker information, and a sound processing unit 102 that performs playback processing of music pieces based on the music information. The marker information storage unit 174 stores marker information in which, when the music piece is a first genre, a marker is set for the music composition of the music piece in a first setting pattern, and in which, when the music piece is a second genre, a marker is set for the music composition of the music piece in a second setting pattern different from the first setting pattern. The first setting pattern and the second setting pattern are patterns regarding which music composition of the music piece a marker is set for. For example, the first setting pattern and the second setting pattern are patterns regarding which music composition of the music piece a marker for the start timing of the mixing period and a marker for the end timing of the mixing period are set for. For example, the music composition of the music piece to which a marker is set differs between when the music piece is a first genre and when the music piece is a second genre.
[0065] Then, when switching from the preceding song to the following song, the sound processor 102 performs mixing processing of the preceding song and the following song in a mixing period set according to the marker information. For example, the sound processor 102 sets a mixing period based on marker information in which markers are set in the song composition in different setting patterns depending on the genre, and performs mixing processing of the preceding song and the following song in the set mixing period. For example, a marker corresponding to the start timing of the mixing period and a marker corresponding to the end timing of the mixing period are set in the song composition of the preceding song and the following song in different setting patterns depending on the genre, and the mixing period is set using these markers. Then, the sound processor 102 performs mixing processing of the preceding song and the following song in the mixing period set by these markers.
[0066] Furthermore, the display processing unit 120 performs display processing for allowing the user to visually recognize the sound waveform of the preceding music piece, the sound waveform of the subsequent music piece, and the timing of the markers in the marker information. For example, the display processing unit 120 performs processing to display an image of the sound waveform of the preceding music piece and an image of the sound waveform of the subsequent music piece on the display unit 190. The display processing unit 120 then performs processing to display an image of a marker for the preceding music piece on the image of the sound waveform of the preceding music piece at the position of the timing set by the marker. The display processing unit 120 also performs processing to display an image of a marker for the subsequent music piece on the image of the sound waveform of the subsequent music piece at the position of the timing set by the marker.
[0067] Furthermore, when the amount of deviation between the timing of the mixing operation by the user and the timing of the marker in the marker information is within a predetermined range, the sound processor 102 performs a process of correcting the timing of the mixing operation to match the timing of the marker. That is, when the user performs a mixing operation such as starting playback of the subsequent music piece, adjusting the volume with a fader, or adjusting the equalizer with an equalizer using the operation unit 160, the sound processor 102 acquires the timing of the mixing operation. Then, the amount of deviation between the timing of the mixing operation and the timing set by the marker in the marker information is measured. Then, when the amount of deviation is within a predetermined allowable range, the sound processor 104 performs a process of correcting the timing of the mixing operation to match the timing of the marker. That is, the sound processor 104 performs a mixing process of the preceding music piece and the subsequent music piece, assuming that the mixing operation by the user was performed at the timing set by the marker.
[0068] The sound processor 104 also performs an evaluation process of the user's mixing operation based on the amount of deviation between the timing of the user's mixing operation and the timing of the marker. For example, when the amount of deviation is within a predetermined range, the sound processor 104 performs an evaluation process of giving a good evaluation (good, excellent) to the mixing operation, and when the amount of deviation exceeds the predetermined range, the sound processor 104 performs an evaluation process of giving a bad evaluation (bad) to the mixing operation. Then, the display processor 120 performs a display process according to the result of this evaluation process. For example, the display processor 120 performs a display process of notifying the user of the evaluation result (good, excellent, bad) of the evaluation of the user's mixing operation. For example, the display processor 120 displays information on the game results based on the result of the evaluation process. Alternatively, the display processor 120 performs a display process for presentation based on the result of the evaluation process. For example, the display processor 120 performs a display process for generating a presentation effect. Specifically, the display processor 120 performs a display process for making a presentation effect display appear around the character.
[0069] The marker information storage unit 174 also stores information on a first marker that sets the timing at which mixing is permitted to start when the song is a preceding song, and information on a second marker that sets the timing at which mixing is permitted to end when the song is a preceding song. The marker information storage unit 174 also stores information on a third marker that sets the timing at which mixing is permitted to start when the song is a following song, and information on a fourth marker that sets the timing at which mixing is permitted to end when the song is a following song. That is, for one song, the marker information storage unit 174 stores information on the first marker and the second marker that are used when the song is a preceding song, and information on the third marker and the fourth marker that are used when the song is a following song, in association with the song. Here, the marker that sets the timing at which mixing is permitted to start is a marker that indicates that mixing is permitted (allowed) at the timing of the transition to the permitted timing.
[0070] The sound processor 102 then determines the playback start timing of the subsequent song based on the second marker of the preceding song and the fourth marker of the subsequent song. Then, the sound processor 102 determines the start timing of the mixing process based on the first marker of the preceding song and the third marker of the subsequent song. For example, the playback start timing of the subsequent song is determined so that the timing of the second marker of the preceding song and the timing of the fourth marker of the subsequent song coincide with each other. Then, after the playback start of the subsequent song, the mixing process of the preceding song and the subsequent song is started at a timing when it is determined that mixing is permitted by the first marker and the third marker.
[0071] The marker information storage unit 174 also stores information on a first marker set of the first to fourth markers used in the first mix mode, and information on a second marker set of the first to fourth markers used in the second mix mode. That is, the marker information storage unit 174 stores information on the first to fourth markers for the first mix mode as information on the first marker set, and stores information on the first to fourth markers for the second mix mode as information on the second marker set. The first mix mode and the second mix mode are the basic mix mode, the short mix mode, and the long mix mode, respectively. In the first mix mode, the sound processing unit 102 performs a mixing process in a mixing period set according to the information on the first marker set. For example, the mixing period is set using the first to fourth markers of the first marker set, and a mixing process is performed on the preceding music piece and the succeeding music piece in the set mixing period. In the second mix mode, the sound processing unit 102 performs a mixing process in a mixing period set according to the information on the second marker set. For example, a mixing period is set using the first to fourth markers of the second marker set, and mixing processing of the preceding music piece and the succeeding music piece is performed in the set mixing period.
[0072] The processing system of this embodiment is realized by an information processing device. That is, in this case, the information processing device includes the processing system of this embodiment. The information processing device is a device that processes various information such as music information and marker information, and is realized by a processor or the like. The information processing device is, for example, a terminal device such as a personal computer, a game device, or a portable information processing device. Alternatively, the information processing device may be a server device. The processing system may also be realized by distributed processing of the terminal device and the server device. In this case, the processing system is realized by multiple information processing devices. In this case, the terminal device may only perform information input processing, reception processing, transmission processing, display processing, etc.
[0073] 2. Method of the Present Invention Next, the method of this embodiment will be described in detail.
[0074] 2.1 Virtual DJ character In this embodiment, as shown in FIG. 2, a virtual DJ (Disc Jockey) character CH appears, and an image of the character CH can be displayed. For example, an object of a DJ device 10 is placed in the virtual space, and the character CH performs a performance of playing music continuously by operating the DJ device 10. For example, the character CH performs a DJ performance by operating the fader, equalizer, turntable, etc. of the DJ device 10 while jumping and dancing to the rhythm and tempo of the music. An image of the character CH performing such a DJ performance is displayed on a large monitor installed in a club or live venue. Also, the sound of the music played one after another by the operation of the character CH is output from the speakers installed in the venue. Alternatively, the image of the DJ character CH in FIG. 2 is distributed to a terminal device such as a user's smartphone or PC (personal computer) via a network such as the Internet. The user enjoys the performance of the DJ character CH by watching the image of the DJ character CH on the display of the smartphone or PC and listening to the music output from the speakers.
[0075] The character CH moves based on motion data recorded by motion capture of the performer who corresponds to the character CH. For example, the character CH moves, such as jumping and dancing, to the rhythm and tempo of the music based on the motion data. When switching from the preceding music to the succeeding music, a mixing operation is performed in which the preceding music is faded out and the succeeding music is faded in, and during this mixing period, the character CH performs a mixing operation. For example, the character CH performs a mixing operation to operate the fader, equalizer, etc. of the virtual DJ device 10. For example, the hand of the character CH performs an operation to move the fader or turn the knob of the equalizer.
[0076] FIG. 3 is a diagram showing a simple example of a DJ device 10 (DJ player, mixing device) operated by a character CH. The faders 11 and 12 adjust the volume of the channels 1 and 2, respectively. In this embodiment, the preceding music is played on the channel 1 (deck 1), and the following music is played on the channel 2 (deck 2). The crossfader 14 adjusts the volume balance between the channels 1 and 2. The equalizers 21 and 22 equalize the music on the channels 1 and 2, respectively, and have knobs for adjusting the treble, midrange, and bass. The trim adjustment knobs 23 and 24 adjust the gain of the channels 1 and 2, respectively. In addition, the play buttons 31 and 32 and the cue buttons 33 and 34 enable the playback, stop, and cue operations of the music on the channels 1 and 2. In addition, the turntables 35 and 36 enable the operation of effects such as scratching. In addition, the sound waveforms of the music are displayed on the displays 41 and 42. 3 is a schematic diagram of an example of the DJ equipment 10, and the DJ equipment 10 may be provided with, for example, a pitch controller, an effect control knob, or a performance pad in addition to the components shown in FIG 3. The DJ equipment 10 is composed of objects in a virtual space, similar to the character CH, and a virtual space image including an image of the character CH and the DJ equipment 10 is generated using the objects of the character CH and the DJ equipment 10.
[0077] 2.2 Mixing process and character movement control according to marker information FIG. 4 is a flow chart for explaining the processing of this embodiment. First, the processing system reads out the music information of the music to be played (step S1). For example, the music information such as the sound data of the music is read out from the music information storage unit 172 in FIG. 1. For example, if a playlist is created for the performance of the DJ, the music information of the multiple music pieces in the playlist is read out. The marker information of the music is also read out (step S2). For example, the marker information associated with the music to be played is read out from the marker information storage unit 174 in FIG. 1. Then, the processing system starts playing the music in accordance with the character's action of playing the music (step S3). For example, when starting to play the first music piece, the character performs a play operation action (animation) such as pressing the play button, so music playback starts in accordance with this play operation action. Specifically, music playback starts, in which multiple music pieces are played one after another in succession while mixing the music pieces together. Then, the motion of the character is controlled based on the information of the music being played (step S4). For example, the motion of the virtual DJ character CH shown in FIG. 2 is controlled to dance and provoke in accordance with the tempo and rhythm of the music being played.
[0078] Next, the processing system judges whether or not it is the mixing period of the preceding music and the following music based on the marker information (step S5). For example, based on the marker information associated with a predetermined music composition of the preceding music or the marker information associated with a predetermined music composition of the following music, it judges whether or not it is the mixing period of the preceding music and the following music. Specifically, based on the marker information corresponding to the start timing of the mixing period or the marker information corresponding to the end timing of the mixing period, it judges whether or not it is the mixing period. Then, when it is the mixing period, it performs a mixing process of the preceding music and the following music (steps S6, S7). For example, it performs a mixing process in which the preceding music is faded out while the following music is faded in. Also, when it is the mixing period, it performs a motion control of the character based on the mixing process (step S8). For example, it moves the character with a motion for mixing. For example, it makes the character perform the motion of the mixing operation by performing a motion correction of the character's hand (fingers) by inverse kinematics processing or the like with respect to the reference motion at that time. That is, the character is made to perform the mixing operation of operating the faders 11, 12, the crossfader 14, the equalizers 21, 22, etc. in Fig. 3. Then, it is determined whether the mixing period has ended (step S9), and if it has ended, the process returns to step S4. This causes the mixing process to be performed with the subsequent song at that time as the preceding song and the next song as the subsequent song. Then, multiple songs are played back continuously until the playback of all songs is completed.
[0079] In this embodiment, as shown in steps S1 to S4 of FIG. 4, a music piece is played based on the music information, and the character's movement is controlled, and the character's image is displayed as shown in FIG. 2. Then, as shown in steps S5 to S9 of FIG. 4, when switching from the preceding music piece to the following music piece, a mixing process of the preceding music piece and the following music piece is performed based on the marker information, and the character's movement is controlled in the mixing period of the preceding music piece and the following music piece based on the marker information. For example, a mixing period is specified based on the marker information, and the preceding music piece and the following music piece are mixed in the specified mixing period, and the character's movement is controlled to perform a motion corresponding to the mixing operation. For example, the mixing process of the preceding music piece and the following music piece in the mixing period and the character's movement control are controlled to be linked based on the marker information, and the character's image is displayed as shown in FIG. 2. In this way, when switching from the preceding music piece to the following music piece, it is possible to realize image generation that effectively uses the marker information that serves as connection information between the preceding music piece and the following music piece. For example, the marker information can be used to realize mixing processing of the preceding and succeeding songs, and an image of a character performing actions such as mixing operations can be generated. In addition, it is possible to realize a smooth transition in the playback of musical sounds by using the marker information, which is connection information between the preceding and succeeding songs. For example, by mixing the preceding and succeeding songs during the mixing period set by the marker information, it is possible to smoothly transition the playback from the preceding song to the succeeding song.
[0080] In this embodiment, the character's behavior is changed according to at least one of the mixing pattern of the preceding and following songs, the mix mode in playing multiple songs, the analysis result of external information, and the song composition of the songs. In this way, the character's behavior changes in various ways according to the mixing pattern, mix mode, the analysis result of external information, or the song composition of the songs, making it possible to realize character behavior control that reflects the mixing pattern, mix mode, the analysis result of external information, or the song composition.
[0081] For example, in Fig. 5, the mixing patterns PMX, PMY, PMZ, etc. of the preceding and succeeding songs correspond to the character's movement patterns MTX, MTY, MTZ, etc. If the mixing pattern is PMX, the character moves according to the movement pattern MTX during the mixing period. Similarly, if the mixing pattern is PMY or PMZ, the character moves according to the movement patterns MTY or MTZ, respectively, during the mixing period.
[0082] Here, the mixing pattern is, for example, at least one of a volume adjustment pattern and an equalizer adjustment pattern in mixing a preceding song and a succeeding song. For example, suppose that the mixing pattern PMX is a pattern in which the volume of the preceding song is gradually decreased while the volume of the succeeding song is gradually increased during the mixing period. In this case, the character's motion pattern MTX corresponding to the mixing pattern PMX is a motion pattern in which the fader 11 for the preceding song in FIG. 2 is gradually lowered downward while the fader 12 for the succeeding song is gradually raised upward. Specifically, the character in FIG. 2 performing a basic motion that moves in accordance with the rhythm and tempo is made to perform the motion of the motion pattern MTX in which the left hand of the character in FIG. 2 moves to the position of the fader 11, the right hand moves to the position of the fader 12, the left hand moves downward, and the right hand moves upward. The motion processing of such a motion pattern MTX can be realized by inverse kinematics processing with the left hand and right hand as end effectors.
[0083] Also, suppose that the mixing pattern PMY is a pattern that adjusts the volume like the mixing pattern PMX, and also adjusts the bass with an equalizer. For example, suppose that the equalizer adjustment pattern attenuates the bass of the following music piece so that the bass drum and the like cannot be heard at the beginning of the mixing period, and increases the bass of the following music piece to return it to the original level at the end of the mixing period so that the bass drum can be heard. In this case, the character's motion pattern MTY corresponding to the mixing pattern PMY is a motion pattern in which the character turns the bass adjustment knob of the equalizer 22 for the following music piece while operating the faders 11, 12 or the crossfader 14. This motion pattern MTY can also be realized by the inverse kinematics processing described above.
[0084] Also, suppose that the mixing pattern PMZ is a pattern that adjusts the volume like the mixing pattern PMX and adjusts the midrange with an equalizer. For example, suppose that the preceding and following songs are vocal songs, and the equalizer adjustment pattern attenuates the midrange of the following song so that the vocals cannot be heard at the beginning of the mixing period, and increases the midrange of the following song to return to the original level at the end of the mixing period so that the vocals can be heard. In this case, the character's movement pattern MTZ corresponding to the mixing pattern PMZ is a movement pattern in which the adjustment knob for the midrange of the equalizer 22 for the following song is turned while operating the faders 11, 12 or the crossfader 14. This movement pattern MTZ can also be realized by the inverse kinematics processing described above. Alternatively, for example, if the mixing pattern is one in which effects are added by the turntables 35, 36 in FIG. 3, the character may be moved in a movement pattern in which the turntables 35, 36 are operated. In this way, it becomes possible to make the character move during the mixing period in conjunction with the volume and equalizer adjustment patterns in the mixing pattern, making it possible to generate a virtual space image that looks as if a real DJ is operating the DJ equipment 10.
[0085] The character's movements during the mixing period are not limited to the inverse kinematics processing method, and various motion processing methods can be used. For example, motion data for the mixing period may be prepared separately, and when the mixing period begins, the motion data may be switched to the motion data for the mixing period.
[0086] In this embodiment, the character's behavior is changed according to the mix mode. The mix mode is a basic mix mode, a short mix mode, or a long mix mode. The short mix mode is a mode in which the following music piece is mixed at an earlier timing than the basic mix mode with respect to the preceding music piece, and the long mix mode is a mode in which the following music piece is mixed at a later timing than the preceding music piece. FIG. 6 is a flowchart for explaining the process of changing the character's behavior according to the mix mode.
[0087] First, the processing system judges whether the mix mode is set (changed), and if the basic mix mode is set, the marker set for the basic mix mode as illustrated in FIG. 21 described later is used to play music in the basic mix mode, and the character's movement control corresponding to the basic mix mode is performed (steps S11, S12, S13, S14). On the other hand, if the short mix mode is set, the marker set for the short mix mode is used to play music in the short mix mode, and the character's movement control corresponding to the short mix mode is performed (steps S15, S16, S17). If the short mix mode is not set, the processing system assumes that the long mix mode is set, and the marker set for the long mix mode is used to play music in the long mix mode, and the character's movement control corresponding to the long mix mode is performed (steps S18, S19).
[0088] For example, in the short mix mode, music pieces are switched one after another at a faster timing within the playing time than in the basic mix mode. Therefore, in the short mix mode, the character is moved in a motion pattern of faster or more intense movements than in the basic mix mode. For example, as the motion data for the short mix mode, motion data that is faster or more intense than the motion data for the basic mix mode is prepared, and motion is played. On the other hand, in the long mix mode, music pieces are switched slowly at a slower timing within the playing time than in the basic mix mode. Therefore, in the long mix mode, the character is moved in a motion pattern of slower or more gentle movements than in the basic mix mode. For example, as the motion data for the long mix mode, motion data that is slower or more gentle than the motion data for the basic mix mode is prepared, and motion is played. In this way, it is possible to move the character in a motion corresponding to the basic mix mode, short mix mode, or long mix mode, and it is possible to display an image of the character performing a motion suitable for each mix mode.
[0089] In this embodiment, the character's behavior is changed according to the analysis result of the external information. For example, in FIG. 7, the terminal devices TM1 to TMn are communicatively connected to the processing system of this embodiment via the network 510. The network 510 is, for example, a communication path using the Internet. The users of the terminal devices TM1 to TMn then vote on the Internet for the performance of the DJ character and the music played, and the results of this Internet voting are analyzed as external information. The processing system then changes the character's behavior according to the analysis result of the external information, which is the result of the Internet voting. For example, if the analysis result shows that many users support the performance of the character, the character's movement becomes an exciting movement or a special movement that will be a privilege for the user. Alternatively, the character may select the next music to be played by the DJ according to the analysis result of the external information, which is the result of the Internet voting. The analysis result of the external information may also be the analysis result of the sound of the external environment or the movement of the audience. For example, when a video of a DJ character as shown in FIG. 2 is displayed on a monitor at a club or a live venue, the cheers of the audience and the movement of the audience excited by the performance of the character are analyzed. Then, the character is made to move in a manner that reflects the analysis result. For example, if the cheers are loud or the audience makes an excited move, the character's movements will become more excited accordingly, or the character will perform a special move that will be a special bonus for the user. In this way, the character's movements will reflect the analysis results of external information, making it possible to generate realistic virtual space images in which it is as if a real DJ were performing in response to the cheers and movements of the audience.
[0090] In this embodiment, the character's movement is changed according to the musical composition of the music. For example, as described later, music in the genre of house music (techno house) has a musical composition of intro, build-up A, drop A, break, build-up B, drop B, and outro, as shown in FIG. 8. In this case, character movement patterns MTA, MTB, MTC, MTD, MTE, MTF, and MTG are associated with these musical compositions. For example, when the musical composition of the intro of the music is being played, the character moves according to the movement pattern MTA. Similarly, when the musical composition of build-up A, drop A, break, build-up B, drop B, and outro is being played, the character moves according to the movement patterns MTB, MTC, MTD, MTE, MTF, and MTG, respectively. For example, when each musical composition of FIG. 8 is being played, the character's motion is played using motion data corresponding to the movement patterns MTA to MTG. In this way, the character can be made to move according to the musical composition of the music, and the character's movement linked to the music can be realized. For example, when a song is being built up, the character CH may move in a motion pattern such as waving his hands or vigorously moving his body, as shown in Fig. 9. This makes it possible to liven up the audience at a club or live concert venue, or a user watching a video stream on a terminal device. Note that the motion patterns MTA to MTG do not all need to be different from each other, and it is sufficient that at least two of them are different motion patterns.
[0091] In this embodiment, at least one of the mixing pattern of the preceding and succeeding songs, the mix mode in playing a plurality of songs, and the marker information used for mixing is changed based on the character information. In this way, it becomes possible to perform mixing processing of the preceding and succeeding songs by the mixing pattern, mix mode, or marker information according to the character information, and it becomes possible to realize various types of song mixing. Figure 10 is a flowchart explaining the processing of changing the mixing pattern, mix mode, and marker information to be used based on the character information.
[0092] First, character information such as parameters such as the character's level, personality information, or control algorithm information is read (step S21). That is, character information is read from the character information storage unit 176 in FIG. 1. The character information is, for example, information on judgment conditions previously set for the character. Next, based on the character information, a mixing pattern for the preceding music piece and the following music piece, a mix mode for playing multiple music pieces, or marker information used for mixing is set (step S22). Then, based on the mixing pattern, mix mode, or marker information thus set, a mixing process for the preceding music piece and the following music piece is executed (step S23). This makes it possible to execute the mixing process using a mixing pattern, mix mode, or marker information that reflects the character information.
[0093] For example, based on character information, a mixing pattern is set, which is at least one of the volume adjustment pattern and the equalizer adjustment pattern in mixing the preceding music piece and the following music piece described in FIG. 5. The character information is, for example, information such as the character's level, ability, and other parameters, the character's personality, or the character's control algorithm. For example, when the character's level or ability is high, a more complex and difficult mixing pattern is set. Or a mixing pattern is set according to the character's personality. For example, when the character has a detailed and meticulous personality, a complex and meticulous mixing pattern is set. Or, the mixing pattern is made different according to the control algorithm for controlling the character. Also, which of the basic mix mode, short mix mode, and long mix mode is used is made different according to the character's level, ability, and other parameters, personality, or control algorithm. For example, when the character's level or ability is low, only the basic mix mode can be used, but when the character's level or ability is high, the short mix mode and long mix mode can also be used. Also, when the character has a fun-loving personality, the short mix mode is used, and when the character has a quiet and calm personality, the long mix mode is used. Also, the marker information to be used is made different depending on the character's level, ability parameters, personality, or control algorithm. For example, when there are multiple marker sets as shown in FIG. 21 described later, which marker set is used is made different depending on the character's information. Alternatively, the marker information set for which of multiple song configurations of a song is to be used is determined depending on the character's level, ability parameters, personality, or control algorithm. For example, if the character has a catchy personality, the marker information set for the part of the catchy song configuration is used, and mixing processing is performed when that song configuration is reached. Also, if the character has a quiet and calm personality, the marker information set for the part of the quiet type song configuration is used, and mixing processing is performed when that song configuration is reached.
[0094] Furthermore, the marker information storage unit 174 stores marker information corresponding to the start timing of the mixing period and marker information corresponding to the end timing of the mixing period. In this way, the start timing and end timing of the mixing period can be set by the marker information stored in the marker information storage unit 174, and mixing processing of the preceding music and the succeeding music can be performed, and the behavior of the character during the mixing period can be controlled.
[0095] For example, in FIG. 11, the preceding song has a song structure of CXA-CXG, and the succeeding song has a song structure of CYA-CYG. In the preceding song, a marker MSX corresponding to the start timing of the mixing period and a marker MEX corresponding to the end timing of the mixing period are set for the song structure CXE. In the succeeding song, a marker MSY corresponding to the start timing of the mixing period and a marker MEY corresponding to the end timing of the mixing period are set for the song structure CYC. In this way, for example, the timing of the markers MSX and MSY is set as the start timing of the mixing period, and the timing of the markers MEX and MEY is set as the end timing of the mixing period, and the mixing process of the preceding song and the succeeding song can be performed. For example, the fade-out of the preceding song starts with the timing of the marker MSX as the start timing, and the fade-out of the preceding song ends with the timing of the marker MEX as the end timing. Also, the fade-in of the succeeding song starts with the timing of the marker MSY as the start timing, and the fade-in of the succeeding song ends with the timing of the marker MEY as the end timing. In this case, by identifying the timing when the markers MSX and MSY match, or the timing when the markers MEX and MEY match, it is possible to identify the timing to start playing the subsequent music piece. For example, playback of the subsequent music piece is started so that the timing of the markers MSX and MSY matches, or playback of the subsequent music piece is started so that the timing of the markers MEX and MEY matches.
[0096] In this embodiment, the markers in the marker information correspond to the song configurations of the songs. A period during which the preceding song has the jth song configuration (j is an integer equal to or greater than 1) and the following song has the kth song configuration (k is an integer equal to or greater than 1) is identified based on the marker information, and mixing processing is performed for the period. In this way, when the preceding song has the jth song configuration and the following song has the kth song configuration, mixing processing of the jth song configuration and the kth song configuration is performed, and the played song can be switched from the preceding song to the following song.
[0097] For example, in FIG. 11, the markers MSX and MEX of the marker information are associated with the composition CXE of the preceding song, and the markers MSY and MEY are associated with the composition CYC of the following song. Then, a period in which the preceding song is the composition CXE (jth composition) and the following song is the composition CYC (kth composition) is specified based on the markers MSX, MEX, MSY, and MEY, and the composition CXE of the preceding song and the composition CYC of the following song are mixed during the mixing period, which is set as a mixing period starting from the timing of the markers MSX and MSY and ending from the timing of the markers MEX and MEY. For example, during the mixing period, the composition CXE of the preceding song and the composition CYC of the following song are mixed. For example, during the mixing period, the composition CXE of the preceding song is faded out while the composition CYC of the following song is faded in. In this case, for example, among the song configurations CXA to CXG of the preceding songs, the song configuration CXE is the song configuration that will not sound unnatural or strange even if it is mixed with the song configuration CYC of the following song. Similarly, among the song configurations CYA to CYG of the following songs, the song configuration CYE is the song configuration that will not sound unnatural or strange even if it is mixed with the song configuration CXE of the preceding song. Therefore, by mixing the song configuration CXE of the preceding song with the song configuration CYC of the following song using the markers MSX, MEX, MSY, and MEY, it is possible to make the playback transition from the preceding song to the following song sound less unnatural or strange.
[0098] 2.3 Marker information for genre-specific patterns In this embodiment, a method is adopted in which marker information of a set pattern according to the genre of the music is used. That is, a mixing process of a preceding music piece and a succeeding music piece is performed using marker information set in the music composition of the music piece with a different set pattern according to the genre of the music piece. Note that the method of this embodiment using such marker information is not limited to the application example in which a virtual DJ character appears as described above, but can also be applied to a system that does not perform, for example, character movement control or display processing.
[0099] Specifically, the processing system of this embodiment using such marker information includes a music information storage unit 172, a marker information storage unit 174, and a sound processing unit 102. The marker information storage unit 174 stores marker information in which markers are set in the music structure in a first setting pattern when the music is in a first genre, and in which markers are set in the music structure in a second setting pattern when the music is in a second genre. That is, it stores information on markers that are set in the music structure of a music piece in different setting patterns depending on the genre. The sound processing unit 102 performs mixing processing of the preceding music piece and the following music piece during a mixing period set according to the marker information when switching from the preceding music piece to the following music piece. FIG. 12 is a flowchart for explaining an example of processing of this embodiment in this case.
[0100] First, the processing system reads out the music information of the music to be played (step S31). Then, the processing system reads out the marker information of the markers set in the music structure of the music in different setting patterns according to the genre (step S32). That is, the marker information storage unit 174 stores marker information in which markers are set in the music structure in a first setting pattern when the music is in a first genre, and in which markers are set in the music structure in a second setting pattern when the music is in a second genre, and reads out the marker information of such setting patterns. Then, the processing system starts playing the music (step S33). For example, the processing system starts playing music in which multiple music pieces are played back one after another in succession while mixing the music pieces together.
[0101] Next, the processing system judges whether or not it is the mixing period of the preceding and subsequent songs based on the marker information. For example, based on the marker information set in the composition of the preceding and subsequent songs in different setting patterns according to the genre, it judges whether or not it is the mixing period of the preceding and subsequent songs. Then, when it is the mixing period, it performs mixing processing of the preceding and subsequent songs (steps S35, S36). For example, it performs mixing processing in which the preceding song is faded out while the subsequent song is faded in. Then, it judges whether or not the mixing period has ended (step S37), and if it has ended, it returns to step S34. As a result, mixing processing is performed in which the subsequent song at that time is the preceding song and the next song is the subsequent song. Then, multiple songs are continuously played until the playback of all songs is completed.
[0102] FIG. 13 is a diagram showing an example of a DJ screen. This DJ screen is displayed on the display of a terminal device such as a PC, a smartphone, or a game device. A user performs DJ operations while viewing this DJ screen. Faders 51 and 52 adjust the volume of channels 1 and 2, respectively. Crossfader 54 adjusts the volume balance between channels 1 and 2. Equalizers 61 and 62 equalize the music of channels 1 and 2, respectively, and are provided with knobs for adjusting treble, midrange, and bass. Trim adjustment knobs 63 and 64 adjust the gain of channels 1 and 2, respectively. Sync buttons 65 and 66 are buttons for synchronizing the tempo of the music of channels 1 and 2. Play buttons 71 and 72 and cue buttons 73 and 74 allow the music of channels 1 and 2 to be played, stopped, and queued. A playlist of music playback is displayed in a display area 80. Display area 81 displays the sound waveform of the preceding song on channel 1, and display area 82 displays the sound waveform of the subsequent song on channel 2. Markers MK1 and MK2 set in the preceding song are displayed in display area 81 in association with the sound waveform of the preceding song. Markers MK3 and MK4 set in the subsequent song are displayed in association with the sound waveform of the subsequent song. Display areas 83 and 84 display enlarged sound waveforms of the preceding song and subsequent song, respectively.
[0103] 14 is an explanatory diagram of markers MK1, MK2, MK3, and MK4 of the marker information of this embodiment. The markers MK1, MK2, MK3, and MK4 are the first marker, the second marker, the third marker, and the fourth marker, respectively.
[0104] For example, marker MK1 is a marker that sets the permitted timing for starting mixing when a song is a preceding song. In other words, mixing is permitted to start at a timing after the timing of marker MK1. Marker MK1 can also be called a mix-out marker for the preceding song.
[0105] The marker MK2 is a marker that sets the mixing end timing when the song is a preceding song. The marker MK2 indicates, for example, the end timing of the song structure of the preceding song that is the subject of mixing. The marker MK2 can also be called a cut marker of the preceding song.
[0106] The marker MK3 is a marker that sets the permitted timing for starting mixing when the song is a subsequent song. In other words, mixing is permitted to start at a timing after the timing of the marker MK3. The marker MK3 can also be called a mix-in marker for the subsequent song.
[0107] The marker MK4 is a marker that sets the mixing end timing when the song is a subsequent song. For example, the marker MK4 indicates the timing of a complete transition from the preceding song to the subsequent song. The marker MK4 can also be called a transition marker that indicates a complete transition to the subsequent song. In this embodiment, the marker information storage unit 174 stores information on these markers MK1, MK2, MK3, and MK4 in association with each song of a plurality of songs, for example.
[0108] By using such information of the markers MK1, MK2, MK3, and MK4, it is possible to appropriately set the mixing period of the preceding song and the following song, and realize the playback transition of switching from the preceding song to the following song while mixing the preceding song and the following song. For example, by using the markers MK1 and MK3, it becomes possible for the processing system (computer) or the user to recognize the timing at which the start of mixing is permitted for each of the preceding song and the following song. For example, when the processing system automatically switches from the preceding song to the following song, the markers MK1 and MK3 enable the processing system to recognize the timing at which the start of mixing is permitted. On the other hand, when the user manually switches from the preceding song to the following song, the markers MK1 and MK3 enable the user to recognize the timing at which the start of mixing is permitted. Furthermore, by using the markers MK2 and MK4, it becomes possible for the processing system or the user to recognize the timing at which the mixing period ends and the transition from the preceding song to the following song is made for each of the preceding song and the following song.
[0109] In this embodiment, the playback start timing of the subsequent song is determined based on the marker MK2 of the preceding song and the marker MK4 of the subsequent song. Then, the start timing of the mixing process is determined based on the marker MK1 of the preceding song and the marker MK3 of the subsequent song. In this way, the start and end timings of the mixing period are determined using the information of the markers MK1, MK2, MK3, and MK4, and a playback transition that automatically switches from the preceding song to the subsequent song can be realized while performing the mixing process of the preceding song and the subsequent song during the mixing period. FIG. 15 is a flowchart explaining an example of the processing of this embodiment in this case.
[0110] First, playback of the preceding song is started (step S41). Then, based on the marker MK2 set in the preceding song and the marker MK4 set in the following song, the playback start timing of the following song is determined (step S42). That is, the playback start timing of the following song to be played after the preceding song is determined using the markers MK2 and MK4 that set the mixing end timing of the preceding song and the following song. For example, the playback start timing of the following song is determined so that the timing of the marker MK2 of the preceding song and the timing of the marker MK4 of the following song are the same. Then, playback of the following song is started at the determined playback start timing (step S43).
[0111] Next, the start timing of the mixing process is determined based on the marker MK1 of the preceding song and the marker MK3 of the following song (step S44). For example, the start of the mixing process is permitted on the condition that both markers MK1 and MK3 that set the permitted timing of the mixing start are in the permitted state. Then, when it is determined that the mixing start timing has arrived, the mixing process of the preceding song and the following song is started (steps S45 and S46). Note that the tempos (BPM) of the preceding song and the following song are automatically adjusted. Then, it is determined whether or not the mixing end timing has arrived, and when the end timing has arrived, the mixing process is ended (step S47). For example, when the timing of the markers MK2 and MK4 of the preceding song and the following song arrives, the mixing process is ended. In this way, according to the process of FIG. 15, the start timing and end timing of the mixing process are automatically determined using the markers MK1, MK2, MK3, and MK4, and the mixing process can be automatically executed during the mixing period.
[0112] Next, a specific example of markers that are set in different setting patterns depending on the genre will be described in detail.
[0113] Figure 16(A) is a representative example of a song structure in the house music (techno house) genre. House music has a song structure of intro, build-up A (first build-up), drop A (first drop), break, build-up B (second build-up), drop B (second drop), and outro. The number of bars in these song structures is 16, 16, 32, 16, 16, 32, and 32, respectively, as shown in Figure 16(A). Note that build-up A may also be a break.
[0114] Figure 16(B) is a representative example of a song structure in the EDM (future bass) genre. EDM has a song structure of intro, verse A (first verse), build-up A (first build-up), chorus A (first chorus), break, verse B (second verse), build-up B (second build-up), chorus B (second chorus), and outro. The number of bars in these song structures is 8, 8, 8, 16, 8, 8, 16, 16, and 8, respectively, as shown in Figure 16(B).
[0115] Figure 16(C) is a typical example of the song structure of the anime song genre known as anisong. Anime songs have a song structure of intro chorus, intro, A melody, B melody, break, chorus A (first chorus), break, chorus B (second chorus), and outro. The number of bars in these song structures is 8, 4, 8, 8, 2, 16, 2, 16, and 2, respectively, as shown in Figure 16(C).
[0116] Fig. 17 is an explanatory diagram of the setting patterns of the markers MK1 to MK4 and the mixing process in house music described in Fig. 16(A). As shown in Fig. 17, the setting patterns of the markers MK1 to MK4 are different between the basic mix mode, the short mix mode, and the long mix mode.
[0117] As shown in FIG. 17, in the basic mix mode of house music, the markers MK1 and MK2 described in FIG. 14 are set for the build-up B of the preceding track, and the markers MK3 and MK4 are set for the build-up A of the following track. As a result, at the timing of the beginning (the beginning of the bar) of the build-up B of the preceding track and the build-up A of the following track set by the markers MK1 and MK3, the mixing process of the preceding track and the following track starts, and the playback transition starts. That is, the fade-out of the preceding track and the fade-in of the following track start. Then, at the timing of the end of the build-up B and the build-up A set by the markers MK2 and MK4, the mixing process ends. That is, the preceding track is completely transitioned to the following track. For example, since the build-ups A and B are both climaxing parts, by mixing the build-up B of the preceding track and the build-up A of the following track, a playback transition from the preceding track to the following track with less awkwardness can be realized.
[0118] In the short mix mode of house music, markers MK1 and MK2 are set for the break of the preceding song, and markers MK3 and MK4 are set for the build-up A of the following song. As a result, the mixing process starts at the timing of the break of the preceding song and the beginning of the build-up A of the following song, and the playback transition begins. Then, the mixing process ends at the timing of the break of the preceding song and the end of the build-up A of the following song, and a complete transition to the following song occurs. That is, in the basic mix mode, the playback transition from the preceding song to the following song is performed after the break of the preceding song ends, but in the short mix mode, in order to maintain the groove, the playback transition to the following song is performed without the listener hearing the break of the preceding song, which is a gentle part.
[0119] In the long mix mode for house music, markers MK1 and MK2 are set for the outro of the preceding track, and markers MK3 and MK4 are set for the intro and build-up A of the succeeding track. This causes the mixing process to start at the beginning of the outro of the preceding track and the intro of the succeeding track, and playback transition begins. Then, at the end of the outro of the preceding track and the build-up A of the succeeding track, the mixing process ends and a complete transition to the succeeding track occurs. This makes it possible to switch tracks at longer time intervals in the long mix mode compared to the short mix mode and basic mix mode.
[0120] FIG. 18 is an explanatory diagram of the setting patterns of the markers MK1 to MK4 in the EDM described with reference to FIG. 16(B) and the mixing process.
[0121] As shown in Fig. 18, in the basic mix mode of EDM, markers MK1 and MK2 are set for the outro of the preceding song, and markers MK3 and MK4 are set for the intro of the succeeding song. As a result, the mixing process of the preceding song and the succeeding song starts at the beginning of the outro of the preceding song and the intro of the succeeding song, and the playback transition begins. Then, at the end of the outro of the preceding song and the intro of the succeeding song, the mixing process ends and a complete transition to the succeeding song occurs.
[0122] In EDM short mix mode, markers MK1 and MK2 are set for the break of the preceding song, and markers MK3 and MK4 are set for the intro of the following song. As a result, the mixing process starts at the timing of the break of the preceding song and the beginning of the intro of the following song, and the playback transition begins. Then, at the timing of the end of the break of the preceding song and the intro of the following song, the mixing process ends and the transition to the following song is complete. As a result, in short mix mode, songs can be switched at shorter time intervals than in basic mix mode.
[0123] FIG. 19 is an explanatory diagram of the setting patterns of the markers MK1 to MK4 in the anime song described in FIG. 16(C) and the mixing process.
[0124] As shown in Fig. 19, in the basic mix mode of anime songs, markers MK1 and MK2 are set to the last bar of the outro of the preceding song, and markers MK3 and MK4 are set to the last bar of the intro of the following song. This causes the mixing process of the preceding song and the following song to start at the beginning of the last bar of the outro of the preceding song and the intro of the following song, and the playback transition begins. Then, at the end of the last bar of the outro of the preceding song and the intro of the following song, the mixing process ends and a complete transition to the following song occurs.
[0125] In the short mix mode of anime songs, markers MK1 and MK2 are set for the last bar of the chorus A of the preceding song, and markers MK3 and MK4 are set for the last bar of the intro of the following song. As a result, the mixing process starts at the beginning of the first bar of the end of the chorus A of the preceding song and the intro of the following song, and the playback transition begins. Then, the mixing process ends at the end of the first bar of the end of the chorus A of the preceding song and the intro of the following song, and the transition to the following song is complete. As a result, in the short mix mode, songs can be switched at shorter time intervals than in the basic mix mode and long mix mode. Also, in the long mix mode, the beginning of the following song comes from the end of the outro bar of the preceding song, and the playback of the following song begins.
[0126] FIG. 20 shows an example of a marker setting pattern and a mixing process in the basic mix mode of house music, similar to FIG. 17. In FIG. 17, the number of bars in the build-up A of the following music piece is 16, but in FIG. 20, the number of bars in the build-up A of the following music piece is 32. Therefore, if the mixing process is started at the timing of the marker MK3 set in the build-up A in this case, appropriate mixing process cannot be realized. In this respect, the markers MK3 and MK1 are markers that set the permission timing for starting mixing, as described in FIG. 14. Therefore, at the timing of the marker MK3, the marker MK1 of the preceding music piece is not in the permission timing, so the mixing process does not start. Then, as shown in FIG. 20, the mixing process starts at the timing when both the markers MK3 and MK1 are in the permission state. This makes it possible to realize appropriate mixing process even when the number of bars in the build-up B of the preceding music piece is different from the number of bars in the build-up A of the following music piece.
[0127] As described above, in this embodiment, the markers in the marker information are associated with the composition of the music. Then, as shown in Figs. 17 to 19, markers are set for the composition of the music in different setting patterns according to the genre of the music. For example, if the music is a first genre (e.g., house music), markers are set for the composition in the first setting pattern, and if the music is a second genre (e.g., EDM), markers are set for the composition in the second setting pattern. Then, using the information of the markers set in different setting patterns according to the genre of the music, a mixing process of the preceding music and the following music is performed. Therefore, by using the marker information, which is the connection information of the preceding music and the following music, it becomes possible to realize a playback transition of musical sounds appropriate for the genre of the music. For example, in house music, by setting the markers MK1 to MK4 in a setting pattern as shown in Fig. 17 and performing a mixing process, a playback transition appropriate for the house music genre can be realized. Also, in EDM, by setting the markers MK1 to MK4 in a setting pattern as shown in Fig. 18 and performing a mixing process, a playback transition appropriate for the EDM genre can be realized. For anime songs, by performing mixing processing by setting markers MK1 to MK4 in a setting pattern as shown in FIG. 19, it is possible to realize a playback transition appropriate for the anime song genre.
[0128] In this embodiment, as shown in FIG. 21, the marker information storage unit 174 stores information on a marker set for the basic mix mode, which is a marker set of markers MK1(N) to MK4(N) used in the basic mix mode, and a marker set for the short mix mode, which is a marker set of markers MK1(S) to MK4(S) used in the short mix mode. It also stores information on a marker set for the long mix mode, which is a marker set of markers MK1(L) to MK4(L) used in the long mix mode. For example, a marker set for the basic mix mode, a marker set for the short mix mode, and a marker set for the long mix mode are set for one song. That is, a plurality of marker sets, each of which is a set of four types of markers MK1 to MK4, are set for one song. In this way, when a mix mode such as the basic mix mode, short mix mode, or long mix mode is set, a marker set corresponding to the set mix mode is used from among a plurality of marker sets set for the song, so that the mixing process can be performed using markers appropriate for the mix mode. One of the basic mix mode, short mix mode, and long mix mode is the first mix mode, and the other is the second mix mode. Also, one of the marker set for the basic mix mode, the marker set for the short mix mode, and the marker set for the long mix mode is the first marker set, and the other is the second marker set.
[0129] In this embodiment, when a predetermined condition is satisfied, switching from a preceding song to a subsequent song or mixing of the preceding song and the subsequent song is performed, but this switching and mixing of songs may be realized fully automatically by a processing system (computer), or may be realized manually or semi-automatically requiring user operation. In the case of semi-automation, for example, an operation of matching the end timing of the preceding song and the subsequent song and an operation of starting playback of the subsequent song are automatically performed by the processing system. Volume adjustment using a fader and equalizer adjustment are manually performed by the user.
[0130] For example, in this embodiment, a DJ screen as shown in FIG. 13 is displayed on the display of the user's terminal device, and the user performs DJ operations to switch from the preceding music to the following music and mix the preceding music. In this case, display processing is performed so that the user can visually recognize the sound waveform of the preceding music, the sound waveform of the following music, and the timing of the markers in the marker information. For example, in FIG. 13, the sound waveform of the preceding music and images of markers MK1 and MK2 set on the preceding music are displayed in a display area 81. Also, the sound waveform of the following music and images of markers MK3 and MK4 set on the following music are displayed in a display area 82. By performing such display processing, the user can easily perform the operation of switching from the preceding music to the following music or the operation of mixing the preceding music and the following music by looking at the sound waveforms of the preceding music and the following music and the images of the markers MK1 to MK4. Therefore, for example, even if there is no special knowledge or skill regarding the operation of switching or mixing music, it becomes possible to realize switching or mixing of music in a way that the listener does not feel uncomfortable.
[0131] In addition, in this embodiment, when the amount of deviation between the timing of the mixing operation of the user and the timing of the marker in the marker information is within a predetermined range, a process may be performed to correct the timing of the mixing operation to match the timing of the marker. In this way, even if the user's skill is insufficient and the user's operation is not performed at the appropriate timing, it is possible to mix or switch songs at the timing set by the marker. This makes it possible to provide an operation interface that is highly convenient for the user. In addition, in this embodiment, an evaluation process of the user's mixing operation may be performed based on the amount of deviation between the timing of the mixing operation of the user and the timing of the marker. This makes it possible to evaluate whether the timing of the mixing operation was appropriate when the user performs the mixing operation while looking at the marker. Therefore, if the user performs the mixing operation at a good time, he or she can get a good evaluation, and becomes engrossed in the mixing operation. This makes it possible to give the user the fun of performing the mixing operation, and to realize an unprecedented DJ game. FIG. 22 is a flowchart showing an example of the processing of this embodiment in this case.
[0132] First, the processing system obtains the amount of deviation between the timing of the user's mixing operation and the timing of the marker (step S51). For example, the amount of deviation between the timing of the mixing operation and the timing of the marker can be measured by performing a monitoring process that samples the timing of the user's mixing operation at short time intervals. Then, based on the obtained amount of deviation, an evaluation process of the user's mixing operation is performed (step S52). For example, if the amount of deviation is small, the evaluation of the user's mixing operation is good (good, excellent), and if the amount of deviation is large, the evaluation of the user's mixing operation is bad (bad). Next, it is determined whether the amount of deviation is within a predetermined range (step S53). For example, it is possible to determine whether the amount of deviation of the user's mixing operation is within an acceptable range. If the amount of deviation is within the predetermined range, the timing of the mixing operation is corrected to match the timing of the marker (step S54). In this way, it is possible to assist the user in the mixing operation, and the user can appropriately mix and switch between the preceding and succeeding songs even if he or she does not have any special skills or knowledge regarding the mixing operation.
[0133] When the evaluation process shown in step S52 in FIG. 22 is performed, evaluations of the switching and mixing of songs may be tallied during the user's DJ time, and the tallied results may be presented to the user, or a special gift may be given to the user according to the tallied results. It is also desirable to perform a process of notifying the user of the evaluation results of the user's mixing operation. For example, an image with the words "good", "excellent", or "bad" written on it may be displayed. If the user's evaluation of the mixing operation is good, a process of generating a dramatic effect may be performed. For example, a process of displaying a sparkling effect display object around the character may be performed.
[0134] Although the present embodiment has been described in detail as above, it will be easily understood by those skilled in the art that many modifications are possible without substantially departing from the novel matters and effects of the present disclosure. Therefore, all such modifications are intended to be included in the scope of the present disclosure. For example, a term described at least once in the specification or drawings together with a different term having a broader meaning or synonymy may be replaced with that different term anywhere in the specification or drawings. In addition, sound playback processing, mixing processing, playback processing, character movement control processing, display processing, marker setting processing, etc. are not limited to those described in the present embodiment, and equivalent methods, processing, and configurations are also included in the scope of the present disclosure. [Explanation of symbols]
[0135] 10...DJ equipment, 11, 12...fader, 14...crossfader, 21, 22...Equalizer, 23, 24...Trim adjustment knob, 31, 32...Play button, 33, 34…cue button, 35, 36…turntable, 41, 42…display, 51, 52…fader, 54…crossfader, 61, 62…equalizer, 63, 64…Trim adjustment knob, 65, 66…Sync button, 71, 72…Play button, 73, 74...cue buttons, 80, 81, 82, 83, 84...display area, 100... processing unit, 102... sound processing unit, 104... sound processing unit, 110... character processing unit, 120: display processing unit, 160: operation unit, 170: storage unit, 172: music information storage unit, 174: marker information storage unit, 175: motion data storage unit, 176: character information storage unit, 177: object information storage unit, 178: drawing buffer, 180: information storage medium, 190: display unit, 192: sound output unit, 194...I / F unit, 195...portable information storage medium, 196...communication unit, 510...Network, CH...Character, CXA~CXG, CYA~CYG...Song composition, MK1, MK2, MK3, MK4, MEX, MEY, MSX, MSY... markers, MTA~MTG, MTX, MTY, MTZ...operation patterns, PMX, PMY, PMZ: mixing patterns, TM1 to TMn: terminal devices
Claims
1. a music information storage unit that stores music information for a plurality of music pieces; a marker information storage unit that stores the marker information; a sound processing unit that performs a playback process of a song based on the song information; a character processing unit that controls the movement of a character; a display processing unit that performs display processing of an image of the character; Including, The sound processing unit includes: When switching from the preceding music piece to the succeeding music piece, a mixing process is performed between the preceding music piece and the succeeding music piece in accordance with the marker information; The character processing unit: When switching from the preceding music piece to the succeeding music piece, controlling the movement of the character during a mixing period of the preceding music piece and the succeeding music piece according to the marker information; The sound processing unit includes: changing at least one of a mixing pattern of the preceding music piece and the succeeding music piece, a mix mode for playing a plurality of music pieces, and the marker information used for mixing based on the information of the character; the mixing pattern is at least one of a volume adjustment pattern and an equalizer adjustment pattern in mixing the preceding music piece and the subsequent music piece, The processing system is characterized in that the mix modes include a short mix mode in which a marker is set in the song to mix the subsequent song at an early timing relative to the preceding song, and a long mix mode in which a marker is set in the song to mix the subsequent song at a late timing relative to the preceding song.
2. In claim 1, The marker information storage unit A processing system comprising: storing information on a marker corresponding to a start timing of the mixing period; and information on a marker corresponding to an end timing of the mixing period.
3. In claim 1 or 2, The markers in the marker information are associated with a musical composition of a musical piece, The sound processing unit includes: A processing system characterized by identifying a period during which the preceding song is a jth song structure (j is an integer equal to or greater than 1) and the succeeding song is a kth song structure (k is an integer equal to or greater than 1) based on the marker information, and performing the mixing process using that period as the mixing period.
4. In any one of claims 1 to 3, The marker information storage unit A processing system characterized by storing the marker information in which, if a song is of a first genre, markers are set for the song's structure in a first setting pattern, and, if the song is of a second genre, markers are set for the song's structure in a second setting pattern different from the first setting pattern.
5. In any one of claims 1 to 4, The marker information storage unit A processing system characterized by storing information of a first marker that sets the permitted timing for starting mixing when a song is the preceding song, information of a second marker that sets the permitted timing for ending mixing when the song is the preceding song, information of a third marker that sets the permitted timing for starting mixing when the song is the following song, and information of a fourth marker that sets the permitted timing for ending mixing when the song is the following song.
6. In claim 5, The sound processing unit includes: A processing system characterized by determining a timing for starting playback of the subsequent song based on the second marker of the preceding song and the fourth marker of the subsequent song, and determining a timing for starting the mixing process based on the first marker of the preceding song and the third marker of the subsequent song.
7. a music information storage unit that stores music information for a plurality of music pieces; a marker information storage unit that stores the marker information; a sound processing unit that performs a playback process of a song based on the song information; a character processing unit that controls the movement of a character; A display processing unit that performs display processing of the image of the character, Make your computer work The sound processing unit includes: When switching from the preceding music piece to the succeeding music piece, a mixing process is performed between the preceding music piece and the succeeding music piece in accordance with the marker information; The character processing unit: When switching from the preceding music piece to the succeeding music piece, controlling the movement of the character during a mixing period of the preceding music piece and the succeeding music piece according to the marker information; The sound processing unit includes: changing at least one of a mixing pattern of the preceding music piece and the succeeding music piece, a mix mode for playing a plurality of music pieces, and the marker information used for mixing based on the information of the character; the mixing pattern is at least one of a volume adjustment pattern and an equalizer adjustment pattern in mixing the preceding music piece and the subsequent music piece, The mix modes include a short mix mode in which a marker is set in the song to mix the subsequent song at an early timing relative to the preceding song, and a long mix mode in which a marker is set in the song to mix the subsequent song at a late timing relative to the preceding song.
Citation Information
Patent Citations
Musical sound reproduction device, method, and program, and recording medium
JP2005331905A
Apparatus for editing / reproducing musical piece
JP2006106103A
Information processing apparatus, information processing method, and program
JP2018011201A
