Moving picture processing device, moving picture processing method, program and information recording medium for providing moving picture indicating beat of music

Abstract

To provide a moving picture indicating beats of reproduced music.SOLUTION: In a moving image processing device 101, a setting part 102 sets a vicinity position, in the vicinity of a predetermined position in a screen. An input part 103 receives input of a beat moving image reproduced in synchronization with music reproduced in response to a control signal output from a DJ controller. The beat moving image is a moving image in which a beat object associated with each beat of the music moves in the screen, and the beat object associated with each of the beats reaches the predetermined position in the screen at a beat timing at which each of the beats of the music is reproduced. A drawing part 104 draws a timing object at a set vicinity position in the input frame each time the frame of the beat moving image is input. An output part 105 outputs a frame in which the timing object is drawn, as the frame of the timing moving image. When an omittable detection part 106 detects scratch operation, the drawing part 104 also draws a scratch object.SELECTED DRAWING: Figure 8

Description

[Technical Field]

[0001] The present invention relates to a moving image processing device, a moving image processing method, a program, and an information recording medium for providing a moving image that indicates the beats of a piece of music being played. [Background technology]

[0002] Breaking (sometimes called breakdancing) has become an official event at the 2028 Paris Olympics. A DJ (Disk Jockey) switches between multiple songs and plays them. Two contestants performed a dance to the rhythm of the music being played. Determine the superiority of the two competitors It's a dance battle.

[0003] The breaking competitions at the Paris Olympics are scheduled to be broadcast live on television and via streaming services. The broadcasts and streams will also be output to large displays at the competition venues, allowing spectators to see the competition from multiple angles.

[0004] Now, the most common way for DJs to switch songs is to adjust the master volume of a mixer that mixes two songs assigned to two turntables (sometimes called "vinyls") on a DJ controller, using software.

[0005] In a DJ controller, each turntable rotates autonomously at a constant speed as music plays, but the DJ can directly touch the turntable to stop, speed up, slow down, or even make it move back and forth vigorously (called "scratch"). This allows the DJ to control the music, stop, speed up, or slow down the music, or repeatedly play short sections forward and backward.

[0006] When the DJ adjusts the rotation of the turntable to align the tempo and beat timing of the two songs while changing the mixing ratio, the audience or listeners watching this will perceive the transition from one song to the other as smooth.

[0007] In a breaking dance battle, one of the key factors in determining the winner is "on-hame," a choreography in which competitors strike "signature poses" in sync with the rhythm of the music and the dance, i.e., when the rhythm of the music is playing on the strong and weak beats.

[0008] However, for audiences unfamiliar with breaking and viewers of broadcasts and live streams, it is difficult to grasp the beat of the music, making it difficult to tell whether the rhythmic timing was successful or not.

[0009] On the other hand, in game devices that offer games that mimic DJing, the player imitating the DJ is required to operate a turntable-type controller in time with the rhythm of the music being played. When moving an object that represents the type of operation the player should perform at a constant speed on the screen, The object reaches a predetermined reference position on the screen at the timing when the operation should be performed. By displaying this on the screen, the timing of operations is notified to the player and spectators watching the play (see Patent Document 1). [Prior art documents] [Patent documents]

[0010] [Patent Document 1] Japanese Patent Publication No. 2023-21142 Summary of the Invention [Problem to be solved by the invention]

[0011] In the game device described in Patent Document 1, the game system controls the playback of music based on a predetermined algorithm, but in breaking, music playback is controlled by a human DJ at the venue where the competition is held, so there is no "predetermined algorithm." Therefore, there is a need for technology that can easily display the beats of music being played under the control of the DJ, while also responding to situations where music is switched by human operation based on the DJ's emotions.

[0012] In addition, the music output controlled by the DJ is provided to competitors and spectators via speakers in the venue via various audio processing devices, and is also provided to the broadcasting and relay system via a route different from the audio processing route in the venue.Then, it is combined with the live audio and video captured and filmed by microphones and video cameras in the venue, and then broadcast and distributed.

[0013] For this reason, A control signal representing the control made by the DJ; A song output by the DJ software / hardware based on the control signal; The music after audio processing, The music after synthesis, In this case, a time lag occurs depending on the processing path.

[0014] Therefore, there is a need for a technology that can adjust the time lag that inevitably occurs depending on the path along which the signal is transmitted, the processing of the signal, the environment in which the device for transmitting the signal is used, and the like.

[0015] The present invention is devised to solve the above-mentioned problems, and relates to a moving image processing device, a moving image processing method, a program, and an information recording medium for providing a moving image that indicates the beats of a piece of music being played. [Means for solving the problem]

[0016] The video processing device according to the present invention comprises: Set the nearby position to the vicinity of the predetermined position in the screen, A beat video is played in synchronization with a song played in response to a control signal output from a DJ controller, a beat object corresponding to each beat of the music piece moves within the screen; At the beat timing when each beat of the music piece is played, a beat object corresponding to each beat reaches the predetermined position on the screen. Accepts beat video input, Each time a frame of the beat moving image is input, a timing object is drawn at the set nearby position within the input frame; The frame in which the timing object is drawn is output as a frame of the timing video. [Effects of the Invention]

[0017] According to the present invention, it is possible to provide a moving image processing device, a moving image processing method, a program, and an information recording medium for providing a moving image that indicates the beats of a piece of music being played. [Brief explanation of the drawings]

[0018] [Figure 1] 1 is an explanatory diagram showing a schematic configuration of a DJ system including a moving image processing device according to an embodiment of the present invention. [Figure 2] 1A and 1B are explanatory diagrams showing examples of frames of a beat moving image used in a moving image processing device according to an embodiment of the present invention. [Figure 3] 1 is an explanatory diagram showing an example of frames of a timing video used in a video processing device according to an embodiment of the present invention; [Figure 4] 10A and 10B are explanatory diagrams showing examples of frames of a timing video used in the video processing device according to the embodiment of the present invention when the current timing is not a beat timing. [Figure 5] 10A and 10B are explanatory diagrams showing examples of frames when the current timing is a beat timing in a timing video used in the video processing device according to the embodiment of the present invention; [Figure 6]10A and 10B are explanatory diagrams showing examples of frames of a timing video used in the video processing device according to the embodiment of the present invention, during which no scratch operation is performed. [Figure 7] 10A and 10B are explanatory diagrams showing examples of frames during a scratch operation in a timing video used in the video processing device according to the embodiment of the present invention; [Figure 8] 1 is an explanatory diagram showing a schematic configuration of a moving image processing device according to an embodiment of the present invention; [Figure 9] 4 is a flowchart showing a control flow of moving image processing executed by the moving image processing device according to the embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0019] The following describes embodiments of the present invention. Note that these embodiments are for illustrative purposes only and do not limit the scope of the present invention. Therefore, those skilled in the art can adopt embodiments in which each or all of the elements of the present embodiments are replaced with equivalents. Furthermore, elements described in each example can be omitted as appropriate depending on the application. In this way, all embodiments constructed in accordance with the principles of the present invention are included in the scope of the present invention.

[0020] (composition) 1 is an explanatory diagram showing the general configuration of a DJ system including a video processing device according to an embodiment of the present invention. The following description will be made with reference to this diagram.

[0021] The DJ system 11 shown in the figure outputs audio based on music and video synchronized with the audio, based on the DJ's operation of a DJ controller 12. A video processing device 101 according to this embodiment forms a part of the DJ system 11.

[0022] The DJ controller 12 has two turntables that control the playback of two songs selected by the DJ, and a mixing volume that determines the mixing ratio of the two songs or an output volume that determines the output volume of the two songs, and outputs a control signal.

[0023] The DJ application 13 running on the computer reads out the music to be processed and the beat video associated with the music from the recording medium based on the control signal output from the DJ controller, and outputs both in synchronization with each other.

[0024] When the DJ selects only one song for playback, the DJ application 13 outputs the audio of the song and a beat video associated with the song.

[0025] On the other hand, when a DJ selects two songs, the two songs are typically mixed at a mixing ratio determined by the DJ and output as one audio signal, but it is also possible to output two audio signals with the volume of the two songs adjusted at an amplification rate according to the mixing ratio.

[0026] Regarding the two beat videos that are output in synchronization with the two songs being played, both may be output, or only one associated with the song with the larger mixing ratio may be output, or, as will be described later, the two beat videos may be synthesized and output. This figure illustrates a mode in which one beat video is output. Figure 2 is an explanatory diagram showing an example of frames of a beat video used in a video processing device according to an embodiment of the present invention.

[0027] As shown in the figure, when the beat video is played at normal playback speed, beat objects 201 representing the strong beats and weak beats of the music move at a constant speed within frames 202 of the beat video. In the figure, the beat objects 201 represent strong beats with long horizontal line segments and weak beats with short horizontal line segments, and these are arranged alternately at approximately equal intervals and move from top to bottom of frames 202. If the distribution of strong beats and weak beats in the music changes or if the tempo of the music changes, the distribution and intervals of the beat objects 201 will also change accordingly.

[0028] Furthermore, when the playback speed of the music is changed by the DJ controller, the playback speed of the beat moving image also changes accordingly, and therefore the movement speed of the beat object 201 also changes.

[0029] The beat video is created in advance so that, if the beat video is synchronized with the music, the moment the beat of the music is played, the beat object 201 associated with the beat reaches a predetermined position 203 within a frame 202. Note that in this diagram, a dotted line is drawn at the predetermined position 203 for ease of understanding, but this dotted line is not included in the actual beat video.

[0030] The beat video is produced, for example, as follows. (1) The creator of the beat video listens to the music and creates MIDI data that matches the tempo of the music. When this MIDI data is played back in sync with the music, sounds (or additional MIDI information) for the specified notes are output at the timing of the strong and weak beats of the music. (2) From the MIDI data created by the beat video creator, a beat video is generated by computer graphics, in which the beat object 201 moves at a constant speed within the frame 202 and reaches a predetermined position 203 within the frame 202 at the moment when the note of the MIDI data is output.

[0031] Therefore, once the beat video is created, there is no need to use the MIDI data itself. Also, since the sounds specified by the MIDI data are not meant to be heard by the audience or viewers, the beat video can be a silent video that does not contain sounds corresponding to the notes specified by the MIDI data.

[0032] The shape and movement direction of the beat object 201 are not limited to this example, and can be changed as appropriate depending on the intended use, presentation intent, and the like.

[0033] As described above, the DJ application 13 runs on a computer. Therefore, even if you try to synchronize and output music and beat video, a certain time lag may occur between the two due to the capabilities and limitations of the computer's hardware and the multimedia devices connected to the computer.

[0034] In a situation where there is a time lag, the beat object 201 will arrive at a position that is shifted from the default position 203 at the moment when the beat of the music is played.

[0035] Therefore, in this embodiment, the video processing device 101 draws a timing object at a nearby position near the default position 203 (which may coincide with the default position 203).

[0036] That is, the video processing device 101 receives as input a beat video output from the DJ application 13, and generates and outputs a timing video by drawing a timing object at a nearby position near the default position 203 in each frame of the beat video. Fig. 3 is an explanatory diagram showing an example of frames of a timing video used in the video processing device according to an embodiment of the present invention. The following description will be made with reference to this diagram.

[0037] As shown in the figure, the timing video is drawn by superimposing a timing object 211 at a nearby position 213 that is near the default position 203 of the beat video. Here, the timing object 211 is intended to indicate that the timing at which it overlaps with the beat object 201 is the timing of the beat of the music. In the figure, the beat object 201 and the timing object 211 do not overlap, which indicates that the timing at which the frame shown in the figure is output is not the timing of the beat of the music.

[0038] In this figure, the default position 203 and the nearby position 213 are drawn with dotted lines to make it easier to understand the positional relationship between them, but in the actual timing video, as shown in Figure 4, these dotted lines are not drawn.

[0039] The process of drawing the timing object in each frame also takes time. Even if the computer corrects for this time, a time lag may occur between the music output by the DJ application 13 and the timing video due to its capabilities and limitations. Even if a time delay filter is applied to the music in consideration of the time required for the drawing process of the timing object to eliminate this time lag, the drawing process time may change depending on the performance of the computer, and a time lag may remain.

[0040] Furthermore, as described above, there may be a time lag between the music and the beat video output by the DJ application 13.

[0041] However, even if a time lag occurs for some reason (including reasons other than those mentioned above), the position at which the timing object 211 is drawn can be corrected by issuing an instruction to adjust the proximity position 213 to the video processing device 101. Therefore, it is possible to set the beat object 201 to reach the timing object 211 at the moment when the beat of the music is output to the audience or viewers. This makes it possible to appropriately indicate the beat of the music to the audience or viewers depending on the environment.

[0042] The shape of the timing object 211 is not limited to this example, and can be changed as appropriate depending on the application.

[0043] In this way, the DJ system 11 outputs the music selected by the DJ and the timing video. The audio signal and video signal are combined into a single multimedia signal. If this multimedia signal is output in accordance with the HDMI (registered trademark) standard, editing for broadcasting and distribution becomes easy.

[0044] That is, as shown in the figure, the voices of announcers and commentators who provide live commentary on the competition are synthesized by an audio post-processing unit 31 into the audio portion of the output multimedia signal.

[0045] Furthermore, the video post-processing unit 32 superimposes the video portion of the output multimedia signal onto the video of the live-filmed competition.

[0046] Therefore, the size of the screen (frame 202) in the beat moving image or timing moving image can be configured to be different from the final screen size for television broadcast or streaming distribution.

[0047] These are then combined again into a single multimedia signal for broadcasting or distribution.

[0048] In this way, by performing post-processing on the audio and video, there is a possibility that a time lag may occur in the audio signal and video signal output from the DJ system 11.

[0049] In addition, the distance from the camera filming the event to the broadcast control room can be tens or even hundreds of meters, and although there is little delay as the camera footage is transmitted using SDI cables, the equipment that switches between multiple cameras passes through the equipment that displays the broadcast captions, so equipment delays are inevitable.

[0050] Furthermore, when the rotation speed information from the DJ controller 12 is received as a MIDI signal (maximum standard transmission distance is 15 meters), a serial signal is essentially used, which can cause a large delay depending on the transmission distance.

[0051] Furthermore, the HDMI signal output by the DJ system 11 is converted to SDI format and sent to the equipment in the control room via an SDI cable, but here too, conversion and transmission delays occur.

[0052] In this embodiment, even if there is a time lag in the processing by the audio post-processing unit 31 or the video post-processing unit 32, or even if a time lag occurs due to various delays as described above, by correcting the nearby position 213 near the default position 203, it is possible to show the audience or viewers the beat of the music currently being played.

[0053] (emphasis on beat timing) In the above example, a timing object 211 is simply drawn as shown in FIG.

[0054] However, the moving image processing device 101 can also indicate whether the current timing is the timing of a beat of music by changing the display form of the timing object 211.

[0055] 5 is an explanatory diagram showing an example of a frame when the current timing is the timing of a beat in a timing video used in a video processing device according to an embodiment of the present invention. As shown in this diagram, while the beat object 201 overlaps with the timing object 211, the shape of the timing object 211 can be changed, an additional pattern can be added, or the color or brightness can be changed, thereby making it easy to understand that the current timing is the timing of a beat in music.

[0056] Before drawing the timing object 211, the moving image processing device 101 can determine whether the current timing is the timing of a beat of music by checking whether there is a pixel of the same (or similar) color as the beat object 201 in an area where the timing object 211 should be drawn in a frame 202 of the beat moving image. Therefore, if the current timing is the timing of a beat of music, the timing object 211 in an emphasized form is drawn in the frame 202, and if not, the timing object 211 in a normal form is drawn in the frame 202, thereby realizing the above form.

[0057] (Performance for scratching) The playback speed of the beat video image is linked to the playback speed of the music piece controlled by the DJ controller 12. Therefore, when the turntable of the DJ controller 12 is rotated quickly forward or reverse, the beat video image is played back fast forward or rewind, and the beat object 201 moves at a speed faster than the above-mentioned constant speed or in the reverse direction.

[0058] However, when the playback speed or direction of the music fluctuates drastically, such as when a DJ is performing a scratch operation, or when attempting to synchronize the beat video with a short, intense movement such as a scratch operation, the beat video may not be decoded in time, resulting in unnatural movement of the beat object 201.

[0059] Furthermore, even if we assume that the processing performance of the DJ application 13 is sufficiently high and that it is possible to move and display the beat object 201 in synchronization with the music being scratched, the output will be one in which the beat object 201 moves violently, making it difficult for the audience and viewers to understand.

[0060] Therefore, as shown in Figure 1, a mode can be adopted in which the video processing device 101 receives input of a control signal from the DJ controller 12 and determines, independently of the DJ application 13, whether or not a scratch operation is currently being performed, and indicates whether or not scratching is being performed by drawing a scratch object of a different form in frame 202 of the timing video depending on whether or not scratching is being performed.

[0061] 6 is an explanatory diagram showing an example of a frame during which a scratch operation is not being performed in a timing video used in a video processing device according to an embodiment of the present invention. In the example shown in this figure, an inactivated scratch object 215 is displayed at a fixed position relative to a timing object 211. In this figure, the scratch object 215 is drawn with a dotted line to indicate that no scratching is being performed.

[0062] 7 is an explanatory diagram showing an example of frames during a scratch operation in a timing video used in a video processing device according to an embodiment of the present invention. In the example shown in this figure, an activated scratch object 215 is displayed. The scratch object 215 is drawn with a thick solid line to indicate that a scratch is being performed.

[0063] If the audience / viewers know that scratching is in progress, the scratch object 215 can be changed in any manner.

[0064] Whether or not scratching is in progress can be determined by checking whether the difference between the rotation speed of the turntable controlling the song with the higher mixing ratio of the two songs and the default rotation speed (the constant rotation speed when the DJ takes his hands off the turntable) exceeds a predetermined threshold.

[0065] During the period in which a scratch operation is detected, the beat video image received by the video processing device 101 immediately before the period may be shifted based on the rotational displacement and speed of the scratch operation. That is, while a scratch operation is being performed, the frame at the start of the scratch operation may be vibrated on the video processing device 101 side in response to the scratch operation. In this case, while the scratch operation is being performed, the beat object 201 moves vigorously within the frames of the timing video image. This allows the audience or viewers to know that a scratch is being performed even if the DJ application 13 cannot decode the beat video image synchronously.

[0066] (Configuration of video processing device) Each unit of the video processing device 101 according to this embodiment is typically realized by a computer that executes a program. A controller that outputs a control signal representing control by a DJ, and an output device such as a music / video output system that is controlled by the controller and provides audio and video, and An input device for receiving video output from the computer and relaying or distributing it It is connected to these devices and sends and receives information to and from them.

[0067] A program executed by a computer can be distributed or sold by a server to which the computer is connected for communication, or it can be recorded on a non-transitory information recording medium such as a CD-ROM (Compact Disk Read Only Memory), flash memory, or EEPROM (Electrically Erasable Programmable ROM), and then the information recording medium can be distributed, sold, etc.

[0068] The program is installed on a non-transitory information recording medium such as a hard disk, solid-state drive, flash memory, EEPROM, etc., of the computer. The video processing device 101 of this embodiment is then realized by the computer. Generally, a computer's central processing unit (CPU) reads the program from the information recording medium into random access memory (RAM) under the control of the computer's operating system (OS), and then interprets and executes the code contained in the program. However, in an architecture in which the information recording medium can be mapped within a memory space accessible by the CPU, explicit loading of the program into RAM may not be necessary. Various pieces of information required during program execution can be temporarily stored in RAM.

[0069] It should be noted that the video processing device 101 of this embodiment can be configured using a dedicated electronic circuit rather than using a general-purpose computer. In this embodiment, the program can also be used as material for generating wiring diagrams, timing charts, and the like for the electronic circuit. In such an embodiment, an electronic circuit that satisfies the specifications defined in the program is configured using an FPGA (Field Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit), and the electronic circuit functions as a dedicated device that performs the functions defined in the program, thereby realizing the video processing device 101 of this embodiment.

[0070] For ease of understanding, the following description will be given assuming that the video processing device 101 is realized by a computer executing a program. Note that the DJ application 13 may run on the computer that realizes the video processing device 101, or may run on another computer connected to the computer that realizes the video processing device 101.

[0071] 8 is an explanatory diagram showing a schematic configuration of a moving image processing device according to an embodiment of the present invention. The following description will be made with reference to this diagram.

[0072] A moving image processing device 101 according to this embodiment includes a setting unit 102, an input unit 103, a drawing unit 104, and an output unit 105. In addition, a detection unit 106 can be included as an optional element.

[0073] In the moving image processing device 101, the setting unit 102 sets a nearby position 213 in the vicinity of the default position 203 on the screen. The screen here corresponds to each frame 202 in the beat moving image.

[0074] Here, the neighboring position 213 is The music is output as audio. The timing video is mixed with other videos and output. The beat objects 201 should be set so that, in a listening environment, the beat objects 201 associated with each beat to be heard will reach the nearby position 213 at the timing when each beat of the output audio is heard.

[0075] For example, when the DJ system 11 including the video processing device 101 is ready, a song is selected from the DJ controller 12 as a test, and the audio and video are output from the DJ system 11. The operator or the like can then make corrections while comparing the output audio and video, thereby setting the nearby position 213.

[0076] On the other hand, the input unit 103 accepts input of a beat moving image to be played in synchronization with the music being played in response to a control signal output from the DJ controller 12. Here, as described above, the beat moving image is configured so that the beat objects 201 associated with each beat of the music move within the screen (frames 202) and reach predetermined positions 203 within the screen (frames 202) at the beat timing at which each beat of the music is played.

[0077] The beat video may be input via hardware wiring, or the beat video frames may be expanded in the RAM of a computer, and the information in this RAM may be treated as the beat video input.

[0078] Then, every time a frame 202 of the beat moving image is input, the drawing unit 104 draws a timing object 211 at a set nearby position 213 within the input frame 202.

[0079] Furthermore, the output unit 105 outputs frame 202 in which the timing object is drawn as a frame of the timing video. The output by the output unit 105 may be via hardware wiring, or may be a form in which a temporary frame is created in the RAM of a computer and left for subsequent processing.

[0080] In addition, the drawing unit 104 It is determined whether the beat object 201 is at the set nearby position 213; Depending on the result of the determination, the appearance of the timing object 211 is changed and drawn. By configuring it in this way, it is possible to clearly show the audience / listener whether or not the current timing is on the beat of the music.

[0081] Furthermore, the detection unit 106 detects, from the control signal output from the DJ controller 12, that a scratch operation is being performed on the DJ controller 12. When the detection unit 106 detects a scratch operation, the drawing unit 104 is configured to further draw a scratch object 215, so that the audience or listeners can know that the DJ is currently scratching without looking at the beat object 201.

[0082] Here, the scratch object 215 can be configured to be drawn at a position that is shifted by a predetermined displacement from the set nearby position 213. Then, by simply paying attention to the timing object 215 and its surroundings, the audience or listeners can easily tell whether the current timing is on a beat of the music or not, and whether a scratch operation is currently being performed or not.

[0083] In addition, while a scratch operation is being detected, the drawing unit 104 may shift the position of the origin of the frame input immediately before the scratch operation is detected according to the strength (displacement or speed) of the scratch operation, transfer it to a frame for timing video, and then draw the timing object 211 and the scratch object 215.

[0084] With this configuration, even if the scratch operation does not allow the beat video to be synchronized with the music, the audience or viewers can naturally sense that scratching is being performed.

[0085] The above processing may be performed by the video processing device 101 on each of the videos, whether the DJ application 13 outputs one song and one beat video, or whether two beat videos are output according to the two turntables of the DJ controller 12, or may be performed based on the beat video with the larger mixing ratio set in the DJ controller 12.

[0086] In addition, when a first song and a second song are mixed and played in response to a control signal output from the DJ controller 12, the input unit 103 may be configured to accept input of a first beat video that is played in synchronization with the first song and a second beat video that is played in synchronization with the second song.

[0087] Then, each time the first frame of the first beat video and the second frame of the second beat video are input simultaneously, the drawing unit 104 may combine the first frame and the second frame in accordance with the mixing ratio of the first music piece and the second music piece, and then draw the timing object 211 at the set nearby position 213.

[0088] When switching from a first song to a second song, DJs often align the beats of both songs. In such cases, the beat object 201 in the first frame and the beat object 201 in the second frame are positioned so that they overlap each other. Therefore, when the two are combined, a frame with almost the same content is obtained.

[0089] On the other hand, if the transition occurs while the beats are out of sync, the mixing ratio of the two pieces of music will change. When both frames are combined at that mixing ratio, a crossfade / crossdissolve transition is made from one beat video to the other.

[0090] (Flow of Control) 9 is a flowchart showing the flow of control of video processing executed by a video processing device according to an embodiment of the present invention. The following description will be made with reference to this diagram. Note that, although the following description will be made assuming that two beat videos are output from the DJ application 13, the following description can also be made to accommodate a case where only one beat video is output by omitting appropriate processing.

[0091] When this process starts, the video processing device 101 first performs various initializations (step S300). This initialization may include, for example, restoring the neighboring position settings from the previous process, acquiring the current mixing ratio from the DJ controller 12, and so on.

[0092] Next, the video processing device 101 receives input from various external devices (step S301) and checks the type of the input (step S302).

[0093] If the received input is an adjustment instruction input to adjust the nearby position given by the operator via the keyboard, mouse, etc. of the video processing device 101 (step S302; adjustment), the video processing device 101 sets the nearby position to the position specified in the adjustment instruction input (step S303), and proceeds to step S321.

[0094] Then, the video processing device 101 obtains the mixing ratio of the first song and the second song in the DJ controller 12 (step S321), obtains the rotation speed of the two turntables in the DJ controller 12 (step S322), and executes other processes that should be repeated as appropriate (step S323), and returns control to step S301.

[0095] On the other hand, if the received input is a frame of each of the first beat video and the second beat video (hereinafter, the first beat video side is referred to as the first frame and the second beat video side is referred to as the second frame) output from the DJ application 13 (step S302; beat video frame), the video processing device 101 combines the first frame and the second frame using the latest acquired mixing ratio and writes the combined frame as a temporary frame in a temporary area in RAM (step S311). Note that, instead of combining the first frame and the second frame, the video processing device 101 may use a frame of the music side with a larger mixing ratio as the temporary frame.

[0096] Then, the moving image processing device 101 checks whether or not a beat object 201 is drawn at a nearby position based on pixel values ​​at nearby positions in the temporary frame or in the frame associated with the music piece having the larger mixing ratio between the first and second frames (step S312).

[0097] If the beat object 201 is not drawn in a nearby position (step S312: No), the moving image processing device 101 draws the timing object 211 in a normal form in a nearby position in the temporary frame by overwriting it (step S313), and proceeds to step S315.

[0098] On the other hand, if the beat object 201 is drawn in a nearby position (step S312; Yes), the timing object 211 in emphasis form is drawn over the nearby position in the temporary frame (step S314).

[0099] Next, the moving image processing device 101 checks whether or not a scratch operation is currently being performed based on the rotation speed of the turntable for the music piece with the larger mixing ratio (step S315).

[0100] If a scratch operation has not been performed (step S315; No), the video processing device 101 overwrites and draws the inactivated scratch object 215 at a position within the temporary frame that is moved by a predetermined relative displacement from a nearby position (step S316), and proceeds to step S318.

[0101] On the other hand, if a scratch operation is being performed (step S315; Yes), the video processing device 101 overwrites and draws the activated scratch object 215 at a position within the temporary frame that is moved by a predetermined relative displacement from a nearby position (step S317).

[0102] Then, the moving image processing device 101 outputs the contents of the temporary frame as a frame of the timing moving image (step S318), and proceeds to step S321.

[0103] If an input other than the above is received (step S302; Other), the process corresponding to the input is executed (step S331), and the process proceeds to step S321.

[0104] By executing such moving image processing, the moving image processing device 101 can generate and output a timing moving image in which the timing object 211 and the scratch object 215 are drawn from the input beat moving image.

[0105] (Adjusting nearby positions) The method for adjusting the neighboring positions will be described below.

[0106] Here, the timing object 211 is drawn in pixel units. If the movement speed of the beat object 201 within the frame 202 is, for example, 200 pixels per second, moving the position of the timing object 211 by one pixel will result in an adjustment of 1 / 200=0.005 seconds.

[0107] The first method is a method in which a human or AI (Artificial Intelligence) operator makes adjustments based on audio. This is suitable for an environment in which there is almost no time lag between audio and video within the DJ system 11, that is, a situation in which the beat of a song is output when the beat object 201 reaches the predetermined position 203 in the output of the DJ system 11.

[0108] The operator: The sound output by DJ System 11 and The audio that is ultimately output for streaming or broadcast using the above audio, If necessary, you can record these sounds again using a microphone or audio cable.

[0109] If the former is delayed, the nearby position is moved one pixel upward in the opposite direction to the movement of the beat object 201, and a delay of 0.005 seconds is applied to the latter. This process is then repeated until the two sounds are synchronized.

[0110] If the latter is delayed, the nearby position is moved by one pixel in the same direction as the movement of the beat object 201, i.e., downward, and a delay of 0.005 seconds is applied to the former. This process is then repeated until the two sounds are synchronized.

[0111] The proximity position at which the two sounds are finally synchronized is determined as the proximity position at which the timing object 211 should be drawn.

[0112] The determination of which of the two sounds is delayed and whether the two sounds are synchronized can be made by a human being, or it can be done automatically and mechanically using signal processing techniques.

[0113] The second method involves a human or AI operator making adjustments based on video footage.

[0114] To measure the time difference, a beat video is prepared in which markers that change color for each pixel are drawn parallel to the movement direction of the beat object 201. These markers function as a kind of "ruler."

[0115] As described above, the timing video output from the DJ system 11 is converted to SDI, processed by the equipment in the broadcast control room, and redrawn on the screen to be broadcast and distributed.

[0116] Therefore, among the finally output moving images, the area derived from the timing moving image and the timing moving image output from the DJ system 11 are arranged side by side to check the display discrepancy of the beat object 201.

[0117] Then, a position displaced from the default position 203 by the number of pixels of the displacement is determined as a neighboring position.

[0118] This process can be performed by a human operator comparing the screens, or by shooting two videos simultaneously with a camera and having a computer or AI that processes the images count the number of pixels that are out of sync.

[0119] The moving speed of the beat object 201, the time required to move one pixel, the number of pixels to be adjusted, the color configuration of the pixels used as the "ruler," etc. can be changed as desired depending on the settings and environment to be applied. The "ruler" may also be expressed by any pattern that can be recognized by humans or machines.

[0120] The third method is to directly adjust the video using sound. In this method, a beat video is created that produces a clicking sound at the timing of the beat (when the beat object 201 reaches the predetermined position 203).

[0121] When generating a timing video from a beat video, when outputting the timing video, or when superimposing the timing video on other video or audio, the click sound is also synthesized.

[0122] The final audio and video output is then recorded along with the audio, and played back along with the audio by either advancing one frame (or multiple frames) at a time or playing back in reverse frame by frame.

[0123] If the frame in which the click sound is heard and the frame in which the beat object 201 and timing object 211 overlap are not aligned, the position of the timing object 211 is corrected.

[0124] Here, three methods for adjusting the proximity position have been described: a method based on audio comparison, a method based on video comparison, and a method that compares audio and video. However, these may be combined in any manner, or other methods may be adopted to make the adjustment.

[0125] (summary) As described above, the video processing device according to this embodiment: a setting unit that sets a nearby position in the vicinity of a predetermined position on the screen; A beat video is played in synchronization with a song played in response to a control signal output from a DJ controller, a beat object corresponding to each beat of the music piece moves within the screen; At the beat timing when each beat of the music piece is played, a beat object corresponding to each beat reaches the predetermined position on the screen. an input unit that accepts input of beat video; a drawing unit that draws a timing object at the set nearby position within the input frame each time a frame of the beat moving image is input; an output unit that outputs a frame in which the timing object is drawn as a frame of a timing video; The device is configured to include the following.

[0126] In addition, in the video processing device according to this embodiment, The nearby location is The music is output as audio, The timing video is mixed with other videos and output. In a listening environment, the beat objects corresponding to the respective beats of the output audio should be set to reach the nearby positions at the timing when the respective beats of the output audio are listened to. It can be configured as follows.

[0127] In addition, in the video processing device according to this embodiment, The drawing unit determining whether the beat object is in the set vicinity position; According to the result of the determination, the appearance of the timing object is changed and drawn. It can be configured as follows.

[0128] In addition, in the video processing device according to this embodiment, A detection unit that detects, from the output control signal, that a scratch operation is being performed on the DJ controller. Furthermore, When the scratch operation is detected, the drawing unit further draws a scratch object. It can be configured as follows.

[0129] In addition, in the video processing device according to this embodiment, The scratch object is drawn at a position shifted by a predetermined displacement from the set nearby position. It can be configured as follows.

[0130] In addition, in the video processing device according to this embodiment, When the scratch operation is detected, the drawing unit shifts the reference position of the frame according to the strength of the scratch operation, and then draws the timing object and the scratch object. It can be configured as follows.

[0131] In addition, in the video processing device according to this embodiment, The first music piece and the second music piece are mixed and played in response to a control signal output from the DJ controller, the input unit receives an input of a first beat video to be played in synchronization with the first music piece and a second beat video to be played in synchronization with the second music piece; The drawing unit, every time a first frame of the first beat video and a second frame of the second beat video are simultaneously input, combines the first frame and the second frame in accordance with a mixing ratio of the first music piece and the second music piece, and then draws the timing object at the set nearby position. It can be configured as follows.

[0132] In the video processing method according to the present embodiment, a video processing device Set the nearby position to the vicinity of the predetermined position in the screen, A beat video is played in synchronization with a song played in response to a control signal output from a DJ controller, a beat object corresponding to each beat of the music piece moves within the screen; At the beat timing when each beat of the music piece is played, a beat object corresponding to each beat reaches the predetermined position on the screen. Accepts beat video input, Each time a frame of the beat moving image is input, a timing object is drawn at the set nearby position within the input frame; Output the frame in which the timing object is drawn as a frame of the timing video. Configure it as follows.

[0133] The program according to this embodiment executes the following steps: a setting unit that sets a nearby position in the vicinity of a predetermined position on the screen; A beat video is played in synchronization with a song played in response to a control signal output from a DJ controller, a beat object corresponding to each beat of the music piece moves within the screen; At the beat timing when each beat of the music piece is played, a beat object corresponding to each beat reaches the predetermined position on the screen. an input unit that accepts input of beat video; a drawing unit that draws a timing object at the set nearby position within the input frame each time a frame of the beat moving image is input; an output unit that outputs a frame in which the timing object is drawn as a frame of a timing video; Configure it to function as a

[0134] A non-transitory computer-readable information recording medium according to this embodiment can be configured to record the above program.

[0135] The program can be distributed or sold by recording it on a non-transitory computer-readable information recording medium, or via a transitory transmission medium such as a computer communication network.

[0136] The present invention allows various embodiments and modifications without departing from the broad spirit and scope of the present invention. Furthermore, the above-described embodiments are intended to explain the present invention and do not limit the scope of the present invention. That is, the scope of the present invention is defined by the claims, not the embodiments. Various modifications made within the scope of the claims and the meaning of the invention equivalent thereto are considered to be within the scope of the present invention. [Industrial Applicability]

[0137] According to the present invention, it is possible to provide a moving image processing device, a moving image processing method, a program, and an information recording medium that output moving images that indicate the beats of a piece of music being played. [Explanation of symbols]

[0138] 11 DJ System 12 DJ controllers 13. DJ App 31 Audio post-processing unit 32 Video post-processing section 101 Video Processing Device 102 Setting section 103 Input section 104 Drawing section 105 Output section 106 Detector

Claims

1. a setting unit that sets a nearby position in the vicinity of a predetermined position on the screen; A beat video is played in synchronization with a song played in response to a control signal output from a DJ controller, a beat object corresponding to each beat of the music piece moves within the screen; At the beat timing when each beat of the music piece is played, a beat object corresponding to each beat reaches the predetermined position on the screen. an input unit that accepts input of beat video; a drawing unit that draws a timing object at the set nearby position within the input frame each time a frame of the beat moving image is input; an output unit that outputs a frame in which the timing object is drawn as a frame of a timing video; A video processing device comprising:

2. The nearby location is The music is output as audio, The timing video is mixed with other videos and output. In a listening environment, the beat objects corresponding to the respective beats of the output audio should be set to reach the nearby positions at the timing when the respective beats of the output audio are listened to.

2. The video processing device according to claim 1,

3. The drawing unit determining whether the beat object is in the set vicinity position; According to the result of the determination, the appearance of the timing object is changed and drawn.

3. The video processing device according to claim 1, wherein the video processing device is a video processing device for processing a video signal.

4. A detection unit that detects, from the output control signal, that a scratch operation is being performed on the DJ controller. Furthermore, When the scratch operation is detected, the drawing unit further draws a scratch object.

2. The video processing device according to claim 1,

5. The scratch object is drawn at a position shifted by a predetermined displacement from the set nearby position.

5. The video processing device according to claim 4.

6. When the scratch operation is detected, the drawing unit shifts the reference position of the frame according to the strength of the scratch operation, and then draws the timing object and the scratch object.

6. The video processing device according to claim 5,

7. The first music piece and the second music piece are mixed and played in response to a control signal output from the DJ controller, the input unit receives an input of a first beat video to be played in synchronization with the first music piece and a second beat video to be played in synchronization with the second music piece; The drawing unit, every time a first frame of the first beat video and a second frame of the second beat video are simultaneously input, combines the first frame and the second frame in accordance with a mixing ratio of the first music piece and the second music piece, and then draws the timing object at the set nearby position.

2. The video processing device according to claim 1,

8. The video processing device Set the nearby position to the vicinity of the predetermined position in the screen, A beat video is played in synchronization with a song played in response to a control signal output from a DJ controller, a beat object corresponding to each beat of the music piece moves within the screen; At the beat timing when each beat of the music piece is played, a beat object corresponding to each beat reaches the predetermined position on the screen. Accepts beat video input, Each time a frame of the beat moving image is input, a timing object is drawn at the set nearby position within the input frame; Output the frame in which the timing object is drawn as a frame of the timing video. A moving image processing method comprising:

9. Computer, a setting unit that sets a nearby position in the vicinity of a predetermined position within the screen; A beat video is played in synchronization with a song played in response to a control signal output from a DJ controller, a beat object corresponding to each beat of the music piece moves within the screen; At the beat timing when each beat of the music piece is played, a beat object corresponding to each beat reaches the predetermined position on the screen. an input unit that accepts input of beat video; a drawing unit that draws a timing object at the set nearby position within the input frame each time a frame of the beat moving image is input; an output unit that outputs a frame in which the timing object is drawn as a frame of a timing video; A program characterized by functioning as

10. A non-transitory computer-readable information recording medium on which the program according to claim 9 is recorded.

Images