Moving image processing device for providing moving image indicating beat of musical piece, moving image processing method, program, and information recording medium

Abstract

A moving image processing device (101) provides a moving image indicating the beat of a musical piece being reproduced. A setting unit (102) sets a vicinity position in the vicinity of a predetermined position in a screen. An input unit (103) receives an input of a beat moving image reproduced in synchronization with the musical piece which is reproduced in accordance with a control signal output from a DJ controller. The beat moving image is a moving image in which a beat object associated with each beat of the musical piece moves in the screen, and the beat object associated with the relevant beat reaches the predetermined position in the screen at a beat timing at which the relevant beat of the musical piece is reproduced. Each time a frame of the beat moving image is input, a drawing unit (104) draws a timing object at the set vicinity position in the input frame. An output unit (105) outputs a frame in which the timing object is drawn as a frame of a timing moving image. If a detection unit (106), which may be omitted, detects a scratch operation, the drawing unit (104) also draws a scratch object.

Description

Video processing device, video processing method, program, and information recording medium for providing video showing beats of music

[0001] The present invention relates to a moving image processing device, a moving image processing method, a program, and an information recording medium for providing a moving image that indicates the beats of a piece of music being played.

[0002] Breaking (sometimes called breakdancing), which will be an official event at the 2028 Paris Olympics, is a dance battle in which a DJ (Disk Jockey) switches between multiple songs and two competitors perform a dance to the rhythm of the songs being played, to determine the superiority of the two competitors.

[0003] The breaking competitions at the Paris Olympics are scheduled to be broadcast live on television and via streaming services. The broadcasts and streams will also be output to large displays at the competition venues, allowing spectators to see the competition from multiple angles.

[0004] Now, the most common way for DJs to switch songs is to adjust the master volume of a mixer that mixes two songs assigned to two turntables (sometimes called "vinyls") on a DJ controller, using software.

[0005] In a DJ controller, each turntable rotates autonomously at a constant speed as music plays, but the DJ can directly touch the turntable to stop, speed up, slow down, or even make it move back and forth vigorously (called "scratch"). This allows the DJ to control the music, stop, speed up, or slow down the music, or repeatedly play short sections forward and backward.

[0006] When the DJ adjusts the rotation of the turntable to align the tempo and beat timing of the two songs while changing the mixing ratio, the audience or listeners watching this will perceive the transition from one song to the other as smooth.

[0007] In a breaking dance battle, one of the key factors in determining the winner is "on-hame," a choreography in which competitors strike "signature poses" in sync with the rhythm of the music and the dance, i.e., when the rhythm of the music is playing on the strong and weak beats.

[0008] However, for audiences unfamiliar with breaking and viewers of broadcasts and live streams, it is difficult to grasp the beat of the music, making it difficult to tell whether the rhythmic timing was successful or not.

[0009] On the other hand, in a game device that provides a game simulating a DJ, the player simulating the DJ is required to operate a turntable-type controller in time with the rhythm of the music being played. In such a game, an object representing the type of operation that the player should perform is moved at a constant speed on the screen, and the object is displayed on the screen so that it reaches a predetermined reference position on the screen at the timing when the operation should be performed, thereby informing the player and the spectators watching the play of the operation timing (see Patent Document 1).

[0010] Japanese Patent Publication No. 2023-21142

[0011] In the game device described in Patent Document 1, the game system controls the playback of music based on a predetermined algorithm, but in breaking, music playback is controlled by a human DJ at the venue where the competition is held, so there is no "predetermined algorithm." Therefore, there is a need for technology that can easily display the beats of music being played under the control of the DJ, while also responding to situations where music is switched by human operation based on the DJ's emotions.

[0012] In addition, the music output controlled by the DJ is provided to competitors and spectators via speakers in the venue via various audio processing devices, and is also provided to the broadcasting and relay system via a route different from the audio processing route in the venue.Then, it is combined with the live audio and video captured and filmed by microphones and video cameras in the venue, and then broadcast and distributed.

[0013] As a result, there will be a time lag depending on the processing path between the control signal representing the control performed by the DJ, the music output by the DJ software / hardware based on that control signal, the music after audio processing, and the music after synthesis.

[0014] Therefore, there is a need for a technology that can adjust the time lag that inevitably occurs depending on the path along which the signal is transmitted, the processing of the signal, the environment in which the device for transmitting the signal is used, and the like.

[0015] The present invention is devised to solve the above-mentioned problems, and relates to a moving image processing device, a moving image processing method, a program, and an information recording medium for providing a moving image that indicates the beats of a piece of music being played.

[0016] The video processing device of the present invention sets a nearby position near a predetermined position on a screen, accepts input of a beat video that is played in synchronization with music that is played in response to a control signal output from a DJ controller, in which beat objects corresponding to each beat of the music move within the screen and the beat objects corresponding to each beat reach the predetermined position on the screen at the beat timing at which each beat of the music is played, draws a timing object at the set nearby position within the input frame each time a frame of the beat video is input, and outputs the frame in which the timing object is drawn as a frame of the timing video.

[0017] According to the present invention, it is possible to provide a moving image processing device, a moving image processing method, a program, and an information recording medium for providing a moving image that indicates the beats of a piece of music being played.

[0018] FIG. 1 is an explanatory diagram showing a schematic configuration of a DJ system including a video processing device according to an embodiment of the present invention. FIG. 2 is an explanatory diagram showing example frames of a beat video used in a video processing device according to an embodiment of the present invention. FIG. 3 is an explanatory diagram showing example frames of a timing video used in a video processing device according to an embodiment of the present invention. FIG. 4 is an explanatory diagram showing example frames of a timing video used in a video processing device according to an embodiment of the present invention when the current timing is not the beat timing. FIG. 5 is an explanatory diagram showing example frames of a timing video used in a video processing device according to an embodiment of the present invention when the current timing is the beat timing. FIG. 6 is an explanatory diagram showing example frames of a timing video used in a video processing device according to an embodiment of the present invention when a scratch operation is not being performed. FIG. 7 is an explanatory diagram showing example frames of a timing video used in a video processing device according to an embodiment of the present invention when a scratch operation is being performed. FIG. 8 is an explanatory diagram showing a schematic configuration of a video processing device according to an embodiment of the present invention. FIG. 9 is a flowchart showing the flow of control of video processing performed by a video processing device according to an embodiment of the present invention.

[0019] The following describes embodiments of the present invention. Note that these embodiments are for illustrative purposes only and do not limit the scope of the present invention. Therefore, those skilled in the art can adopt embodiments in which each or all of the elements of the present embodiments are replaced with equivalents. Furthermore, elements described in each example can be omitted as appropriate depending on the application. In this way, all embodiments constructed in accordance with the principles of the present invention are included in the scope of the present invention.

[0020] (Configuration) Fig. 1 is an explanatory diagram showing the general configuration of a DJ system including a video processing device according to an embodiment of the present invention. The following description will be made with reference to this diagram.

[0021] The DJ system 11 shown in the figure outputs audio based on music and video synchronized with the audio, based on the DJ's operation of a DJ controller 12. A video processing device 101 according to this embodiment forms a part of the DJ system 11.

[0022] The DJ controller 12 has two turntables that control the playback of two songs selected by the DJ, and a mixing volume that determines the mixing ratio of the two songs or an output volume that determines the output volume of the two songs, and outputs a control signal.

[0023] The DJ application 13 running on the computer reads out the music to be processed and the beat video associated with the music from the recording medium based on the control signal output from the DJ controller, and outputs both in synchronization with each other.

[0024] When the DJ selects only one song for playback, the DJ application 13 outputs the audio of the song and a beat video associated with the song.

[0025] On the other hand, when a DJ selects two songs, the two songs are typically mixed at a mixing ratio determined by the DJ and output as one audio signal, but it is also possible to output two audio signals with the volume of the two songs adjusted at an amplification rate according to the mixing ratio.

[0026] Regarding the two beat videos that are output in synchronization with the two songs being played, both may be output, or only one associated with the song with the larger mixing ratio may be output, or, as will be described later, the two beat videos may be synthesized and output. This figure illustrates a mode in which one beat video is output. Figure 2 is an explanatory diagram showing an example of frames of a beat video used in a video processing device according to an embodiment of the present invention.

[0027] As shown in the figure, when the beat video is played at normal playback speed, beat objects 201 representing the strong beats and weak beats of the music move at a constant speed within frames 202 of the beat video. In the figure, the beat objects 201 represent strong beats with long horizontal line segments and weak beats with short horizontal line segments, and these are arranged alternately at approximately equal intervals and move from top to bottom of frames 202. If the distribution of strong beats and weak beats in the music changes or if the tempo of the music changes, the distribution and intervals of the beat objects 201 will also change accordingly.

[0028] Furthermore, when the playback speed of the music is changed by the DJ controller, the playback speed of the beat moving image also changes accordingly, and therefore the movement speed of the beat object 201 also changes.

[0029] The beat video is created in advance so that, if the beat video is synchronized with the music, the moment the beat of the music is played, the beat object 201 associated with the beat reaches a predetermined position 203 within a frame 202. Note that in this diagram, a dotted line is drawn at the predetermined position 203 for ease of understanding, but this dotted line is not included in the actual beat video.

[0030] A beat video is created, for example, as follows: (1) A beat video creator listens to a piece of music and creates MIDI data that matches the tempo of the music. When this MIDI data is played back in synchronization with the music, sounds (or additional MIDI information) for specified notes are output at the timing of the strong and weak beats of the music. (2) From the MIDI data created by the beat video creator, a beat video is generated using computer graphics, in which a beat object 201 moves at a constant speed within a frame 202 and reaches a predetermined position 203 within the frame 202 at the moment the note in the MIDI data is output.

[0031] Therefore, once the beat video is created, there is no need to use the MIDI data itself. Also, since the sounds specified by the MIDI data are not meant to be heard by the audience or viewers, the beat video can be a silent video that does not contain sounds corresponding to the notes specified by the MIDI data.

[0032] The shape and movement direction of the beat object 201 are not limited to this example, and can be changed as appropriate depending on the intended use, presentation intent, and the like.

[0033] As described above, the DJ application 13 runs on a computer. Therefore, even if you try to synchronize and output music and beat video, a certain time lag may occur between the two due to the capabilities and limitations of the computer's hardware and the multimedia devices connected to the computer.

[0034] In a situation where there is a time lag, the beat object 201 will arrive at a position that is shifted from the default position 203 at the moment when the beat of the music is played.

[0035] Therefore, in this embodiment, the video processing device 101 draws a timing object at a nearby position near the default position 203 (which may coincide with the default position 203).

[0036] That is, the video processing device 101 receives as input a beat video output from the DJ application 13, and generates and outputs a timing video by drawing a timing object at a nearby position near the default position 203 in each frame of the beat video. Fig. 3 is an explanatory diagram showing an example of frames of a timing video used in the video processing device according to an embodiment of the present invention. The following description will be made with reference to this diagram.

[0037] As shown in the figure, the timing video is drawn by superimposing a timing object 211 at a nearby position 213 that is near the default position 203 of the beat video. Here, the timing object 211 is intended to indicate that the timing at which it overlaps with the beat object 201 is the timing of the beat of the music. In the figure, the beat object 201 and the timing object 211 do not overlap, which indicates that the timing at which the frame shown in the figure is output is not the timing of the beat of the music.

[0038] In this figure, the default position 203 and the nearby position 213 are drawn with dotted lines to make it easier to understand the positional relationship between them. Fig. 4 is an explanatory diagram showing an example of a frame when the current timing is not a beat timing in a timing video used by a video processing device according to an embodiment of the present invention. As shown in this figure, these dotted lines do not need to be drawn in an actual timing video.

[0039] The process of drawing the timing object in each frame also takes time. Even if the computer corrects for this time, a time lag may occur between the music output by the DJ application 13 and the timing video due to its capabilities and limitations. Even if a time delay filter is applied to the music in consideration of the time required for the drawing process of the timing object to eliminate this time lag, the drawing process time may change depending on the performance of the computer, and a time lag may remain.

[0040] Furthermore, as described above, there may be a time lag between the music and the beat video output by the DJ application 13.

[0041] However, even if a time lag occurs for some reason (including reasons other than those mentioned above), the position at which the timing object 211 is drawn can be corrected by issuing an instruction to adjust the proximity position 213 to the video processing device 101. Therefore, it is possible to set the beat object 201 to reach the timing object 211 at the moment when the beat of the music is output to the audience or viewers. This makes it possible to appropriately indicate the beat of the music to the audience or viewers depending on the environment.

[0042] The shape of the timing object 211 is not limited to this example, and can be changed as appropriate depending on the application.

[0043] In this way, the DJ system 11 outputs the music selected by the DJ and the timing video. The audio signal and video signal are combined into a single multimedia signal. If this multimedia signal is output in accordance with the HDMI (registered trademark) standard, editing for broadcasting and distribution becomes easy.

[0044] That is, as shown in FIG. 1, the voice of an announcer or commentator who provides live commentary on the competition is synthesized into the audio portion of the output multimedia signal by an audio post-processing unit 31.

[0045] Furthermore, the video post-processing unit 32 superimposes the video portion of the output multimedia signal onto the video of the live-filmed competition.

[0046] Therefore, the size of the screen (frame 202) in the beat moving image or timing moving image can be configured to be different from the final screen size for television broadcast or streaming distribution.

[0047] These are then combined again into a single multimedia signal for broadcasting or distribution.

[0048] In this way, by performing post-processing on the audio and video, there is a possibility that a time lag may occur in the audio signal and video signal output from the DJ system 11.

[0049] In addition, the distance from the camera filming the event to the broadcast control room can be tens or even hundreds of meters, and although there is little delay as the camera footage is transmitted using SDI cables, the equipment that switches between multiple cameras passes through the equipment that displays the broadcast captions, so equipment delays are inevitable.

[0050] Furthermore, when the rotation speed information from the DJ controller 12 is received as a MIDI signal (maximum standard transmission distance is 15 meters), a serial signal is essentially used, which can cause a large delay depending on the transmission distance.

[0051] Furthermore, the HDMI (registered trademark) signal output by the DJ system 11 is converted into SDI format and sent to the equipment in the control room via an SDI cable, but here too, conversion and transmission delays occur.

[0052] In this embodiment, even if there is a time lag in the processing by the audio post-processing unit 31 or the video post-processing unit 32, or even if a time lag occurs due to various delays as described above, by correcting the nearby position 213 near the default position 203, it is possible to show the audience or viewers the beat of the music currently being played.

[0053] (Emphasis on the Timing of the Beat) In the above example, as shown in FIG. 4, the timing object 211 is simply drawn.

[0054] However, the moving image processing device 101 can also indicate whether the current timing is the timing of a beat of music by changing the display form of the timing object 211.

[0055] 5 is an explanatory diagram showing an example of a frame when the current timing is the timing of a beat in a timing video used in a video processing device according to an embodiment of the present invention. As shown in this diagram, while the beat object 201 overlaps with the timing object 211, the shape of the timing object 211 can be changed, an additional pattern can be added, or the color or brightness can be changed, thereby making it easy to understand that the current timing is the timing of a beat in music.

[0056] Before drawing the timing object 211, the moving image processing device 101 can determine whether the current timing is the timing of a beat of music by checking whether there is a pixel of the same (or similar) color as the beat object 201 in an area where the timing object 211 should be drawn in a frame 202 of the beat moving image. Therefore, if the current timing is the timing of a beat of music, the timing object 211 in an emphasized form is drawn in the frame 202, and if not, the timing object 211 in a normal form is drawn in the frame 202, thereby realizing the above form.

[0057] (Effects for Scratching) The playback speed of the beat video is linked to the playback speed of the music controlled by the DJ controller 12. Therefore, when the turntable of the DJ controller 12 is rotated quickly forward or reverse, the beat video is played back fast forward or rewind, and the beat object 201 moves at a speed faster than the above-mentioned constant speed or in the reverse direction.

[0058] However, when the playback speed or direction of the music fluctuates drastically, such as when a DJ is performing a scratch operation, or when attempting to synchronize the beat video with a short, intense movement such as a scratch operation, the beat video may not be decoded in time, resulting in unnatural movement of the beat object 201.

[0059] Furthermore, even if we assume that the processing performance of the DJ application 13 is sufficiently high and that it is possible to move and display the beat object 201 in synchronization with the music being scratched, the output will be one in which the beat object 201 moves violently, making it difficult for the audience and viewers to understand.

[0060] Therefore, as shown in Figure 1, a mode can be adopted in which the video processing device 101 receives input of a control signal from the DJ controller 12 and determines, independently of the DJ application 13, whether or not a scratch operation is currently being performed, and indicates whether or not scratching is being performed by drawing a scratch object of a different form in frame 202 of the timing video depending on whether or not scratching is being performed.

[0061] 6 is an explanatory diagram showing an example of a frame during which a scratch operation is not being performed in a timing video used in a video processing device according to an embodiment of the present invention. In the example shown in this figure, an inactivated scratch object 215 is displayed at a fixed position relative to a timing object 211. In this figure, the scratch object 215 is drawn with a dotted line to indicate that no scratching is being performed.

[0062] 7 is an explanatory diagram showing an example of frames during a scratch operation in a timing video used in a video processing device according to an embodiment of the present invention. In the example shown in this figure, an activated scratch object 215 is displayed. The scratch object 215 is drawn with a thick solid line to indicate that a scratch is being performed.

[0063] As long as the audience / viewer knows that scratching is in progress, the scratch object 215 can be changed in any manner.

[0064] Whether or not scratching is in progress can be determined by checking whether the difference between the rotation speed of the turntable controlling the song with the higher mixing ratio of the two songs and the default rotation speed (the constant rotation speed when the DJ takes his hands off the turntable) exceeds a predetermined threshold.

[0065] During the period in which a scratch operation is detected, the beat video image received by the video processing device 101 immediately before the period may be shifted based on the rotational displacement and speed of the scratch operation. That is, while a scratch operation is being performed, the frame at the start of the scratch operation may be vibrated on the video processing device 101 side in response to the scratch operation. In this case, the beat object 201 moves vigorously within the frames of the timing video image while the scratch operation is being performed. This allows the audience or viewers to know that a scratch is being performed even if the DJ application 13 cannot decode the beat video image synchronously.

[0066] (Configuration of Video Processing Device) Each unit of the video processing device 101 according to this embodiment is typically realized by a computer that executes a program. The computer is connected to a controller that outputs a control signal indicating control by a DJ, an output device such as a music / video output system that is controlled by the controller to provide audio and video, and an input device that receives input of video output from the computer and relays or distributes it, and transmits and receives information to and from these devices.

[0067] A program executed by a computer can be distributed or sold by a server to which the computer is connected for communication, or it can be recorded on a non-transitory information recording medium such as a CD-ROM (Compact Disk Read Only Memory), flash memory, or EEPROM (Electrically Erasable Programmable ROM), and then the information recording medium can be distributed, sold, etc.

[0068] The program is installed on a non-transitory information recording medium such as a hard disk, solid-state drive, flash memory, or EEPROM of the computer. The computer then implements the video processing device 101 of this embodiment. Generally, a computer's central processing unit (CPU) reads the program from the information recording medium into random access memory (RAM) under the control of the computer's operating system (OS), and then interprets and executes the code contained in the program. However, in an architecture in which the information recording medium can be mapped within a memory space accessible by the CPU, explicit loading of the program into RAM may not be necessary. Various pieces of information required during program execution can be temporarily stored in RAM.

[0069] It should be noted that the video processing device 101 of this embodiment can be configured using a dedicated electronic circuit rather than using a general-purpose computer. In this embodiment, the program can also be used as material for generating wiring diagrams, timing charts, and the like for the electronic circuit. In such an embodiment, an electronic circuit that satisfies the specifications defined in the program is configured using an FPGA (Field Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit), and the electronic circuit functions as a dedicated device that performs the functions defined in the program, thereby realizing the video processing device 101 of this embodiment.

[0070] For ease of understanding, the following description will be given assuming that the video processing device 101 is realized by a computer executing a program. Note that the DJ application 13 may run on the computer that realizes the video processing device 101, or may run on another computer connected to the computer that realizes the video processing device 101.

[0071] 8 is an explanatory diagram showing a schematic configuration of a moving image processing device according to an embodiment of the present invention. The following description will be made with reference to this diagram.

[0072] A moving image processing device 101 according to this embodiment includes a setting unit 102, an input unit 103, a drawing unit 104, and an output unit 105. In addition, a detection unit 106 can be included as an optional element.

[0073] In the moving image processing device 101, the setting unit 102 sets a nearby position 213 in the vicinity of the default position 203 on the screen. The screen here corresponds to each frame 202 in the beat moving image.

[0074] Here, the nearby position 213 should be set so that in an environment where the music is output as sound and the timing video is output after being composited with other video and viewed, the beat objects 201 corresponding to each beat of the output sound will reach the nearby position 213 at the timing when each beat of the output sound is viewed.

[0075] For example, when the DJ system 11 including the video processing device 101 is ready, a song is selected from the DJ controller 12 as a test, and the audio and video are output from the DJ system 11. The operator or the like can then make corrections while comparing the output audio and video, thereby setting the nearby position 213.

[0076] On the other hand, the input unit 103 accepts input of a beat moving image to be played in synchronization with the music being played in response to a control signal output from the DJ controller 12. Here, as described above, the beat moving image is configured so that the beat objects 201 associated with each beat of the music move within the screen (frames 202) and reach predetermined positions 203 within the screen (frames 202) at the beat timing at which each beat of the music is played.

[0077] The beat video may be input via hardware wiring, or the beat video frames may be expanded in the RAM of a computer, and the information in this RAM may be treated as the beat video input.

[0078] Then, every time a frame 202 of the beat moving image is input, the drawing unit 104 draws a timing object 211 at a set nearby position 213 within the input frame 202.

[0079] Furthermore, the output unit 105 outputs frame 202 in which the timing object is drawn as a frame of the timing video. The output by the output unit 105 may be via hardware wiring, or may be a form in which a temporary frame is created in the RAM of a computer and left for subsequent processing.

[0080] The drawing unit 104 is configured to determine whether the beat object 201 is at the set nearby position 213 or not, and to change the appearance of the timing object 211 and draw it depending on the result of the determination, thereby making it possible to clearly show the audience / listener whether the current timing is the timing of the beat of the music.

[0081] Furthermore, the detection unit 106 detects, from the control signal output from the DJ controller 12, that a scratch operation is being performed on the DJ controller 12. When the detection unit 106 detects a scratch operation, the drawing unit 104 is configured to further draw a scratch object 215, so that the audience or listeners can know that the DJ is currently scratching without looking at the beat object 201.

[0082] Here, the scratch object 215 can be configured to be drawn at a position that is shifted by a predetermined displacement from the set nearby position 213. Then, by simply paying attention to the timing object 215 and its surroundings, the audience or listeners can easily tell whether the current timing is on a beat of the music or not, and whether a scratch operation is currently being performed or not.

[0083] In addition, while a scratch operation is being detected, the drawing unit 104 may shift the position of the origin of the frame input immediately before the scratch operation is detected according to the strength (displacement or speed) of the scratch operation, transfer it to a frame for timing video, and then draw the timing object 211 and the scratch object 215.

[0084] With this configuration, even if the scratch operation does not allow the beat video to be synchronized with the music, the audience or viewers can naturally sense that scratching is being performed.

[0085] The above processing may be performed by the video processing device 101 on each of the videos, whether the DJ application 13 outputs one song and one beat video, or whether two beat videos are output according to the two turntables of the DJ controller 12, or may be performed based on the beat video with the larger mixing ratio set in the DJ controller 12.

[0086] In addition, when a first song and a second song are mixed and played in response to a control signal output from the DJ controller 12, the input unit 103 may be configured to accept input of a first beat video that is played in synchronization with the first song and a second beat video that is played in synchronization with the second song.

[0087] Then, each time the first frame of the first beat video and the second frame of the second beat video are input simultaneously, the drawing unit 104 may combine the first frame and the second frame in accordance with the mixing ratio of the first music piece and the second music piece, and then draw the timing object 211 at the set nearby position 213.

[0088] When switching from a first song to a second song, DJs often align the beats of both songs. In such cases, the beat object 201 in the first frame and the beat object 201 in the second frame are positioned so that they overlap each other. Therefore, when the two are combined, a frame with almost the same content is obtained.

[0089] On the other hand, if the transition occurs while the beats are out of sync, the mixing ratio of the two pieces of music will change. When both frames are combined at that mixing ratio, a crossfade / crossdissolve transition is made from one beat video to the other.

[0090] (Control flow) Fig. 9 is a flowchart showing the control flow of video processing executed by the video processing device according to an embodiment of the present invention. The following description will be made with reference to this diagram. Note that the following description will be made assuming that two beat videos are output from the DJ application 13, but by omitting appropriate processing from the following description, it is also possible to deal with the case where one beat video is output.

[0091] When this process starts, the video processing device 101 first performs various initializations (step S300). This initialization may include, for example, restoring the neighboring position settings from the previous process, acquiring the current mixing ratio from the DJ controller 12, and so on.

[0092] Next, the video processing device 101 receives input from various external devices (step S301) and checks the type of the input (step S302).

[0093] If the received input is an adjustment instruction input to adjust the nearby position given by the operator via the keyboard, mouse, etc. of the video processing device 101 (step S302; adjustment), the video processing device 101 sets the nearby position to the position specified in the adjustment instruction input (step S303), and proceeds to step S321.

[0094] Then, the video processing device 101 obtains the mixing ratio of the first song and the second song in the DJ controller 12 (step S321), obtains the rotation speed of the two turntables in the DJ controller 12 (step S322), and executes other processes that should be repeated as appropriate (step S323), and returns control to step S301.

[0095] On the other hand, if the received input is a frame of each of the first beat video and the second beat video (hereinafter, the first beat video side is referred to as the first frame and the second beat video side is referred to as the second frame) output from the DJ application 13 (step S302; beat video frame), the video processing device 101 combines the first frame and the second frame using the latest acquired mixing ratio and writes the combined frame as a temporary frame in a temporary area in RAM (step S311). Note that, instead of combining the first frame and the second frame, the video processing device 101 may use a frame of the music side with a larger mixing ratio as the temporary frame.

[0096] Then, the moving image processing device 101 checks whether or not a beat object 201 is drawn at a nearby position based on pixel values ​​at nearby positions in the temporary frame or in the frame associated with the music piece having the larger mixing ratio between the first and second frames (step S312).

[0097] If the beat object 201 is not drawn in a nearby position (step S312: No), the moving image processing device 101 draws the timing object 211 in a normal form in a nearby position in the temporary frame by overwriting it (step S313), and proceeds to step S315.

[0098] On the other hand, if the beat object 201 is drawn in a nearby position (step S312; Yes), the timing object 211 in emphasis form is drawn over the nearby position in the temporary frame (step S314).

[0099] Next, the moving image processing device 101 checks whether or not a scratch operation is currently being performed based on the rotation speed of the turntable for the music piece with the larger mixing ratio (step S315).

[0100] If a scratch operation has not been performed (step S315; No), the video processing device 101 overwrites and draws the inactivated scratch object 215 at a position within the temporary frame that is moved by a predetermined relative displacement from a nearby position (step S316), and proceeds to step S318.

[0101] On the other hand, if a scratch operation is being performed (step S315; Yes), the video processing device 101 overwrites and draws the activated scratch object 215 at a position within the temporary frame that is moved by a predetermined relative displacement from a nearby position (step S317).

[0102] Then, the moving image processing device 101 outputs the contents of the temporary frame as a frame of the timing moving image (step S318), and proceeds to step S321.

[0103] If an input other than the above is received (step S302; Other), the process corresponding to the input is executed (step S331), and the process proceeds to step S321.

[0104] By executing such moving image processing, the moving image processing device 101 can generate and output a timing moving image in which the timing object 211 and the scratch object 215 are drawn from the input beat moving image.

[0105] (Adjustment of Nearby Positions) A method for adjusting nearby positions will be described below.

[0106] Here, the timing object 211 is drawn in pixel units. If the movement speed of the beat object 201 within the frame 202 is, for example, 200 pixels per second, moving the position of the timing object 211 by one pixel will result in an adjustment of 1 / 200=0.005 seconds.

[0107] The first method is a method in which a human or AI (Artificial Intelligence) operator makes adjustments based on audio. This is suitable for an environment in which there is almost no time lag between audio and video within the DJ system 11, that is, a situation in which the beat of a song is output when the beat object 201 reaches the predetermined position 203 in the output of the DJ system 11.

[0108] The operator compares the audio output from the DJ system 11 with the audio that is ultimately output for streaming or broadcasting using the audio. If necessary, these audio signals may be picked up again using a microphone or audio cable.

[0109] If the former is delayed, the nearby position is moved one pixel upward in the opposite direction to the movement of the beat object 201, and a delay of 0.005 seconds is applied to the latter. This process is then repeated until the two sounds are synchronized.

[0110] If the latter is delayed, the nearby position is moved by one pixel in the same direction as the movement of the beat object 201, i.e., downward, and a delay of 0.005 seconds is applied to the former. This process is then repeated until the two sounds are synchronized.

[0111] The proximity position at which the two sounds are finally synchronized is determined as the proximity position at which the timing object 211 should be drawn.

[0112] The determination of which of the two sounds is delayed and whether the two sounds are synchronized can be made by a human being, or it can be done automatically and mechanically using signal processing techniques.

[0113] The second method involves a human or AI operator making adjustments based on video footage.

[0114] To measure the time difference, a beat video is prepared in which markers that change color for each pixel are drawn parallel to the movement direction of the beat object 201. These markers function as a kind of "ruler."

[0115] As described above, the timing video output from the DJ system 11 is converted to SDI, processed by the equipment in the broadcast control room, and redrawn on the screen to be broadcast / distributed.

[0116] Therefore, among the finally output moving images, the area derived from the timing moving image and the timing moving image output from the DJ system 11 are arranged side by side to check the display discrepancy of the beat object 201.

[0117] Then, a position displaced from the default position 203 by the number of pixels of the displacement is determined as a neighboring position.

[0118] This process can be performed by a human operator comparing the screens, or by shooting two videos simultaneously with a camera and having a computer or AI that processes the images count the number of pixels that are out of sync.

[0119] The moving speed of the beat object 201, the time required to move one pixel, the number of pixels to be adjusted, the color configuration of the pixels used as the "ruler," etc. can be changed as desired depending on the settings and environment to be applied. The "ruler" may also be expressed by any pattern that can be recognized by humans or machines.

[0120] The third method is to directly adjust the video using sound. In this method, a beat video is created that produces a clicking sound at the timing of the beat (when the beat object 201 reaches the predetermined position 203).

[0121] When generating a timing video from a beat video, when outputting the timing video, or when superimposing the timing video on other video or audio, the click sound is also synthesized.

[0122] The final audio and video output is then recorded along with the audio, and played back along with the audio by either advancing one frame (or multiple frames) at a time or playing back in reverse frame by frame.

[0123] If the frame in which the click sound is heard and the frame in which the beat object 201 and timing object 211 overlap are not aligned, the position of the timing object 211 is corrected.

[0124] Here, three methods for adjusting the proximity position have been described: a method based on audio comparison, a method based on video comparison, and a method that compares audio and video. However, these may be combined in any manner, or other methods may be adopted to make the adjustment.

[0125] (Summary) As described above, the video processing device according to this embodiment is configured to include: a setting unit that sets a nearby position near a predetermined position on the screen; an input unit that accepts input of a beat video that is played in synchronization with music that is played in response to a control signal output from a DJ controller, wherein beat objects associated with each beat of the music move within the screen and the beat objects associated with each beat reach the predetermined position on the screen at the beat timing at which each beat of the music is played; a drawing unit that draws a timing object at the set nearby position within the input frame each time a frame of the beat video is input; and an output unit that outputs the frame in which the timing object is drawn as a frame of the timing video.

[0126] Furthermore, in the video processing device according to this embodiment, the nearby position can be configured to be set so that, in an environment in which the music is output as audio and the timing video is composited with other video, output, and viewed, beat objects associated with each beat of the output audio will reach the nearby position at the timing at which each beat of the output audio is viewed.

[0127] Furthermore, in the video processing device according to this embodiment, the drawing unit can be configured to determine whether the beat object is in the set nearby position or not, and to change the appearance of the timing object and draw it according to the result of the determination.

[0128] Furthermore, the video processing device according to this embodiment may further include a detection unit that detects from the output control signal that a scratch operation is being performed on the DJ controller, and when the scratch operation is detected, the drawing unit may be configured to further draw a scratch object.

[0129] Furthermore, in the video processing device according to this embodiment, the scratch object can be configured to be drawn at a position shifted by a predetermined displacement from the set nearby position.

[0130] Furthermore, in the video processing device according to this embodiment, when the scratch operation is detected, the drawing unit can be configured to shift the reference position of the frame according to the strength of the scratch operation, and then draw the timing object and the scratch object.

[0131] Furthermore, in the video processing device according to this embodiment, a first music piece and a second music piece are mixed and played in response to a control signal output from the DJ controller; the input unit receives input of a first beat video that is played in synchronization with the first music piece and a second beat video that is played in synchronization with the second music piece; and the drawing unit can be configured to, each time a first frame of the first beat video and a second frame of the second beat video are input simultaneously, combine the first frame and the second frame in response to a mixing ratio at which the first music piece and the second music piece are mixed, and then draw the timing object at the set nearby position.

[0132] The video processing method according to this embodiment is configured so that the video processing device sets a nearby position near a predetermined position on a screen, accepts input of a beat video that is played in synchronization with a piece of music that is played in response to a control signal output from a DJ controller, in which beat objects corresponding to each beat of the music move within the screen and the beat objects corresponding to each beat reach the predetermined position on the screen at the beat timing at which each beat of the music is played, draws a timing object at the set nearby position within the input frame each time a frame of the beat video is input, and outputs the frame in which the timing object is drawn as a frame of the timing video.

[0133] The program of this embodiment configures a computer to function as: a setting unit that sets a nearby position near a predetermined position on the screen; an input unit that accepts input of a beat video that is played in synchronization with a song that is played in response to a control signal output from a DJ controller, wherein beat objects corresponding to each beat of the song move within the screen and the beat objects corresponding to each beat reach the predetermined position on the screen at the beat timing at which each beat of the song is played; a drawing unit that draws a timing object at the set nearby position within the input frame each time a frame of the beat video is input; and an output unit that outputs the frame in which the timing object is drawn as a frame of the timing video.

[0134] A non-transitory computer-readable information recording medium according to this embodiment can be configured to record the above program.

[0135] The program can be distributed or sold by recording it on a non-transitory computer-readable information recording medium, or via a transitory transmission medium such as a computer communication network.

[0136] The present invention allows for various embodiments and modifications without departing from the broad spirit and scope of the present invention. Furthermore, the above-described embodiments are intended to illustrate the present invention and do not limit the scope of the present invention. That is, the scope of the present invention is defined by the claims, not the embodiments. Various modifications made within the scope of the claims and their equivalents are considered to be within the scope of the present invention. This application claims priority based on patent application No. 2024-017765, filed in Japan on Thursday, February 8, 2024, and the contents of that basic application are incorporated herein to the extent permitted by the laws and regulations of the designated countries.

[0137] According to the present invention, it is possible to provide a moving image processing device, a moving image processing method, a program, and an information recording medium that output moving images that indicate the beats of a piece of music being played.

[0138] 11 DJ system 12 DJ controller 13 DJ application 31 Audio post-processing unit 32 Video post-processing unit 101 Video processing device 102 Setting unit 103 Input unit 104 Drawing unit 105 Output unit 106 Detection unit

Claims

1. A video processing device comprising: a setting unit that sets a nearby position near a predetermined position on a screen; an input unit that accepts input of a beat video that is played in synchronization with a song that is played in response to a control signal output from a DJ controller, wherein beat objects corresponding to each beat of the song move within the screen and the beat objects corresponding to each beat reach the predetermined position on the screen at the beat timing at which each beat of the song is played; a drawing unit that draws a timing object at the set nearby position within the input frame each time a frame of the beat video is input; and an output unit that outputs a frame in which the timing object is drawn as a frame of the timing video.

2. The video processing device according to claim 1, characterized in that the nearby position is set so that, in an environment in which the music is output as audio and the timing video is output after being composited with other video and viewed, beat objects associated with each beat of the output audio will reach the nearby position at the timing when each beat of the output audio is viewed.

3. The video processing device according to claim 1 or 2, characterized in that the drawing unit determines whether the beat object is in the set nearby position, and changes the appearance of the timing object and draws it depending on the result of the determination.

4. The video processing device according to claim 1, further comprising a detection unit that detects from the output control signal that a scratch operation is being performed on the DJ controller, and when the scratch operation is detected, the drawing unit further draws a scratch object.

5. The video processing device according to claim 4, characterized in that the scratch object is drawn at a position shifted by a predetermined displacement from the set nearby position.

6. The video processing device according to claim 5, characterized in that when the scratch operation is detected, the drawing unit shifts the reference position of the frame according to the strength of the scratch operation, and then draws the timing object and the scratch object.

7. The video processing device according to claim 1, wherein a first music piece and a second music piece are mixed and played in response to a control signal output from the DJ controller; the input unit receives input of a first beat video that is played in synchronization with the first music piece and a second beat video that is played in synchronization with the second music piece; and the drawing unit, each time a first frame of the first beat video and a second frame of the second beat video are input simultaneously, combines the first frame and the second frame in response to a mixing ratio at which the first music piece and the second music piece are mixed, and then draws the timing object at the set nearby position.

8. A video processing method characterized in that a video processing device sets a nearby position near a predetermined position on a screen, accepts input of a beat video that is played in synchronization with music that is played in response to a control signal output from a DJ controller, in which beat objects corresponding to each beat of the music move within the screen and the beat objects corresponding to each beat reach the predetermined position on the screen at the beat timing at which each beat of the music is played, draws a timing object at the set nearby position within the input frame each time a frame of the beat video is input, and outputs the frame in which the timing object is drawn as a frame of the timing video.

9. A program that causes a computer to function as: a setting unit that sets a nearby position near a predetermined position on the screen; an input unit that accepts input of a beat video that is played in synchronization with music that is played in response to a control signal output from a DJ controller, wherein beat objects associated with each beat of the music move within the screen and the beat objects associated with each beat reach the predetermined position on the screen at the beat timing at which each beat of the music is played; a drawing unit that draws a timing object at the set nearby position within the input frame each time a frame of the beat video is input; and an output unit that outputs the frame in which the timing object is drawn as a frame of the timing video.

10. A non-transitory computer-readable information recording medium on which the program according to claim 9 is recorded.

Images