Information processing device and information processing method

Abstract

This information processing device is provided with a display unit and a control unit. When an application having a recording function using a camera and a microphone and an editing function for recorded data obtained by the recording function is running, the display unit displays visual information relating to the application. The control unit executes the application, and when sound source data, the sound quality of which is to be improved, is selected from the recorded data by a user via the application, executes processing for improving the sound quality of the selected sound source data, and causes the display unit to display an operation component related to the editing function so that the sound source data and the video data included in the recorded data can be matched and edited, including the case where the sound quality improvement processing has been completed.

Description

Information processing device and information processing method

[0001] The present disclosure relates to an information processing device and an information processing method.

[0002] In recent years, the popularity of portable information processing devices such as smartphones and tablet terminals has been remarkable. Furthermore, the processing capabilities of these portable information processing devices have been rapidly improving, enabling users to easily create various media content, such as music and video, on their own using such information processing devices.

[0003] In the personal production of music content, various music production application software (hereinafter referred to as "apps") that run on information processing devices are often used. Some of these music production apps have a recording function, allowing users to use such apps to freely edit audio data recorded by themselves singing or playing, or audio data purchased from external sources.

[0004] In addition, some music production apps support the collaborative production of music content by allowing multiple users to edit audio data recorded on multiple information processing devices via a network (see, for example, Patent Document 1).

[0005] International Publication No. 2019 / 102730

[0006] However, users often have the opportunity to publish music content with video that they have created by recording themselves singing or playing music, etc., using social networking services (SNSs), video distribution services, etc. The above-mentioned conventional techniques have room for further improvement in terms of improving the convenience of creating such music content with video.

[0007] Therefore, the present disclosure proposes an information processing device and an information processing method that can further improve the convenience when a user creates music content with video.

[0008] In order to solve the above-described problems, an information processing device according to one embodiment of the present disclosure includes a display unit and a control unit. When an application having a recording function using a camera and a microphone and an editing function for data recorded by the recording function is running, the display unit displays visual information related to the application. The control unit executes the application, and when a user selects sound source data to be improved in sound quality from the recorded data via the application, the control unit performs a sound quality improvement process on the sound source data and causes the display unit to display operation components related to the editing function so that the sound source data and video data included in the recorded data, including those that have already been improved in sound quality, can be edited in a consistent manner.

[0009] FIG. 1 is a diagram illustrating an overview of an information processing method according to an embodiment of the present disclosure. FIG. 2 is a block diagram illustrating a configuration example of a smartphone according to an embodiment of the present disclosure. FIG. 3 is a block diagram illustrating a configuration example of a server device according to an embodiment of the present disclosure. FIG. 4 is a diagram illustrating a UI example during recording (part 1). FIG. 5 is a diagram illustrating a UI example during recording (part 6). FIG. 6 is a diagram illustrating a UI example during recording (part 7). FIG. 7 is a diagram illustrating a UI example during recording (part 8). FIG. 8 is a diagram illustrating a UI example during recording (part 9). FIG. 9 is a diagram illustrating a UI example during editing. FIG. 10 is a diagram illustrating a UI example during export. FIG. 11 is a diagram illustrating a UI example related to a sound quality improvement processing function. FIG. 12 is a diagram illustrating a UI example related to a sound quality improvement processing function. FIG. 13 is a diagram illustrating a UI example related to a sound quality improvement processing function. FIG. 14 is a diagram illustrating a UI example related to a sound quality improvement processing function. FIG. 5 is a diagram (part 5) showing an example of a UI related to the sound quality improvement processing function. FIG. 1 is a diagram (part 1) showing an example of a UI related to the trimming function. FIG. 2 is a diagram (part 2) showing an example of a UI related to the trimming function. FIG. 3 is a diagram (part 3) showing an example of a UI related to the trimming function. FIG. 4 is a diagram (part 4) showing an example of a UI related to the trimming function. FIG. 5 is a diagram (part 6) showing an example of a UI related to the trimming function. FIG. 7 is a diagram (part 7) showing an example of a UI related to the trimming function. FIG. 8 is a diagram (part 8) showing an example of a UI related to the trimming function. FIG. 1 is a diagram (part 1) showing an example of a UI related to the shift playback function. FIG. 2 is a diagram (part 3) showing an example of a UI related to the shift playback function. FIG. 4 is a diagram (part 1) showing an example of a UI related to the fifth modified example. FIG. 2 is a diagram (part 2) showing an example of a UI related to the fifth modified example. 10 is a diagram (part 3) showing an example of a UI according to the fifth modified example. FIG. 11 is a diagram (part 4) showing an example of a UI according to the fifth modified example. FIG. 12 is a hardware configuration diagram showing an example of a computer that realizes the functions of a smartphone.

[0010] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In the following embodiments, the same components are designated by the same reference numerals, and redundant description will be omitted.

[0011] In the following description, the information processing device according to an embodiment of the present disclosure (hereinafter referred to as "the present embodiment") will be mainly described as a smartphone 10 used by a user. In the following description, a music production app that runs on the smartphone 10 will be referred to as a "music production app." The music production app according to the present embodiment is assumed to have a recording function.

[0012] In the following description, video data, which is the moving image portion of the recorded data recorded by the smartphone 10, will be referred to as a "video frame" where appropriate. Audio source data, which is the audio portion of the recorded data, will be referred to as a "track" where appropriate. Audio source data purchased from an external source and stored in the smartphone 10's storage will be referred to as a "loaded file" where appropriate. An example of a loaded file is a karaoke audio source.

[0013] In the following description, the function of improving the sound quality of tracks in the music production app according to this embodiment will be referred to as the "sound quality improvement function." The sound quality improvement function may be illustrated as "Studio tuning" or "STUDIO TUNING" in the drawings shown below. The processing executed by the sound quality improvement function will be referred to as the "sound quality improvement processing."

[0014] The present disclosure will be described in the following order: 1. Overview 2. Example of a Smartphone Configuration 3. Example of a Server Device Configuration 4. Specific Examples of Various UIs for Producing Music Content with Video 4-1. Basic Example 4-1-1. During Recording 4-1-2. During Editing 4-1-3. During Export 4-2. High-Quality Sound Processing Function 4-3. Trimming Function 4-4. Shift Playback Function 5. Modifications 5-1. First Modification 5-2. Second Modification 5-3. Third Modification 5-4. Fourth Modification 5-5. Fifth Modification 5-6. Sixth Modification 6. Hardware Configuration 7. Conclusion

[0015] <<1. Overview>> Fig. 1 is a diagram illustrating an overview of an information processing method according to an embodiment of the present disclosure. The information processing method according to the present embodiment is executed by an information processing system 1. As shown in Fig. 1, the information processing system 1 includes a smartphone 10 and a server device 100.

[0016] The smartphone 10 is a portable information processing device used by a user. The smartphone 10 has a platform capable of executing the music production application according to this embodiment.

[0017] 1, the smartphone 10 has a camera 11, a microphone 12, a speaker 13, and a display unit 16. The camera 11 is provided at one end (hereinafter referred to as "top") of the smartphone 10 near the earpiece in the longitudinal direction. The camera 11 is a front camera that enables the user to take selfies.

[0018] The microphone 12 is provided at the other end (hereinafter referred to as "lower") closer to the mouthpiece than the one end. The speaker 13 is provided at a position corresponding to the earpiece, for example.

[0019] The display unit 16 is realized as a touch screen integrated with the operation unit 15 (see FIG. 2 ), which will be described later, and has a rectangular shape. The touch screen detects touch operations by the user. Known detection methods for the touch screen include a capacitance method, a resistive film method, a surface acoustic wave method (or an ultrasonic method), an infrared method, an electromagnetic induction method, and a load detection method.

[0020] The touch screen displays various user interfaces (UIs) related to the music production app when the music production app is running on the smartphone 10. The touch screen also detects user touch operations on these UIs.

[0021] When an operation to start recording is performed in the music production app, the camera 11 captures a scene, for example, of a user singing and playing the guitar. Similarly, when an operation to start recording is performed, the microphone 12 collects sounds emitted from a sound source (e.g., the user or an instrument) and records them as sound source data of a track type designated in advance for recording. When an operation to play sound source data is performed in the music production app, the speaker 13 outputs the sound source data selected for playback.

[0022] During playback, the music production app allows the user to adjust various parameters, such as the volume and reverb strength of each track. The "volume" parameter is sometimes referred to as "Volume" in the drawings shown below. Similarly, "reverb" is sometimes referred to as "Reverb."

[0023] However, conventional music production apps have room for further improvement in terms of providing greater convenience for users when producing music content with video.

[0024] As already mentioned, in recent years, the processing power of portable information processing devices such as smartphones 10 has rapidly improved, and users are now able to easily create music content with video on their own using music production apps. This has also led to an increase in opportunities for users to make the music content with video they have created publicly available, for example, via social networking sites or video distribution services.

[0025] For this reason, there is a growing need for users to easily create high-quality video-accompanied music content using a user interface that is easier to use and has superior operability. "High quality" here refers to, for example, high-quality audio and high synchronization between audio and video.

[0026] Conventional music production apps have not yet fully expanded their UIs to meet these needs. In particular, there is room for improvement in the operability of music production apps on devices such as smartphones 10, where the size of the display unit 16 is smaller than that of a typical PC display.

[0027] In the information processing method according to the present embodiment, when a user selects audio data to be enhanced in sound quality from recorded data via a music production app, the smartphone 10 performs a sound quality enhancement process on the audio data. The smartphone 10 also displays operation components related to an editing function on the display unit 16 so that the audio data, including the audio data that has already been enhanced in sound quality, and the video data included in the recorded data can be edited in a consistent manner.

[0028] In the information processing method according to the present embodiment, the smartphone 10 displays operation components that enable collective or individual editing of audio source data and video data, including data that has already been subjected to high-quality sound processing, on the display unit 16. The above-mentioned "consistent editing" refers to the ability to edit audio source data and video data that are consistent overall and consistent, while performing sound quality processing and a trimming function (described later) using such collective or individual editing operation components.

[0029] 1, the smartphone 10 first executes a music production app with a recording function (step S1). When the user uses the sound quality improvement function of the music production app, the smartphone 10 transmits sound source data selected as the target for sound quality improvement processing from the recorded data to the server device 100 (step S2).

[0030] At this time, an option is specified for the selected sound source data. Examples of the option include "sound source separation" for separating multiple different sound sources from one sound source data recorded using the microphone 12, "noise reduction" for removing noise, and "dereverberation" for removing unnecessary reverberation. Other examples of the option include "microphone simulation" for reproducing the microphone characteristics of a high-performance microphone, and "studio simulation" for adding reverberations unique to a recording studio or the like.

[0031] The server device 100 is a device that is provided so as to be able to communicate with the smartphone 10 via a network N such as a mobile phone network or the Internet. The server device 100 is realized as, for example, a cloud server.

[0032] The server device 100 has a sound quality improvement processing model 102a (see FIG. 3 ) and is configured to be able to execute sound quality improvement processing using this sound quality improvement processing model 102a. The sound quality improvement processing model 102a is, for example, a deep neural network (DNN) model trained using a deep learning algorithm.

[0033] The server device 100 performs sound quality improvement processing on the sound source data acquired from the smartphone 10 by AI (Artificial Intelligence) processing using the sound quality improvement processing model 102a (step S3).The server device 100 then transmits the sound quality improved data to the smartphone 10, and the smartphone 10 receives it (step S4).

[0034] Furthermore, since the server device 100 executes the high-quality sound processing, which has a high processing load, in an integrated manner using a cloud server model, the smartphone 10 can continue to use the music production app or run other apps during the high-quality sound processing, as long as it does not affect the high-quality sound processing.

[0035] The smartphone 10 then executes an editing function via a UI that enables editing so that the audio data and the video data are consistent with each other (step S5). Specifically, the smartphone 10 provides a UI related to a trimming function that, for example, trims the audio data and the video data collectively so that the audio data and the video data are consistent with each other. The smartphone 10 also provides a UI that, for example, changes or moves the audio data individually with respect to the video data so that the audio data and the video data are consistent with each other.

[0036] The smartphone 10 provides these UIs as UIs that are easy for the user to intuitively operate and that are easy to use even on a device with a relatively small display unit 16, such as the smartphone 10. In addition to these, the smartphone 10 also provides a UI for guidance on recording methods, so that, for example, vocal sound sources can be more easily produced with higher sound quality when singing and playing the guitar.

[0037] Specific examples of these various UIs in this embodiment will be described later with reference to FIGS.

[0038] <<2. Configuration Example of Smartphone>> Next, a configuration example of the smartphone 10 will be described. Fig. 2 is a block diagram showing a configuration example of the smartphone 10 according to an embodiment of the present disclosure. Note that Fig. 2 and Fig. 3 shown later show only components necessary for explaining the features of the embodiment of the present disclosure, and descriptions of general components are omitted.

[0039] 2 and 3 are functional concepts and do not necessarily have to be physically configured as shown. For example, the specific form of distribution and integration of each block is not limited to that shown, and all or part of them can be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc.

[0040] In addition, in the description using FIGS. 2 and 3, the description of components that have already been described may be simplified or omitted.

[0041] As shown in FIG. 2 , the smartphone 10 includes a camera 11 , a microphone 12 , a speaker 13 , a communication unit 14 , an operation unit 15 , a display unit 16 , a storage unit 17 , and a control unit 18 .

[0042] The camera 11, microphone 12, and speaker 13 have already been described, and therefore will not be described here. The communication unit 14 is realized by a communication module or the like. The communication unit 14 is wirelessly connected to the network N described above, and realizes communication with the server device 100 via the network N.

[0043] The operation unit 15 is an operation component mounted on the smartphone 10 and is realized by, for example, a touch panel. The display unit 16 is a display component mounted on the smartphone 10 and is realized by, for example, a display.

[0044] The operation unit 15 and the display unit 16 are, for example, integrated to form the touch screen described above. Therefore, the operation unit 15 may be a software component displayed on the display unit 16, or may be, for example, a GUI (Graphical User Interface) that a music production application displays on the display unit 16 in an operable manner.

[0045] The storage unit 17 is realized by, for example, a semiconductor memory element such as a random access memory (RAM), a read only memory (ROM), a flash memory, etc. In the example shown in Fig. 2, the storage unit 17 stores application information 17a, recording information 17b, and editing information 17c.

[0046] The application information 17a includes the program of the music production application, various parameters used during operation of the music production application, and the like.

[0047] The recording information 17b includes video data recorded by the recording function of the music production application. The video data includes video data captured and recorded by the camera 11. The video data also includes audio data collected and recorded by the microphone 12.

[0048] The editing information 17c is information including music content with moving images that is being edited or has been edited by the user using the editing function of the music production application.

[0049] The music production app according to this embodiment simultaneously records and films in MP4 format, but because the MP4 recording data is compressed using AAC (Advanced Audio Coding), the music production app saves the audio data separately in uncompressed WAV (RIFF waveform Audio Format) format. The music production app then processes the audio data in WAV format to improve sound quality and adjust parameters, then converts it to AAC when exporting it as MP4 audio data.

[0050] The control unit 18 corresponds to a so-called processor or controller. The control unit 18 is realized by, for example, a central processing unit (CPU), a micro processing unit (MPU), a graphical processing unit (GPU), or the like executing a program according to this embodiment stored in the storage unit 17 using RAM as a work area. The control unit 18 can also be realized by, for example, an integrated circuit such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA).

[0051] The control unit 18 has an application execution unit 18a, a display control unit 18b, and a communication control unit 18c, and realizes or executes the functions and actions of information processing described below.

[0052] The application execution unit 18a executes the music production application by reading the application information 17a stored in the storage unit 17. The application execution unit 18a executes the sound quality improvement process in accordance with, for example, the operation content for executing the sound quality improvement process in the music production application input from the operation unit 15. The application execution unit 18a also executes the editing function in accordance with, for example, the operation content for executing the editing function in the music production application input from the operation unit 15.

[0053] Furthermore, the application execution unit 18a causes the display control unit 18b to control the display of visual information, including various UIs, in accordance with the content of various operations performed when executing the music production application, including the sound quality improvement processing and editing functions. Specific examples of the various UIs in the music production application according to this embodiment will be described later with reference to FIGS. 4 to 31.

[0054] The display control unit 18b controls the display of visual information on the display unit 16 when the music production app is executed based on instructions from the app execution unit 18a. The communication control unit 18c controls communication with the server device 100 when the music production app is executed.

[0055] The communication control unit 18c transmits, for example, sound source data to be subjected to sound quality improvement processing to the server device 100 via the communication unit 14. In addition, the communication control unit 18c receives, for example, sound quality improved data that has been subjected to sound quality improvement processing in the server device 100 from the server device 100 via the communication unit 14.

[0056] <<3. Configuration Example of Server Device>> Next, a description will be given of a configuration example of the server device 100. Fig. 3 is a block diagram showing a configuration example of the server device 100 according to an embodiment of the present disclosure.

[0057] As shown in FIG. 3, the server device 100 includes a communication unit 101 , a storage unit 102 , and a control unit 103 .

[0058] The communication unit 101 is realized by a communication module or the like, similar to the above-described communication unit 14. The communication unit 101 is connected to the above-described network N by wire or wirelessly, and realizes communication with the smartphone 10 via the network N.

[0059] The storage unit 102, like the storage unit 17 described above, is realized by, for example, a semiconductor memory element such as a RAM, a ROM, or a flash memory. The storage unit 102 is also realized by a disk device such as a hard disk device or an optical disk device. In the example shown in Fig. 3, the storage unit 102 stores a sound quality improvement processing model 102a.

[0060] As already mentioned, the sound quality improvement processing model 102a is a DNN model trained using, for example, a deep learning algorithm. Note that the learning algorithm of the sound quality improvement processing model 102a is not limited to deep learning. The sound quality improvement processing model 102a is read by the sound quality improvement processing unit 103b (described later), and when sound source data to be processed is input, the sound quality improvement processing model 102a performs sound quality improvement processing on the sound source data using specified settings (options) and outputs the sound quality processed data.

[0061] The control unit 103 corresponds to a so-called processor or controller, similar to the above-described control unit 18. The control unit 103 is realized by, for example, a CPU, an MPU, a GPU, or the like executing various programs (not shown) stored in the storage unit 102 using a RAM as a work area. The control unit 103 can also be realized by, for example, an integrated circuit such as an ASIC or an FPGA.

[0062] The control unit 103 has an acquisition unit 103a, a sound quality improvement processing unit 103b, and a transmission unit 103c, and realizes or executes the functions and actions of information processing described below.

[0063] The acquisition unit 103a acquires sound quality improvement processing options specified on the smartphone 10 and sound source data to be processed via the communication unit 101. The acquisition unit 103a also outputs the acquired data to the sound quality improvement processing unit 103b.

[0064] The sound quality improvement processing unit 103b reads data from the storage unit 102 into the sound quality improvement processing model 102a, and inputs each piece of data received from the acquisition unit 103a to the sound quality improvement processing model 102a. The sound quality improvement processing unit 103b also acquires the sound quality processed data output from the sound quality improvement processing model 102a, and outputs it to the transmission unit 103c.

[0065] The transmitting unit 103 c transmits the data that has been subjected to the high-quality sound processing, received from the high-quality sound processing unit 103 b, to the smartphone 10 via the communication unit 101 .

[0066] <<4. Specific Examples of Various UIs Used When Producing Music Content with Moving Image>> Next, specific examples of various UIs used when producing music content with moving image in the music production application according to this embodiment will be described in order using FIGS. 4 to 31. FIG.

[0067] <4-1. Basic Example> [4-1-1. During Recording] First, an example of a UI during recording will be described. Figures 4 to 12 are diagrams (No. 1) to (No. 9) showing examples of a UI during recording.

[0068] The display control unit 18b can display a project list screen on the display unit 16, as shown in Fig. 4. This project list screen displays a list of projects that the user has created so far. When the "Audio" tab is selected, a list of projects based on audio data recorded using only the microphone 12 is displayed. On the other hand, when the "Video" tab T1 is selected, a list of projects based on video data recorded using the camera 11 and the microphone 12 is displayed.

[0069] The project list screen displays a button B1 labeled "+ New Project" for adding a new project. When the user touches this button B1, a dialog box appears asking whether the new project will be an "Audio" (sound recording) or "Video" (video recording) project. To record, the user selects the "Video" button B2, enters a project name in the new project creation dialog box that appears, and touches the "OK" button B3 to create a new project for recording.

[0070] On the recording screen for a new project by recording, a preview screen of the camera 11 is displayed in the preview area R1, as shown in Fig. 5. Basically, when the user touches the record button B4 on this recording screen, recording starts.

[0071] When a user sings and plays the guitar, in order to improve the quality of the sound recorded, the recording must be made not too far from the sound source (vocals or instruments) to the microphone 12. At the same time, when singing and playing the guitar, the sound of the instruments tends to be louder, so it is desirable to place the microphone closer to the vocals than to the instruments.

[0072] Specifically, assume that user U1 is singing and playing the guitar, as shown in Figure 6. If smartphone 10 is set up in the normal orientation (camera 11 facing up and microphone 12 facing down) for recording, as shown in the upper diagram of Figure 6, the guitar will not be captured in the field of view AV1 of camera 11. Furthermore, because microphone 12 faces downward, there is a risk that the sound of the guitar will be recorded louder than the sound of the vocals at vocal position VP1.

[0073] To resolve this, it is desirable to rotate the smartphone 10 180 degrees. When the smartphone 10 is turned upside down, the guitar is more likely to be captured in the field of view AV1 of the camera 11, as shown in the lower diagram of Figure 6. In addition, because the microphone 12 faces upward, the vocal sound at the vocal position VP1 is captured well, and because the guitar is located in the opposite direction from the microphone 12, it is recorded quietly, allowing the sounds of each sound source to be recorded in a well-balanced manner. Furthermore, a good volume balance between the vocals and the instruments can also improve the accuracy of sound source separation in, for example, sound quality enhancement processing.

[0074] The music production app according to this embodiment can provide guidance to the user taking this into consideration. Specifically, when the user touches button B3 shown in Fig. 4, the display control unit 18b displays a dialog box D1 on the display unit 16, which provides guidance on "how to record singing and playing the guitar," as shown in the left diagram of Fig. 7.

[0075] The dialog D1 displays a button B5 for when the user is singing along with playing the guitar, and when the user touches this button B5, a dialog D2 is displayed on the display unit 16, as shown in the right diagram of Fig. 7, which provides guidance regarding the content described with reference to Fig. 6.

[0076] Then, when the user touches the "OK" button B6 of this dialog D2 and rotates the smartphone 10 180 degrees as shown by the arrow a1 in Figure 8, the display control unit 18b displays a dialog D3 on the display unit 16, asking whether to flip the screen display upside down.

[0077] In response to this, if the user selects "invert" in the dialog D3, the display control unit 18b performs display control to invert the screen display upside down as shown in the right diagram of Fig. 8. In this way, by providing the user with a so-called "inverted UI" as described with reference to Figs. 6 to 8, it is possible to provide the user with guidance on the optimal recording method for scenes where the user is singing and playing the guitar.

[0078] Note that the display control unit 18b may cause the display unit 16 to display the dialog D3 again when the user rotates the smartphone 10 by 180 degrees from the state shown in the right diagram of FIG.

[0079] Furthermore, if the user returns the smartphone 10 to the state before rotation without pressing "Cancel" or "Invert" while the dialog D3 is displayed, the display control unit 18b may hide (erase) the dialog D3.

[0080] Furthermore, during recording, the display control unit 18b may be configured not to display the dialog D3 even when the smartphone 10 is rotated.

[0081] Furthermore, the display of the inversion UI may be set to be linked to the "Auto rotation" setting of the smartphone 10, forced inversion, or a dialog display. When linked to "Auto rotation" or forced inversion is selected, the display control unit 18b automatically inverts the screen display without displaying a dialog.

[0082] Next, on the recording screen, a load file button B7 indicated by "+ Load track" is displayed, as shown in the left diagram of Fig. 9. When the user touches this button B7, the load file specified by the user can be loaded. Once the load file is loaded, the waveform of the load file is displayed in the waveform display area corresponding to the musical note icon, as shown in the center diagram of Fig. 9.

[0083] When the user touches the record button B4, recording begins. During recording, the waveform of the audio being recorded is displayed in the waveform display area corresponding to the microphone icon, as shown on the right side of Fig. 9. Also, during recording, a record stop button B8 is displayed on the display unit 16. When the user touches this record stop button B8, recording stops.

[0084] When the recording is stopped, the display control unit 18b transitions the display on the display unit 16 to an editing screen, as shown in Fig. 10. On the editing screen, the recorded video frames are displayed in the video frame display area R2, and the recorded tracks are displayed in the track display area R3.

[0085] A re-record button B9 is also displayed on the edit screen. When the user touches this button B9, the display returns to the state shown in the center of Fig. 9, and the user can re-record by touching the record button B4. Overdubbing is not possible.

[0086] If the re-record button B9 is touched in the state shown in Fig. 10, before the sound quality improvement process is performed, a dialog D4 is displayed asking about the re-recording mode according to the state before the process is performed, as shown in Fig. 11. Also, if the sound quality improvement process is performed after the process is performed, a dialog D5 is displayed asking about the re-recording mode according to the state after the process is performed, as shown in Fig. 12.

[0087] Dialogue D5 differs from dialogue D4 in that it clearly indicates to the user that discarding recorded data will also delete data that has undergone sound quality enhancement processing. This is made clear to the user in consideration of the fact that sound quality enhancement processing is a paid service.

[0088] 13 is a diagram showing an example of a UI during editing. On the editing screen, as shown in FIG. 13, the display control unit 18b displays an icon I1 for adjusting parameters of each track. When the user touches this icon I1, the display control unit 18b displays a parameter adjustment component area overlapping the waveform display area of ​​each track.

[0089] As shown in the center diagram of Fig. 13, when the volume tab is selected by default, the display control unit 18b displays a volume adjustment component area R4. When the user selects the reverb tab T2, the display control unit 18b displays reverb adjustment component areas R5-1 and R5-2, as shown in the right diagram of Fig. 13.

[0090] The volume control component area R4 displays a slider that allows for volume adjustment for each track, and the reverb control component area R5-1 displays a pull-down menu and a switch.

[0091] When the user touches the pull-down menu in the reverb adjustment component area R5-1, various pre-prepared reverb presets are displayed. The user can set the reverb using a preset by selecting any one of these presets. The global switch in the reverb adjustment component area R5-1 is displayed as a switch that allows the overall reverb adjustment function to be turned on or off.

[0092] The reverb adjustment component area R5-2 displays sliders that allow reverb adjustment for each track, and switches that allow the reverb adjustment function for each track to be turned on or off.

[0093] The user can easily adjust the parameters of each track by intuitively operating the operation components displayed in the volume adjustment component area R4 and the reverb adjustment component areas R5-1 and R5-2.

[0094] [4-1-3. Exporting] Next, Fig. 14 is a diagram showing an example of a UI during exporting. Exporting refers to outputting the video-accompanied music content edited via the editing screen. As shown in the left diagram of Fig. 14, when the user touches the export button B10 indicated by "EXPORT" on the editing screen, the display control unit 18b transitions the display on the display unit 16 to the export screen shown in the center diagram of Fig. 14.

[0095] When the user touches the EXPORT button B11 on this export screen, the display control unit 18b displays a dialog D6 for exporting, as shown in the right diagram of Fig. 14. If the user specifies a file name and resolution in the dialog D6 and touches the "EXPORT" button, the edited music content with video is output and stored in the editing information 17c.

[0096] <4-2. Sound Quality Improvement Processing Function> Next, an example of a UI related to the sound quality improvement processing function will be described. Figs. 15 to 19 are diagrams (1) to (5) showing examples of a UI related to the sound quality improvement processing function. As shown in the left diagram of Fig. 15, when the user touches the sound quality improvement processing execution button B12 indicated by "STUDIO TUNING" on the editing screen, the display control unit 18b displays a menu M1 that allows the user to select the type of sound source to be targeted, as shown in the center diagram of Fig. 15.

[0097] Here, when the user selects, for example, the "Vocals & Guitar" button B13 and touches the "Next" button B14, the display control unit 18b displays a menu M2 for specifying sound quality improvement processing options on the display unit 16, as shown in the right diagram of Fig. 15. When the user specifies a desired option in the menu M2 and touches the execute button B15, the target sound source data is transmitted to the server device 100.

[0098] Then, when the sound quality improvement processing is performed in the server device 100 and the sound source data that has been processed for high quality sound is received from the server device 100, the display control unit 18b displays the waveform of the sound source data that has been processed for high quality sound in the track display areas R3-1 and R3-2, as shown in Figure 16.

[0099] 15, if the user selects "vocals & guitar" for singing and playing as the type of sound source, the sound source data transmitted to the server device 100 is separated into a vocal track and a guitar track by sound source separation, which is part of the sound quality improvement processing. Then, as shown in FIG. 16, the waveform of the vocal track corresponding to the microphone icon is displayed in the track display area R3-1. Also, the waveform of the guitar track corresponding to the guitar icon is displayed in the track display area R3-2.

[0100] Furthermore, once the sound quality improvement process has been performed, the display control unit 18b grays out the sound quality improvement process execution button B12 (see the dashed line in the figure) and disables it so that it cannot be operated. If the user wants to redo the sound quality improvement process, as shown in the left diagram of Fig. 17, when the user touches the waveform display area R6 of the track that has already been processed for sound quality improvement, the display control unit 18b displays a menu M3 on the display unit 16, as shown in the center diagram of Fig. 17.

[0101] Menu M3 is a menu including buttons such as "Change sound source," "Move left," and "Move right." When the user selects the "Change sound source" button B16, the display control unit 18b causes the display unit 16 to display a menu M4 related to changing the sound source, as shown in the right diagram of FIG.

[0102] "Tuned" in the menu M4 corresponds to sound source data that has been processed for high sound quality. "Original" corresponds to sound source data before the sound quality processing. When the user selects the "Original" button B17, as shown in FIG. 18, the display control unit 18b displays the waveform of the sound source data before the sound quality processing (i.e., before sound source separation) in the track display area R3, instead of the display of the waveform of each track that has been processed for high sound quality. In addition, the display control unit 18b ungrays out the sound quality processing execution button B12, making it operable. By operating this button B12, the user can redo the sound quality processing.

[0103] As shown in Fig. 19, if there is more than one piece of sound source data that has been subjected to sound quality enhancement processing in the past, the menu M4 can be used to select it (see "Tuned(1)" and "Tuned(2)" in the figure). By selecting the desired sound source data from the menu M4 in Fig. 19, the user can change the sound source to one of the sound source data that has been subjected to sound quality enhancement processing in the past.

[0104] <4-3. Trimming Function> Next, an example of a UI related to the trimming function will be described. Figures 20 to 27 are diagrams (part 1) to (part 8) showing examples of the UI related to the trimming function. The trimming function according to this embodiment is a function that specifies the portion to be trimmed from the beginning or end of a video frame, a recording track, or an imported file, and processes it so that only the portion to be kept is retained in the video-accompanied music content that the user creates.

[0105] As shown in the left figure of Figure 20, when a user touches the video frame portion R7 in the video frame display area R2 (see Figure 10) on the editing screen, the display control unit 18b displays the trimming frame Tr0 on the display unit 16, as shown in the right figure of Figure 20.

[0106] The trimming frame Tr0 is an operation component that allows the user to specify the portion to be cut from the beginning or end of a video frame, audio track, or loaded file all at once.

[0107] 21 , when the user moves their finger to the right while touching the left edge or left corner of the trimming frame Tr0, the portions to be cropped from the beginning of the video frame, audio track, and imported file are specified according to the amount of finger movement (specified amount). On the other hand, although not shown, when the user moves their finger to the left while touching the right edge or right corner of the trimming frame Tr0, the portions to be cropped from the end of the video frame, audio track, and imported file are specified according to the amount of finger movement (specified amount).

[0108] At this time, the display control unit 18b disables the portion designated to be cut out by the trimming frame Tr0 by displaying it in gray, as shown in the right diagram of Fig. 21. Note that when the user performs an operation to expand the trimming frame Tr0 to the left or right, the display control unit 18b can restore the disabled portion, cancel the grayed-out display, and enable it.

[0109] As shown in the right diagram of Fig. 20, when the display control unit 18b displays the trimming frame Tr0, it hides the seek bar SK1 during playback so that it does not interfere with the operability of the trimming function. Furthermore, as shown as "15:00:00" or "14:00:00" in Fig. 21, the display control unit 18b constantly displays the length of the valid data currently valid through the trimming function. Furthermore, the preview area R1 described above displays the video frame corresponding to the position of the user's closest finger.

[0110] Such a UI allows the user to intuitively specify the portion to be cut from the beginning or end of a video frame, audio track, or loaded file all at once.

[0111] Also, as shown in Figure 22, if the loaded file is longer than the recorded video frame and audio track, the display control unit 18b automatically trims and cuts out the portion that is longer than the video frame and audio track using a trimming frame Tr0.

[0112] 21 and 22, the portion cropped by the trimming frame Tr0 is actually applied only to the video-accompanied music content that is output by the export process described above. In other words, the video frames, recording tracks, high-quality audio recording tracks, and the original data of the imported file before trimming that were recorded using the recording function of the music production application are maintained unchanged when exported.

[0113] For example, as shown in FIG. 23, it is assumed that there is a video frame, a vocal track (Tuned Vocal track) that has been subjected to sound quality enhancement processing, a guitar track (Tuned Guitar track), and a trimming state of a loaded file using a trimming frame Tr1.

[0114] In this trimming state, when the user taps on the waveform display area R6 (see Figure 17) of the vocal track or guitar track that has been processed for high-quality sound on the editing screen, the display control unit 18b displays a menu M3 on the display unit 16.

[0115] Then, as shown in Figure 24, if the user selects the "Move Right" button B18, the user can touch and hold the vocal track or guitar track that has been processed for high-quality sound, and move it to the right, thereby simultaneously moving the positions of these tracks to the right. By performing this position change operation, the user can, for example, adjust the synchronization of the audio track with the video frame. Note that, as shown in Figure 24, even if the position of the track that has been processed for high-quality sound is changed, the trimming state using the trimming frame Tr1 is maintained.

[0116] Then, from this trimmed state in Figure 24, when the user taps the waveform display area R6 of the vocal track or guitar track that has been processed for high sound quality, the display control unit 18b causes the menu M3 to be displayed again on the display unit 16, as shown in Figure 25.

[0117] When the user selects the "Change Audio Source" button B16, the display control unit 18b displays a menu M4 related to this audio source change on the display unit 16. When the user selects the "Original" button B17 from the menu M4, the display control unit 18b switches the display of each track that has been subjected to sound quality enhancement processing to a display of the audio source data (original track) before the sound quality enhancement processing. As shown in FIG. 25, even when this audio source change operation is performed, the trimming state using the trimming frame Tr1 is maintained. Furthermore, not only the trimming frame Tr1 but also the positions changed using the "Move Right" button B18 and the "Move Left" button B19 (see FIG. 26) are maintained.

[0118] Next, when the user performs sound quality improvement processing from this trimming state in Fig. 25 as shown in Fig. 26, the display control unit 18b switches the display of the sound source data before the sound quality improvement processing to a display of each track that has already been processed for sound quality improvement. At this time, the sound quality improvement processing also targets the area outside the trimming frame as shown in Fig. 26. However, even when this sound quality improvement processing is performed, the trimming state by the trimming frame Tr1 is maintained.

[0119] Then, in this trimmed state, when the user taps on the waveform display area R6 of the vocal track or guitar track that has been subjected to sound quality enhancement processing, the display control unit 18b causes the display unit 16 to display the menu M3 again.

[0120] Then, when the user selects the "move left" button B19, the user can move the relative positions of the vocal track or guitar track that has been processed for high sound quality to the left all at once by touching and moving the track to the left.

[0121] At this time, as mentioned above, the area outside the trimming frame was also subject to the high-quality sound processing, so as shown in Figure 26, the vocal track and guitar track that have been processed for high-quality sound will have "no data loss" even if the position is changed.

[0122] On the other hand, as shown in FIG. 27, it is assumed that the user expands the trimming frame Tr1 toward the leading edge of the image, changing the trimming state to a trimming frame Tr2.

[0123] Even in this case, as mentioned above, the entire track, including the area outside the trimming frame, was the subject of the high-quality sound processing, so as shown in Figure 27, the vocal track and guitar track that have been processed for high-quality sound will have "no data loss" even if the trimming frame is changed.

[0124] <4-4. Shift Playback Function> Next, an example of a UI related to the shift playback function will be described. FIGS. 28 to 31 are diagrams (1) to (4) showing examples of a UI related to the shift playback function. Note that the term "position" in the description of the shift playback function may be read as "timing." The shift playback function according to this embodiment is a function that enables recording to start before the playback start position of a loaded file, such as a karaoke sound source. In other words, the shift playback function is a function that delays the playback start timing of a loaded file relative to the recording start timing when recording is performed using the recording function while playing back the loaded file.

[0125] If the shift playback function is not available or is OFF, touching the record button B4 will immediately start recording and playback of the loaded file. In other words, as shown in "Shift Playback OFF" in Figure 28, the "recording start position" and the "audio playback start position" of the loaded file will be simultaneous.

[0126] In this case, the user must touch the record button B4 and then hastily move to the shooting position, which may result in recording failure. The shift playback function solves this problem by allowing recording to start before the playback start position of the loaded file.

[0127] As shown in "Shift Playback ON" in Fig. 28, if the shift playback setting is 4 / 4 time and the length of shift playback is 4 bars, the playback start position of the imported file will be the beginning of the 5th bar, which is 4 bars from the recording start position using the record button B4. In other words, in this case, the music production application inserts a silent section of 4 bars at the beginning of the imported file.

[0128] This shift playback function eliminates the need for the user to hastily move to the shooting position after touching the record button B4, reducing the chance of a failed recording. Also, if the user wants to record the movement and preparation actions leading up to the start of playback of the loaded file for dramatic effect, this function can reliably capture such actions.

[0129] An example of a UI for setting the shift playback function will be described. When the user specifies use of the shift playback function by performing a predetermined operation on the music production application, the display control unit 18b displays a setting screen for the shift playback function on the display unit 16, as shown in FIG.

[0130] This setting screen has a section for setting the tempo (see "Tempo" in the figure), a section for setting the beat TS1, and a section for setting the shift playback function OFF / ON and the length of shift playback CI1. The tempo can be set by specifying an arbitrary value.

[0131] Furthermore, the user can select and set 4 / 4 time, 3 / 4 time, 6 / 8 time, etc. from the setting section TS1. For example, the user can set 4 / 4 time by touching "4 / 4" shown in section B21 of the setting section TS1.

[0132] In addition, the user can select and set the number of bars (e.g., 1 bar, 2 bars, 4 bars, or 8 bars) that indicates the length of shift playback from a list that is displayed by touching setting section CI1.

[0133] As shown in "Shift Playback ON" in Figure 28, for example, if the shift playback setting is 4 / 4 time and the shift playback length is 4 bars, the user will select "4 / 4" as the time signature and "4 bars" as the number of bars, as shown in parts B20 and B21 in Figure 29.

[0134] When the display control unit 18b causes the display unit 16 to display the setting screen for the shift playback function, the display control unit 18b may display a dialogue D7 as shown in FIG. 30 that provides guidance regarding the shift playback function.

[0135] As shown in dialog D7, the shift playback function is only valid when a load file exists. If a load file exists, the shift playback function can be set before the load file is read. If the shift playback function is set to ON before the load file is read, a silent section according to the shift playback function setting will be inserted at the beginning of the load file when the file is read.

[0136] Furthermore, the display control unit 18b may display a countdown from when the record button B4 is touched to when playback of the loaded file starts, depending on the settings of shift playback.

[0137] For example, as shown in Fig. 31, when the shift playback setting is 4 / 4 time and the length of the shift playback is 4 bars, the display control unit 18b displays a countdown from the recording start position where the record button B4 is touched. The countdown is displayed overlapping the recording screen in an easy-to-see position, such as the center of the screen.

[0138] The display control unit 18b counts up from negative values ​​such as "-4," "-3," and "-2" in accordance with the set tempo from the recording start position to one bar immediately before the sound source playback start position. As shown in Fig. 31, "-4" corresponds to four bars back, "-3" corresponds to three bars back, and "-2" corresponds to two bars back.

[0139] Furthermore, the display control unit 18b displays a count-up display from "1" in the measure immediately preceding the sound source playback start position. In the case of the 4 / 4 time signature setting shown in Fig. 31, the display control unit 18b displays a count-up display from 1 to 4, i.e., "1," "2," "3," and "4," for each beat in accordance with the tempo.

[0140] This allows the user to reliably know when the loaded file will be played back, and allows the user to appropriately perform preparatory actions before starting to play or sing, for example.

[0141] <<5. Modifications>> <5-1. First Modification> Incidentally, several modifications can be made to the present embodiment described above. For example, a music production app may not record audio data in a "Video" project, but instead load audio data that has been shot and edited separately in an "Audio" project and perform editing by overlaying it on video data.

[0142] In this case, the display control unit 18b may display a dialog box asking the user whether or not to record sound source data in a "Video" project, whether or not to mute the sound source data, etc., on the display unit 16. Furthermore, the display control unit 18b may display a setting screen or operation components for setting whether or not to record sound source data when a "Video" project is generated, etc.

[0143] <5-2. Second Modification> In the above embodiment, it has been described that recording can be redone using the re-record button B9, but it may also be possible to re-record only the video data or only the audio data.

[0144] In this case, when the re-recording button B9 is touched, for example, the display control unit 18b may display on the display unit 16 a dialog or the like that allows the user to select video data and audio data, video data only, or audio data only as the target for re-recording.

[0145] 5-3. Third Modification Example In the above embodiment, the sound quality improvement process is executed by the server device 100, which is a cloud server. However, it is preferable that the amount of data to be processed is as small as possible.

[0146] Therefore, the display control unit 18b may display a dialog or the like on the display unit 16 to allow the user to select the range of sound quality improvement processing to be performed on the sound source data, for example, whether the entire sound source data or only a trimmed portion.

[0147] <5-4. Fourth Modification> In addition, in the shift playback function described above, the audio playback start position of the loaded file may be controlled by a smartwatch, voice interaction, or the like. Specifically, for example, in the case of a smartwatch, when the user touches a record button displayed on the smartwatch, recording starts. Also, when the user touches a play button displayed on the smartwatch, playback of the loaded file starts.

[0148] 17, when the user touches the waveform display area R6 of a track that has been subjected to sound quality enhancement processing, the display control unit 18b displays the menu M3 including buttons such as "Change sound source" on the display unit 16. However, this is merely an example. The manner in which the menu including buttons such as "Change sound source" is displayed is not limited to this example.

[0149] In the above-described embodiment, for example, as shown in Fig. 20 , when the user touches the video frame portion R7 on the editing screen, the display control unit 18b displays the trimming frame Tr0 on the display unit 16. However, this is merely an example. The manner of operation leading up to displaying the trimming frame Tr0 is not limited to this example. For example, the trimming frame Tr0 may be selected from a menu and displayed.

[0150] 32 to 35 are diagrams (parts 1) to (part 4) showing UI examples according to the fifth modified example. As shown in Fig. 32, when a waveform display area R8 of a loaded file on the recording screen is touched, the display control unit 18b may display a menu M5 that allows the loaded file to be "deleted" or "changed," for example.

[0151] 33, when the waveform display region R9 of the loaded file on the editing screen is touched, the display control unit 18b may display a menu M6 that allows "Trim," "Change," "Move Left," "Move Right," etc. In this case, when the user touches "Trim," a trimming frame Tr0 is displayed.

[0152] Furthermore, as shown in FIG. 34, when the waveform display area R10 of the recording track before the high-quality sound processing on the editing screen is touched, the display control unit 18b may display a menu M7 that allows "trim," "move left," "move right," etc.

[0153] Furthermore, as shown in FIG. 35, when the waveform display area R6 of a track that has been processed for high-quality sound on the editing screen is touched, the display control unit 18b may display a menu M8 that allows for "trim," "change sound source," "move left," "move right," etc.

[0154] <5-6. Sixth Modification> In the above-described embodiment, the camera 11 used for the recording function of the music production app is a front camera, but a main camera provided on the back side of the smartphone 10 may also be used.

[0155] When the main camera is used, the display control unit 18b may be configured to allow the user to know, for example, the shift playback count, by the color, blinking speed, etc. of the flash of the main camera or an LED (light-emitting diode), even if the user cannot see the display unit 16 during shooting.

[0156] Furthermore, among the processes described in the above-described embodiments of the present disclosure, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically using known methods. In addition, the information including the processing procedures, specific names, various data, and parameters shown in the above documents and drawings can be changed as desired unless otherwise specified. For example, the various information shown in each drawing is not limited to the information shown in the drawings.

[0157] Furthermore, the components of each device shown in the figure are conceptual functional components and do not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or part of them can be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc.

[0158] The above-described embodiments of the present disclosure can be combined as appropriate within the scope of the present disclosure without causing any inconsistency in the processing content. The order of the steps shown in the sequence diagrams or flowcharts of the present embodiments can be changed as appropriate.

[0159] <<6. Hardware Configuration>> The smartphone 10 and server device 100 according to the embodiments of the present disclosure described above are realized by, for example, a computer 1000 configured as shown in FIG. 36 . The smartphone 10 will be described as an example. FIG. 36 is a hardware configuration diagram showing an example of the computer 1000 that realizes the functions of the smartphone 10. The computer 1000 includes a CPU 1100, a RAM 1200, a ROM 1300, a secondary storage device 1400, a communication interface 1500, and an input / output interface 1600. The components of the computer 1000 are connected by a bus 1050.

[0160] The CPU 1100 operates and controls each component based on programs stored in the ROM 1300 or the secondary storage device 1400. For example, the CPU 1100 loads the programs stored in the ROM 1300 or the secondary storage device 1400 into the RAM 1200 and executes processing corresponding to the various programs.

[0161] The ROM 1300 stores boot programs such as a Basic Input Output System (BIOS) that is executed by the CPU 1100 when the computer 1000 is started, and programs that depend on the hardware of the computer 1000 .

[0162] The secondary storage device 1400 is a computer-readable recording medium that non-temporarily records programs executed by the CPU 1100 and data used by such programs. Specifically, the secondary storage device 1400 is a recording medium that records a program according to an embodiment of the present disclosure that corresponds to the application information 17a, which is an example of the program data 1450.

[0163] The communication interface 1500 is an interface for connecting the computer 1000 to an external network 1550. For example, the CPU 1100 receives data from other devices and transmits data generated by the CPU 1100 to other devices via the communication interface 1500.

[0164] The input / output interface 1600 is an interface for connecting the input / output device 1650 and the computer 1000. For example, the CPU 1100 receives data from an input device such as a microphone or a touch panel via the input / output interface 1600. The CPU 1100 also transmits data to an output device such as a display or a speaker via the input / output interface 1600. The input / output interface 1600 may also function as a media interface for reading programs and the like recorded on a predetermined recording medium. Examples of the media include optical recording media such as a DVD (Digital Versatile Disc) or a PD (Phase Change Rewritable Disc), magneto-optical recording media such as an MO (Magneto-Optical Disk), tape media, magnetic recording media, and semiconductor memories.

[0165] For example, when the computer 1000 functions as the smartphone 10 according to an embodiment of the present disclosure, the CPU 1100 of the computer 1000 executes a program loaded onto the RAM 1200 to realize the function of the control unit 18. The secondary storage device 1400 stores the program according to the present disclosure and data in the storage unit 17. The CPU 1100 reads and executes the program data 1450 from the secondary storage device 1400, but as another example, the CPU 1100 may obtain these programs from another device via an external network 1550.

[0166] <<7. Conclusion>> As described above, according to one embodiment of the present disclosure, smartphone 10 (corresponding to an example of an “information processing device”) includes display unit 16 and control unit 18. When a music production app (corresponding to an example of an “app having a recording function using a camera and a microphone and an editing function for data recorded by the recording function”) is running, display unit 16 displays visual information related to the music production app. When a user selects audio data from the recorded data via the music production app to be enhanced in sound quality, control unit 18 performs a sound quality enhancement process on the selected audio data and causes display unit 16 to display operation components related to the editing function so that the audio data, including data that has already been enhanced in sound quality, and the video data included in the recorded data can be edited in a consistent manner. This improves the convenience for users when creating video-accompanied music content.

[0167] Although the embodiments of the present disclosure have been described above, the technical scope of the present disclosure is not limited to the above-described embodiments, and various modifications are possible within the scope of the gist of the present disclosure. Furthermore, components of different embodiments and modifications may be combined as appropriate.

[0168] Furthermore, the effects of each embodiment described in this specification are merely examples and are not intended to be limiting, and other effects may also be obtained.

[0169] Note that the present technology can also be configured as follows. (1) An information processing device comprising: a display unit that displays visual information related to an app having a recording function using a camera and a microphone and an editing function for data recorded by the recording function when the app is running; and a control unit that executes the app; and when a user selects sound source data to be subjected to sound quality enhancement from the recorded data via the app, performs sound quality enhancement processing on the selected sound source data, and causes the display unit to display operation components related to the editing function so that the sound source data, including data that has already been subjected to sound quality enhancement processing, and video data included in the recorded data can be edited in a consistent manner. (2) The information processing device described in (1), wherein the control unit displays the operation components on the display unit so that the video data and the sound source data can be edited collectively or individually. (3) The information processing device described in (1) or (2), wherein the control unit displays the operation components on the display unit so that a plurality of the sound source data can be edited collectively or individually. (4) The information processing device according to (1), (2), or (3), wherein the sound source data includes an imported file of a sound source that is not to be recorded by the video recording function. (5) The information processing device according to (4), wherein the control unit causes the display unit to display a trimming frame that allows the user to specify trimming ranges for the video data and the sound source data collectively. (6) The information processing device according to (5), wherein the control unit specifies a portion to be cropped from the beginning of the video data and the sound source data in accordance with an amount specified by the user from the left end of the trimming frame to the right, and specifies a portion to be cropped from the end of the video data and the sound source data in accordance with an amount specified by the user from the right end of the trimming frame to the left. (7) The information processing device according to (5) or (6), wherein the control unit automatically crops the end of the imported file using the trimming frame in accordance with the lengths of the video data and the sound source data when the data length of the imported file is longer than the lengths of the video data and the sound source data.(8) The information processing device according to (5), (6), or (7), wherein the control unit causes the display unit to display the operation component that enables the sound source data to be changed while maintaining the trimming state using the trimming frame. (9) The information processing device according to any one of (5) to (8), wherein the control unit causes the display unit to display the operation component that enables the sound source data to be moved relative to the position of the video data while maintaining the trimming state using the trimming frame. (10) The information processing device according to any one of (5) to (9), wherein the control unit is capable of performing the sound quality improvement process on the entire sound source data while maintaining the trimming state using the trimming frame. (11) The information processing device according to any one of (1) to (10), wherein the control unit causes the display unit to display a dialog that provides guidance on the placement of the camera and the microphone when the user sings and plays the guitar. (12) The information processing device according to (11), wherein the control unit provides guidance on the placement of the microphone so as to increase the volume of vocals. (13) The information processing device according to (11) or (12), wherein, when a user rotates a main body on which the camera and the microphone are mounted in accordance with the placement guidance, the control unit rotates the display related to the app on the display unit in accordance with the rotation of the main body. (14) The information processing device according to (13), wherein, when detecting that the main body has been rotated in accordance with the placement guidance, the control unit displays a dialog on the display unit asking the user whether or not to rotate the display related to the app. (15) The information processing device according to (4), wherein, when recording using the recording function while playing back the loaded file, the control unit is capable of executing a shift playback function that delays a playback start timing of the loaded file relative to a recording start timing. (16) The information processing device according to (15), wherein, when executing the shift playback function, the control unit inserts a silent section at the beginning of the loaded file in accordance with settings related to the shift playback function.(17) The information processing device according to (16), wherein the control unit causes the display unit to perform a count-up display according to a setting related to the shift playback function during the silent section. (18) The information processing device according to (17), wherein the control unit causes the count-up display to be performed in units of one bar during the silent section, and causes the count-up display to be performed in units of one beat during the bar immediately before the timing at which playback of the loaded file starts. (19) The information processing device according to any one of (1) to (18), which is portable. (20) An information processing method executed by an information processing device, comprising: when an application having a recording function using a camera and a microphone and an editing function for data recorded by the recording function is being executed, displaying visual information related to the application; executing the application; when sound source data to be subjected to sound quality improvement is selected from the recorded data by a user via the application, performing sound quality improvement processing on the selected sound source data; and displaying operation components related to the editing function so that the sound source data and video data included in the recorded data, including when the sound quality improvement processing has been performed, are consistent and editable. (21) A program that causes a computer to execute the following operations: when an application having a recording function using a camera and a microphone and an editing function for the recorded data by the recording function is being executed, displaying visual information about the application; executing the application; when sound source data to be subjected to sound quality improvement is selected from the recorded data by a user via the application, executing sound quality improvement processing for the selected sound source data; and displaying operation components related to the editing function so that the sound source data and video data included in the recorded data, including when the sound quality improvement processing has been performed, are consistent and editable.(22) A computer-readable recording medium having recorded thereon a program for causing a computer to execute the following operations: when an application having a recording function using a camera and a microphone and an editing function for the recorded data by the recording function is being executed, displaying visual information about the application; executing the application; when sound source data to be subjected to sound quality improvement is selected from the recorded data by a user via the application, executing sound quality improvement processing for the selected sound source data; and displaying operation components related to the editing function so that the sound source data and video data included in the recorded data, including when the sound quality improvement processing has been performed, are consistent and editable.

[0170] REFERENCE SIGNS LIST 1 Information processing system 10 Smartphone 11 Camera 12 Microphone 13 Speaker 14 Communication unit 15 Operation unit 16 Display unit 17 Storage unit 17a Application information 17b Recording information 17c Editing information 18 Control unit 18a Application execution unit 18b Display control unit 18c Communication control unit 100 Server device 101 Communication unit 102 Storage unit 102a Sound quality improvement processing model 103 Control unit 103a Acquisition unit 103b Sound quality improvement processing unit 103c Transmission unit

Claims

1. An information processing device comprising: a display unit that displays visual information related to an app having a recording function using a camera and a microphone and an editing function for data recorded by the recording function when the app is being executed; and a control unit that executes the app, and when sound source data to be improved in sound quality is selected from the recorded data by a user via the app, performs sound quality improvement processing on the selected sound source data, and displays operation components related to the editing function on the display unit so that the sound source data and video data included in the recorded data, including cases where the sound quality improvement processing has been performed, are consistent and can be edited.

2. The information processing device according to claim 1, wherein the control unit causes the display unit to display the operation components so that the video data and the audio data can be edited collectively or individually.

3. The information processing device according to claim 1, wherein the control unit causes the display unit to display the operation components so that the plurality of sound source data can be edited collectively or individually.

4. The information processing device according to claim 1, wherein the sound source data includes a read file from a sound source that is not subject to recording by the recording function.

5. The information processing device according to claim 4, wherein the control unit causes the display unit to display a trimming frame that allows the trimming ranges of the video data and the audio data to be specified collectively.

6. The information processing device of claim 5, wherein the control unit specifies the portion to be cut from the beginning of the video data and the audio data in accordance with an amount specified by the user from the left end of the trimming frame to the right, and specifies the portion to be cut from the end of the video data and the audio data in accordance with an amount specified by the user from the right end of the trimming frame to the left.

7. The information processing device according to claim 5, wherein the control unit, when the data length of the loaded file is longer than the lengths of the video data and the audio data, automatically cuts off the end portion of the loaded file using the trimming frame in accordance with the lengths of the video data and the audio data.

8. The information processing device according to claim 5, wherein the control unit causes the display unit to display the operation component that enables the sound source data to be changed while maintaining the trimming state set by the trimming frame.

9. The information processing device according to claim 5, wherein the control unit causes the display unit to display the operation component that enables moving the position of the sound source data relative to the video data while maintaining the trimming state by the trimming frame.

10. The information processing device according to claim 5, wherein the control unit is capable of executing the high-quality sound processing on the entire sound source data while maintaining the trimming state by the trimming frame.

11. The information processing device according to claim 1, wherein the control unit causes the display unit to display a dialogue providing guidance on the placement of the camera and the microphone when a user sings and plays the guitar.

12. The information processing device according to claim 11, wherein the control unit provides guidance on the placement of the microphone so as to increase the volume of vocals.

13. The information processing device according to claim 11, wherein the control unit rotates the display relating to the app on the display unit in accordance with the rotation of the main body when the user rotates the main body on which the camera and the microphone are mounted in accordance with the placement guidance.

14. The information processing device according to claim 13, wherein the control unit, when detecting that the main body has been rotated in accordance with the placement guidance, causes the display unit to display a dialog asking the user whether or not to rotate the display related to the app.

15. The information processing device according to claim 4, wherein the control unit is capable of executing a shift playback function that delays the start timing of playback of the loaded file relative to the start timing of recording when recording using the recording function while playing back the loaded file.

16. The information processing device according to claim 15, wherein the control unit, when executing the shift playback function, inserts a silent section at the beginning of the loaded file according to a setting related to the shift playback function.

17. The information processing device according to claim 16, wherein the control unit causes the display unit to perform a count-up display in accordance with a setting related to the shift playback function during the silent section.

18. The information processing device according to claim 17, wherein the control unit causes the count-up display to be performed in one-bar units during the silent section, and causes the count-up display to be performed in one-beat units during the bar immediately prior to the timing at which playback of the loaded file begins.

19. The information processing device according to claim 1, which is portable.

20. An information processing method executed by an information processing device, comprising: when an app having a recording function using a camera and a microphone and an editing function for data recorded by the recording function is being executed, displaying visual information related to the app; executing the app; when sound source data to be targeted for sound quality improvement is selected from the recorded data by a user via the app, performing sound quality improvement processing on the selected sound source data; and displaying operation components related to the editing function so that the sound source data and video data included in the recorded data, including when the sound quality improvement processing has been performed, are consistent and can be edited.

Images