Program and system

Abstract

To provide a program that can reduce the audio editing work when adding audio data to a video.SOLUTION: A computer is made to function as object detection means that inputs drawing data into a first trained model and detects objects included in the drawing data, and as audio data addition means that generates an input sentence for outputting a script that plays audio data corresponding to the type of object on the basis of the result of the object detection, inputs the input sentence into a second trained model to obtain the obtained script, and adds the script to the drawing data.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a program and a system. [Background technology]

[0002] As an example of a device that assists in video creation, Patent Document 1 discloses a video creation device that "creates videos including avatars, and is provided with an automatic schedule information generation means for automatically generating at least a portion of schedule information that consists of a plurality of motion instructions arranged over time to cause the avatar to perform specific motions so that the avatar's mouth moves in sync with the sound (summary excerpt)." [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 7024027 Summary of the Invention [Problem to be solved by the invention]

[0004] Patent document 1 makes it possible to move an avatar's mouth in sync with the audio, but adding other sounds, such as footsteps or sound effects appropriate to the avatar's environment, to the video must be done manually by the video creator, posing the issue that adding audio to video is a time-consuming process.

[0005] The present invention has been made in consideration of the above circumstances, and an object of the present invention is to provide a program and a system capable of reducing the amount of audio editing work required when adding audio data to a video. [Means for solving the problem]

[0006] In order to solve the above problem, the program of the present invention causes a computer to function as an object detection means that inputs drawing data into a first trained model and detects objects contained in the drawing data, and as an audio data addition means that generates an input sentence for outputting a script that plays audio data corresponding to the type of object based on the result of the object detection, inputs the input sentence into a second trained model to obtain the obtained script, and adds the script to the drawing data.

[0007] The present invention also provides a system comprising a user terminal, an image classification AI server having a first trained model, and a script generation AI server having a second trained model, wherein the user terminal comprises an object detection means for transmitting drawing data to the image classification AI server and receiving detection result data of an object contained in the drawing data, and an audio data addition means for generating an input sentence for outputting a script that plays audio data according to the type of the object based on the detection result data of the object, and transmitting input sentence data including the input sentence to the script generation AI server, wherein the F server receives the drawing data, inputs the drawing data into the first trained model to detect the object contained in the drawing data, and transmits the detection result data of the object to the user terminal, the script generation AI server receives the input sentence data, inputs the input sentence data into the second trained model to generate the script, and transmits script data including the script to the user terminal, and the audio data addition means receives the script data and adds the script to the drawing data. Effect of the Invention

[0008] According to the present invention, it is possible to provide a program and a system that can reduce the amount of audio editing work required when adding audio data to a video. [Brief description of the drawings]

[0009] [Figure 1]FIG. 2 is a hardware configuration diagram of a user terminal according to the present embodiment. [Diagram 2] FIG. 2 is a functional block diagram of a user terminal. [Diagram 3] FIG. 3 shows an example of a screen display (frame t1) of a video edited by an audio editing program. [Figure 4] FIG. 4 shows an example of a screen display (frame t2) of a video edited by an audio editing program. [Diagram 5] FIG. 13 is a diagram showing an example of a screen display (frame t3) of video edited by an audio editing program. [Figure 6] FIG. 13 is a diagram showing an example of a screen display (frame t4) of video edited by an audio editing program. [Figure 7] FIG. 13 is a diagram showing an example of a screen display (frame t5) of video edited by an audio editing program. [Figure 8] 10 is a flowchart showing a process flow for performing audio editing by executing an audio editing program on a user terminal. [Figure 9] FIG. 13 is a diagram showing an example of an editing screen of an audio editing program. [Figure 10] FIG. 13 is a diagram showing an example of a frame displayed in a scene display area. [Figure 11] 11 is a screen example showing the results of executing an object detection process on the frame of FIG. 10. [Figure 12] FIG. 4 illustrates an example of a data structure of a sound library. [Figure 13] FIG. 13 is a diagram showing the component balance of an effect parameter set. [Figure 14] FIG. 11 is a functional block diagram of a system according to a second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. Note that the embodiment of the present invention described below is an example of how the present invention is embodied, and the scope of the present invention is not limited to the scope of the description of the embodiment. Therefore, the present invention can be implemented by adding various modifications to the embodiment.

[0011] First, the terms used in this embodiment will be explained. "Drawing data" refers to data that is only image data and does not include audio data. Drawing data includes two-dimensional and three-dimensional drawing data. "Video data" refers to data in which audio data is associated with drawing data. Therefore, when the video data is played, a screen based on the drawing data is displayed, and audio data that matches the content of the screen is played.

[0012] First Embodiment [Configuration of user terminal 10] FIG. 1 is a hardware configuration diagram of a user terminal 10 according to this embodiment. The user terminal 10 is a terminal operated by a video creator of a video, a virtual space video, an animation, a game, or the like at a production site of a 2D or 3D video. For example, the user terminal 10 is realized by a general-purpose computer such as a workstation or a personal computer. As shown in FIG. 1, the user terminal 10 mainly includes a processor 11, a memory 12, a storage 13, an input / output interface 14, and a communication interface 15. Each component of the user terminal 10 is connected to a communication bus 16.

[0013] The processor 11 performs processes described below by executing a series of instructions included in an audio editing program 13P stored in the memory 12 or the storage 13. The processor 11 is realized as, for example, a central processing unit (CPU), a graphics processing unit (GPU), a micro processing unit (MPU), a field-programmable gate array (FPGA), or other devices.

[0014] The memory 12 temporarily holds an audio editing program 13P and data. The audio editing program 13P is loaded from, for example, the storage 13. The data includes data input to the user terminal 10 and data generated by the processor 11. For example, the memory 12 is realized as a RAM (Random Access Memory) or other volatile memory.

[0015] The storage 13 permanently holds the audio editing program 13P and data. The storage 13 is realized, for example, as a ROM (Read-Only Memory), a hard disk drive, a flash memory, or other non-volatile storage device. The storage 13 may also be realized as a removable storage device such as a memory card. As yet another example, instead of being built into the user terminal 10, the storage 13 may be stored in a file server as an external storage device, and the user terminal 10 may be connected to the file server. With this configuration, for example, in a situation where a plurality of user terminals 10 are used in a studio, the audio editing program 13P, drawing data to be edited, and an audio library can be stored and updated collectively.

[0016] The input / output interface 14 is an interface for connecting external devices, such as an input device 17 (e.g., a keyboard, a pointing device, etc.), a display 18, and an audio output device 19 (a speaker, a headphone, etc.), to the user terminal 10. The processor 11 communicates with the external devices through the input / output interface 14. The input / output interface 14 is realized using, for example, a Universal Serial Bus (USB), a Digital Visual Interface (DVI), a High-Definition Multimedia Interface (HDMI, registered trademark), or other terminals.

[0017] The communication interface 15 communicates with other devices via a communication network. The communication interface 15 is realized as, for example, a wired communication interface such as a Local Area Network (LAN) or a wireless communication interface such as Wireless Fidelity (Wi-Fi), Bluetooth (registered trademark), or Near Field Communication (NFC).

[0018] [Functional block diagram of user terminal 10] 2 is a functional block diagram of the user terminal 10. As shown in FIG. 2, the audio editing program 13P loaded into the memory 12 causes the user terminal 10 to function as an editing screen control means 210, a drawing data reading means 220, an object detection means 230, an audio data adding means 240, a display control means 250, an audio output control means 260, and a playback means 270. The storage 13 also includes an editing screen data storage means 310, a drawing data storage means 320, a trained model storage means 330, an audio library storage means 340, and an image data storage means 350. In addition, when an avatar is used when generating a virtual space image, an avatar control means 280 may be provided.

[0019] The editing screen control means 210 reads out the editing screen data from the editing screen data storage means 310, and displays the editing screen 900 (see FIG. 9) on the display 18 via the display control means 250. Then, the editing screen 900 accepts an operation from the input device 17, and operates the audio editing program 13P in response to the user's operation.

[0020] The drawing data reading means 220 reads out the drawing data stored in the drawing data storage means 320 as drawing data to be edited. When an avatar is to be placed in the virtual space in the drawing data depicting the virtual space, a character image of the avatar may also be read out as a type of drawing data.

[0021] The object detection means 230 inputs drawing data to be edited and detects objects included in the drawing data. In this embodiment, an image classification AI (Artificial Intelligence) 231 is used as the object detection means 230. The image classification AI 231 is realized using an image trained model trained using teacher data that associates image data with the object type (or a label indicating the object type). The trained model may be of any type, such as CNN (Convolutional Neural Networks) or R-CNN (Region Based Convolutional Neural Networks), as long as it can classify and detect images. The image classification AI 231 inputs each frame tn of the drawing data to be edited into the image trained model 231M read from the trained model storage means 330, and outputs the result of classifying the object drawn in the frame tn. This allows the object to be detected. By using R-CNN for the image classification AI 231, it becomes easier to detect what object is drawn in which area of ​​each frame. The voice data adding means 240 obtains the result of associating the area in the frame with the object drawn there, and determines which area in the frame should have voice data corresponding to which object added to it, and executes the voice data adding process. The image trained model 231M corresponds to an example of the first trained model.

[0022] The audio data adding means 240 reads out audio data corresponding to an object included in the drawing data from the audio library storage means 340 and associates it with the drawing data. In this embodiment, a script generation AI 241 is used as the audio data adding means 240. The script generation AI 241 is realized by, for example, a generation AI including a large language model (LLM) 241M. The script generation AI 241 inputs an input sentence generated by the audio data adding means 240 into the LLM 241M read out from the trained model storage means 330, and generates a script (a collection of commands) that reproduces audio corresponding to the reflection and absorption characteristics of the object included in the drawing data. The LLM 241M corresponds to an example of a second trained model. In this embodiment, the "reflection and absorption characteristics" may be expressed using, for example, one of absorption, reflection, and transmission, or any combination thereof.

[0023] The audio data adding means 240 adds the script generated by the script generation AI 241 to the drawing data. As one mode of adding audio data to the drawing data, in the drawing data that depicts a virtual space, an effect parameter may be added to an area in which an object that constitutes the background of the virtual space is drawn. The effect parameter is a numerical value related to an audio effect for reflecting the reflection and absorption characteristics of the object in the standard sound quality of the sound source data.

[0024] When playing back sound source data in one area of ​​the virtual space, the script generation AI 241 refers to effect parameters added according to the object in the area in which the sound source data is played back, and generates a script that applies an audio effect defined in the effect parameter data to the standard sound quality of the sound source data to play back a sound effect with changed sound quality. This allows an audio effect according to the reflection and absorption characteristics of the object drawn in the area to be applied to the standard sound quality when the area in which the sound source data is played back to be different. The audio data adding means 240 generates video data by synchronizing the drawing data with the script, so that the script synchronized with the drawing data is executed when the video data is played back, and a sound effect that matches the reflection and absorption characteristics of the object in the area is played back.

[0025] The sound data adding means 240 adds the script generated by the script generation AI 241 to the object included in the drawing data. For example, in a virtual space expressed by drawing data, if the background of the virtual space itself is not a sound source, sound source data is not attached, and only effect parameters according to the reflection and absorption characteristics of the object constituting the background are attached.

[0026] On the other hand, the audio data adding means 240 attaches a script to the object serving as the sound source, which refers to the sound source data and the effect parameters associated with the area where the sound is emitted and causes the standard sound quality of the sound source data to be affected. Taking an avatar as an example, a script is attached to the avatar, which reads out the effect parameters associated with the area where the avatar exists and causes the standard sound quality of the sound source data of the sound emitted by the avatar to be affected. This makes it possible to play a sound effect according to the area where the avatar exists. Taking an item used in the virtual space represented by the drawing data as an example, a script is attached to the item, which reads out the sound source data emitted by the item and the effect parameters associated with the area where the item is used and causes the standard sound quality to be affected. This makes it possible to play a sound effect according to the area where the item is used.

[0027] The display control means 250 controls the display 18 to display the editing screen 900, drawing data, video data, and the like.

[0028] The audio output control means 260 causes the audio output device 19 to generate audio based on the audio data.

[0029] The playback means 270 plays back video data, i.e., displays video and plays audio. The playback means 270 executes a script synchronized with the drawing data when playing back video data, and plays back sound effects that match the reflection and sound absorption characteristics of the objects in the area. The sound effects are generated from the audio output device 19.

[0030] The avatar control means 280 moves an avatar placed in a virtual space depicted by the drawing data in response to an operation received by the input device 17. The avatar control means 280 is not essential for adding audio data to the drawing data, but may be used, for example, by a video creator operating the user terminal 10 to operate the input device 17 to move an avatar and check whether a script attached to the avatar is executed as planned.

[0031] Next, with reference to Figs. 3 to 7, a screen display example of a video edited by the audio editing program 13P according to this embodiment will be described. Fig. 3 is a diagram showing a screen display example (frame t1) of a video edited by the audio editing program 13P. Fig. 4 is a diagram showing a screen display example (frame t2) of a video edited by the audio editing program 13P. Fig. 5 is a diagram showing a screen display example (frame t3) of a video edited by the audio editing program 13P. Fig. 6 is a diagram showing a screen display example (frame t4) of a video edited by the audio editing program 13P. Fig. 7 is a diagram showing a screen display example (frame t5) of a video edited by the audio editing program 13P.

[0032] For example, in frame t1 shown in FIG. 3, avatar A is outside the dungeon 410, and avatar B is inside the dungeon 410, and avatar B is attempting to destroy the door 420 with a bomb 400.

[0033] 4, avatar B throws bomb 400 at door 420 inside dungeon 410, and the explosion event triggers the reproduction of an explosion sound effect "BOOM" associated with bomb 400. At this time, avatar B, who is inside dungeon 410, hears the explosion sound effect reverberating inside dungeon 410, so an explosion sound effect is reproduced for the player of avatar B at the maximum volume for the explosion sound effect of bomb 400, with maximum echo effect to represent reverberation between walls, maximum refrain in a small time unit due to the reverberation of sound between walls, treble emphasis, and bass emphasis applied as sound effects.

[0034] On the other hand, for Avatar A outside the dungeon 410, the explosion sound effect from inside the dungeon 410 is blocked by the structures, so the explosion sound effect from the bomb 400 is played back to the player of Avatar A at the lowest volume, with the high frequencies cut so that it sounds muffled, and at a lower volume without echo or refrain.

[0035] In frame t3 in Fig. 5, avatar A is holding a bomb 400 in a grassland 430 to defeat monster M. This bomb 400 is the same bomb 400 as the one used in frames t1 and t2, but since the bomb 400 has not yet been thrown, no explosion sound effect is played. Additionally, while frame t3 is being displayed, the cry of monster M is being played.

[0036] 6, avatar A throws bomb 400 at monster M, and the explosion sound effect "BOOM" is played. The explosion sound effect at this time is absorbed by grassland 430, and because there are no structures nearby that can reverberate the sound, it is played at a lower volume than the explosion sound effect played inside dungeon 410 at the time of playback of frame t2, and at a higher volume than the explosion sound effect heard by avatar A who was outdoors at frame t2, because the explosion sound effect was not blocked or absorbed by the structures of dungeon 410.

[0037] Frame t5 in FIG. 7 depicts a stone-paved walkway 450, crossing a wooden bridge 440 over a stream 460 from a footpath in a grassland 430. A distinctive feature of frame t5 is that the background object includes a sound-emitting stream 460. The display of frame t5 triggers the reproduction of a babbling sound effect. Also, the footsteps of avatar A are different depending on whether they are on the dirt footpath, the wooden bridge 440, or the stone-paved walkway 450. In other words, even if avatar A's walking motion is the same, different sound effects are reproduced depending on the footing environment.

[0038] In the above example, the explosion sound effect "BOOM" of the same item, the bomb 400, is reproduced in the same environment, but when the environment in which the explosion occurs is different, the sound quality and volume of the explosion sound effect change according to the environment. Therefore, even if the same item is used, the sound effect of the user operating the avatar is heard differently depending on the location of the avatar, so that the immersive feeling of the virtual space can be further enhanced. In this way, in order to generate sound effects according to the environment of the virtual space constructed by the drawing data, it is necessary to add audio data according to the reflection and absorption characteristics of the objects in each area included in the drawing data. However, if the video creator manually adds the audio data, the work load is high. Therefore, by using the audio editing program 13P according to this embodiment, the work of adding audio data according to the objects included in the drawing data can be automated, and the load of the video creation work and audio editing work by the video creator can be reduced. Next, the procedure for automating the work of adding audio data will be described.

[0039] FIG. 8 is a flowchart showing a process flow in which the user terminal 10 executes the audio editing program 13P to perform audio editing.

[0040] When the user starts the audio editing program 13P on the user terminal 10, the editing screen control means 210 displays the editing screen 900 of the audio editing program 13P on the display 18 (S01).

[0041] 9 is a diagram showing an example of an editing screen of the audio editing program 13P. The editing screen 900 includes an operation button area 910, a scene display area 920, a timeline 930, and a sound box 940. The operation button area 910 includes an "import" button 911, a "recognition" button 912, an "Add sound" button 913, and an "export" button 914.

[0042] The user operates the "import" button 911 on the editing screen 900 to issue an instruction to read the drawing data to be edited (S02). The drawing data reading means 220 reads the instructed drawing data from the drawing data storage means 320, and loads it into the work area of ​​the memory 12. If the drawing data is video data, the frame at the time indicated by the pointer 931 on the timeline 930 is displayed in the scene display area 920. The frame being displayed in the scene display area 920 becomes the drawing data to be processed for the audio editing work. The drawing data read in this step forms background data for the video, and is prepared separately from the item and avatar data. Multiple pointers 931 on the timeline 930 may be displayed, and the frames between the pointers may be specified as the range of frames to be processed.

[0043] Fig. 10 is a diagram showing an example of a frame displayed in the scene display area 920. In the scene display area 920 of the editing screen 900, a frame tn at a time tn shown in Fig. 10 is displayed. The frame tn is background data in which a structure is constructed in a grassland 430. The inside of the structure forms a so-called dungeon 410, and is separated from the outside world, that is, the grassland 430, the sky, and the distant view.

[0044] When the user operates the “recognition” button 912 on the editing screen 900, the object detection means 230 executes an object detection process on the frame tn displayed in the scene display area 920, and detects what is drawn at what position in the frame (S03).

[0045] Fig. 11 is an example of a screen showing the result of executing the object detection process on the frame of Fig. 10. As a result of the object detection means 230 inputting the frame tn to the image-learned model, the frame tn is roughly classified into four regions, specifically, region 1 is a structure (lab:01), region 2 is the inside of the dungeon (lab:02), region 3 is the grassland 430 (lab:03), and region 4 is a distant view (lab:04).

[0046] The user may correct the position of the label boundary line 950 of the frame tn* using the input device 17. For example, if the position of the boundary line between the grassland 430 and the background is in the background, the user may operate the input device 17 to correct the position of the boundary line.

[0047] Furthermore, when the user places the cursor 980 on the label number tag 960 and clicks it, the name indicated by the label number is displayed. For example, when "lab:03" is clicked, "Grassland" is displayed in a pop-up. This allows the user to check whether the object detection process was performed correctly. Furthermore, if corrections are required, the user can select the correct label from the pull-down menu 970 and correct the tag 960.

[0048] When the user confirms that object detection has been performed correctly, he or she operates the "Add sound" button 913 on the editing screen 900. The sound data adding means 240 refers to the sound library storage means 340, and adds sound data corresponding to the label added to each area of ​​the frame in association with the area (S04).

[0049] Fig. 12 is a diagram showing an example of the data structure of a sound library. In the sound library shown in Fig. 12, "label number", "category name", "sound data (parameter set)", "trigger", and "sound data (sound source data)" are stored in association with each other. "Sound data (parameter set)" corresponds to effect parameter data.

[0050] The "label number" is the same number as the number labeled by the object detection means 230. The "classification name" indicates the recognized object corresponding to the label number. This allows the voice data addition means 240 to associate voice data with the object using the result of labeling the background object in the frame by the object detection means 230. The voice library also includes voice data of appearing objects such as the avatar, the monster M, and the bomb 400.

[0051] "Sound data (parameter set)" is a type of sound data that is associated with an object that is not a sound source, such as a background object. Parameter set data includes an effect parameter set that adjusts sound quality and volume, and cannot be played by itself and does not produce sound effects. However, when a sound effect is generated near an object with which the parameter set is associated, the sound source data can be played with a sound quality and volume that suits the playback environment by applying the effect indicated in the parameter set to the sound effect. The sound quality of the sound data included in the sound source data played without applying any effects corresponds to the standard sound quality. The file format of the parameter set is also not limited to txt.

[0052] "trigger" indicates a trigger variable that causes the sound source object to generate a sound effect.

[0053] "Sound data (sound source data)" is a type of sound source data, and is audio data associated with an object that is a sound source. The format of the audio data for sound effects can be any format, such as mp3, WAV, OGG, FLAC, etc.

[0054] FIG. 13 is a diagram showing the component balance of an effect parameter set. The effect parameters include the type and value of an audio effect for reflecting the reflection and absorption characteristics of an object in the standard sound quality. The effect parameter set is a data set including a plurality of effect parameters. Examples of audio effects include five effects: "echo", "refrain", "treble emphasis", "bass emphasis", and "volume level". Each effect parameter set includes the type and each value of "echo", "refrain", "treble emphasis", "bass emphasis", and "volume level", and the effect parameter set is stored in association with the label number. For example, in effect set_02.txt, "echo", "refrain", "treble emphasis", "bass emphasis", and "volume level" are specified in the order of (5,5,5,5,5). Therefore, the sound effect generated inside the dungeon 410 is reproduced with echo and maximum volume. In addition, in effect set_03.txt, "echo", "refrain", "treble emphasis", "bass emphasis", and "volume level" are specified in the order of (0,0,3,3,3). In Grassland 430, sound is absorbed and reverberates, resulting in standard sound quality being reproduced as is.

[0055] 11, the audio data adding means 240 reads out effect set_01.txt associated with label 01 in area 1, and associates and stores effect set_01.txt with area 1 of frame t1. Similarly, the audio data adding means 240 associates effect set_02.txt associated with label 02 with area 2, and effect set_03.txt and grassland01.mp3 with area 3. Area 4 is not associated with either effect set or sound source data.

[0056] Since the trigger variable of grassland01.mp3 associated with area 3 is specified as "isWalking", the sound effect in which the effect indicated by the effect parameter set defined in effect set_03.txt is applied to the standard sound quality of grassland01.mp3 is played back when the avatar walks through area 3. This creates a sense of realism as if the avatar is walking through grassland 430. Note that grassland02.mp3 may be used as the sound source data instead of grassland01.mp3 in area 3. As an example, after the sound data adding means 240 adds grassland01.mp3 to area 3, the video creator may replace it with grassland02.mp3 on the editing screen 900 (see FIG. 9), thereby realizing the replacement of the sound source data.

[0057] Moreover, since no sound source data, trigger variables, or effect parameter sets are associated with the distant view of area 4, no sound effects are generated from area 4.

[0058] 7, the audio data adding means 240 associates stream01.mp3 (babbling sound) associated with label 05 with the area in which the stream 460 is drawn, and stores it. Since the trigger variable of stream01.mp3 is specified as "display", the display of frame t5 is used as a trigger to play the babbling sound. In this way, even if it is a background object, if there is an object that serves as a sound source, sound source data is added.

[0059] For example, in the case of a game video, the script generation AI 241 of the sound data adding means 240 uses a scripting language used in the game engine to generate a script for playing a sound suited to the environment of the area, using an area and sound data associated therewith as input. More specifically, the sound data adding means 240 generates an input sentence (prompt) to be input to the script generation AI 241, and obtains the output. As an example, effect data D consisting of an effect parameter set according to the reflection and absorption characteristics of an object present in the area X is prepared, and the sound data adding means 240 generates a prompt such as "When a trigger variable B appears in the area X, please write a script using a scripting language E to play a sound in which the effect data D is applied to the sound source data C." The sound data adding means 240 then inputs the generated prompt to the script generation AI 241 and obtains the output script. The script generation AI 241 automatically generates this script, so that sound data for playing a sound effect reflecting the reflection and absorption characteristics of an object present in the area X can be automatically added to the drawing data.

[0060] For example, in the example of Fig. 11, the sound data adding means 240 generates a prompt saying "Please generate a script for applying the effect indicated by the effect parameter set in effect set_03.txt to the sound in grassland01.mp3 when the avatar walks in area 1 and playing it", and inputs it to the script generation AI 241. The script generation AI 241 generates a script for playing the sound effect grassland01.mp3 used in area 1 (grassland 430) as a walking sound corresponding to the area in which the avatar is walking, according to the trigger variable "isWalking" that generates the sound. This script is a code for controlling the avatar (Avatar Controller), so it is attached to the drawing data of the avatar.

[0061] Similarly, when the bomb 400 in FIG. 6 is exploded in the grassland 430, a script is generated that applies the effect parameters defined in effect set_03.txt to bomb01.mp3 in response to "explosion" and plays it back.

[0062] Also, when the bomb 400 is exploded in the dungeon 410, a script is generated for the user of avatar B to apply the effect parameters defined in effect set_02.txt to bomb01.mp3 in response to "explosion" and play the same. On the other hand, when the bomb 400 is exploded in the dungeon 410, a script is generated for the user of avatar A to apply the minimum value of each item defined in effect set_01.txt, effect set_02.txt, and effect set_03txt to bomb01.mp3 in response to "explosion". As a result, when the bomb 400 is exploded in the dungeon 410, the reverberating explosion sound effect is heard by users inside the dungeon 410, and the explosion sound effect absorbed and attenuated by structures is heard by users outside the dungeon 410, and audio editing with a sense of realism can be realized with reduced operation man-hours by the user. In this way, the position of the avatar may also be used as a variable in the script, and a script may be generated that changes the sound effects depending on the distance from the sound source to the avatar's position, obstructions, and other environmental factors.

[0063] If there is an unprocessed target frame (S05: Yes), the process returns to step S02 and continues the process for the next frame.

[0064] If there are no unprocessed frames (S05: No), the user operates the "export" button 914 on the editing screen 900 to output the rendering data with the script written therein and write it into the video data storage means 350 (S06).

[0065] According to the audio editing program 13P of this embodiment, the object detection means 230 (e.g., image classification AI 231) recognizes objects drawn in each area of ​​the frame of drawing data. Then, the audio data addition means 240 (e.g., script generation AI 241) selects audio data corresponding to the recognized object, automatically generates a script for playing the audio data corresponding to the object with an effect according to the area, and adds the script to the object. This automates the task of adding audio data to drawing data and supports the work of video creators.

[0066] <Second embodiment> The second embodiment relates to a system 1 including a user terminal 10, an image classification AI server 232, and a script generation AI server 242. FIG. 14 is a functional block diagram of the system 1 according to the second embodiment. The difference from the user terminal 10 according to the first embodiment is that the second embodiment does not include a trained model storage means 330 and a voice library storage means 340 in order to reduce the hardware load of the user terminal 10, and is configured to have the necessary functions in an external server. Specifically, the user terminal 10 is communicatively connected to each of the image classification AI server 232, the voice library server 341, and the script generation AI server 242.

[0067] Then, the object detection means 230 transmits the drawing data to the image classification AI server 232, and the image classification AI server 232 receives the drawing data, inputs it to the image trained model 232M, and outputs the object detection result. The object detection means 230 acquires the object detection result from the image classification AI server 232.

[0068] The voice data adding means 240 acquires voice data corresponding to an object included in the drawing data from the voice library server 341 according to the object detection result by the object detection means 230. The voice data adding means 240 generates an input sentence (prompt) for causing the script generation AI server 242 to generate a script for playing a voice corresponding to the object included in the drawing data, and inputs the input sentence data to the script generation AI server 242. The script generation AI server 242 then inputs the received input sentence data to the LLM 242M, and transmits script data including the output script to the voice data adding means 240. The voice data adding means 240 receives the script data and adds the script to the drawing data.

[0069] The configuration of other parts is the same as in the first embodiment, so the description will be omitted.

[0070] According to the second embodiment, it is possible to configure the AI ​​processing and the voice library storage means in a cloud environment, so that the processing load and storage capacity of the user terminal 10 can be reduced.

[0071] The program according to the present invention is not limited to a single program, but may be a collection of multiple programs. Moreover, the program according to the present invention is not limited to being executed by a single device, but may be executed by multiple devices in a shared manner. Furthermore, the division of roles between the user terminal 10 and the various servers, i.e., the image classification AI server 232, the voice library server 341, and the script generation AI server 242, is not limited to the above example. That is, a part of the processing of each server may be executed by the user terminal 10, or a part of the processing of the user terminal 10 may be executed by each server.

[0072] Furthermore, a part or all of the means realized by the program can be realized by hardware such as an integrated circuit. Furthermore, the program may be provided by being recorded on a non-transitory recording medium that can be read by a computer. Examples of the recording medium include a hard disk, an SD card, a DVD, and a server on the Internet. [Explanation of symbols]

[0073] 1: System, 10: User terminal, 11: Processor, 12: Memory, 13: Storage, 13P: Audio editing program, 14: Input / output interface, 15: Communication interface, 16: Communication bus, 17: Input device, 18: Display, 19: Audio output device, 210: Editing screen control means, 220: Drawing data reading means, 230: Object detection means, 231M: Image trained model, 232: Image classification AI server, 232M: Image trained model, 240: Audio data addition means, 242: Script generation AI server, 250: Display control control means, 260: audio output control means, 270: playback means, 280: avatar control means, 310: editing screen data storage means, 320: drawing data storage means, 330: learned model storage means, 340: audio library storage means, 341: audio library server, 350: video data storage means, 400: bomb, 410: dungeon, 420: door, 430: grassland, 440: bridge, 450: sidewalk, 460: stream, 900: editing screen, 910: operation button area, 911: import button, 912: recognition button, 913: Add sound button, 914: export button, 920: scene display area, 930: timeline, 931: pointer, 940: sound box, 950: label boundary, 960: tag, 970: pull-down menu, 980: cursor, A: avatar, B: avatar, M: monster

Claims

1. Computer, An object detection means for inputting drawing data into a first trained model and detecting an object included in the drawing data; A program that functions as an audio data addition means that generates an input sentence for outputting a script that plays audio data corresponding to the type of object based on the detection result of the object, obtains the script obtained by inputting the input sentence into a second trained model, and adds the script to the drawing data.

2. The program according to claim 1, the audio data includes sound source data to be added to the object, and effect parameter data for changing a standard sound quality of the sound source data; The audio data adding means is a program which, when playing back the sound source data in a specific area of ​​the drawing data, refers to the effect parameter data associated with the object drawn in the specific area, and acquires a script which applies an audio effect defined in the effect parameter data to the standard sound quality to play a sound effect with changed sound quality.

3. The program according to claim 2, A program, wherein the effect parameters include types and values ​​of audio effects for reflecting the sound reflection and absorption characteristics of objects included in the specific area in the standard sound quality.

4. A system including a user terminal, an image classification AI server including a first trained model, and a script generation AI server including a second trained model, The user terminal, an object detection means for transmitting drawing data to the image classification AI server and receiving detection result data of an object included in the drawing data; a voice data adding means for generating an input sentence for outputting a script for reproducing voice data corresponding to the type of the object based on the object detection result data, and transmitting input sentence data including the input sentence to the script generation AI server, The F server is receiving the drawing data, inputting the drawing data into the first learned model to detect the object included in the drawing data, and transmitting detection result data of the object to the user terminal; The script generation AI server includes: receiving the input sentence data, inputting the input sentence data into the second trained model to generate the script, and transmitting script data including the script to the user terminal; The voice data adding means receives the script data and adds the script to the drawing data.

Images