A method for adding an animation sound effect to an AR model and a playing device

Abstract

The application relates to the AR technical field and provides a method for adding animation sound effects to an AR model and a playing device. For an AR applet, a SLAM plug-in is used to separately play a soundless animation of an AR model and a completely transparent video containing sound, a timer is used to coordinate a synchronous playing time of the animation and the completely transparent video, an audio-visual unified sound animation effect is achieved, the rendering of the animation effect is improved, the picture of the completely transparent video is invisible and does not interfere with the playing of the AR animation, so that the animation quality is ensured and the user experience is improved.

Description

Technical Field

[0001] This application relates to the field of augmented reality (AR) technology, and provides a method and playback device for adding animation and sound effects to AR models. Background Technology

[0002] Augmented Reality (AR) technology is a technology that cleverly integrates virtual information with the real world. It widely uses various technologies such as multimedia, 3D modeling, real-time tracking and registration, intelligent interaction, and sensing to add simulated AR models to real-world scenes. Users can interact with the AR models, enhancing the fun of the visuals.

[0003] To make added AR models more vivid and lifelike, matching animations and sound effects are usually added to them, creating audio-visual content. However, if the playback times of the AR model and the animation / sound are not synchronized, it will directly affect the user's enjoyment experience. Summary of the Invention

[0004] This application provides a method and playback device for adding sound effects to AR animations, which enables unified playback of video and sound in AR animations.

[0005] On one hand, embodiments of this application provide a method for adding animated sound effects to an AR model, including:

[0006] Launch the AR mini-program, draw an AR scene in the target window, and load an AR model and a fully transparent video. The fully transparent video is a pre-generated animation video based on the AR model and contains sound that matches the animation of the AR model. Areas in the fully transparent video that are not part of the AR model are set to be completely transparent, so that areas that are not part of the AR model are not visible when the fully transparent video is played.

[0007] In response to an AR model placement command, the AR model is placed at the target location in the AR scene and displayed.

[0008] When the AR model is triggered to play an animation, the model animation plugin is used to play the animation video of the AR model in the target window and start a timer;

[0009] Based on the animation time of the AR model, a video playback plugin is used to play the fully transparent video in the target window to provide narration for the AR model animation.

[0010] On the other hand, this application embodiment provides a playback device, wherein the client software installed on the playback device includes an AR mini-program, and the playback device includes a processor, a memory, a display screen, a speaker, and a camera, wherein the camera, the speaker, the display screen, the memory, and the processor are connected via a bus;

[0011] The memory stores a computer program, and the processor performs the following operations according to the computer program:

[0012] Launch the AR mini-program, draw the AR scene captured by the camera in the target window, and load the AR model and the fully transparent video. The fully transparent video is a pre-generated animation video based on the AR model and contains sound that matches the animation of the AR model. The area of ​​the AR model in the fully transparent video is set to be completely transparent, so that the area of ​​the AR model is not visible when the fully transparent video is played.

[0013] In response to an AR model placement command, the AR model is placed at the target location in the AR scene and displayed on the display screen;

[0014] When the AR model is triggered to play an animation, the model animation plugin is used to draw the animation video of the AR model in the target window and display it on the display screen. At the same time, a timer is started.

[0015] Based on the animation time of the AR model, a video playback plugin is used to draw the fully transparent video in the target window, play the fully transparent video through the display screen, and play the sound from the fully transparent video through the speaker to provide narration for the AR model animation.

[0016] On the other hand, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions for causing a computer device to perform the method steps of adding animation and sound effects to an AR model provided in embodiments of this application.

[0017] This application provides a method and playback device for adding animation sound effects to AR models. For AR mini-programs, based on the animated video of the AR model, a fully transparent video containing sound matching the AR model's animation is pre-generated. The area of ​​the AR model in the fully transparent video is set to be completely transparent, making the AR model area invisible during playback. After the AR mini-program starts, the AR model and the fully transparent video are loaded into memory. By setting a timer, two independent players are used to draw the AR model's animated video and the fully transparent video on the same target window, achieving unified audiovisual playback and improving the animation playback effect. Furthermore, because the AR model area in the fully transparent video is set to be completely transparent, the AR model area is invisible during playback, preventing interference with the AR model's animation display and ensuring the animation display quality. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1A An AR scene illustration provided for an embodiment of this application;

[0020] Figure 1B A flowchart illustrating the method for generating fully transparent video as provided in this application embodiment;

[0021] Figure 1C A comparison diagram of a fully transparent video frame and an original video frame provided in an embodiment of this application;

[0022] Figure 2 A schematic diagram illustrating a method for generating a completely transparent video according to an embodiment of this application;

[0023] Figure 3 A schematic diagram illustrating another method for generating a completely transparent video according to an embodiment of this application;

[0024] Figure 4 A schematic diagram illustrating another method for generating a completely transparent video according to an embodiment of this application;

[0025] Figure 5 A schematic diagram illustrating another method for generating a completely transparent video according to an embodiment of this application;

[0026] Figure 6 This is a flowchart illustrating a method for adding animation and sound effects to an AR model, as provided in an embodiment of this application.

[0027] Figure 7 This is a schematic diagram illustrating the process of placing an AR model as provided in an embodiment of this application;

[0028] Figure 8 An example diagram of adding animation and sound effects to an AR model provided in this application embodiment;

[0029] Figure 9 Another effect diagram for adding animation and sound effects to an AR model provided in this application embodiment;

[0030] Figure 10 An interactive flowchart for adding animation and sound effects to an AR model is provided for an embodiment of this application;

[0031] Figure 11 This is a structural diagram of the playback device provided in an embodiment of this application. Detailed Implementation

[0032] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments of this application. Obviously, the described embodiments are only some embodiments of the technical solutions of this application, and not all embodiments. Based on the embodiments recorded in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the technical solutions of this application.

[0033] Currently, when adding animation and sound effects to 3D models in ordinary AR clients, professional animation production tools (such as Unity, Maya, etc.) are usually used. Starting from a certain frame, corresponding audio files in WAV or MP3 formats are created for the 3D model. After the audio files are combined with the video files of the 3D model, they are exported as an AR file for playback, achieving the effect of playing sound animation.

[0034] However, some AR mini-programs are typically developed using Simultaneous Localization and Mapping (SLAM) plugins, which lack the ability to play audio files in formats such as WAV or MP3. Therefore, conventional methods for adding animation and sound effects to AR models are not suitable for AR mini-programs.

[0035] In addition, some animation production tools (such as Maya) do not have the function of adding voice-over to animations, and it may not be possible to implement this function in the short term. Furthermore, adding animation sound effects to AR models through animation production tools requires a high level of experience from the animators.

[0036] In view of this, embodiments of this application provide a method and playback device for adding animation sound effects to an AR model. The method pre-generates a fully transparent MP4 video with a completely transparent area of ​​the AR model, containing the animation sound to be added to the AR model, which is independent of the original AR model animation video. When playing the AR model animation video, the fully transparent MP4 video is played synchronously using a set scene timer, thus achieving unified audiovisual playback. Since the AR model's area is completely transparent in the MP4 fully transparent video, this area is invisible to the user during playback and will not interfere with the AR model's playback. Furthermore, the SLAM plugin in the AR mini-program has the ability to play MP4 format videos. Therefore, this method is applicable not only to ordinary client software with AR functionality on the playback device but also to AR mini-programs within the client software on the playback device.

[0037] See Figure 1A This is an AR scene illustration provided in an embodiment of this application. When using navigation software on a mobile phone for navigation, and the distance to the intersection is within a preset distance range, a flowing triangular arrow model for navigation is displayed, and an animated sound of "Please go straight" is added to the triangular arrow to guide the user to go straight.

[0038] It should be noted that, Figure 1A This is merely an example of adding animation and sound effects. This application does not impose restrictive requirements on AR scenarios. For example, in addition to the navigation scenario mentioned above, it can also be for museum scenarios, smart home scenarios, etc.

[0039] In the embodiments of this application, for each AR model's animation, a fully transparent video containing sound matching the AR model's animation is pre-generated. The method for generating the fully transparent video is described in [link to documentation]. Figure 1B It mainly includes the following steps:

[0040] S101: Export animated videos of AR models.

[0041] Export the animated video of the AR model to a video editor for video editors to edit.

[0042] In one alternative implementation, to make the difference between the before and after video editing more obvious, the animated video of the AR model is imported into two symmetrical video frames, and the video editor processes one of the video frames.

[0043] S102: For each video frame in the video animation, identify the region of the AR model, set the identified region as a completely transparent region, and generate a new video frame with transparency.

[0044] In order not to interfere with the playback of the AR model's animated video, the AR model area in the newly generated completely transparent video needs to be made transparent.

[0045] Specifically, for each video frame in the video animation, the region of the AR model is identified, and the alpha channel value of the pixels in that region is set to 0, making it a completely transparent region, thereby generating a new video frame with transparency.

[0046] For example, such as Figure 1C As shown, in the exported symmetrical video frames, the right side is the original video frame of the AR model's animation video, and the left side is the new video frame after the AR model's region of the video frame has been made completely transparent.

[0047] S103: Add sound to the new video frame that matches the animation of the AR model.

[0048] After obtaining a new video frame, add sound to that new video frame that matches the animation of the AR model.

[0049] S104: Generate a fully transparent video from multiple new video frames with transparency and sound.

[0050] In the embodiments of this application, the duration of the completely transparent video is less than or equal to the animation duration of the AR model's animated video.

[0051] Optionally, in one implementation, if the AR model's animated video is divided into N (N is an integer greater than or equal to 2) animation segments, and M (M is an integer greater than or equal to 1 and less than or equal to N) animation segments require voice-over, then when the duration of the completely transparent video is less than or equal to the animation duration of the animated video, each of the M animation segments corresponds to a video segment within the completely transparent video. The durations of the different animation segments can be the same or different.

[0052] For example, such as Figure 2 As shown, the AR model's animated video has an animation duration of T = 10s, divided into 4 animation segments. The sub-animation durations for animation segment 1 are t1 = 2s, t2 = 1.5s, t3 = 4.5s, and t4 = 2s. When all 4 animation segments require voice-over, the generated completely transparent video also has a duration of T′ of 10s, divided into 4 video segments. Specifically, video segment 1 provides voice-over for animation segment 1, with a sub-duration t1′ = 2s; video segment 2 provides voice-over for animation segment 2, with a sub-duration t2′ = 1.5s; video segment 3 provides voice-over for animation segment 3, with a sub-duration t3′ = 4.5s; and video segment 4 provides voice-over for animation segment 4, with a sub-duration t4′ = 2s.

[0053] In the embodiments of this application, the M animation segments can be consecutive animation segments or non-consecutive animation segments.

[0054] For example, such as Figure 3 As shown, the AR model's animated video has an animation duration of T = 10s, divided into 4 animation segments. The sub-animation duration of animation segment 1 is t1 = 2s, that of animation segment 2 is t2 = 1.5s, that of animation segment 3 is t3 = 4.5s, and that of animation segment 4 is t4 = 2s. When consecutive animation segments 1 and 2 require voice-over, the generated completely transparent video has a duration of T′ of 3.5s, divided into 2 video segments. Video segment 1 is used to provide voice-over for animation segment 1, with a sub-duration of t1′ = 2s, and video segment 2 is used to provide voice-over for animation segment 2, with a sub-duration of t2′ = 1.5s.

[0055] For example, such as Figure 4 As shown, the AR model's animated video has an animation duration of T = 10s, divided into 4 animation segments. The sub-animation durations for animation segment 1 are t1 = 2s, t2 = 1.5s, t3 = 4.5s, and t4 = 2s. When non-contiguous animation segments 1 and 3 require voice-over, the generated completely transparent video has a duration of T′ of 6.5s, divided into 2 video segments. Video segment 1 is used to provide voice-over for animation segment 1, with a sub-duration of t1′ = 2s, and video segment 2 is used to provide voice-over for animation segment 3, with a sub-duration of t2′ = 4.5s.

[0056] In the embodiments of this application, each video segment can be stored independently or spliced ​​together into a complete video for storage.

[0057] Optionally, in one implementation, if the animation duration of the AR model's animation video is divided into N animation segments, and M animation segments require voice-over, then when the duration of the completely transparent video is equal to the animation duration of the animation video, the sound in the completely transparent video is empty within the sub-animation duration of the NM animation segments that do not require voice-over.

[0058] For example, such as Figure 5As shown, the animation video of the AR model has an animation duration of T = 10s and is divided into 4 animation segments. The sub-animation duration of animation segment 1 is t1 = 2s, the sub-animation duration of animation segment 2 is t2 = 1.5s, the sub-animation duration of animation segment 3 is t3 = 4.5s, and the sub-animation duration of animation segment 4 is t4 = 2s. When animation segments 2 and 4 do not require dubbing, the duration of the completely transparent video is T′ = 10s. There is no sound between the 2nd and 3.5s and between the 8th and 10ths in the completely transparent video, that is, there is no sound.

[0059] Optionally, in this embodiment, the completely transparent video is in MP4 format.

[0060] It should be noted that this application embodiment does not impose restrictive requirements on the format of completely transparent video, and can be set according to the format supported by the target application in the actual scenario.

[0061] Based on a pre-generated AR model, its animation, and a fully transparent video, this application provides a method for adding animation and sound effects to an AR model, as follows: Figure 6 As shown, this process is executed by a playback device with AR capabilities. The client software installed on the playback device includes an AR mini-program. The process mainly includes the following steps:

[0062] S601: Launch the AR mini-program, draw the AR scene in the target window, and load the AR model and fully transparent video.

[0063] In one optional implementation, the user opens an AR mini-program in the client software to enter a SLAM AR scene. Using a Renderer control, the constructed AR scene is drawn into the target window, thereby displaying the environment within the camera's field of view on the playback device. The pre-generated AR model and fully transparent video from the AR mini-program's service platform are asynchronously loaded into the SLAM AR scene. Initially, the loaded AR model and fully transparent video are invisible.

[0064] The fully transparent video is a pre-generated animated video based on an AR model, containing sound that matches the AR model's animation. The AR model's area in the fully transparent video is set to be completely transparent, making it invisible during playback and thus avoiding interference with the AR model's animated video. The specific generation method for the fully transparent video is described in the aforementioned embodiments and will not be repeated here.

[0065] In this application embodiment, the number of AR models can be one or multiple. This application embodiment does not impose any limiting requirements. The three-dimensional reconstruction process of the AR models is not the focus of this application and will not be described in detail here.

[0066] S602: In response to the AR model placement command, place the AR model at the target location in the AR scene and display it.

[0067] In one alternative implementation, the playback device receives an AR model placement instruction triggered by a user's click operation, takes the user's click location as the target location, and places the AR model at the target location.

[0068] For example, such as Figure 7 As shown, in the SLAM AR scene, the SLAM plugin detects planes in the current AR scene in real time. After detecting a plane, it gives the plane "Place AR model" instruction to guide the user to place the AR model. According to the instruction, the user finds a horizontal table and clicks the corresponding position on the display screen to trigger the AR model placement instruction. After receiving the AR model placement instruction, the playback device places two AR spheres with a certain distance between them in the AR scene selected by the user on the horizontal table clicked by the user.

[0069] In another alternative implementation, the playback device detects the current scene and determines the target location for placing the AR model. Once the target location is determined, an AR model placement command is triggered to place the AR model at the target location.

[0070] For example, the SLAM plugin in the playback device can detect planes in the current scene captured by the camera in real time. When the desktop is detected, it immediately triggers the AR model placement command and automatically places two AR spheres that are a certain distance apart in the AR scene selected by the user on the desktop.

[0071] S603: When the AR model is triggered to play an animation, use the model animation plugin to play the AR model's animation video in the target window and start a timer.

[0072] In one alternative implementation, in response to the movement of the playback device, the SLAM plugin in the AR mini-program is used to collect the distance between the mobile device and the AR model in real time during the movement of the playback device. When the distance meets the preset distance range, the model animation plugin is used to start playing the AR model's animation video in the target window.

[0073] In practice, after the AR model is placed at the target location, the SLAM AR scene provides the user with prompts to interact with the AR model. The user moves the playback device according to these prompts to adjust the distance between the playback device (i.e., the camera) and the AR model displayed on the screen. During this distance adjustment process, the distance perception module in the SLAM plugin monitors the distance between the playback device and the AR model in real time, stopping the adjustment when the distance is within the optimal interaction range. This optimal interaction range can be set according to the actual AR scene.

[0074] For example, assuming the optimal interaction distance in the current scene is 10-15 centimeters (cm), when the SLAM plugin detects that the distance between the playback device and the AR model is greater than 15cm, the display shows a prompt to shorten the distance to the AR model; when the SLAM plugin detects that the distance between the playback device and the AR model is less than 10cm, the display shows a prompt to increase the distance to the AR model.

[0075] When the distance between the playback device and the AR model falls within the optimal interaction range by moving the playback device, the AR model is triggered to play animation, and a timer is started at the same time.

[0076] As an example, when using a model animation plugin to play animated videos of AR models, it essentially uses the three.js framework as a 3D model player. Through the loader control within the three.js framework, the animations of AR models in formats such as GLB are loaded into the AR scene, and operations such as adjusting color and lighting, adjusting coordinates, switching perspectives, and playing animations are performed on the AR models. The three.js framework can play video frames of each animation segment of the AR model and render the latest calculated scene data in the target window, allowing users to see the dynamic changes in the animation and scene, thus achieving the playback of the animated video.

[0077] It should be noted that the embodiments of this application do not impose restrictive requirements on the triggering method for playing the AR model animation. For example, the playback device may respond to the user's click operation and use the model animation plugin to start playing the AR model animation in the target window. Alternatively, after the AR model is placed, the model animation plugin may be used to immediately play the AR animation video in the target window.

[0078] S604: Based on the animation time of the AR model, use a video playback plugin to play a completely transparent video in the target window to provide audio for the AR model animation.

[0079] As an example, the video playback plugin in the AR mini-program supports the 3D drawing protocol (WebGraphics Library, WebGL) in HTML5. Based on this protocol, the corresponding video frames in the completely transparent video are drawn in the target window according to the animation time of the AR model to provide audio for the AR model animation.

[0080] In the embodiments of this application, the playback method of the completely transparent video differs depending on the animation of the AR model. Specifically, the model animation component of the AR model's animated video and the video playback component of the completely transparent video are different components. Since the transparency channel value of pixels within the AR model's region in the completely transparent video is 0, the user will not see the AR model in the completely transparent video. Therefore, the playback of the completely transparent video will not interfere with the AR model's animation playback process.

[0081] In one optional implementation, when the AR model's animation video is divided into N animation segments and M animation segments require dubbing, if each of the M animation segments corresponds to a video segment in a completely transparent video (i.e., the duration of the completely transparent video is less than or equal to the animation duration), when the time for the animation segment requiring dubbing arrives, the video segment in the completely transparent video corresponding to the current animation segment is played to provide dubbing for the AR model animation.

[0082] by Figure 7 Taking the SLAMAR scene shown as an example, the AR sphere animation video is divided into three animation segments. Segment A shows the two spheres approaching each other, lasting 1 second (i.e., the sub-animation duration of segment A is 1 second). Segment B shows the two spheres colliding, flying out, and landing, lasting 2 seconds (i.e., the sub-animation duration of segment B is 2 seconds). Segment C shows the two spheres moving away from each other, lasting 1 second (i.e., the sub-animation duration of segment C is 1 second). The sound in the completely transparent video is processed to simulate the collision, flying out, and landing sound effects of segment B, used to provide narration for segment B, lasting 2 seconds. The playback process is as follows... Figure 8 As shown, when animation clip A starts playing using the model animation plugin, a timer is simultaneously started. After a 1-second delay (the duration of animation clip A), the fully transparent video playback component is invoked to play the fully transparent video. The fully transparent video plays for 2 seconds (the duration of animation clip B). In other words, at the last frame after animation clip A finishes playing, the timer delay logic starts, and the fully transparent video plays synchronously with animation clip B for 2 seconds, matching the animation's movement to achieve audiovisual unity. After animation clip B finishes playing, animation clip C, showing the two spheres moving away from each other, begins playing.

[0083] In the method for adding animation sound effects to AR models provided in this application embodiment, based on the SLAM plugin in the AR mini-program, when playing a silent AR model animation video using the model animation plugin, a timer function is used to call the video playback component of the fully transparent video from the moment the animation starts playing, and play a fully transparent video with sound that matches the scene and actions of the animation segment, thereby achieving audiovisual unity, improving the playback effect of the animation, and thus improving the user experience.

[0084] In another optional implementation, when the AR model's animation video is divided into N animation segments, and M of these segments require voice-over, if the duration of the completely transparent video is equal to the animation duration of the animation video, then the completely transparent video is played to provide voice-over for the AR model's animation process, starting from the beginning of the AR model's animation. Where at least one animation segment in the AR model's animation video does not require voice-over, the sub-animation duration of the animation segment in the completely transparent video that does not require voice-over is silent.

[0085] Still with Figure 7 Taking the SLAM AR scene shown as an example, the AR sphere animation video is divided into three animation segments. Segment A shows the two spheres approaching each other, lasting 1 second (i.e., the sub-animation duration of segment A is 1 second). Segment B shows the two spheres colliding, flying out, and landing, lasting 2 seconds (i.e., the sub-animation duration of segment B is 2 seconds). Segment C shows the two spheres moving away from each other, lasting 1 second (i.e., the sub-animation duration of segment C is 1 second). The duration of the completely transparent video is 4 seconds. In the completely transparent video, the first second is silent, the middle two seconds contain the sounds of collision, flying out, and landing, and the last second is silent. The playback process is as follows... Figure 9 As shown, when the model animation plugin starts playing animation clip A, a timer is simultaneously started, calling the fully transparent video playback component to play the fully transparent video. That is, in the first frame of animation clip A, the timer starts, and the fully transparent video plays synchronously with animation clip B for 4 seconds, until the last frame of animation clip C, where the two spheres move away from each other. Since the fully transparent video only contains valid sound for the middle 2 seconds, with no sound in the first and last seconds, although the fully transparent video lasts for 4 seconds, only the middle 2 seconds contain sound for animation clip B, achieving audiovisual consistency.

[0086] In the method for adding animation sound effects to AR models provided in this application embodiment, based on the SLAM plugin in the AR mini-program, for the animation process of the silent AR model, a timer function is used to call the video playback component of the fully transparent video from the moment the animation starts playing, and the fully transparent video is played independently. Since the duration and content of the sound in the fully transparent video are adapted to the animation clips of the AR model, audiovisual unity is achieved, the playback effect of the animation is improved, and the user experience is enhanced.

[0087] The following describes the method flow for adding animation and sound effects to an AR model according to the embodiments of this application, starting from the process of user interaction with the playback device. See also Figure 10 It mainly includes the following steps:

[0088] S1001: The user clicks to play the AR mini-program on the device.

[0089] The AR mini-program includes a SLAM plugin.

[0090] S1002: The playback device launches an AR mini-program, loading an AR model and a fully transparent video containing sound that matches the animation of the AR model.

[0091] The playback device launches the AR mini-program and asynchronously loads the pre-generated AR model and fully transparent video from the AR mini-program's service platform into the SLAM AR scene. Initially, the loaded AR model and fully transparent video are invisible in the scene, awaiting playback.

[0092] S1003: The SLAM plugin in the playback device performs real-time planar detection on the current screen and displays the placement position of the AR model based on the detection results.

[0093] S1004: The user clicks the display screen as prompted.

[0094] S1005: The playback device places the AR model at the location clicked by the user and displays the AR model.

[0095] S1006: The SLAM plugin in the playback device monitors the distance between the playback device and the AR model in real time.

[0096] S1007: Determine if the current distance is within the optimal interaction distance range. If not, execute S1008; if yes, execute S1009.

[0097] S1008: The playback device provides a movement prompt so that the user can adjust the distance between the playback device and the AR model according to the prompt until the adjusted distance is within the optimal interaction range.

[0098] S1009: Use the model animation plugin to play the animation video of the AR model and start the timer.

[0099] S1010: When the moment for the animated segment that requires voice-over arrives, use the video playback plugin to play the video clip corresponding to the current animated segment in a completely transparent video to provide voice-over for the AR model animation.

[0100] The method for adding animation and sound effects to an AR model provided in this application embodiment separates the silent animation of the AR model from the completely transparent video containing sound. The model animation plugin and video playback plugin of the AR mini-program are used respectively to draw the animated video and the completely transparent video in the same target window. A timer is used to coordinate the synchronous playback timing of the animated video and the completely transparent video to achieve a unified audiovisual sound animation effect, improve the rendering power of the animation effect, and since the AR model is invisible in the completely transparent video, it will not interfere with the playback of the AR animation, thereby ensuring the animation quality and improving the user experience.

[0101] Based on the same technical concept, this application provides a playback device, which includes, but is not limited to, smartphones, AR glasses, tablets, laptops, vehicle terminals, etc., and can implement the above-described method steps for adding animation and sound effects to AR models, and can achieve the same technical effect.

[0102] See Figure 11 The client software installed on the playback device includes an AR mini-program. The playback device includes a processor 1101, a memory 1102, a display screen 1103, a speaker 1104, and a camera 1105. The camera 1105, the speaker 1104, the display screen 1103, the memory 1102, and the processor 1101 are connected via a bus 1106.

[0103] The memory 1102 stores a computer program, and the processor 1101 performs the following operations according to the computer program:

[0104] Launch the AR mini-program, draw the AR scene captured by the camera 1105 in the target window, and load the AR model and the fully transparent video. The fully transparent video is a pre-generated animation video based on the AR model and contains sound that matches the animation of the AR model. The area of ​​the AR model in the fully transparent video is set to be completely transparent, so that the area of ​​the AR model is not visible when the fully transparent video is played.

[0105] In response to the AR model placement command, the AR model is placed at the target location in the AR scene and displayed on the display screen 1103;

[0106] When the AR model is triggered to play an animation, the model animation plugin is used to draw the animation video of the AR model in the target window and display it on the display screen 1103. At the same time, a timer is started.

[0107] Based on the animation time of the AR model, the fully transparent video is drawn in the target window using a video playback plugin, and the fully transparent video is played through the display screen 1103. The sound from the fully transparent video is played through the speaker 1104 to provide narration for the AR model animation.

[0108] Optionally, the triggering method for the AR model to play animations includes:

[0109] In response to the movement of the playback device, the SLAM plugin in the AR mini-program is used to collect the distance between the mobile device and the AR model in real time during the movement of the playback device;

[0110] When the distance meets the preset distance range, the model animation plugin is used to start drawing the animation video of the AR model in the target window and display it on the display screen 1103.

[0111] Optionally, the triggering method for the AR model to play animations also includes:

[0112] In response to a user's click, the model animation plugin is used to begin drawing an animation video of the AR model in the target window and displaying it on the display screen 1103; or

[0113] After the AR model is placed, the model animation plugin is used to immediately draw an animated video of the AR model in the target window and display it on the display screen 1103.

[0114] Optionally, when the AR model's animation video is divided into N animation segments, and M animation segments require dubbing, and the M animation segments each correspond to a video segment in the completely transparent video, then N is an integer greater than or equal to 2, and M is an integer greater than or equal to 1 and less than or equal to N.

[0115] The processor 1101, based on the animation time of the AR model, uses a video playback plugin to draw the completely transparent video in the target window, plays the completely transparent video through the display screen 1103, and plays the sound from the completely transparent video through the speaker 1104 to provide narration for the AR model animation. Specifically, the operation is as follows:

[0116] When the moment for the animated segment requiring voiceover arrives, the video playback plugin is used to draw the video segment corresponding to the current animated segment in the fully transparent video in the target window and display it on the display screen. At the same time, the speaker 1104 plays the sound of the corresponding video segment to provide voiceover for the AR model animation.

[0117] Optionally, the duration of the fully transparent video is equal to the animation duration of the AR model's animated video;

[0118] The processor 1101, based on the animation time of the AR model, uses a video playback plugin to draw the completely transparent video in the target window, plays the completely transparent video through the display screen 1103, and plays the sound from the completely transparent video through the speaker 1104 to provide narration for the AR model animation. Specifically, the operation is as follows:

[0119] From the start of the AR model animation, a video playback plugin is used to draw the completely transparent video in the target window and display it on the display screen 1103. At the same time, the sound from the completely transparent video is played through the speaker 1104 to provide narration for the AR model animation process.

[0120] Optionally, when at least one animation segment in the AR model's animation video does not require voice-over, the sub-animation duration of the animation segment that does not require voice-over in the fully transparent video is empty.

[0121] Optionally, in response to an AR model placement command, the processor 1101 places the AR model at a target location in the AR scene, specifically as follows:

[0122] The system receives an AR model placement command triggered by a user clicking on the display screen 1103, sets the user's click location as the target location, and places the AR model at the target location; or

[0123] The current scene is detected to determine the target position for the AR model. Once the target position is determined, an AR model placement command is triggered to place the AR model at the target position.

[0124] Optionally, the fully transparent video is in MP4 format.

[0125] It should be noted that, Figure 11 This is merely an example illustrating the hardware necessary for a playback device to perform the method steps for adding sound effects to AR animations as provided in the embodiments of this application. Not shown, the playback device also includes conventional components of a display device such as a microphone and power supply.

[0126] Examples of this application Figure 11 The processor involved can be a central processing unit (CPU), a general-purpose processor, a graphics processing unit (GPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof.

[0127] This application also provides a computer-readable storage medium for storing instructions that, when executed, can perform the method of adding sound effects to AR animations as described in the foregoing embodiments.

[0128] This application also provides a computer program product for storing a computer program that executes the method for adding sound effects to AR animations as described in the foregoing embodiments.

[0129] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0130] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in one or more blocks of the flowchart illustrations and / or one or more blocks of the block diagrams.

[0131] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means that implement the functions specified in one or more flowcharts and / or one or more block diagrams.

[0132] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide steps for implementing the functions specified in one or more flowcharts and / or one or more block diagrams.

[0133] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A method for adding animated sound effects to an AR model, characterized in that, include: Launch the AR mini-program, draw the AR scene in the target window, and load the AR model and the fully transparent video; the AR mini-program includes a simultaneous localization and mapping (SLAM) plugin, the SLAM plugin does not have the ability to play audio files, the fully transparent video is pre-generated based on the animation video of the AR model, and contains sound that matches the animation of the AR model, the area of ​​the AR model in the fully transparent video is set to be completely transparent, so that the area of ​​the AR model is not visible when the fully transparent video is played, and the format of the fully transparent video is a video format supported by the SLAM plugin; In response to an AR model placement command triggered after the SLAM plugin detects a plane, the AR model is placed at the target location in the AR scene and displayed. When the SLAM plugin detects that the AR model is triggered to play animation, it uses the model animation plugin to play the animation video of the AR model in the target window and starts a timer; Based on the animation time of the AR model, a video playback plugin is used to play the fully transparent video in the target window to play sound through the fully transparent video to provide narration for the AR model animation; The model animation plugin and the video playback plugin are independent playback components. The AR model's animation video and the completely transparent video are drawn in the same target window, and the image of the completely transparent video is not visible.

2. The method as described in claim 1, characterized in that, The triggering methods for the AR model to play animations include: In response to the movement of the playback device, the SLAM plugin in the AR mini-program is used to collect the distance between the playback device and the AR model in real time during the movement of the playback device; When the distance meets the preset distance range, the model animation plugin is used to start playing the animation video of the AR model in the target window.

3. The method as described in claim 1, characterized in that, The triggering methods for the AR model to play animations also include: In response to a user's click, the model animation plugin is used to start playing an animated video of the AR model in the target window; or After the AR model is placed, the model animation plugin is used to immediately play the AR model's animation video in the target window.

4. The method as described in claim 1, characterized in that, When the AR model's animated video is divided into N animation segments, and M animation segments require dubbing, and each of the M animation segments corresponds to a video segment in the completely transparent video, then N is an integer greater than or equal to 2, and M is an integer greater than or equal to 1 and less than or equal to N. The step of using a video playback plugin to play the fully transparent video in the target window to provide audio for the AR model animation, based on the animation time of the AR model, includes: When the time for the animated segment requiring voiceover arrives, a video playback plugin is used to play the video clip corresponding to the current animated segment in the fully transparent video in the target window to provide voiceover for the AR model animation.

5. The method as described in claim 1, characterized in that, The duration of the completely transparent video is equal to the duration of the animated video. The step of using a video playback plugin to play the fully transparent video in the window to provide narration for the animation process of the AR model, based on the animation time of the AR model, includes: From the start of the AR model animation, a video playback plugin is used to play the fully transparent video in the window to provide narration for the AR model animation process.

6. The method as described in claim 5, characterized in that, When at least one animation segment in the animated video does not require dubbing, the audio is empty within the sub-animation duration of the animation segment that does not require dubbing in the completely transparent video.

7. The method according to any one of claims 1-6, characterized in that, The process of generating the fully transparent video is as follows: Export the animated video of the AR model; For each video frame in the video animation, the region of the AR model is identified, and the identified region is set as a completely transparent region to generate a new video frame with transparency. Add sound to the new video frame that matches the animation of the AR model; Generate a fully transparent video from multiple new video frames, each with its own transparency and sound.

8. The method according to any one of claims 1-6, characterized in that, The step of placing the AR model at the target location in the AR scene in response to the AR model placement command includes: Receive the AR model placement instruction triggered by a user click operation, take the user click location as the target location, and place the AR model at the target location; or The current scene is detected to determine the target position for the AR model. Once the target position is determined, an AR model placement command is triggered to place the AR model at the target position.

9. The method according to any one of claims 1-6, characterized in that, The completely transparent video is in MP4 format.

10. A playback device, characterized in that, The client software installed on the playback device includes an AR mini-program. The playback device includes a processor, a memory, a display screen, a speaker, and a camera. The camera, the speaker, the display screen, the memory, and the processor are connected via a bus. The memory stores a computer program, and the processor performs the following operations according to the computer program: Launch the AR mini-program, draw the AR scene captured by the camera in the target window, and load the AR model and the fully transparent video; the AR mini-program includes a simultaneous localization and mapping (SLAM) plugin, the SLAM plugin does not have the ability to play audio files, the fully transparent video is pre-generated based on the animation video of the AR model, and contains sound matching the animation of the AR model, the area of ​​the AR model in the fully transparent video is set to be completely transparent, so that the area of ​​the AR model is not visible when the fully transparent video is played, and the format of the fully transparent video is a video format supported by the SLAM plugin; In response to an AR model placement command triggered after the SLAM plugin detects a plane, the AR model is placed at the target location in the AR scene and displayed on the display screen; When the SLAM plugin detects that the AR model is triggered to play an animation, it uses the model animation plugin to draw the animation video of the AR model in the target window and display it on the display screen. At the same time, a timer is started. Based on the animation time of the AR model, a video playback plugin is used to draw the fully transparent video in the target window, play the fully transparent video through the display screen, and play the sound in the fully transparent video through the speaker to provide sound effects for the AR model animation. The model animation plugin and the video playback plugin are independent playback components. The AR model's animation video and the completely transparent video are drawn in the same target window, and the image of the completely transparent video is not visible.

Images