Video generation method and device, equipment, medium and product

Abstract

The embodiment of the invention provides a video generation method and device, equipment, a medium and a product. The method comprises the following steps: in a video picture display process, determining a first track based on collected audio information; determining a first special effect track associated with the audio information according to the first track and a preset first material image; determining a second special effect track of the first track based on a target mask graph and a second material image associated with the first track; and based on the first special effect track and the second special effect track, determining and displaying a target track displayed in the video picture. According to the technical scheme provided by the embodiment of the invention, special effect processing is carried out on the first track corresponding to the audio information from different dimensions, the smooth target track is obtained, and correspondingly, when the target track is displayed, the content richness of a picture and the interactivity with a user can be improved.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of image processing technology, and in particular to a video generation method, apparatus, device, medium and product. Background Art

[0002] With the popularization of intelligent devices, users' demand for interaction with smart terminals is increasing.

[0003] Currently, users can interact with smart terminals based on audio information, but at this time, the content of the audio information is mainly analyzed and processed to determine the corresponding interactive content. That is, the interaction mainly relies on the content of the audio information itself, and the interactive content is not generated by combining the richer content of the audio information. There is a problem of limitations on the use of audio information. Summary of the invention

[0004] The present disclosure provides a video generation method, apparatus, device, medium and product, which generate a corresponding first trajectory based on audio information, and process the first trajectory with special effects from at least two dimensions to obtain a target trajectory displayed in a video screen, thereby improving the richness of the video screen content and the interactivity between the user and the terminal device based on the audio information.

[0005] In a first aspect, an embodiment of the present disclosure provides a video generation method, the method comprising:

[0006] During the video display, a first trajectory is determined based on the collected audio information;

[0007] Determining a first special effect track associated with the audio information according to the first track and a preset first material image; wherein display information corresponding to the first special effect track is related to pixel information of the first material image;

[0008] Determining a second special effect track of the first track based on a target mask image and a second material image associated with the first track;

[0009] Based on the first special effect track and the second special effect track, a target track to be displayed in the video screen is determined and displayed; wherein change information of the target track is associated with the audio information.

[0010] In a second aspect, an embodiment of the present disclosure further provides a video generating device, the device comprising:

[0011] A first trajectory determination module, used to determine a first trajectory based on the collected audio information during the video screen display process;

[0012] A first special effect track generating module, configured to determine a first special effect track associated with the audio information according to the first track and a preset first material image; wherein display information corresponding to the first special effect track is related to pixel information of the first material image;

[0013] A second special effect trajectory generating module, configured to determine a second special effect trajectory of the first trajectory based on a target mask image and a second material image associated with the first trajectory;

[0014] A target trajectory generation module is used to determine and display the target trajectory displayed in the video picture based on the first special effect trajectory and the second special effect trajectory; wherein the change information of the target trajectory is associated with the audio information.

[0015] In a third aspect, an embodiment of the present disclosure further provides an electronic device, the electronic device comprising:

[0016] one or more processors;

[0017] a storage device for storing one or more programs,

[0018] When the one or more programs are executed by the one or more processors, the one or more processors implement the video generating method as described in any one of the embodiments of the present disclosure.

[0019] In a fourth aspect, the embodiments of the present disclosure further provide a storage medium comprising computer executable instructions, which, when executed by a computer processor, are used to execute the video generation method as described in any one of the embodiments of the present disclosure.

[0020] In a fifth aspect, the embodiments of the present disclosure further provide a computer program product, including a computer program, which, when executed by a processor, implements the video generation method as described in any one of the embodiments of the present disclosure.

[0021] The technical solution of the disclosed embodiment can analyze and process the collected audio information during the video screen display process to determine the first track. Furthermore, the first material image can be used to perform special effects processing on the first track to obtain the first special effects track, which enriches the screen effect of the first track. In order to further improve the richness and enjoyability of the video screen content, the mask image corresponding to the first track can be rendered based on the second material image to obtain the second special effects track. Based on the first special effects track and the second special effects track, the target track displayed in the video screen can be obtained, which solves the current problem that when interacting with smart terminals based on audio information, the content of the audio information itself is mainly analyzed and processed to obtain the corresponding interactive results, and the richer content of the audio information is not used. There are limitations in the use of audio information, resulting in poor interactive effects. The corresponding first track is generated based on the audio information, and the first track is processed with special effects from multiple dimensions and then integrated and displayed, which improves the richness of the video screen content and the interactivity with the user. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that components and elements are not necessarily drawn to scale.

[0023] Figure 1 A schematic diagram of a flow chart of a video generation method provided by an embodiment of the present disclosure;

[0024] Figure 2 A schematic diagram of a flow chart of a video generation method provided by an embodiment of the present disclosure;

[0025] Figure 3 A schematic diagram of a process for determining trajectory control points provided by an embodiment of the present invention;

[0026] Figure 4 A schematic diagram of a flow chart of a first trajectory provided by an embodiment of the present invention;

[0027] Figure 5 A schematic diagram of a video generation method provided by an embodiment of the present disclosure

[0028] Figure 6 A first material image provided by an embodiment of the present disclosure;

[0029] Figure 7 A schematic diagram of a first special effect track provided by an embodiment of the present disclosure;

[0030] Figure 8 A schematic diagram of a first special effect trajectory after Gaussian blur processing provided in an embodiment of the present disclosure;

[0031] Fig. 9 A schematic diagram of generating a target mask map provided by an embodiment of the present disclosure;

[0032] Fig.10 The target mask image after Gaussian blur processing provided in the embodiment of the present disclosure;

[0033] Fig.11 A schematic diagram of a second material image provided by an embodiment of the present disclosure;

[0034] Fig.12 A schematic diagram of a second special effect track provided by an embodiment of the present disclosure;

[0035] Fig.13 A schematic diagram of the trajectory to be fused provided by an embodiment of the present disclosure;

[0036] Fig.14 A schematic diagram of a target trajectory provided by an embodiment of the present disclosure;

[0037] Fig.15 A schematic diagram of the structure of a video generating device provided by an embodiment of the present disclosure;

[0038] Fig.16 A schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0039] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0040] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0041] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0042] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0043] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0044] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0045] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0046] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0047] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0048] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet the relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0049] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.

[0050] Before introducing the technical solution provided by the embodiment of the present invention, an exemplary description of the application scenario can be given. Users can use application software to shoot videos, make video calls with other users, and use certain application software that supports games or integrate the solution provided by the embodiment of the present invention as a game function into existing applications. The solution provided by the embodiment of the present invention can be used to generate a target trajectory, which is the special effect trajectory finally displayed.

[0051] In practical applications, if a target trajectory corresponding to audio information is to be generated, it can be implemented by referring to the solution provided in the embodiment of the present disclosure.

[0052] It can be understood that the above application software can be a type of software for image / video processing. The specific application software will not be described one by one again. It can mainly realize image / video processing. Of course, it is also possible to develop forwarding applications to realize the software of adding special effects and displaying special effects, or integrate them into the corresponding pages. Users can realize the processing of special effect videos through the pages integrated on the PC.

[0053] Exemplarily, if the technical solution provided by the embodiment of the present invention is integrated into an application software that can shoot videos or video calls, the solution provided by the embodiment of the present invention can be integrated into a special effects prop and installed in the corresponding application software. When the user's trigger operation on the special effects prop is detected, audio information can be collected and the target trajectory displayed in the video screen can be generated based on the audio information; if the solution provided by the embodiment of the present invention is integrated as a function in any existing supported game application, it can be displayed as a function control in the main interface or a sub-interface of the application. When the trigger operation on the function control is detected, the game map corresponding to the game can be called, and the target trajectory can be generated based on the game map and the collected audio information. Alternatively, when it is detected that certain conditions are met during the game, the solution provided by the embodiment of the present invention can be automatically called to form a target trajectory.

[0054] The specific implementation method of generating the target trajectory can be found in the detailed description of the following embodiment.

[0055] Figure 1 This is a flow chart of a video generation method provided by an embodiment of the present disclosure. The embodiment of the present disclosure is applicable to the scene of generating a target trajectory in the currently displayed video screen. The video generation method provided by the embodiment of the present invention can be executed by the client, can be executed by the server, or can be executed by the client and the server in cooperation. Of course, a video generation device corresponding to the video generation method can be integrated in the client and / or the server.

[0056] like Figure 1As shown, the method of this embodiment may specifically include:

[0057] S110: During the video display process, determine a first trajectory based on the collected audio information.

[0058] The video screen may be a video screen captured after a special effect prop is triggered, or a game screen displayed after a game function control is triggered. In other words, the video screen is a screen displayed after a special effect prop or a game function is triggered according to an embodiment of the present invention.

[0059] It should be noted that the video screen may include a lot of background content, which may be pre-set content. Optionally, the background content may be flowers, plants, and trees, or display elements corresponding to a certain festival. Optionally, the festival may be the Mid-Autumn Festival, and the display elements may be rabbit elements, moon elements, and so on.

[0060] In the process of displaying the video screen, audio information can be collected in real time or periodically. Alternatively, when the presence of audio information is detected, it can be collected. At this time, the audio information is mainly the audio emitted by the object not in the video screen. However, when the presence of audio information is detected, the lip shape of the object in the video screen can be made consistent with the lip shape of the collected audio information to simulate the effect that the audio information is emitted based on the object in the video screen, so as to achieve the effect of authenticity and appreciation of the picture effect.

[0061] After the audio information is collected, the audio information can be analyzed and processed to obtain a first track corresponding to the audio information. The first track at this time is a track that has not been further rendered with special effects. The length of the first track and the display height in the video screen are related to the duration and pitch of the audio information.

[0062] It should be noted that the first trajectory is not displayed in the video screen at this time, and the first trajectory can be processed again to obtain a target trajectory with a cool effect and display it in the video screen.

[0063] Specifically, after detecting the triggering of special effect props, a video screen can be displayed. During the video screen display process, the microphone array of the terminal device to which the application belongs can be started to collect audio information based on the microphone array. Alternatively, if the game function control corresponding to the embodiment of the present invention integrated in the application is triggered, a pre-set game screen (video screen) can be displayed. The user can play games based on the video screen, and obtain the audio information of the object in the non-video screen in real time during the game (displaying the video screen). Of course, it can also be a live video or a video call. If the function control corresponding to the solution provided by the embodiment of the present invention is triggered, the audio information of the object of the terminal to which the application belongs can also be collected. After the audio information collection is completed, the audio information can be analyzed and processed to obtain the first trajectory.

[0064] In this embodiment, during the process of displaying the video screen, the method also includes: when it is detected that the trajectory generation condition is met, collecting audio information of the first object, wherein the trajectory generation condition includes: determining that the second object moves to a preset position based on a preset map, wherein the second object is a movable object displayed in the video screen; triggering the trajectory generation control in the video screen or the application software to which the video screen belongs.

[0065] Among them, the screen content displayed in the video screen can be the content designed in the design stage, or it can be the screen content based on the screen content collected by the camera device, or it can be the screen after the content designed in the design stage and the screen collected by the camera device are superimposed. The user corresponding to the terminal device to which the application software to which the video screen belongs can be taken as the first object. For example, if "I" triggers the function control in the application to generate the target trajectory, "I" is the first object. In other words, the first object is an object that sends audio information in the real environment and can be collected by the terminal to which the application belongs. The second object is a movable object displayed in the video screen. The second object can be a pre-set object or an object virtualized based on the image of the first object.

[0066] In this embodiment, if it is a game scene, a game map can be created. The game map can be used as a preset map. The game map may include screen elements such as mountains, streams, and gullies that the second object needs to cross. When the above elements appear, it may be necessary to generate a target trajectory to cross the elements in the above screen based on the target trajectory to achieve the effect of moving from one point to another. At the same time, the coordinate information of the above elements can be marked in the game map. When the second object moves to the preset position corresponding to the above elements at a preset moving speed, the audio information of the first object can be collected. In other words, the preset position is the starting position where the above elements appear.

[0067] If it is a non-game scene, such as a live broadcast scene, a special effect video generation scene, or a video call scene. The track generation control can be displayed in the video screen, or the application software to which the video screen belongs can display the track generation control. When the above track generation control is detected to be triggered, the audio information of the first object can be collected. Alternatively, the audio information can be collected all the time, and when there is a need for track generation, the audio information collected at the current moment and after the current moment is analyzed and processed to obtain the first track.

[0068] The above may be understood as: if the trajectory generation condition is met, the audio information of the first object may be collected.

[0069] S120: Determine a first special effect track associated with the audio information according to the first track and a preset first material image.

[0070] In order to improve the picture quality, the obtained first trajectory may be further processed to obtain a first special effect trajectory. That is, the first special effect trajectory is a trajectory obtained after special effect rendering of the first trajectory.

[0071] The first material image is used to adjust the display information of each pixel point in the first track. A base map with a gradient of color and / or transparency can be made as the first material image. The size of the first material image is consistent with the size of the video screen. After obtaining the first track, the first material image can be traversed according to the horizontal coordinate of each point on the first track to obtain the display information of each track point on each first track. The first track after the display information is updated is used as the first special effect track.

[0072] It can be understood that: a first material image with color and / or transparency gradient can be produced according to actual needs. According to the horizontal coordinate of each point in the first track, the pixel information in the first material image is traversed, and the pixel information of each point in the first track is determined and updated to obtain the first special effect track.

[0073] S130: Determine a second special effect track of the first track based on the target mask image and the second material image associated with the first track.

[0074] Among them, an initial trajectory corresponding to the first trajectory can be created, and the initial mask image can be expanded to obtain a target mask image. It can be understood that the shape of the target mask image is related to the first trajectory, and the width is greater than the width of the first trajectory. The second material image is a pre-set material image that matches the target effect. The second material image can be an image that simulates a certain phenomenon. Optionally, the certain phenomenon can be an aurora phenomenon, that is, the second material image is an image that simulates the aurora. The second special effect trajectory is the trajectory obtained after processing the mask area associated with the first trajectory in the target mask image.

[0075] Specifically, after obtaining the target mask map, the display information of the pixels corresponding to the mask area in the target mask map can be adjusted according to the pixel values ​​of each pixel in the second material image. After the mask area of ​​the target mask map is filled with pixels, the mask area can be extracted to obtain the second special effect track.

[0076] S140: Based on the first special effect track and the second special effect track, determine and display a target track to be displayed in the video picture; wherein the track length of the target track is associated with the audio information.

[0077] It can be understood that after obtaining the first special effect track and the second special effect track, the first special effect track and the second special effect track can be merged together to obtain the target track. The target track is displayed in the video screen.

[0078] It should also be noted that the display starting point of the target trajectory can be pre-set, or it can be a position related to the element in the video screen. Optionally, if the element is a stream, the display starting point can be the bank of the stream to which the second object currently belongs; if the element is a gully, the display starting point can be any position on one side of the gully to which the second object currently belongs. The final display end point of the target trajectory (i.e., the pre-set constraint position) can be pre-set, optionally, the right edge of the video screen, or the ground on the other side of the peak, the ground on the other side of the stream, or the ground on the other side of the gully. However, the real-time display position of the trajectory point in the target trajectory in the video screen changes dynamically, and the main factor of its change is determined according to the tone of the audio information. It can be understood that the change information of the target trajectory is related to the audio information.

[0079] The technical solution provided by the embodiment of the present invention can determine a first trajectory by analyzing and processing the collected audio information during the display of the video screen, and then render the first trajectory according to the first trajectory and the first material image to obtain a first special effect trajectory. In order to further improve the display effect of the screen, the target mask image and the second material image corresponding to the first trajectory can be processed again to obtain a second special effect trajectory. Based on the first special effect trajectory and the second special effect trajectory, the target trajectory corresponding to the audio information can be determined and displayed in the video screen, achieving the effect of rich and natural content of the screen.

[0080] Figure 2 A flow chart of a video generation method provided in an embodiment of the present invention. Based on the aforementioned embodiment, "determining a first trajectory based on the collected audio information" can be further refined. The specific implementation method can refer to the detailed description of the technical solution, wherein the technical terms that are the same as or corresponding to the aforementioned embodiment are not repeated in this embodiment.

[0081] like Figure 2 As shown, the method includes:

[0082] S210: Determine at least one trajectory control point according to the pitch information of the valid audio frame in the audio information.

[0083] The audio information is composed of a plurality of audio frames, and optionally, there are 25 audio frames per second. Some audio frames may be invalid, and based on this, the validity analysis of the audio frames can be performed to determine the first track based on the valid audio frames. The track control point is used to adjust the direction of the first track, and optionally, the direction can be understood as the track shape.

[0084] Specifically, the audio information may be analyzed and processed to determine valid audio frames in the audio information, and then the trajectory control points may be determined according to the valid audio frames.

[0085] In this embodiment, the method of determining the valid audio frame in the audio information may be: after extracting the audio frame in the audio information, the volume and tone of the audio frame may be obtained. If the volume is greater than the volume in the first volume condition and the tone satisfies the tone in the first tone condition, the audio frame may be used as a valid audio frame.

[0086] Exemplarily, the first volume condition is that the volume is greater than 0.002, and the first pitch condition is that the pitch is greater than 0. If the volume in the audio frame is greater than 0.002 and the pitch is greater than 0, it indicates that the audio frame is a valid audio frame.

[0087] In this embodiment, determining at least one trajectory control point based on the tone information of the valid audio frame in the audio information can be: determining the tone offset based on the tone information and tone control information of the valid audio frame in the audio information, and adjusting the tone information of the valid audio frame according to the tone offset to obtain the tone information to be used; determining the target screen coordinates of the tone information to be used in the video picture, and determining the trajectory control point based on the target screen coordinates and the historical screen coordinates in the coordinate set.

[0088] Typically, when the first object is of the first type, optionally, the first type is a male type with a lower pitch, then the display height of the first track in the video screen will approach the bottom of the screen; when the first object is of the second type, optionally, the second type is a female type with a higher pitch, then the display height of the first track in the video screen will approach the top of the screen. That is to say, the first type is an object type with a lower pitch, and the second type is an object type with a higher pitch. Before use, it is possible to pre-set which objects are of the first type and which objects are of the second type, and determine the tone control information based on this. In order to avoid the problem of poor effect of the first track due to the above-mentioned object types, tone control information can be generated based on the first tone corresponding to the first type and the second tone corresponding to the second type, so as to facilitate the adjustment of the tone of any object type based on the tone control information, thereby improving the harmony of the target track with the video screen.

[0089] In this embodiment, the tone control information is determined based on the first tone of the first type and the second tone of the second type. Optionally, the average of the first tone and the second tone can be calculated and used as the tone control information. The advantage of such a setting is that the influence of the high tone and the low tone on the first trajectory can be neutralized. In other words, the tone control information is used to control the tone information to fluctuate within a preset range.

[0090] The pitch offset is the offset information obtained after the sound effect information of the valid audio frame is constrained according to the pitch control information. The pitch information of the valid audio frame can be adjusted based on the pitch offset, which is equivalent to normalizing all valid audio frames based on the pitch offset. The pitch information adjusted based on the pitch offset is used as the pitch information to be used.

[0091] The target screen coordinates are coordinate information corresponding to the mapping of the to-be-used tone information to the video screen. According to the target screen coordinates and the historical screen coordinates determined in the coordinate set, a trajectory control point for adjusting the direction of the Bezier curve is determined. The trajectory control point is mainly used to adjust the trajectory direction of the first trajectory. Optionally, the trajectory direction may include an extension angle of the first trajectory, and the extension angle may be an angle corresponding to the ground of the video screen.

[0092] The target screen coordinates of the valid audio processed before the current valid audio are used as the historical screen coordinates, that is, the coordinate set mainly stores the target screen coordinates of the valid audio frame before the current valid audio frame.

[0093] It should be noted that the solution provided in this embodiment can be processed with the first valid audio frame. Of course, in order to improve the processing accuracy and the transition effect of the picture, the analysis and processing of the determined valid audio frames can be started only when the number of valid audio frames reaches the first threshold, thereby starting to generate the trajectory.

[0094] Specifically, the pitch offset is first determined according to the pitch information and pitch control information of the valid audio frame in the audio information. The pitch information of the valid audio is then adjusted according to the pitch offset to obtain the pitch information to be used. Finally, the trajectory control point can be determined according to the target screen coordinates corresponding to the pitch information to be used in the video screen.

[0095] It should be noted that the processing method for each valid audio frame is the same, and the analysis and processing of a valid audio frame is mainly used as an example for explanation. That is to say, for each valid audio frame, the corresponding tone information to be used is dynamically determined.

[0096] For example, the first type is male type, the first tone is male lowest tone lowPitch, the second type is female type, the second tone is female highPitch. The tone control information can be the average of the first tone and the second tone, midPitch = (lowPitch + highPitch) / 2, where midPitch is the tone control information.

[0097] It should also be noted that the tone control information may also be determined according to the preset first type of tone weight and the first tone, the second type of tone weight and the second tone.

[0098] S220: Determine a first trajectory according to at least one trajectory control point and a preset Bessel function.

[0099] The preset Bezier function may be a function for generating a Bezier curve, and accordingly, the first trajectory corresponds to the generated Bezier curve.

[0100] Specifically, the target screen coordinates corresponding to at least one trajectory control point can be substituted into the Bezier function to obtain a Bezier function with determined coefficients. Next, the independent variable can be adjusted to generate a large number of trajectory points. All trajectory points connected together form a Bezier curve, which is the final first trajectory.

[0101] The technical solution provided by the embodiment of the present invention can determine the valid audio frame in the audio information, and determine at least one trajectory control point according to the pitch information of the valid audio frame. Then, the first trajectory is determined according to the trajectory control point and the preset Bessel function, thereby improving the effectiveness of determining the first trajectory.

[0102] Figure 3The flowchart for determining the trajectory control points provided in the embodiment of the present invention can be further refined on the basis of the aforementioned embodiment on how to determine the trajectory control points in the aforementioned embodiment. The specific implementation method can refer to the detailed description of the present technical solution, wherein the technical terms that are the same as or corresponding to the aforementioned embodiment are not repeated in this embodiment.

[0103] like Figure 3 As shown, the method includes:

[0104] S310: When the number of valid audio frames reaches a first threshold, determine initial tone information according to tone information of the valid audio frames.

[0105] The first threshold is a pre-set natural number. The first threshold is mainly used to determine how many valid audio frames are needed as a reference to determine the initial pitch information. The first threshold is also used to determine the number of valid audio frames after which the trajectory control point is determined based on the valid audio frames. Optionally, the number of the first threshold can be 10, that is, if the number of valid audio frames reaches 10, the initial pitch information can be determined based on these 10 valid audio frames.

[0106] Specifically, after determining the valid audio frame in the audio information, the first track may not be determined first, but may be used as a reference data to determine the initial tone information.

[0107] Exemplarily, the first threshold is 10, and the pitch information is a pitch value. After 10 valid audio frames are determined based on the audio information, the pitch values ​​of the 10 valid audio frames may be averaged to obtain initial pitch information (initial pitch value) initPitch.

[0108] S320: Determine a pitch offset according to the initial pitch information and the pitch control information.

[0109] The initial tone information includes an initial tone value, and the tone control information is a tone control value.

[0110] Specifically, by calculating the difference between the initial pitch value and the pitch control value, the pitch offset corresponding to each valid audio frame can be obtained. The advantage of determining the pitch offset is that it is convenient for subsequent pitch control processing of the first type and the second type to ensure the trajectory stability of the first trajectory relative to the video picture.

[0111] S330 . Starting from the valid audio frame after the first threshold, average value processing is performed on the tone information of the valid audio frame and a preset number of historical valid audio frames before the valid audio frame to determine the tone information to be adjusted of the valid audio frame.

[0112] It should be noted that, for each scenario, when the number of valid audio frames reaches a first threshold, the pitch offset is determined according to the pitch value of the valid audio frame corresponding to the first threshold. That is, the first track is generated starting from the valid audio frames determined after the first threshold. The processing method of the valid audio frames generated after the first threshold can be referred to the detailed description below.

[0113] It should also be noted that the valid audio frames generated after the first threshold are all determined in the following manner, which is described here by taking one of the valid audio frames as an example. The valid audio frame described here can be used as the current valid audio frame.

[0114] Among them, the preset number is a value pre-set according to actual needs, and the value is mainly used to constrain how many historical valid audio frames need to be combined to constrain the current valid audio frame. Optionally, the preset number is 5 or 10 historical valid audio frames. The valid audio frames determined before the current valid audio frame can be called historical valid audio frames to be selected. Each historical valid audio frame has a corresponding historical timestamp, and the interval duration can be determined based on the historical timestamp and the current timestamp of the current valid audio frame. According to the interval duration, a preset number of historical valid audio frames to be selected can be obtained as the target historical valid audio frames of the current audio frame. The tone information to be used is based on the tone information of the target historical valid audio frame.

[0115] Exemplarily, the first threshold is 10 valid audio frames, the preset value is 9 historical valid audio frames, and the current valid audio frame is the 11th valid audio frame. The pitch information of the 11th valid audio frame and the pitch information of the 2nd to 10th target historical valid audio frames are obtained. The pitch information to be adjusted avgPitch of the 11th valid audio frame is obtained by processing the average of the pitch information of the 2nd to 11th valid audio frames.

[0116] That is, in order to improve the smoothness of the first trajectory, the pitch information to be adjusted is obtained by processing the average of the pitch information of the current valid audio frame and the previous nine target historical valid audio frames.

[0117] S340: Determine the tone information to be used of the valid audio frame according to the tone information to be adjusted and the tone offset.

[0118] It can be understood that: for the valid audio frame after the first threshold, the to-be-used tone information of the valid audio frame can be obtained according to the to-be-adjusted tone information and the tone offset of the valid audio frame.

[0119] Exemplarily, after obtaining the to-be-adjusted pitch information avgPitch of the 11th valid audio frame, the sum of the to-be-adjusted pitch information avgPitch and the pitch offset pitchOffset may be calculated to obtain the to-be-used pitch information curPitch of the 11th valid audio frame.

[0120] S350: Determine, according to the first tone, the second tone, and the tone information to be used, mapping the tone information of the valid audio frame to the target screen coordinates in the video picture.

[0121] The target screen coordinates are coordinates obtained after mapping the valid audio frame to the video screen.

[0122] Exemplarily, the video screen includes a plurality of pixels, each of which corresponds to a screen coordinate. Usually, the screen vertical coordinate is 0 to 1 from top to bottom in the video screen, and the pitch range is mainly from the first pitch lowPitch to the second pitch highPitch. Based on this, the pitch information to be used is mapped to the target screen coordinate in the following manner: curHight = (curPitch-lowPitch) / (highPitch-lowPitch). curHight is the vertical coordinate of the target screen coordinate. The horizontal coordinate in the target screen coordinate is determined according to the preset trajectory moving speed.

[0123] S360: Use the newly added historical screen coordinates in the coordinate set as the screen coordinates to be used, and determine the difference between the vertical coordinates of the screen coordinates to be used and the vertical coordinates of the target screen coordinates.

[0124] The coordinate set includes a plurality of historical screen coordinates, and the historical screen coordinates that are most recently added to the coordinate set can be used as the screen coordinates to be used.

[0125] Specifically, after obtaining the target screen coordinates of the valid audio frame, the latest historical screen coordinates added in the coordinate set may be obtained, and the difference between the ordinates of the target screen coordinates and the historical screen coordinates may be calculated.

[0126] S370. Update the coordinate set according to the difference, so as to determine the at least one trajectory control point based on the updated coordinate set.

[0127] The number of at least one trajectory control point may be one or more, and the specific number is related to the order of the Bessel function.

[0128] Specifically, according to the difference, it can be determined how to process the target screen coordinates, so as to update the coordinate set based on the processed target screen coordinates. After the coordinate set is updated, at least one trajectory control point can be determined.

[0129] In this embodiment, the coordinate set is updated according to the difference to determine at least one trajectory control point based on the updated coordinate set, including: according to the preset conditions satisfied by the difference, calling the coordinate processing method corresponding to the preset conditions to process the target screen coordinates, and updating the processed target screen coordinates as historical screen coordinates in the coordinate set; when the number of historical screen coordinates in the coordinate set reaches a preset number threshold, adding the historical screen coordinates to the coordinate set The time information corresponding to the preset number threshold is obtained; and at least one trajectory control point is determined based on the multiple screen coordinates to be applied.

[0130] It can be understood that: according to the preset conditions satisfied by the difference, the coordinate processing method corresponding to the preset conditions can be called to process the target screen coordinates, and the processed target screen coordinates are updated as historical screen coordinates into the coordinate set.

[0131] In this embodiment, the preset conditions include at least three types, and correspondingly, the coordinate processing methods also include at least three types. Next, each preset condition and the coordinate processing method corresponding to the preset condition are described in detail.

[0132] The first type: the preset condition is that the difference is less than the first difference threshold, and the coordinate processing method is to keep the target screen coordinates unchanged and delete the screen coordinates to be used from the coordinate set.

[0133] It can be understood that if the coordinate difference is less than the first difference threshold, the target screen coordinates can be kept unchanged, and the screen coordinates to be used can be deleted from the coordinate set.

[0134] The second type: the preset condition is that the difference is greater than the second difference threshold, and the coordinate processing method is to correct the target screen coordinates according to the second difference threshold.

[0135] It can be understood that if the difference is greater than the second difference threshold, the second difference threshold can be superimposed on the vertical coordinate of the target screen coordinate to obtain the updated target screen coordinate.

[0136] The third type: the preset condition is that the difference is greater than the first difference threshold and less than the second difference threshold, and the coordinate processing method is to keep the target screen coordinates unchanged.

[0137] It can be understood that: if the difference is greater than the first difference threshold and less than the second difference threshold, the target screen coordinates can be kept unchanged.

[0138] After the coordinate set is updated, a plurality of screen coordinates to be applied that are consistent with a preset number threshold can be determined based on the number of historical screen coordinates in the coordinate set and the time information added to the coordinate set. At least one trajectory point can be determined by analyzing and processing the screen coordinates to be applied. Optionally, the difference between the time information when the historical screen coordinates are added to the coordinate set and the current time is calculated. According to the order of the difference from small to large, a preset number of historical screen coordinates are selected as the screen coordinates to be applied.

[0139] The technical solution provided by the embodiment of the present invention can process the tone information and tone offset of the valid audio frame to obtain the tone information to be used of the valid audio frame, and then determine the trajectory control point according to the target screen coordinates corresponding to the tone information to be used and the historical screen coordinates in the coordinate set, thereby improving the accuracy of determining the trajectory control point. Furthermore, when the trajectory control point has a high accuracy, when determining the first trajectory according to the trajectory control point, the smoothness of the first trajectory can be improved, thereby improving the picture display effect.

[0140] Figure 4 The flowchart of generating trajectory control points provided in the embodiment of the present invention can be further refined on the basis of the aforementioned embodiment: "determining the at least one trajectory control point based on the multiple screen coordinates to be applied" and "determining the first trajectory according to the at least one trajectory control point and the preset Bessel function". The specific implementation method can refer to the detailed description of the present technical solution, wherein the technical terms that are the same as or corresponding to the above-mentioned embodiments are not repeated in this embodiment.

[0141] like Figure 4 As shown, the method includes:

[0142] S410 , determining a slope value between two adjacent screen coordinates to be applied starting from the second screen coordinate to be applied according to the time sequence in which the multiple screen coordinates to be applied are added to the coordinate set.

[0143] It should be noted that the preset Bessel function is a second-order Bessel function, so the number of the multiple screen coordinates to be applied may be four, and accordingly, the number of the control points finally determined is three.

[0144] Among them, the time when each screen coordinate to be applied is added to the coordinate set is different. According to the time sequence of adding to the coordinate set, the slope value between two adjacent screen coordinates to be applied can be determined starting from the second screen coordinate to be applied. Exemplarily, the points corresponding to the screen coordinates to be applied are marked as P1, P2, P3, and P4. According to the order from early to late in adding to the coordinate set, P1 is earlier than P2, P2 is earlier than P3, and P3 is earlier than P4. Starting from P2, the slope K1 between P2 and P3 and the slope K2 between P3 and P4 can be calculated.

[0145] S420: Determine a trajectory control point from the plurality of screen coordinates to be applied according to the slope value.

[0146] Specifically, after obtaining the two slope values ​​obtained above, the relationship between the two slope values ​​may be analyzed and processed, so that the trajectory control point may be determined from a plurality of screen coordinates to be applied according to the analysis result.

[0147] In this embodiment, the method of determining the trajectory control point according to the slope value can be: if the difference between the slope values ​​is greater than the slope difference threshold, the to-be-applied screen coordinates other than the to-be-applied screen coordinates with the latest time sequence are used as the trajectory control point; if the difference between the slope values ​​is less than the slope difference threshold, the to-be-applied screen coordinates other than the second to-be-applied screen coordinates are used as the trajectory control point.

[0148] It can be understood that: if the difference between the slope values ​​is greater than the slope difference threshold, the screen coordinates to be applied of P1, P2, and P3 can be used as trajectory control points. If the difference between the slope values ​​is less than or equal to the slope difference threshold, P1, P2, and P4 can be used as trajectory control points. The above trajectory control points are mainly used for subsequent processing of the preset Bessel function to generate the first trajectory.

[0149] S430: Substitute the target screen coordinates of the trajectory control point into a preset Bessel function, obtain the trajectory points of the valid audio frame in the video picture by adjusting the independent variables of the preset Bessel function, and generate a first trajectory according to the trajectory points.

[0150] Among them, the preset Bessel function can be S t =(1-t) 2 S0+2t(1-t)S1+t 2 S2 t∈(0,1); S0 is the first trajectory control point in time sequence, S1 is the second trajectory control point in time sequence, S2 is the third trajectory control point in time sequence, and t is the independent variable.

[0151] Take the trajectory control points P1, P2, and P3 as an example, P1 corresponds to S0, P2 corresponds to S1, and P3 corresponds to S2. Substitute the target screen coordinates corresponding to the above trajectory control points into the preset Bessel function. Next, adjust the value of the independent variable t to obtain multiple trajectory points. Connect all the trajectory points to obtain the first trajectory.

[0152] That is to say, the independent variable of t≥0 and t<0.5 can be substituted into the above preset Bessel function to obtain trajectory points, and the trajectory points are displayed in the video screen, and the splicing of all trajectory points is the first trajectory.

[0153] It should also be noted that the coordinate information of the trajectory point obtained after substituting the t=0.5 independent variable into the preset Bessel function can be used to update the second screen coordinate to be used. Optionally, the screen coordinate to be used of the P2 trajectory control point is updated to the coordinate information obtained when t=0.5.

[0154] That is to say, the coordinate information of the track point corresponding to the target independent variable is used as the second screen coordinate to be applied and updated into the coordinate set.

[0155] In this embodiment, for valid audio frames after the first threshold, they can be processed in a manner to determine the track points. The valid audio frames can be continuously analyzed and processed to update the length of the first track.

[0156] The technical solution provided by the embodiment of the present invention can further combine the determined trajectory control points and the preset Bessel function to determine the first trajectory, improve the smoothness of the first trajectory, and thus improve the picture display effect.

[0157] Figure 5 A flow chart of a video generation method provided in an embodiment of the present disclosure, based on the aforementioned embodiment, can further refine: "Determine a first special effect track associated with the audio information according to the first track and a pre-set first material image", and its specific implementation method can refer to the detailed description of the present technical solution, wherein the technical terms that are the same as or corresponding to the aforementioned embodiment are not repeated in this embodiment.

[0158] like Figure 5 As shown, the method includes:

[0159] S510: During the video display process, determine a first trajectory based on the collected audio information.

[0160] S520: Update display information of each track point in the first track according to coordinate information of each track point in the first track and pixel information of each pixel point in the first material image to obtain a first special effect track.

[0161] The first material image is a material image set according to a preset effect. Optionally, the first material image can be Figure 6 ,. A color / transparency gradient base map can be created. For example, a first color is set at one end of the first material image and a second color is set at the other end. The color information of each pixel in the first material image is obtained by interpolating the first color and the second color. Furthermore, transparency information can be set. By adjusting the transparency of each pixel in the first material image, an updated first material image is obtained.

[0162] After obtaining the first trajectory, the first material image can be traversed based on the horizontal coordinates of the trajectory points in the first trajectory to determine the pixel points corresponding to the trajectory points in the first material image, and the display information of the trajectory points can be updated according to the pixel values ​​of the pixel points, and the first trajectory after the display information is updated is used as the first special effect trajectory. Figure 6 A schematic diagram of the first special effect track after processing the first track can be seen in Figure 7 .

[0163] It should be noted that Figure 6 The color and transparency shown are merely exemplary. The image content of the first material image may be set according to actual needs during the design phase, and the specific content is not limited in this embodiment.

[0164] The rendering map is queried based on the position information of each track point in the first track, so as to update the display information of the track point according to the transparency and the actual color in the rendering map. The size of the first material image is consistent with the size of the video screen, and the display information of any two pixels of the first material image is different.

[0165] In this embodiment, in order to further improve the display effect of the first special effects trajectory, the obtained first special effects trajectory can be further processed. Optionally, the method also includes: Gaussian blur processing on the first special effects trajectory to obtain the special effects trajectory to be mixed; and fusing the special effects trajectory line to be mixed with the first special effects trajectory before Gaussian blur processing to obtain the first special effects trajectory after the display information is updated.

[0166] It can be understood that: the first special effect track can be Gaussian blurred to obtain the special effect track to be mixed. Next, the special effect track to be mixed and the first special effect track before Gaussian blurred are mixed to obtain the first special effect track after the display information is updated.

[0167] For example, when obtaining Figure 7 After the first special effect track is shown. The Gaussian blur algorithm is used to blur the first special effect track, and the following is obtained: Figure 8The effect schematic diagram shown is as follows: Pixel point superposition processing is performed on the first special effect track before the mixed special effect track and the Gaussian blur processing to update the display information of each pixel in the first special effect track.

[0168] S530: Determine a target mask map according to the first trajectory.

[0169] The shape of the track in the target mask image is related to the shape of the first track, and there is an image after dilation processing.

[0170] In this embodiment, a method for determining a target mask map may be: obtaining an initial trajectory corresponding to the first trajectory; offsetting the pixel points in the initial trajectory by a preset number of pixels along a first direction to obtain a trajectory to be disturbed; perturbing the trajectory to be disturbed according to a preset perturbation pixel range to obtain the target perturbed trajectory; and using an area formed by the initial trajectory and the target perturbed trajectory as a mask area corresponding to the first trajectory in the target mask map.

[0171] Can be combined Fig. 9 To understand how to determine the target mask area, the trajectory corresponding to the first trajectory can be copied as the initial trajectory. The initial trajectory is moved along the first direction by a preset number of pixels, optionally, the preset number of pixels is 60px, and the pixel points on the initial trajectory are moved 60px along the first direction perpendicular to the horizontal plane to obtain Fig. 9 The effect diagram shown in (a) is shown in FIG. 1 , and the trajectory obtained at this time is used as the trajectory to be disturbed. The disturbance pixel range can be a pixel fluctuation range set according to actual needs. Optionally, the pixel fluctuation range can be a random fluctuation of 0px-25px along the first direction or the second direction. Each pixel point in the trajectory to be disturbed can be randomly disturbed to obtain the following Fig. 9 The red curve in (b) is a schematic diagram of the effect. The trajectory corresponding to the red curve is used as the target perturbation trajectory after perturbation. The area formed by the initial trajectory and the target perturbation trajectory is used as the mask area corresponding to the first trajectory in the target mask map, see Fig. 9 (c) in .

[0172] On the basis of the above technical solution, the second special effect track may be generated based on the obtained target mask map, and the target mask map may be further processed to improve the display effect of generating the second special effect track.

[0173] Optional, yes Fig. 9 (c) performs Gaussian blur processing and obtains Fig.10 The effect diagram shown in FIG. 1 can be understood as that the target mask image is Gaussian blurred to obtain an updated target mask image.

[0174] S540: Determine a second special effect track of the first track based on the target mask image and the second material image.

[0175] The second material image is also a material image produced according to a preset effect. Optionally, the second material may be a material image in a simulated aurora scene, such as Fig.11 The material image shown may be a second material image.

[0176] It should be noted that the first material image and the second material image can be changed according to actual needs. In this embodiment, the material images shown in the accompanying drawings are only used for illustrative purposes and do not limit this solution. The core of the technical solution provided by the embodiment of the present disclosure is that this technology can be used to generate special effect tracks.

[0177] Specifically, by processing the target mask image and the second material image, a second special effect track corresponding to the first track can be obtained.

[0178] In this embodiment, the method of generating the second special effect trajectory according to the target mask map and the second material image can be: according to the pixel information of at least one pixel point in the target mask map in the second material image, the pixel information of at least one pixel point in the target mask map is updated to obtain the second special effect trajectory.

[0179] Specifically, the pixel to be used corresponding to each pixel in the target mask image in the second material image can be determined, and the pixel information of the pixel to be used can be updated to the pixel of the target mask image. After processing each pixel in the target mask image, the second special effect track can be obtained, see Fig.12 .

[0180] S550: Based on the first special effect trajectory and the second special effect trajectory, determine and display a target trajectory to be displayed in the video picture.

[0181] In this embodiment, the end points of the first special effect track and the second special effect track may be kept consistent and merged to obtain the target track displayed in the video screen.

[0182] Based on the above technical solution, if the end point of the target trajectory is a preset screen coordinate, the trajectory length of the target trajectory is no longer updated; or, if no audio information is collected within a preset time length, the trajectory length of the target trajectory is no longer updated.

[0183] It can be understood that if the terminal of the target trajectory is a preset screen coordinate, optionally, the preset screen coordinate can be the screen coordinate corresponding to the ground on the other side of the hill, or the preset screen coordinate corresponding to the ground at the other end of the gully, then the estimated trajectory length of the target may no longer be updated, that is, the collected audio may no longer be processed. It is also possible to continue to analyze and process the audio information to generate the target trajectory. If within the preset time length, optionally, no audio information is collected in 20S, the trajectory length of the target trajectory may no longer be updated. Of course, if audio information is collected in 20S, the trajectory length of the target trajectory may continue to be updated. Based on the above technical solution, the target trajectory may also include a sub-line. Multiple copies of the first special effect trajectory can be copied to obtain the trajectory to be fused, and the thickness value of the trajectory to be fused can be adjusted to obtain the following. Fig.13 The trajectories to be fused are shown.

[0184] By combining the fusion track, the first special effect track and the second special effect track and displaying them in the video screen, the obtained video screen can be seen Fig.14 .

[0185] The technical solution provided by the embodiment of the present invention can obtain a first special effect track by processing a first special effect with a first material image, and obtain a second special effect track by processing a mask image corresponding to the first track with a second material image. Based on the first special effect track and the second special effect track, the target track displayed in the video screen can be determined, so that not only the target track corresponding to the audio information is displayed, but the target track is obtained after multiple special effect processing, which improves the richness of the video screen content and the interactivity between the video screen and the user.

[0186] Fig.15 A schematic diagram of the structure of a video generating device provided by an embodiment of the present disclosure is shown in FIG. Fig.15 As shown, the device includes: a first trajectory determination module 610 , a first special effect trajectory generation module 620 , a second special effect trajectory generation module 630 and a target trajectory generation module 640 .

[0187] The first trajectory determination module 610 is used to determine the first trajectory based on the collected audio information during the display of the video screen; the first special effect trajectory generation module 620 is used to determine the first special effect trajectory associated with the audio information according to the first trajectory and a preset first material image; wherein the display information corresponding to the first special effect trajectory is related to the pixel information of the first material image; the second special effect trajectory generation module 630 is used to determine the second special effect trajectory of the first trajectory based on the target mask map and the second material image associated with the first trajectory; the target trajectory generation module 640 is used to determine and display the target trajectory displayed in the video screen based on the first special effect trajectory and the second special effect trajectory; wherein the change information of the target trajectory is associated with the audio information.

[0188] According to the technical solution of the disclosed embodiment, during the display of the video screen, the collected audio information can be analyzed and processed to determine the first trajectory. Furthermore, the first material image can be used to perform special effects processing on the first trajectory to obtain the first special effects trajectory, which enriches the picture effect of the first trajectory. In order to further improve the richness and enjoyability of the video screen content, the mask image corresponding to the first trajectory can be rendered based on the second material image to obtain the second special effects trajectory. Based on the first special effects trajectory and the second special effects trajectory, the target trajectory displayed in the video screen can be obtained, which solves the problem that the prior art only converts audio information into text, has a narrow scope of use of audio information and low interactivity with terminal devices, and realizes the generation of the corresponding first trajectory based on audio information, and the first trajectory is processed by special effects from multiple dimensions and then integrated and displayed, thereby improving the richness of the video screen content and the interactivity with users.

[0189] On the basis of the above technical solution, the device comprises: an audio acquisition module, configured to acquire audio information of the first object when it is detected that the trajectory generation condition is met;

[0190] The trajectory generation condition includes at least one of the following:

[0191] Determining, based on a preset map, that a second object moves to a preset position, wherein the second object is a movable object displayed in the video screen;

[0192] Trigger the video screen or a trajectory generation control in the application software to which the video screen belongs.

[0193] On the basis of the above technical solutions, the first trajectory determination module includes:

[0194] a trajectory control point determination unit, configured to determine at least one trajectory control point according to pitch information of a valid audio frame in the audio information, wherein the valid audio frame is an audio frame whose volume satisfies a first volume condition and whose pitch satisfies a first pitch condition, and the trajectory control point is used to adjust a trajectory shape of the first trajectory;

[0195] The first trajectory determining unit is configured to determine the first trajectory according to the at least one trajectory control point and a preset Bessel function.

[0196] On the basis of the above technical solutions, the trajectory control point determination unit includes:

[0197] a tone offset determination subunit, configured to determine a tone offset according to tone information and tone control information of a valid audio frame in the audio information; wherein the tone control information is determined according to a first tone of a first type and a second tone of a second type, and the tone control information is used to control the tone information to fluctuate within a preset range;

[0198] a tone information to be used determining subunit, configured to adjust the tone information of the valid audio frame according to the tone offset to obtain the tone information to be used;

[0199] The trajectory control point determination subunit is used to determine the target screen coordinates of the to-be-used tone information in the video picture, and determine the trajectory control point according to the target screen coordinates and the historical screen coordinates in the coordinate set.

[0200] On the basis of the above technical solutions, the tone offset determination subunit includes:

[0201] an initial tone determination subunit, configured to determine initial tone information according to tone information of the valid audio frames when the number of the valid audio frames reaches a first threshold;

[0202] an offset determination subunit, configured to determine a tone offset according to the initial tone information and the tone control information;

[0203] The tone information to be used determining subunit includes:

[0204] A first information determination unit is used to determine, for a valid audio frame determined after the first threshold, the to-be-adjusted tone information of the valid audio frame according to the valid audio frame and a target historical valid audio frame; wherein the target historical valid audio frame is determined according to the interval duration between all to-be-selected historical valid audio frames of the audio information and the valid audio frame and a preset number of the historical valid video frames;

[0205] The second tone determination unit is configured to determine the tone information to be used of the valid audio frame according to the tone information to be adjusted and the tone offset.

[0206] On the basis of the above technical solutions, the trajectory control point determination subunit includes:

[0207] A first coordinate determination subunit, configured to determine, according to the first tone, the second tone and the tone information to be used, the target screen coordinates of mapping the tone information of the valid audio frame to the video picture;

[0208] a difference determination subunit, configured to use the newly added historical screen coordinates in the coordinate set as the screen coordinates to be used, and determine the difference between the vertical coordinates of the screen coordinates to be used and the vertical coordinates of the target screen coordinates; wherein the screen coordinates to be used are the coordinates in the historical screen coordinates;

[0209] The trajectory point determination subunit is used to update the coordinate set according to the difference, so as to determine the at least one trajectory control point based on the updated coordinate set.

[0210] On the basis of the above technical solutions, the trajectory point determination subunit includes:

[0211] a coordinate set updating subunit, for invoking a coordinate processing method corresponding to a preset condition to process the target screen coordinates according to the preset condition satisfied by the difference, and updating the processed target screen coordinates as historical screen coordinates into the coordinate set;

[0212] a screen coordinate determination subunit to be applied, configured to, when the number of historical screen coordinates in the coordinate set reaches a preset number threshold, obtain a plurality of screen coordinates to be applied corresponding to the preset number threshold according to the time information of the historical screen coordinates added to the coordinate set;

[0213] The trajectory point determination subunit is used to determine the at least one trajectory control point based on the multiple screen coordinates to be applied.

[0214] On the basis of the above technical solutions, the coordinate processing method corresponding to the preset condition includes at least one of the following:

[0215] The preset condition is that the difference is less than a first difference threshold, and the coordinate processing method is to keep the target screen coordinates unchanged and delete the screen coordinates to be used from the coordinate set=;

[0216] The preset condition is that the difference is greater than a second difference threshold, and the coordinate processing method is to correct the target screen coordinates according to the second difference threshold;

[0217] The preset condition is that the difference is greater than a first difference threshold and less than a second difference threshold, and the coordinate processing method is to keep the target screen coordinates unchanged.

[0218] On the basis of the above technical solutions, the trajectory point determination subunit is used to

[0219] Determine the slope value between two adjacent screen coordinates to be applied starting from the second screen coordinate to be applied according to the time sequence in which the multiple screen coordinates to be applied are added to the coordinate set;

[0220] According to the slope value, a trajectory control point is determined from the multiple screen coordinates to be applied.

[0221] On the basis of the above technical solutions, the trajectory point determination subunit is used to

[0222] If the difference between the slope values ​​is greater than the slope difference threshold, taking the screen coordinates to be applied other than the screen coordinates to be applied which are the latest in time sequence as the track control points;

[0223] If the difference between the slope values ​​is less than the slope difference threshold, the screen coordinates to be applied other than the second screen coordinates to be applied are used as the trajectory control points.

[0224] On the basis of the above technical solutions, the first trajectory determination unit is also used to

[0225] Substituting the target screen coordinates of the trajectory control point into the preset Bessel function, and obtaining the trajectory point of the valid audio frame in the video picture by adjusting the independent variable of the preset Bessel function; and determining the first trajectory based on the trajectory point.

[0226] Based on the above technical solutions, the first special effect trajectory generation module is also used to

[0227] According to the coordinate information of each trajectory point in the first trajectory and the pixel information of each pixel point in the first material image, the display information of each trajectory point in the first trajectory is updated to obtain the first special effect trajectory; wherein the size of the first material image is consistent with the size of the video screen, and there is a difference in the display information of any two pixels of the first material image.

[0228] Based on the above technical solutions, the device also includes: a special effect update module, which is used to perform Gaussian blur processing on the first special effect trajectory to obtain a special effect trajectory to be mixed; and obtain the first special effect trajectory after the display information is updated by fusing the special effect trajectory line to be mixed with the first special effect trajectory before Gaussian blur processing.

[0229] On the basis of the above technical solutions, the device further includes:

[0230] An initial trajectory corresponding to the first trajectory is obtained; pixels in the initial trajectory are offset by a preset number of pixels along a first direction to obtain a trajectory to be disturbed; the trajectory to be disturbed is disturbed according to a preset range of disturbed pixels to obtain the target disturbed trajectory; and an area formed by the initial trajectory and the target disturbed trajectory is used as a mask area in a target mask map corresponding to the first trajectory.

[0231] Based on the above technical solutions, the second special effect trajectory generation module is also used to update the pixel information of at least one pixel point in the target mask image according to the pixel information of at least one pixel point in the target mask image in the second material image to obtain the second special effect trajectory.

[0232] The video generating device provided in the embodiments of the present disclosure can execute the video generating method provided in any embodiment of the present disclosure, and has the corresponding functional modules and beneficial effects of the execution method.

[0233] It is worth noting that the various units and modules included in the above-mentioned device are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the protection scope of the embodiments of the present disclosure.

[0234] Fig.16 This is a schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure. Fig.16 , which shows an electronic device (eg, Fig.16 The terminal device in the embodiment of the present disclosure may include but is not limited to mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Fig.16 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0235] like Fig.16As shown, the electronic device 700 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage device 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the electronic device 700 are also stored. The processing device 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An edit / output (I / O) interface 705 is also connected to the bus 704.

[0236] Typically, the following devices may be connected to the I / O interface 705: an input device 706 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 707 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 708 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 709. The communication device 709 may allow the electronic device 700 to communicate with other devices wirelessly or by wire to exchange data. Although Fig.16 The electronic device 700 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0237] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 709, or installed from a storage device 708, or installed from a ROM 702. When the computer program is executed by the processing device 701, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0238] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0239] The electronic device provided in the embodiment of the present disclosure and the video generation method provided in the above embodiment belong to the same inventive concept. The technical details not fully described in the embodiment of the present disclosure can be referred to the above embodiment, and the present embodiment has the same beneficial effects as the above embodiment.

[0240] The embodiment of the present disclosure provides a computer storage medium on which a computer program is stored. When the program is executed by a processor, the video generation method provided by the above embodiment is implemented.

[0241] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0242] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0243] The computer-readable medium may be included in the electronic device, or may exist independently without being installed in the electronic device.

[0244] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device:

[0245] During the video display, a first trajectory is determined based on the collected audio information;

[0246] Determining a first special effect track associated with the audio information according to the first track and a preset first material image; wherein display information corresponding to the first special effect track is related to pixel information of the first material image;

[0247] Determining a second special effect track of the first track based on a target mask image and a second material image associated with the first track;

[0248] Based on the first special effect track and the second special effect track, a target track to be displayed in the video screen is determined and displayed; wherein change information of the target track is associated with the audio information.

[0249] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0250] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0251] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware. The name of a unit does not limit the unit itself in some cases. For example, the first acquisition unit may also be described as a "unit for acquiring at least two Internet Protocol addresses".

[0252] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0253] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0254] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0255] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0256] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. A video generation method, characterized in that: include: During the video display, a first trajectory is determined based on the collected audio information; Determining a first special effect track associated with the audio information according to the first track and a preset first material image; wherein display information corresponding to the first special effect track is related to pixel information of the first material image; Determining a second special effect track of the first track based on a target mask image and a second material image associated with the first track; Based on the first special effect track and the second special effect track, a target track to be displayed in the video screen is determined and displayed; wherein change information of the target track is associated with the audio information.

2. The method according to claim 1, characterized in that During the video display process, the method further includes: When it is detected that the trajectory generation condition is met, collecting audio information of the first object; The trajectory generation condition includes at least one of the following: Determining, based on a preset map, that a second object moves to a preset position, wherein the second object is a movable object displayed in the video screen; Trigger the video screen or a trajectory generation control in the application software to which the video screen belongs.

3. The method according to claim 1, characterized in that: The step of determining a first track based on the collected audio information includes: Determining at least one trajectory control point according to the pitch information of a valid audio frame in the audio information, wherein the valid audio frame is an audio frame whose volume satisfies a first volume condition and whose pitch satisfies a first pitch condition, and the trajectory control point is used to adjust the trajectory shape of the first trajectory; The first trajectory is determined according to the at least one trajectory control point and a preset Bessel function.

4. The method according to claim 3, characterized in that The step of determining at least one trajectory control point according to the pitch information of the valid audio frame in the audio information comprises: Determine a pitch offset according to pitch information and pitch control information of a valid audio frame in the audio information; wherein the pitch control information is determined according to a first pitch of a first type and a second pitch of a second type, and the pitch control information is used to control the pitch information to fluctuate within a preset range; Adjusting the tone information of the valid audio frame according to the tone offset to obtain the tone information to be used; The target screen coordinates of the to-be-used tone information in the video picture are determined, and a trajectory control point is determined according to the target screen coordinates and the historical screen coordinates in the coordinate set.

5. The method according to claim 4, characterized in that The step of determining the pitch offset according to the pitch information and the pitch control information of the valid audio frame in the audio information comprises: When the number of the valid audio frames reaches a first threshold, determining initial tone information according to the tone information of the valid audio frames; Determining a pitch offset according to the initial pitch information and the pitch control information; The step of adjusting the tone information of the valid audio frame according to the tone offset to obtain the tone information to be used includes: For a valid audio frame determined after the first threshold, determining the to-be-adjusted tone information of the valid audio frame according to the valid audio frame and a target historical valid audio frame; wherein the target historical valid audio frame is determined according to the interval duration between all to-be-selected historical valid audio frames of the audio information and the valid audio frame and a preset number of the historical valid video frames; The to-be-used tone information of the valid audio frame is determined according to the to-be-adjusted tone information and the tone offset.

6. The method according to claim 4, characterized in that The step of determining the target screen coordinates of the to-be-used tone information in the video picture, and determining the trajectory control point according to the target screen coordinates and the historical screen coordinates in the coordinate set, includes: Determine, according to the first tone, the second tone and the tone information to be used, mapping the tone information of the valid audio frame to the target screen coordinates in the video picture; Using the newly added historical screen coordinates in the coordinate set as the screen coordinates to be used, and determining the difference between the vertical coordinates of the screen coordinates to be used and the vertical coordinates of the target screen coordinates; wherein the screen coordinates to be used are the coordinates in the historical screen coordinates; The coordinate set is updated according to the difference, so as to determine the at least one trajectory control point based on the updated coordinate set.

7. The method according to claim 6, characterized in that The updating of the coordinate set according to the difference to determine the at least one trajectory control point based on the updated coordinate set includes: According to the preset condition satisfied by the difference, calling the coordinate processing method corresponding to the preset condition to process the target screen coordinates, and updating the processed target screen coordinates as the historical screen coordinates into the coordinate set; When the number of historical screen coordinates in the coordinate set reaches a preset number threshold, adding time information in the coordinate set according to the historical screen coordinates to obtain a plurality of screen coordinates to be applied corresponding to the preset number threshold; Based on the multiple screen coordinates to be applied, the at least one trajectory control point is determined.

8. The method according to claim 7, characterized in that The coordinate processing method corresponding to the preset condition includes at least one of the following: The preset condition is that the difference is less than a first difference threshold, and the coordinate processing method is to keep the target screen coordinates unchanged and delete the screen coordinates to be used from the coordinate set; The preset condition is that the difference is greater than a second difference threshold, and the coordinate processing method is to correct the target screen coordinates according to the second difference threshold; The preset condition is that the difference is greater than a first difference threshold and less than a second difference threshold, and the coordinate processing method is to keep the target screen coordinates unchanged.

9. The method according to claim 7, characterized in that: The determining the at least one trajectory control point based on the multiple screen coordinates to be applied includes: Determine the slope value between two adjacent screen coordinates to be applied starting from the second screen coordinate to be applied according to the time sequence in which the multiple screen coordinates to be applied are added to the coordinate set; According to the slope value, a trajectory control point is determined from the multiple screen coordinates to be applied.

10. The method according to claim 9, characterized in that The step of determining a trajectory control point from the plurality of screen coordinates to be applied according to the slope value comprises: If the difference between the slope values ​​is greater than the slope difference threshold, taking the screen coordinates to be applied other than the screen coordinates to be applied which are the latest in time sequence as the track control points; If the difference between the slope values ​​is less than the slope difference threshold, the screen coordinates to be applied other than the second screen coordinates to be applied are used as the trajectory control points.

11. The method according to claim 3, characterized in that The determining the first trajectory according to the at least one trajectory control point and a preset Bessel function comprises: Substituting the target screen coordinates of the trajectory control point into the preset Bessel function, and obtaining the trajectory point of the valid audio frame in the video picture by adjusting the independent variable of the preset Bessel function; Based on the trajectory points, the first trajectory is determined.

12. The method according to claim 1, characterized in that The step of determining a first special effect track associated with the audio information according to the first track and a preset first material image includes: According to the coordinate information of each track point in the first track and the pixel information of each pixel point in the first material image, the display information of each track point in the first track is updated to obtain the first special effect track; The size of the first material image is consistent with the size of the video screen, and display information of any two pixels of the first material image is different.

13. The method according to claim 1, characterized in that After obtaining the first special effect trajectory, the method further includes: Performing Gaussian blur processing on the first special effect trajectory to obtain a special effect trajectory to be mixed; The first special effect trajectory after display information update is obtained by fusing the special effect trajectory line to be mixed with the first special effect trajectory before Gaussian blur processing.

14. The method according to claim 1, characterized in that Also includes: Acquire an initial trajectory corresponding to the first trajectory; Shifting the pixel points in the initial trajectory by a preset number of pixels along a first direction to obtain a trajectory to be disturbed; The to-be-disturbed trajectory is disturbed according to a preset disturbance pixel range to obtain the target disturbance trajectory; The area formed by the initial trajectory and the target perturbation trajectory is used as the mask area corresponding to the first trajectory in the target mask map.

15. A video generation method, characterized in that: include: A first trajectory determination module, used to determine a first trajectory based on the collected audio information during the video screen display process; A first special effect track generating module, configured to determine a first special effect track associated with the audio information according to the first track and a preset first material image; wherein display information corresponding to the first special effect track is related to pixel information of the first material image; A second special effect trajectory generating module, configured to determine a second special effect trajectory of the first trajectory based on a target mask image and a second material image associated with the first trajectory; A target trajectory generation module is used to determine and display the target trajectory displayed in the video picture based on the first special effect trajectory and the second special effect trajectory; wherein the change information of the target trajectory is associated with the audio information.

16. An electronic device, characterized in that: The electronic device comprises: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the video generating method as described in any one of claims 1-14.

17. A storage medium containing computer executable instructions, characterized in that: When the computer executable instructions are executed by a computer processor, they are used to perform the video generation method according to any one of claims 1 to 14.

18. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the computer program implements the video generation method according to any one of claims 1 to 14.

Images