Video generation method, electronic device and medium thereof
By using image foreground segmentation and depth of field estimation algorithms in electronic devices, combined with audio interval division, matching the theme and range of motion of dynamic poster images, the problem of insufficient correlation between dynamic poster images and background music is solved, and the user experience is improved.
Patent Information
- Application Number
- CN202110841263.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-07-23
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2041-07-23
AI Technical Summary
In the prior art, the background music of the dynamic poster image has no connection with the theme of the poster image, which causes the user to fail to generate audio-visual linkage when viewing, reducing the user experience.
By using image foreground segmentation algorithm and depth of field estimation algorithm in electronic devices, we determine the range of motion of the foreground object in the poster image, and detect the rhythm changes in background music, divide the audio intervals, and follow the theme and range of motion of the audio intervals with the poster image. Match the theme and range of motion to generate dynamic poster clip videos to fit the background music in theme and motion.
It improves the user's audio-visual experience. Through audio-visual linkage, dynamic poster images match background music in the theme content and range of motion, improving user immersion and satisfaction.
Smart Images

Figure CN115695899B_ABST
Abstract
Description
Technical Field
[0001] The present application belongs to the field of electronic technology, and in particular, relates to a video generation method, an electronic device and a medium thereof. Background Art
[0002] At present, major video application platforms, TV series, film distributors or advertisers have launched special effects poster images such as dynamic posters and naked-eye 3D (dimension) in order to attract users. Compared with static poster images of a single form, dynamic poster images, naked-eye 3D and other poster images can visually attract users and bring users a good sense of immersion.
[0003] However, when playing dynamic poster images, the background music of the dynamic poster images is not linked to the theme of the dynamic poster images, and users cannot have an audio-visual connection. For example, some dynamic poster images with relatively mild themes have a relatively intense background music rhythm, or some dynamic poster images have a relatively large foreground object movement range, but the background music rhythm is relatively gentle. This audio-visual contrast will cause user discomfort and reduce user experience. Summary of the invention
[0004] In view of this, an embodiment of the present application provides a method for generating a video. In the method for generating a video of the present application, the electronic device will add a poster image theme tag to each poster image according to the theme content of each poster image, and use the image foreground segmentation algorithm and the depth of field estimation algorithm to determine the motion range of the foreground object in each poster image; at the same time, the time node when the rhythm in the background music changes is detected, and the theme corresponding to the audio interval is determined according to the rate value of the audio change during this period of time, and an audio theme tag is added to the audio interval according to the theme of the audio interval, and then the motion range of the audio interval is determined according to the duration of the audio interval and the above-mentioned audio change rate value; thereafter, the audio interval is associated with a dynamic poster with a similar theme tag and a similar motion range as the audio interval, and is played in the playback order of the audio interval to form a dynamic poster image card point video. The aforementioned method is used to make the dynamic poster image and the background music match in theme content. In addition, the method also associates the motion range of the foreground object of the dynamic poster image with the motion range of the audio interval, so that when the foreground object of the dynamic poster image ends its motion, the audio interval also ends playing, so that when the user watches the video at the dynamic poster image point, an audio-visual linkage can be generated, thereby improving the user experience.
[0005] The following describes a method for generating the video of this application.
[0006] In a first aspect, an embodiment of the present application provides a method for generating a video, which can be applied to an electronic device, the method comprising: dividing the background music used to generate the video into N continuous audio intervals, where N is a positive integer; matching the N images to the corresponding audio intervals according to the audio rhythm changes of each audio interval and the matching degree between the N images to generate a video, wherein during the playback of the video, the image matching each audio interval is displayed when each audio interval is played.
[0007] It can be understood that in order to realize the card point video of the above-mentioned dynamic poster image, so that each dynamic poster image and the corresponding audio interval are not only adapted in terms of subject matter, but also the motion range of the foreground object of the dynamic poster image is adapted to the motion range of the corresponding audio interval, the electronic device needs to first divide the background music into multiple continuous audio intervals according to the audio rhythm changes of the background music, and then match the image to the corresponding audio interval according to the matching degree between the rhythm changes of each audio interval and the image, so as to generate the card point video of the dynamic poster image. Moreover, it is not difficult to understand that during the playback of the card point video of the dynamic poster image, whenever a certain audio interval is played, an image matching the rhythm changes of the audio interval will be displayed.
[0008] Since the rhythm change of the audio interval can not only reflect the content of the audio interval, but also reflect the motion range of the audio interval to a certain extent, in some embodiments, the matching degree between the rhythm change of each audio interval and the image includes the first matching degree between the rhythm change of each audio interval and the image content and / or the second matching degree between the rhythm change of each audio interval and the dynamic motion range of the foreground object in the image. The first matching degree can be obtained by calculating the matching degree between the rhythm tag reflecting the rhythm change of the audio interval and the content tag reflecting the image content, and the second matching degree can be obtained by calculating the matching degree between the motion range reflecting the rhythm change of the audio interval and the playback duration and the motion range of the foreground object in the image.
[0009] Specifically, in some embodiments, the first matching degree can be calculated in the following manner: using a content recognition neural network model to perform content recognition on an image, adding a content tag reflecting the image content to the image based on the recognized content, and adding a rhythm tag reflecting the audio rhythm change to the audio interval based on the audio rhythm change of the audio interval; calculating the matching degree between the content tag and the rhythm tag to obtain the first matching degree. In some embodiments, the second matching degree can be calculated in the following manner: calculating the motion range of the audio interval based on the audio rhythm change and the duration of the audio interval; calculating the matching degree between the motion range of the audio interval and the dynamic motion range of the foreground object in the image to obtain the second matching degree. Among them, the rhythm tag reflecting the rhythm change of the audio interval corresponds to the theme tag of the audio interval in the specific embodiment section below, and the content tag reflecting the image content corresponds to the theme tag of the image in the specific embodiment section below.
[0010] In combination with the first aspect and the above-mentioned possible implementation, in another possible implementation of the first aspect, the dynamic range of the foreground object of the image can be calculated in the following manner: the image is segmented by using a foreground segmentation neural network model to obtain the foreground object and the background object of the image, and the dynamic motion range of the foreground object in the image is obtained according to the positional relationship of the foreground object relative to the background object. It can be understood that the positional relationship of the foreground object relative to the background object refers to the distance of the outermost side contour from the edge of the image in each direction of the foreground object, and the distance of the outermost side contour from the edge of the image in each direction of the foreground object is the dynamic motion range of the foreground object. For example, the leftmost side of the foreground object of the image is 2 pixels away from the left edge of the image, the rightmost side of the foreground object is 3 pixels away from the right edge of the image, the uppermost side of the foreground object is 2 pixels away from the upper edge of the image, and the lowermost side of the foreground object is 1 pixel away from the lower edge of the image. At this time, the dynamic motion range of the foreground object is: the leftward motion range is 2 pixels, the rightward motion range is 3 pixels, the upward motion range is 2 pixels, and the downward motion range is 1 pixel. Specifically, how to segment the foreground of an image and determine the motion range of the foreground object of the image will be introduced in the specific embodiment section below and will not be described in detail here.
[0011] In combination with the above, in some embodiments, the methods of matching a suitable image for each audio interval according to the rhythm changes of each audio interval may include the following: 1) First, N images are respectively assigned to N consecutive audio intervals in a first order, and then it is determined whether the first matching degree and the second matching degree between each audio interval and the image meet the requirements. If the first matching degree and the second matching degree between each audio interval and the image meet the requirements, it is determined whether the sum of the first matching degree and the second matching degree of all audio intervals and the images meets the requirements. If the sum of the first matching degree and the second matching degree of all audio intervals and the images meets the requirements, N images are matched for the N consecutive audio intervals in the first order to generate a video.
[0012] Specifically, N images are respectively assigned to N consecutive audio intervals in a first order, and the above method is used to calculate multiple first matching degrees and multiple second matching degrees between each audio interval and the assigned image; when multiple first matching degrees are all greater than the first matching degree threshold, and multiple second matching degrees are all greater than the second matching degree threshold, the sum of multiple first matching degrees and multiple second matching degrees is calculated; and when the sum of multiple first matching degrees and multiple second matching degrees is greater than the total matching degree threshold, the N images are matched to the corresponding audio intervals in the first order. The first matching degree threshold can be set according to the overall style of the N images and the overall style of the background music. For example, if the overall style of the background music is "passionate", the overall style of the image is also "passionate", indicating that the image and the background music are easy to match, so the first matching degree threshold can be set higher. If the overall style of the background music is "urgent" and the overall style of the image is "passionate", it means that the image and the background music are not easy to match, so the first matching degree threshold can be set lower. The setting method of the second matching degree threshold is similar to the setting method of the first matching degree threshold in principle, and will not be repeated here. However, it should be understood that neither the first matching threshold nor the second matching threshold can be too low. If the first matching threshold and the second matching threshold need to be set before N images can be matched to N audio intervals, it means that the background music and the N images are not suitable for generating a video, and the background music can be replaced at this time.
[0013] Furthermore, in order to avoid the situation where the first matching degree and the second matching degree of a certain audio interval with the assigned image meet the conditions, but the sum of the first matching degree and the second matching degree of the audio interval with the assigned image does not meet the conditions, for example, taking the first matching degree and the second matching degree between audio interval 1 and image 1 as 1.1 and the first matching degree as 1.1 as an example, assuming that the first matching degree threshold is 1 and the second matching degree threshold is also 1, it is obvious that the first matching degree and the second matching degree of audio interval 1 with image 1 both meet the requirements, but the sum of the first matching degree and the second matching degree of audio interval 1 with image 1 is only 2.2, indicating that audio interval 1 and image 1 are still not particularly matched. In some embodiments, it is also necessary to calculate whether the sum of the first matching degree and the second matching degree of each audio interval with the image meets the conditions. Specifically, when multiple first matching degrees are all greater than the first matching degree threshold, and multiple second matching degrees are all greater than the second matching degree threshold, the sum of multiple first matching degrees and multiple second matching degrees is calculated, including: respectively calculating the sum of the first matching degree and the second matching degree between each audio interval and the assigned image, and, when the sum of the first matching degree and the second matching degree between each audio interval and the assigned image is greater than the third matching degree threshold, calculating the sum of multiple first matching degrees and multiple second matching degrees. The third matching degree threshold corresponds to the first preset value in the specific embodiment section below. The setting of the first preset value can be referred to the specific embodiment section below, and no further description is given here.
[0014] 2) Since the method in 1) requires a lot of calculations, it is not conducive to saving the power consumption of electronic devices. Therefore, in some embodiments, the electronic device can directly calculate whether the sum of the first matching degree and the second matching degree of all audio intervals and images meets the requirements after assigning N images to N consecutive audio intervals in the first order. If the sum of the first matching degree and the second matching degree of all audio intervals and images meets the requirements, N images are matched to N consecutive audio intervals in the first order to generate a video. In this way, while saving the power consumption of electronic devices, a card point video of a dynamic poster image that can meet the viewing needs of users is generated to the greatest extent.
[0015] Specifically, in some embodiments, according to the audio rhythm change of each audio interval and the matching degree between N images, N images are matched to the corresponding audio interval, including: N images are respectively assigned to N consecutive audio intervals in a first order, and multiple first matching degrees and second matching degrees are calculated between each audio interval and the assigned image; the sum of multiple first matching degrees and multiple second matching degrees is calculated, and, when the sum of multiple first matching degrees and multiple second matching degrees is greater than the total matching degree threshold, N images are matched to the corresponding audio interval in the first order. The total matching degree threshold corresponds to the second preset value below. The setting of the second preset value can be referred to the specific embodiment section below, which will not be repeated here.
[0016] Furthermore, in order to avoid the situation where the total matching degree meets the requirement, but the sum of the first matching degree and the sum of the second matching degree between each audio interval and the image do not meet the requirement, for example, in the first order, the sum of the first matching degree and the second matching degree between all audio intervals and the image is 4, which is greater than the total matching degree threshold of 3, but the sum of the first matching degree and the second matching degree between a certain audio interval and the image is very small (for example, 0.1). In this case, the generated video will also have the problem of audio-visual incoordination. Therefore, in some embodiments, before calculating the sum of multiple first matching degrees and multiple second matching degrees, the sum of the first matching degree and the second matching degree between each audio interval and the assigned image can be calculated respectively. When the sum of the first matching degree and the second matching degree between each audio interval and the assigned image is greater than the third matching degree threshold, the sum of multiple first matching degrees and multiple second matching degrees is calculated again. Then, when the sum of multiple first matching degrees and multiple second matching degrees is greater than the total matching degree threshold, N images are matched to the corresponding audio intervals in the first order to generate a video that meets the requirements.
[0017] In combination with the first aspect and the above-mentioned possible implementation methods, in another possible implementation method of the first aspect, the background music used to generate the video is divided into N continuous audio intervals, including: according to the rhythm changes of the background music, the background music is divided into N continuous audio intervals. It can be understood that when the rhythm of the background music changes, there is a corresponding time node, that is, an audio node, and each audio node constitutes an audio interval. Since the audio interval is divided according to the rhythm changes of the background music, the audio interval can reflect the style or theme content of a certain audio segment in the background music. After obtaining multiple continuous audio intervals of the background music, according to the style or theme content of the audio interval and the rhythm changes of the audio interval, according to the above method, a suitable image can be matched to the audio interval to generate a card point video of the dynamic poster image.
[0018] In a second aspect, an embodiment of the present application further provides an electronic device, comprising a memory storing computer program instructions; and a processor, wherein the processor and the memory are coupled, and when the computer program instructions stored in the memory are executed by the processor, the electronic device implements the video generation method described in any one of the above-mentioned first aspects.
[0019] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and wherein when the computer program is executed by a processor, the method for generating a video described in any one of the first aspects above is implemented.
[0020] In a fourth aspect, an embodiment of the present application provides a computer program product. When the computer program product is run on an electronic device, the electronic device executes the video generation method described in any one of the above-mentioned first aspects.
[0021] It can be understood that the beneficial effects of the second to fourth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0023] Figure 1 is an application scenario diagram of a method for generating a video provided in some embodiments;
[0024] Figure 2 is a schematic diagram of a dynamic poster display provided by some embodiments;
[0025] Figure 3 is another example of a dynamic poster display schematic diagram provided by some embodiments;
[0026] Figure 4 is a flowchart of a method for generating a video provided by some embodiments;
[0027] Figure 5 is a schematic diagram of an example of calculating a poster foreground motion range provided by some embodiments;
[0028] Figure 6 is a schematic diagram of a time node for detecting a rhythm change in an audio node provided by some embodiments;
[0029] Figure 7It is a schematic diagram of a method for adjusting the poster playback order by using the video generation method of the present application provided in some embodiments;
[0030] Figure 8 is a schematic diagram of a hardware structure of a smart TV provided in some embodiments;
[0031] Fig. 9 is a schematic diagram of a software structure of a smart TV provided in some embodiments;
[0032] Fig.10 is a flowchart of a method for generating a video provided by some embodiments;
[0033] Fig.11 This is a schematic diagram of the structure of a server provided by some embodiments. DETAILED DESCRIPTION
[0034] The following will use terms commonly used by those skilled in the art to describe various aspects of the illustrative embodiments. It is understood that the illustrative embodiments of the present application include but are not limited to methods for generating videos, electronic devices, storage media, and the like.
[0035] Figure 1 According to an embodiment of the present application, an application scenario diagram of a dynamic poster is provided.
[0036] like Figure 1 As shown, the display interface of the smart TV 100 is divided into a dynamic poster image display area 110 and a movie introduction area 120. The dynamic poster image display area 110 is used to present video information such as recently released movies or TV series to users in the form of dynamic poster images, so as to attract users to click to watch a certain video. The movie introduction area 120 displays various movies in the form of Figure 1 The layout shown is listed so that users can choose a video to watch according to their own preferences through the introductions of various videos in the movie introduction area 120. Among them, the dynamic poster image is a depth estimation map of the poster image obtained by the electronic device after performing depth estimation on the poster image, and then converting the depth value and color (RGB) value of each pixel of the poster image according to the specified motion mode (motion range and duration) according to the depth estimation map, to form a video of the foreground object of the poster image moving. For example, assuming that the foreground object in the poster image moves to the left, when the foreground object moves to the left, the pixel value in the area where the foreground object passes will be converted into a pixel value consistent with the depth value and color of the foreground object pixel. When the motion process becomes continuous, it can visually give the user a feeling that the foreground object moves to the left, that is, a dynamic poster image is formed.
[0037] It should be understood that the dynamic poster image of the present application can be formed by the above-mentioned depth of field estimation method, and can also be formed by other methods, such as processing the poster image through view simulation (view synthesis), light field simulation (light field synthesis), etc., simulating changes in the poster image content to form a dynamic poster image. The present application does not limit the method of forming the dynamic poster image.
[0038] It should also be understood that the video generation method of the present application can be applied to various electronic devices, such as tablet computers, mobile phones, servers, portable game consoles, portable music players, reader devices, or other electronic devices that can access the Internet. The present application does not limit the type of electronic devices to which the video generation method can be applied. For ease of description, the following will continue to be described using the smart TV 100 as an example.
[0039] Currently, the background music of dynamic poster images provided by major video application platforms has no correlation with the theme of the poster image or the motion pattern of the foreground object of the poster image, which cannot generate audio-visual linkage for users. Figure 2 The dynamic poster image of Movie 1 and the dynamic poster image of Movie 2 are shown, and the background music of these two dynamic poster images is the audio interval 1 of the background music 1. Assuming that the theme of Movie 1 is "comfortable", the theme of Movie 2 is "passionate", and the theme or style of the audio interval 1 of the background music 1 is "urgent", then when the user watches the video composed of the dynamic poster image of Movie 1 and the dynamic poster image of Movie 2, he will find that the background music and the image are very inconsistent; moreover, the movement of the foreground object 101 of the dynamic poster image of Movie 1 is relatively gentle (rotating around the center), and the movement of the foreground object 201 of the dynamic poster image of Movie 2 is also relatively gentle (moving along the right diagonal direction of the dynamic poster image of Movie 2), but the rhythm of the audio interval 1 of the background music 1 is relatively intense, and this contrast will also make the video picture and the background music appear inconsistent.
[0040] In order to improve the playback mode of dynamic poster images and enable users to obtain a better audio-visual experience, some embodiments of the present application provide a method for generating a video. This method detects the audio nodes where the audio rhythm in the background music changes, divides the background music into multiple audio intervals, and also divides the video into multiple video clips, wherein the number of video clips is equal to the number of audio intervals, and then matches different video clips in different audio intervals, and adjusts the playback duration of each audio interval and the corresponding video clip to make the playback duration of the two equal, and then plays the video clips in the playback order of the audio intervals to generate a card point video. For example, taking background music 1 and video 1 as an example, assuming that 5 audio nodes are detected in background music 1, then background music 1 will be divided into 6 audio intervals, and similarly, video 1 will also be divided into 6 video clips, and the playback duration of each video clip and the audio interval matching it is equal. Specifically, the corresponding relationship between each audio interval of background music 1 and each video clip of video 1 is shown in Table 1 below:
[0041] Table 1
[0042]
[0043] It can be seen from Table 1 above that although the above method can associate the video with the background music to a certain extent, that is, each audio interval corresponds to a video clip, and the playback duration of each audio interval is equal to the playback duration of the video clip corresponding to the audio interval, this method can only be edited for existing pictures or video materials, and in order to ensure that the playback duration of each audio interval is equal to that of the video clip, it is difficult to make the theme of each audio interval correspond to the theme of each video clip at the same time. For example, assuming that the theme of video clip 1 is "Passion" and the playing time is 3 seconds; the theme of video clip 2 is "Emergency" and the playing time is 7 seconds; the playing time of audio interval 1 was originally 5 seconds, and the theme is "Passion", and the playing time of audio interval 2 was originally 5 seconds, and the theme is "Emergency". In order to make the playing time of audio interval 1 equal to the playing time of video clip 1, and the playing time of audio interval 2 and video clip 2 equal, it is necessary to adjust the playing time of audio interval 1 (or the playing time of video clip 1). Finally, the playing time of audio interval 1 is adjusted to 3 seconds, and the playing time of audio interval 2 becomes 7 seconds. However, it can be understood that at this time the theme of audio interval 2 becomes: the theme of the first 2 seconds is "Passion", and the theme of the last 5 seconds is "Emergency". When video clip 2 with the theme of "Emergency" is played in audio interval 2, in the first 2 seconds, there is still an incoordination problem between the video picture and background music.
[0044] In order to solve the above technical problems, some other embodiments of the present application provide a method for generating a video. Specifically, in the method, the smart TV 100 adds a poster image theme tag to each poster image according to the theme content to be expressed by each poster image in the M posters, and uses an image foreground segmentation algorithm, a depth of field estimation algorithm, etc. to determine the motion range of the foreground object in each poster image.
[0045] At the same time, the smart TV 100 detects N audio nodes with rhythm changes in at least one music X in the music library, and then divides the music X into (N+1) continuous audio intervals according to the N audio nodes, and determines the theme and the corresponding theme tag corresponding to each audio interval according to the audio change rate value in each audio interval, and then determines the movement range of each audio interval according to the playback time of each audio interval and the audio change rate value in the above audio interval.
[0046] Afterwards, the smart TV 100 will calculate the subject label, motion range and matching degree of each dynamic poster image of each audio interval of music X, obtain the matching degree of M*N groups of dynamic poster images and audio intervals, calculate the total matching degree of each combination of dynamic poster images and each audio interval, and then determine a combination of dynamic posters and audio intervals with the highest total matching degree and higher than a preset value from multiple combinations to obtain a dynamic poster card point video. For example, with dynamic poster image 1 (hereinafter referred to as "image 1"), dynamic poster image 2 (hereinafter referred to as "image 2"), audio interval 1 of music 1 (hereinafter referred to as "interval 1"), and audio interval 2 of music 1 (hereinafter referred to as "interval 2"), it is assumed that the matching degree of the aforementioned dynamic poster image and each audio interval of background music 1 is as shown in Table 2 below:
[0047] Table 2
[0048] Image 1 Image 2 Interval 1 1.25 1.20 Interval 2 1.22 1.24
[0049] From Table 2, we can see that the matching degree between image 1 and interval 1 is 1.25, the matching degree between image 2 and interval 1 is 1.20, the matching degree between image 1 and interval 2 is 1.22, and the matching degree between image 2 and interval 2 is 1.24. Then, the total matching degree of each of the two combinations of dynamic poster images and each audio interval of music 1 is shown in Table 3 below:
[0050] Table 3
[0051]
[0052] As can be seen from Table 3, the total matching degree of combination 1 {interval 1-image 1, interval 2-image 2} is 2.49, and the total matching degree of combination 2 {interval 1-image 2, interval 2-image 1} is 2.42. The total matching degree of combination 1 is the highest and is assumed to be higher than the preset value. Then the smart TV 100 will play the dynamic poster image 1 in the audio interval 1 of music 1 and the dynamic poster image 2 in the audio interval 2 of music 2 in the manner of combination 1, and generate a dynamic poster image card point video. Among them, the matching degree calculation of the dynamic poster image and the audio interval, the combination method of the dynamic poster image and the audio interval, etc. will be introduced in detail below and will not be repeated here.
[0053] It can be understood that in the above method, there may be a total matching score of various combinations of the above M dynamic poster images and all audio intervals of music X that is less than a preset value, that is, music X does not meet the conditions for generating a dynamic poster image card point video composed of M dynamic poster images. In this case, the smart TV 100 can replace music X, that is, replace music X with music Y, and re-determine the combination of each audio interval of music Y with the above M dynamic poster images in the above method with the highest total matching degree and higher than the preset value, so as to obtain a new dynamic poster image card point video. Among them, the way in which the smart TV 100 changes music will be described in detail below and will not be repeated here.
[0054] It can also be understood that in the above method, there may also be a situation where a new dynamic poster image is added to the M dynamic poster images or a dynamic poster image is replaced. In this case, the smart TV 100 will re-determine the combination of the various audio intervals of music X and the various combinations of all the new dynamic poster images in the above manner according to all the new dynamic poster images, and the combination with the highest total matching degree and higher than the preset value to obtain a new dynamic poster image card point video.
[0055] It should be noted that the number of dynamic poster images is generally the same as the number of audio intervals, that is, assuming that there are M dynamic poster images, then correspondingly, the smart TV 100 will randomly select M continuous audio intervals from the (n+1) audio intervals of the music X, and then when the total matching degree of all combinations of the M audio intervals and the M dynamic poster images is less than the preset value, the smart TV 100 will start from the previous audio interval or the next audio interval of any audio interval in the M audio intervals, re-acquire the M audio intervals, and then repeat the above method until the M audio intervals that meet the conditions are determined, and the audio composed of the M audio intervals is used as the background music for generating the dynamic poster image card point video. It should be understood that in some embodiments, the number of dynamic poster images can also be greater than the number of audio intervals. At this time, the smart TV 100 will re-determine new background music for the extra dynamic poster images to form the card point video of the extra dynamic poster images. Among them, the specific implementation details will be introduced below and will not be repeated here.
[0056] The dynamic poster image card point video formed by the above method not only makes the dynamic poster image and the background music match in theme content, but also associates the motion range of the foreground object of the dynamic poster image with the motion range of each audio interval of the background music, thus solving the problem of the incoordination between the motion range of the foreground object of the dynamic poster image and the background music, providing users with audio-visual linkage and improving user experience. Figure 3 The theme tag of the movie 1 dynamic poster image shown is "comfortable", the theme tag of the movie 2 dynamic poster image is "passionate", the theme tag of the audio interval 1 of the background music 1 is "urgent", and the theme tag of the audio interval 2 of the background music 1 is "relaxed". The smart TV 100 uses the method of generating dynamic poster card point videos of the present application to associate the movie 1 dynamic poster image and the audio interval 1 of the background music 1 with the same or similar theme tags and the same or similar motion ranges, and associates the movie 2 dynamic poster image and the audio interval 2 of the background music 1 with the same or similar theme tags and the same or similar motion ranges, and then plays the movie 1 dynamic poster image and the movie 2 dynamic poster image in the audio playback order (audio interval 1-audio interval 2). In this way, the movie 1 dynamic poster image and the audio interval 1 of the background music 1 associated with the movie 1 dynamic poster image, or the movie 2 dynamic poster image and the audio interval 2 of the background music 1 associated with the movie 2 dynamic poster image can bring audio-visual linkage to the user, which can not only improve the user experience, but also arouse the user's interest in the movie corresponding to the dynamic poster image.
[0057] It can be understood that in the embodiment of the present application, the above-mentioned background music and poster images can be music and poster images downloaded from the cloud or other servers and stored locally by the smart TV 100 after the smart TV 100 is connected to the Internet. For example, after the smart TV 100 is connected to the Internet, it downloads the latest poster images, music, etc. of various film and television dramas from major video application platforms, and stores the downloaded music and poster images locally; or, the smart TV 100 obtains the above-mentioned background music and poster images from the music library and image library stored by the user in the smart TV 100 according to the user's preferences.
[0058] In order to better understand the implementation process of the above video generation method, the following will combine Figures 4 to 11 The implementation details of the method for generating a video of the present application are introduced in detail. It should be noted that the following method can be executed by the processor of the smart TV 100, or by an application installed on the smart TV 100, such as a video application, a gallery application, etc. installed on the smart TV 100. The present application does not limit this. For the sake of ease of description, the following method is introduced with the processor of the smart TV 100 as the execution subject.
[0059] It should also be understood that the above method can be targeted at the determined M poster images and then select appropriate background music for the M poster images, and can be targeted at the selected N background music and then respectively match the N background music with poster images that are suitable for the background music. The principles and implementation processes of the two methods are similar. The following takes the selection of appropriate background music for the determined M poster images as an example to introduce the method of the present application.
[0060] Specifically, Figure 4 FIG. 1 is a flow chart of a method for generating a video provided by some embodiments. Figure 4 As shown, the method 400 includes:
[0061] Step 402: Obtain M poster images and determine the subject label of each poster image.
[0062] It can be understood that in order to match the M poster images with background music that is compatible with the content and theme of the poster images, it is first necessary to determine the theme tag of each poster image based on the content of the poster image, and then match the appropriate background music for all the poster images based on the theme tag of each poster image. In some embodiments, the M poster images can be obtained by the smart TV 100 from the poster images stored in the video application or other application installed on the smart TV 100, for example, the smart TV 100 obtains the M poster images from the gallery installed on the smart TV 100. Among them, the M poster images can be poster images of videos pushed by the video application according to the user's preferences, or poster images of videos that have been recently launched, and this application does not limit this.
[0063] Specifically, in some embodiments, the smart TV 100 can use an image recognition method to identify the content of each poster image, and determine the theme of each poster image based on the identified poster image content, and add a theme tag that is suitable for the poster image theme to each poster image. For example, assuming that the smart TV 100 uses an image recognition method to identify that a poster image content contains elements such as action and cars, the theme of the poster image is an action movie, and the theme tag added by the smart TV 100 to the poster image is "urgent"; if the smart TV 100 uses an image recognition method to identify that a poster image content contains elements such as tables and rooms, the theme of the poster image is a family movie, and the theme tag added by the smart TV 100 to the poster image is "warmth". In some embodiments, the smart TV 100 can pre-train an image recognition model, and then use the trained image recognition model to perform content recognition on the acquired M poster images, and add a theme tag to each poster image based on the identified content. Among them, the method of training the image recognition model of the smart TV 100 and the method of adding theme tags to the poster image will be described in detail below.
[0064] Step 404, determining the motion range of the foreground object of each poster image.
[0065] It can be understood that if the background music is matched for the poster image only according to the theme tag of the poster image, it is possible to match music that is close to or consistent with the theme of the poster image but has a shorter or longer music duration. At this time, if the music is intercepted or other operations are performed on the music, it may cause the entire dynamic poster image to be stuck when the video is played, and the background music will be discontinuous, and there will also be problems of audio-visual incoordination. Therefore, in some embodiments of the present application, it is also necessary to determine the motion range of the foreground object of the poster image, and then match the poster image with appropriate background music based on the motion range of the foreground object of the poster image and the theme of the poster image, so that the background music is not only adapted to the theme of the poster image, but also adapted to the foreground motion range of the poster image in terms of playback duration and audio amplitude changes.
[0066] Specifically, in some embodiments, the smart TV 100 performs foreground segmentation on the content of each poster image to obtain the foreground object of the poster image, and then the smart TV 100 can preliminarily determine the motion range of the foreground object relative to the background object of the poster image according to the position of the foreground object of the poster image in the poster image, that is, the distance from the outermost end of the outline of the foreground object in each direction to the edge of the poster image (up, down, left, and right) is the motion range of the foreground object relative to the background object of the poster image. For example, the leftmost side of the foreground object of the poster image is 2 pixels away from the left edge of the poster image, the rightmost side of the foreground object is 3 pixels away from the right edge of the poster image, the uppermost side of the foreground object is 2 pixels away from the upper edge of the poster image, and the lowermost side of the foreground object is 1 pixel away from the lower edge of the poster image. At this time, the motion range of the foreground object relative to the background object of the poster image is: the leftward motion range is 2 pixels, the rightward motion range is 3 pixels, the upward motion range is 2 pixels, and the downward motion range is 1 pixel. Among them, the specific implementation details of the smart TV 100 performing foreground segmentation on the poster image and determining the motion range of the foreground object of the poster image will be introduced below and will not be repeated here.
[0067] Step 406: Acquire music as background music to be matched, divide the background music to be matched into audio intervals, and determine a theme tag for each audio interval of the background music to be matched.
[0068] It can be understood that when matching appropriate background music for a specific poster image, it is first necessary to determine whether the style or theme of the background music is compatible with the poster image, and then use the music with a theme that is compatible with the poster image as the background music for the poster image. Similarly, when selecting appropriate background music for multiple poster images at the same time, it is also necessary to determine whether the background music is compatible with the themes of all or most of the poster images in the multiple poster images, so that the dynamic poster image card point video formed in this way can achieve preliminary consistency in audio-visual effects. Therefore, after obtaining the music as the background music to be matched, it is necessary to divide the background music into audio intervals, and then perform a preliminary match between the audio intervals and the poster images based on the theme tags of each audio interval.
[0069] Specifically, in some embodiments, the smart TV 100 can first obtain at least one piece of music from the music library as the background music to be selected, and then detect n time nodes corresponding to the change of the music rhythm of the background music to be selected according to the change of its music rhythm, and divide the background music into (n+1) continuous audio intervals according to the n time nodes, and then determine the theme label of each audio interval according to the audio change of each audio interval, and add a theme label to each audio interval. Among them, the specific implementation details of the smart TV 100 detecting the audio nodes of the background music to be matched and adding theme labels to each audio interval will be introduced below and will not be repeated here.
[0070] Step 408: determine the motion range of each audio interval in the background music to be matched.
[0071] As described in step 404, when matching background music for a poster image only based on the theme tag of the poster image, it is possible to match music that is close to or consistent with the theme of the poster image, but has a shorter or longer duration, which may lead to audio-visual incoordination. Therefore, when matching background music for a poster image, it is also necessary to consider whether the background music is compatible with the motion range of the poster image. Therefore, in some embodiments of the present application, it is also necessary to calculate the motion range of each audio interval of the above-mentioned background music to be selected, and then, based on the preliminary matching of the poster image and the audio interval based on the theme tag of the poster image and the theme tag of each audio interval, the motion range of the poster image is matched with the motion range of the audio interval, so as to obtain the background music described above, which is not only compatible with the theme of the poster image in terms of theme, but also compatible with the motion range of the foreground object of the poster image in terms of motion range.
[0072] Specifically, in some embodiments, the smart TV 100 can determine the movement range of the audio interval according to the audio interval duration (ie, the audio interval playback duration) and the audio change rate value of the audio interval.
[0073] For example, the smart TV 100 calculates the motion range of a certain audio interval by formula (1):
[0074] The range of motion of the audio interval = audio change rate value × audio interval duration (1)
[0075] Taking the calculation of the motion range of the audio interval 2 with an audio change rate value of 7 and an audio interval duration of 6 as an example, the motion range of the audio interval 2=7×6=42.
[0076] Assuming that the interval length and audio change rate value of some audio intervals in the music ID number "000012" are: the audio interval length of audio interval 1 is 3 seconds, the audio change rate value is 4, and the theme label is "passion"; the audio interval length of audio interval 2 is 6 seconds, the audio change rate value is 7, and the theme label is "urgent"; the interval length of audio interval 3 is 4 seconds, the audio change rate value is 1, and the theme label is "comfortable", then the smart TV 100 uses formula (1) to calculate the movement range of some audio intervals in the music ID number "000012" and the theme labels corresponding to each audio interval as shown in Table 4 below:
[0077] Table 4
[0078]
[0079] It can be seen from Table 6 above that the movement range of audio interval 1 is 12; the movement range of audio interval 2 is 42; and the movement range of audio interval 3 is 4.
[0080] Step 410, calculating the matching degree between each poster image and each audio interval of the background music to be matched.
[0081] It can be understood that when matching a suitable background music for a specific poster image, it is only necessary to match the poster image with a certain audio interval of the background music. However, when matching the suitable background music for the above M poster images at the same time, it is necessary to match each poster image with each audio interval respectively, and then determine the most suitable combination. In this way, it is necessary to calculate the matching degree of each poster image with each audio interval, and then determine the total matching degree of each combination of the M poster images and each audio interval based on the matching degree of each poster image with each audio interval.
[0082] Specifically, in some embodiments, the smart TV 100 first calculates the matching degree between each poster image and the subject label of each audio interval, then calculates the matching degree between each poster image and the motion range of each audio interval, and finally adds the matching degree between each poster image and the subject label of each audio interval and the matching degree between each image and the motion range of each audio interval to obtain the matching degree between each poster image and each audio interval.
[0083] Exemplarily, taking the calculation of the matching degree between a poster image and an audio interval as an example, the smart TV 100 first calculates the matching degree between the poster image and the theme tag of the audio interval, for example, 0.8, and then calculates the matching degree between the poster image and the motion range of the audio interval, for example, 0.9, and then adds the matching degree between the poster image and the theme tag of the audio interval and the matching degree between the poster image and the motion range of the audio interval to obtain the matching degree between the poster image and the audio interval: 0.8+0.9(1.7). Among them, other specific implementation details of this step will be introduced below and will not be repeated here.
[0084] Step 412: Arrange and combine the M poster images and the audio intervals of the background music to be matched, and calculate the total matching degree between the poster images and the music to be matched in each combination.
[0085] It can be understood that there are multiple combinations of the above M poster images and each audio interval. In each combination, each poster image has a corresponding matching degree with the corresponding audio interval. In order to balance the matching degree between each poster image and the audio interval, the smart TV 100 needs to calculate the total matching degree between the poster image and the background music to be matched in each combination. It can be understood that the total matching degree can represent the optimal matching method between each poster image and each audio interval to a certain extent. For example, continuing to take images 1 and 2 and audio intervals 1 and 2 shown in Table 2 as an example, assuming that the matching degree between images 1 and 2 and audio intervals 1 and 2 is as shown in Table 5 below:
[0086] Table 5
[0087] Image 1 Image 2 Interval 1 1.30 1.20 Interval 2 1.25 1.10
[0088] From Table 5, we can see that the matching degree between image 1 and interval 1 is 1.30, the matching degree between image 2 and interval 1 is 1.20, the matching degree between image 1 and interval 2 is 1.25, and the matching degree between image 2 and interval 2 is 1.10. Then, the total matching of each of the two combinations of images and each audio interval of music 1 is shown in Table 6 below:
[0089] Table 6
[0090]
[0091] From Table 6, we can see that the total matching degree of combination 1 {interval 1-image 1, interval 2-image 2} is 2.40, and the total matching degree of combination 2 {interval 1-image 2, interval 2-image 1} is 2.45. It can be seen that in combination 1, although the matching degree between image 1 and interval 1 is very high, the matching degree between image 2 and interval 2 is low, which is 1.10. In combination 2, although the matching degree between images 1 and 2 and intervals 1 and 2 is not greater than the matching degree between image 1 and interval 1 in combination 1, the matching degree between the poster image and the audio interval in combination 2 is relatively close, and the total matching degree is also greater than that of combination 1. Obviously, the combination method of combination 2 can better comprehensively consider the matching between the poster image and the audio interval.
[0092] It can be understood that if only the total match meets the condition, it may happen that although the total match meets the condition, some individual poster images and audio intervals are particularly mismatched. For example, continuing to take the poster images and intervals shown in Table 5 as an example, assuming that the total match of combination 1 {interval 1-image 1, interval 2-image 2} is 2.40, and the total match of combination 2 {interval 1-image 2, interval 2-image 1} is 2.30, the total match of combination 1 is greater than the total match of combination 2, but the match between interval 1 and image 1 in combination 1 is 0.10, and the total match between interval 2 and image 2 is 2.30. Obviously, the match between interval 1 and image 1 in combination 1 is too low. In order to avoid this situation, in some embodiments, the smart TV 100 can only calculate the total match under each combination mode when the match between each poster image and audio interval is greater than the first preset value, that is, only when each poster image and audio interval meet the condition, consider whether the combination mode meets the condition.
[0093] Specifically, in some embodiments, the smart TV 100 may arrange and combine the M poster images and the audio intervals, and then calculate the number of arrangements and combinations of the poster images and the audio intervals using the following formula (2):
[0094]
[0095] Wherein, i represents the number of poster images, j represents the number of audio intervals, and A(i,j) represents the number of permutations and combinations of the number of i poster images and j audio intervals. Afterwards, the smart TV 100 calculates the total matching degree between the poster image and the background music to be matched under each combination, and then executes step 414. The specific implementation details of step 412 will be described in detail below.
[0096] Step 414, determining the highest total matching degree from the various combinations mentioned above, and then determining whether the highest total matching degree is greater than or equal to a second preset value.
[0097] It can be understood that the combination method with the highest total matching degree determined in step 412 may not necessarily meet the requirements. For example, the highest total matching degree determined in step 412 is only 0.21, so the smart TV 100 also needs to compare the highest matching degree determined in step 412 with the second preset value. Only when the highest total matching degree is greater than the second preset value, the corresponding combination method is used as a combination method that can form a dynamic poster image card point video. If the highest total matching degree is less than the second preset value, it means that it is not suitable as the background music to be matched, and the smart TV 100 will change the background music to be matched. Specifically, in some embodiments, the smart TV 100 needs to determine the combination method with the highest total matching degree from multiple combinations, and compare the highest total matching degree with the preset value. When the highest total matching degree is greater than or equal to the second preset value, step 416 is executed, that is, the card point video of the dynamic poster image is generated according to the combination method with the highest total matching degree; when the highest total matching degree is less than the preset value, step 406 is executed, that is, the background music to be matched is re-acquired.
[0098] Step 416: Generate a card point video of the dynamic poster image according to the combination with the highest total matching degree.
[0099] The smart TV 100 generates a card point video of the dynamic poster image of the M poster images according to the combination method with the highest total matching degree and the order of playing the audio intervals.
[0100] Step 418, play the card point video of the dynamic poster image.
[0101] After generating the dynamic poster image card point video, the smart TV 100 Figure 1 The dynamic poster image card point video is played on the home page of the video application shown. In some embodiments, the card point video of the dynamic poster image can be played when the user turns on the smart TV 100 and enters the main interface of the smart TV 100, that is, as the startup video of the smart TV 100; it can also be played on the main interface of the smart TV 100 after the user turns on the smart TV 100 and enters the main interface of the smart TV 100; it can also be played on the main interface of a video application installed on the smart TV 100 after the user opens the video application, and this application does not limit this.
[0102] The above introduces the general process of the method for generating the video of the present application. The following introduces the specific implementation details of each step in the above method 400 in conjunction with the accompanying drawings.
[0103] In some embodiments, corresponding to step 404, Figure 5Taking the poster image 1, the poster image 2, and the poster image 3 as an example, the method for the smart TV 100 to determine the motion range of the poster image 1, the poster image 2, and the poster image 3 includes:
[0104] The method for the smart TV 100 to segment the foreground of the poster image and determine the motion range of the foreground object of the poster image is as follows: Figure 5 After the poster image 1, poster image 2, and poster image 3 are respectively segmented for the foreground, a binary image of the poster image is obtained, in which the white area is the foreground object and the black area is the background object. Therefore, the foreground object of poster image 1 is P1, and the background object is P1′, the foreground object of poster image 2 is P2, and the background object is P2′, and the foreground object of poster image 3 is P3, and the background object is P3′. Afterwards, the smart TV 100 can preliminarily determine the motion range of the foreground object P1 of poster image 1, the foreground object P2 of poster image 2, and the foreground object P3 of poster image 3 relative to their respective background objects as shown in Table 7 below:
[0105] Table 7
[0106]
[0107] In Table 7 above, "ID number" represents the ID number of the poster image, and the left-right (horizontal) and up-down (vertical) movement range represents the range of left-right and up-down movement of the foreground object of the poster image relative to the background object; wherein, x_l represents the range of leftward movement of the foreground object relative to the background object, x_r represents the range of rightward movement of the foreground object relative to the background object, y_t represents the range of upward movement of the foreground object relative to the background object, and y_d represents the range of leftward movement of the foreground object relative to the background object; the movement range represents the maximum value of the horizontal movement range and the vertical movement range of the foreground object of the poster image relative to the background object of the poster image, for example, the horizontal movement range of poster image 1 is: x_l+x_r=4+3=7 pixels (pixel), and the vertical movement range of poster image 1 is: y_t+y+d=1+0=1p, the horizontal movement range of poster image 1 is greater than the vertical movement range of poster image 1, so the movement range of poster image 1 is 7p; the theme label represents the theme of the poster image.
[0108] Optionally, the movement of the foreground object of the poster image relative to the background object includes but is not limited to left-right and up-down movement. The foreground object may also perform rotational movement relative to the background object, for example Figure 2 The way the foreground object moves relative to the background object in the dynamic poster image of Movie 1, or the foreground object moves diagonally up or down relative to the background object, for example Figure 2The movement mode of the foreground object relative to the background object in the dynamic poster image of Movie 2. This application does not restrict the movement mode of the foreground object of the poster image relative to the background object.
[0109] Combination Figure 5 ,From Table 7 above, it can be seen that the theme label of poster image 1 is “passionate”, and the motion range of the foreground object P1 is 7, the theme label of poster image 2 is “comfortable”, and the motion range of the foreground object P2 is 1p, and the theme label of poster image 3 is “urgent”, and the motion range of the foreground object P3 is 7p.
[0110] Then, the smart TV 100 performs depth estimation on the poster image to obtain a depth estimation map of the poster image, wherein the depth estimation Figure 1 Generally, it is a grayscale image, and the darker the pixel grayscale, the larger the grayscale value, the smaller the depth value, the lighter the pixel grayscale, the smaller the grayscale value, the larger the depth value, and white represents the maximum depth value. After the smart TV 100 obtains the poster image depth estimation image, it will use the depth estimation method described above and combine the motion range of the foreground object to convert the poster image into a dynamic poster image.
[0111] Because the foreground objects in some poster images have a large range of motion, and some of the image pixel values in the background objects in these poster images are similar to the foreground object pixel values, when the foreground object moves within a large range of motion, the image of the background object with similar pixel values also moves with the foreground object, and a part of the background object will be missing. If there is no other poster image content to refer to around this part of the missing, when the smart TV 100 fills this part of the missing with color, it will form an area that is not compatible with other content in the poster image, affecting the user's visual experience. Figure 5Taking the poster image 1 shown as an example, the partial image of the background object P1′ similar to the foreground object P1 in the poster image 1 is P1″. Since the movement range of the foreground object P1 of the poster image 1 is 7 pixels horizontally (4 pixels to the left and 3 pixels to the right), when the foreground object P1 moves 3 pixels to the right together with the image P1″, a vacant part of P1″′ will appear in the background object. At this time, since there is no other reference image content on the left side of the poster image 1, the color and content of the image formed after the smart TV 100 fills P1″′ are not particularly matched with other parts of the poster image 1, which will affect the user's visual experience. It can be understood that the above-mentioned 7 pixels, 4 pixels, and 3 pixels are only exemplary and do not constitute a limitation on the movement range of the foreground object of the poster image in the present application. In some embodiments, when the poster image size is relatively large, such as 1920 pixels * 1080 pixels, the above-mentioned movement range will be adaptively increased, such as 50 pixels, 100 pixels, etc., and the present application does not impose any limitation on this.
[0112] Therefore, in some embodiments, the smart TV 100 uses the depth estimation map of the poster image to obtain the depth value of each pixel in the poster image, that is, the pixel grayscale value of the poster image, and determines the background object portion close to the foreground object pixel grayscale value according to the pixel grayscale value of each pixel in the poster image, and then further adjusts the motion range of the foreground object according to the position of the background object portion close to the foreground object pixel grayscale value, so that the foreground object of the poster image can move as far as possible without affecting the visual effect presented by the dynamic poster image to the audience. For example, assuming that the center of the background object portion close to the foreground object pixel grayscale value is located at the image center of the poster image, since there is image content available for reference around the image center of the poster image, the smart TV 100 may not adjust the motion range of the foreground object, or may make a small adjustment.
[0113] Specifically, continue with Figure 5Taking the poster image 1 shown as an example, the rightward movement range of the poster image can be adjusted from 3p to 1p, and the area P1″′ will be narrowed from 3p to 1p. It can be understood that at this time, the foreground object P1 can still move in the horizontal range of 1p relative to the background object P1′, and the color filling of 1 pixel will not cause much visual discomfort to the user. Optionally, in some embodiments, the smart TV 100 can make appropriate adjustments to the movement range of the foreground object of the poster image in each direction according to the length, width, and color of the poster image, and the adjustment change value can be smaller than 1 pixel or larger than 1 pixel. For example, if the poster image 1 itself is a black and white poster image, the rightward movement range of the foreground object P1 can be adjusted from 3 pixels to 2 pixels, or even no adjustment is made. When the poster image 1 is a poster image of a pure color system other than black, white and gray, such as a pure green poster image, even a slightly smaller color difference may cause visual discomfort to the user. At this time, the rightward movement range of the foreground object P1 of the poster image 1 can be adjusted from 3 pixels to 0.1 pixels. It should be understood that the present application does not limit the adjustment method and the adjustment change value of the poster image movement range.
[0114] In some embodiments, the motion range of the foreground object of the poster image after the smart TV 100 adjusts the motion range of the foreground object of the poster image is as shown in Table 8 below:
[0115] Table 8
[0116]
[0117] Combination Figure 5 ,From Table 8 above, it can be seen that the theme label of poster image 1 is “passionate”, and the motion range of the foreground object P1 is 7p to 5p, the theme label of poster image 2 is “comfortable”, and the motion range of the foreground object P2 is 1p, and the theme label of poster image 3 is “urgent”, and the motion range of the foreground object P3 is 7p.
[0118] In some embodiments, corresponding to the above step 406, the method in which the smart TV 100 detects the audio nodes of the background music to be matched, divides the audio intervals, and sets a theme tag for each audio interval includes:
[0119] The smart TV 100 randomly obtains at least one piece of music from the music library installed on the smart TV 100. Assume that the smart TV 100 obtains a piece of music from the music library, and the ID number of the music is "000012". Optionally, the smart TV 100 can also select at least one of the above-mentioned music as the background music to be selected according to the user's preferences. For example, the smart TV 100 can use a big data statistical method to arrange the music that the user usually likes to listen to according to the number of times the user listens to the music, and then the smart TV 100 selects one or more of them as the background music to be selected according to the number of times each piece of music is played. Specifically, assuming that the user usually listens to 10 songs, their ID numbers are: {"00003", "00004", "00005", "00006", "00007", "00008", "00009" "000010", "000011", "000012"}, and they are sorted by the number of plays: {"000012", "00006", "00003", "00004", "000010", "000011", "00009", "00008", "00005", "00007"}, so the smart TV 100 will give priority to the music with ID number "000012" as the background music to be selected. It can be understood that when the playback order of ID numbers "000012" and "00006" is the same, the smart TV 100 will select ID numbers "000012" and "00006" as the background music to be selected, participate in the calculation of the matching degree with the dynamic poster image below, and finally determine one of them as the background music.
[0120] In other embodiments, the smart TV 100 may also pre-add the style label of the entire music to each piece of music in the music library, and then, based on the theme that accounts for the largest proportion in the above-mentioned M poster images, preferentially select the music that is consistent with or similar to the theme that accounts for the largest proportion from the music library as the background music to be selected. For example, assuming that there are 4 pieces of music in the music library, their style labels are: music 1-pop, music 2-rock, music 3-heavy metal, music 4-jazz, and the theme label that accounts for the largest proportion in the M poster images is "emergency", then the smart TV 100 will preferentially select the music that is compatible with the theme label of "emergency" from the music library as the background music to be selected. For example, the general soundtrack or interlude type of the movie with the theme label of "emergency" is rock or heavy metal, so the smart TV 100 will select music 2 with the style label of "rock" and music 3 with the style label of "heavy metal" as the background music to be selected to participate in the calculation of the matching degree with the dynamic poster image below, and determine one of them as the background music.
[0121] Then, the smart TV 100 will detect the n time nodes corresponding to the audio rhythm change according to the audio rhythm distribution of the entire piece of music, that is, n audio nodes, and divide the aforementioned music into (n+1) continuous audio intervals according to the n audio nodes. The duration of the audio interval is the playback duration of the audio in the interval.
[0122] In some embodiments, after the smart TV 100 obtains the (n+1) audio intervals of a certain background music to be matched, for example, after the smart TV 100 obtains the (n+1) audio intervals of the background music to be matched with the ID number "000012", the smart TV 100 will first randomly select the same number of audio intervals as the poster image from the (n+1) audio intervals as the audio intervals to be matched, and participate in the calculation of the matching degree between the audio intervals and the sea wave image below to determine the audio intervals that meet the conditions. For example, taking the poster image shown in Table 8 above as an example, the smart TV 100 will randomly select 3 audio intervals from the (n+1) intervals of the background music to be matched with the ID number "000012" as the audio intervals to be matched, and then participate in the matching degree calculation of poster images 1, 2, and 3. In other embodiments, the smart TV 100 may also start from the first audio interval of a certain piece of music to be selected, and sequentially select the same number of audio intervals to be matched as the poster image; or select the same number of audio intervals to be matched as the poster image at intervals; or, start from any audio interval of a certain piece of music to be selected, and sequentially select the same number of audio intervals to be matched as the poster image, and the present application does not limit this. Taking the smart TV 100 selecting the same number of audio intervals as the poster images 1, 2, and 3 as an example, Figure 6 FIG. 4 is a schematic diagram of an example of an audio interval provided by some embodiments. Figure 6 As shown, the smart TV 100 can detect the audio nodes of the music with ID number "000012" through the audio detection algorithm: A, B, ..., E, F, ..., J, K. The audio node A and the audio node B are in the audio interval 1 (00:01:20-00:01:23), and the audio playing time is 3 seconds; the audio node E and the audio node F are in the audio interval 2 (00:01:32-00:01:38), and the audio playing time is 6 seconds; the audio node J and the audio node K are in the audio interval 3 (00:02:11-00:02:15), and the audio playing time is 4 seconds. Among them, the way in which the smart TV 100 detects the background music audio nodes using the node detection algorithm will be introduced below, and will not be described in detail here.
[0123] After determining each audio interval in the background music with ID number "000012", the smart TV 100 will determine the audio change rate value of each audio interval based on the variance of the audio amplitude change in the audio interval. For example, each moment in the audio interval corresponds to the audio amplitude at that moment. The smart TV 100 can obtain the audio change rate value of the audio interval by calculating the variance of all audio amplitudes in the audio interval.
[0124] Then, the theme of each audio interval is determined according to the audio change rate value and a theme tag is added to each audio interval. In some embodiments, each audio change rate value corresponds to a theme tag. These theme tags can be the types of theme tags obtained by the smart TV 100 based on the types of general film and television drama posters obtained according to big data statistics. For example, general movie theme tags include "urgent"... "passionate", "warmth", "light"... "comfortable", etc., and because the audio rate change value can represent the intensity of the audio rhythm in a certain audio interval, the higher the audio rate value of a certain audio interval, the more intense the audio rhythm of the audio interval, so the smart TV 100 can add the aforementioned theme tags {"urgent"... "passionate", "warmth", "light"... "comfortable"} for the audio change rate value in the order of the audio rate change value from large to small. For example, the audio theme tags added by the smart TV 100 to the audio change rate value and the audio interval can be shown in Table 9 below:
[0125] Table 9
[0126]
[0127] Among them, the background music ID number indicates the number of the currently detected background music; the audio interval is divided according to the rhythm of audio changes, so the audio interval can reflect the distribution of audio nodes, and the interval duration indicates the playback duration of a certain audio interval.
[0128] Combination Figure 6 ,It can be seen from Table 9 that the interval length of audio interval 1 is 3 seconds, the audio change rate value is 4, and the theme label is "passion"; the interval length of audio interval 2 is 7 seconds, the audio change rate value is 7 seconds, and the theme label is "urgent"; the interval length of audio interval 3 is 4 seconds, the audio change rate value is 1, and the theme label is "comfortable".
[0129] In some embodiments, corresponding to the above step 410, the method for the smart TV 100 to calculate the matching degree between each poster image and each audio interval to be matched includes:
[0130] Continuing with the poster images shown in Table 8 and the audio intervals shown in Table 9 as examples, in some embodiments, the smart TV 100 calculates the matching degree of the topic tags between each poster image and the audio interval based on the topic tags of the three poster images and the topic tags of the audio intervals 1, 2, and 3 in the music with ID number "000012". In some embodiments, the smart TV 100 can use a matching degree calculation formula to calculate the matching degree between a poster image and a topic tag of an audio interval. Optionally, the matching degree calculation formula can be a Euclidean distance calculation formula, a Manhattan distance calculation formula, a Minkowski distance formula, or a Pearson correlation coefficient calculation formula. It should be understood that the present application does not impose any restrictions on the method of calculating the poster image topic tags and the audio interval topic tags. For example, in some embodiments, the smart TV 100 uses the following Euclidean distance formula (3) to calculate the matching degree between the poster image topic tags and the audio interval topic tags:
[0131]
[0132] Among them, d n (p, q) represents the matching degree between the poster image topic label and the audio interval topic label, p represents each poster image topic label, n represents the number of feature values of each poster image topic label or audio interval topic label, p i represents the feature values of each poster image theme label, q represents the theme label of each audio interval, and q i Representing the various feature values of the topic label of each audio interval, it can be understood that if the topic label is represented as text, the text has text features that can be represented by text features, such as the string length, encoding, etc. of the text. This application does not limit the number of features of the topic label.
[0133] Since the motion range of the audio interval may not match the motion range of the foreground object of the poster image, as mentioned above, the motion range of the audio interval is related to the duration of the audio interval and the rate of change of the audio, and the motion range of the foreground object of the poster image represents the maximum horizontal or vertical motion range of the foreground object. If only the poster image and the theme tag of the audio interval are calculated to see whether they match, and then only the poster image with a theme tag matching degree higher than the preset theme tag matching degree is associated with the audio interval, it may happen that the foreground object of the poster image has finished moving within the maximum motion range, but the audio interval has not been played, or the audio interval has been played, but the foreground object of the poster image has not finished moving within the maximum motion range. It is understandable that this audio-visual disharmony will also affect the user experience.
[0134] Therefore, in some embodiments, the smart TV 100 not only calculates the matching degree between the theme label of the poster image and the theme label of the audio interval, but also calculates the matching degree between the motion range of the poster image and the motion range of the audio interval. In some embodiments, the smart TV 100 can calculate the motion range of the foreground object of the poster image and the motion range of the audio interval using formula (4):
[0135] Motion range matching degree = audio interval motion range / poster image interval motion range (4)
[0136] Taking the motion range of the foreground object of poster image 1 and the motion range of audio interval 2 of the selected background music as examples, it can be seen from Table 8 above that the motion range of the foreground object of poster image 1 is 5, and from Table 4 above, it can be seen that the motion range of audio interval 2 is 42, then the motion range matching degree between poster image 1 and audio interval 2 is: 5 / 42=0.11.
[0137] Furthermore, it can be seen from the motion range of the background music audio interval to be selected and the motion range of the foreground object of the poster image that the motion range of the foreground object of the poster image is generally in the single digit, while the motion range of the background music audio interval to be selected is generally in the double digit. The motion range of the background music audio interval to be selected and the motion range of the foreground object of the poster image differ too much in value. If the two are directly compared using formula (4), the matching result obtained each time is relatively small, and no subsequent calculation is used.
[0138] Therefore, in some embodiments, the smart TV 100 may multiply the motion range of the foreground object of the poster image by a preset coefficient, so that the motion range of the foreground object of the poster image and the motion range of the audio interval of the selected background music are in the same order of magnitude. For example, continuing to take the motion range of the foreground object of the poster image 1 and the motion range of the audio interval 2 of the selected background music as examples, the smart TV 100 multiplies the motion range of the foreground object of the poster image 1 by a preset coefficient of 8, and obtains that the motion range of the foreground object of the poster image 1 is 5×8=40. Then, the smart TV calculates the motion range of the foreground object of the poster image 1 and the motion range of the audio interval 2 using formula (5):
[0139] Motion range matching degree = 1-|Motion range of poster image foreground object-Motion range of audio interval| / Motion range of poster image interval (5)
[0140] Then the matching degree between the motion range of the foreground object of poster image 1 and the motion range of the audio interval = 1-|40-42| / 40 = 1-0.5 = 0.95
[0141] Finally, the smart TV 100 associates the matching degree between the motion range of the foreground object of the poster image and the motion range of the audio interval with the matching degree between the poster image theme label and the audio interval theme label to obtain the matching degree between the poster image and the audio interval. In some embodiments, the smart TV 100 can calculate the matching degree between the poster image and the audio interval by formula (6):
[0142] The matching degree between the poster image and the audio interval = the matching degree between the poster image theme label and the audio interval theme label + the matching degree between the poster image foreground motion range and the background music interval motion range (6)
[0143] For example, the smart TV 100 adds the matching degree between the motion range of the foreground object of poster image 1 and the motion range of audio interval 2 and the matching degree between the topic label of poster image 1 and the topic label of audio interval 2 to obtain a total matching score between poster image 1 and audio interval 2 = 0.95 + 0.3 = 1.25.
[0144] In other embodiments, the smart TV 100 may also set weights for the motion range matching and the topic label matching respectively. For example, the importance of whether the poster image matches the topic label of the audio interval is higher than the importance of whether the motion range matches. Therefore, the weight of the matching degree of the poster image and the topic label of the audio interval can be set higher. For example, the smart TV 100 sets the weight of the matching degree of the poster image and the topic label of the audio interval to 80%, and then sets the weight of the matching degree of the motion range of the poster image and the audio interval to 20%, and then uses formula (7) to calculate the matching degree of the poster image and the audio interval:
[0145] The matching degree between the poster image and the audio interval = 80% × the matching degree between the poster image topic label and the audio interval topic label + 20% × the matching degree between the poster image foreground motion range and the audio interval motion range (7)
[0146] For example, the smart TV 100 multiplies the matching degree between the motion range of the foreground object of poster image 1 and the motion range of audio interval 2 by 80%, and multiplies the matching degree between the theme label of poster image 1 and the theme label of audio interval 2 by 20%, and then adds them together to obtain the matching degree between poster image 1 and audio interval 2 = 20% × 0.95 + 80% × 0.3 = 0.43. It should be understood that the present application does not limit the manner in which the smart TV 100 calculates the matching degree between the poster image and the audio interval based on the matching degree between the motion range of the poster image and the audio interval and the matching degree between the theme label of the poster image and the audio interval.
[0147] In some embodiments, corresponding to step 412, the smart TV 100 calculates the combination of each poster image and each audio interval, and the total matching degree of the poster image and the background music to be matched in each combination, including:
[0148] Continuing with the poster images shown in Table 8 and the audio intervals shown in Table 9 as an example, when the smart TV 100 arranges and combines the above three poster images with the three audio intervals of the music with the ID number "000012", i=3, j=3 in the above formula (2), so A(3,3)=6, so there are 6 combinations of the above poster image 1, poster image 2, poster image 3 and audio interval 1, audio interval 2, audio interval 3. Assume that the smart TV 100 calculates the matching degree of poster image 1, poster image 2, poster image 3 and audio interval 1, audio interval 2, audio interval 3 with the ID number "000012" through the above step 410 as shown in Table 10 below:
[0149] Table 10
[0150] Poster Image 1 Poster Image 2 Poster Image 3 Audio interval 1 1.26 1.23 1.10 Audio Zone 2 1.15 1.16 1.81 Audio Zone 3 1.26 1.35 1.38
[0151] Among the six permutations and combinations of the poster image and the audio interval calculated by the smart TV 100, the total matching degree between the poster image and the music with ID number "000012" in each combination is shown in Table 11 below:
[0152] Table 11
[0153]
[0154]
[0155] It can be seen from Table 11 that the total matching degree of combination 1 {audio interval 1-poster image 1, audio interval 2-poster image 2, audio interval 3-poster image 3} is 3.59, the total matching degree of combination 2 {audio interval 1-poster image 1, audio interval 2-poster image 3, audio interval 3-poster image 2} is 4.42, the total matching degree of combination 3 {audio interval 1-poster image 2, audio interval 2-poster image 1, audio interval 3-poster image 2} is 4.19, the total matching degree of combination 4 {audio interval 1-poster image 2, audio interval 2-poster image 2, audio interval 3-poster image 1} is 3.41, the total matching degree of combination 5 {audio interval 1-poster image 3, audio interval 2-poster image 1, audio interval 3-poster image 2} is 4.30, and the total matching degree of combination 6 {audio interval 1-poster image 1, audio interval 2-poster image 2, audio interval 3-poster image 1} is 3.52.
[0156] In some embodiments, the smart TV 100 may first determine whether the matching degree between each poster image and the audio interval in each combination is greater than or equal to a first preset value, and then, when the matching degree between each poster image and the audio interval is greater than or equal to the first preset value, calculate the total matching degree under this combination. For example, continuing with Table 10 and Table 11 as examples, assuming that the first preset value is 1.15, since the matching degree between the poster image and the audio interval in combination 1, combination 3, combination 4, and combination 6 is less than 1.15, the smart TV 100 will directly calculate the total matching degree in combination 2 and combination 5. Among them, the first preset value is a value set by the R&D personnel using big data statistics, and the first preset value can be taken as the average value of the matching degree between each poster image and the audio interval in each combination. In this way, it is possible to exclude the situation where a small number of poster images and audio intervals in each combination have a low matching degree.
[0157] Continuing with Table 10 and Table 11 as examples, in some embodiments, corresponding to step 414, the highest total matching degree is determined from the various combinations mentioned above, and judging whether the highest total matching degree is greater than or equal to the second preset value includes: the smart TV 100 determines that the total matching degree of combination 2 {audio interval 1-poster image 1, audio interval 2-poster image 3, audio interval 3-poster image 2} is 4.42, which is the highest, so the smart TV 100 compares the total matching degree of combination 2 with the second preset value. Among them, the second preset value is a value preset by R&D personnel using big data statistics.
[0158] Assuming that the second preset value is 4, the total matching degree of combination 2 is higher than the second preset value, and the smart TV 100 generates a dynamic poster image card point video according to combination 2. That is, the smart TV 100 intercepts the audio interval 1 to the audio interval 3 of the music with ID number "000012" as the background music of poster images 1, 2, and 3, and plays the poster images in the order of poster image 1, poster image 3, and poster image 2.
[0159] Assuming that the second preset value is 5, the total matching degree of combination 2 is also less than the second preset value, and the smart TV 100 will reselect three audio intervals from other audio intervals with ID number "000012" and repeat the above steps 410 to 412 until an audio interval that meets the above conditions is determined. Figure 7As shown, the smart TV 100 will start from the previous audio interval X of the audio interval 1, reselect three audio intervals, audio interval X, audio interval 1, and audio interval 2 as new audio intervals to be matched, and then calculate the total matching of all combinations of audio interval X, audio interval 1, audio interval 2 and poster image 1, poster image 2, and poster image 3, and determine a group of combinations with the highest total matching degree and higher than the second preset value 5 to form a dynamic poster card point video. In some embodiments, the smart TV 100 will store the foreground objects, background objects, theme tags, motion ranges of each poster image obtained through the above steps 402 to 410, and the rate change values of each audio interval, audio interval theme tags, and audio interval of each background music, as well as the matching degree of each poster image and each audio interval of each background music in the memory of the smart TV 100, so when determining and calculating the total matching degree of various combinations of new audio intervals and poster images, the smart TV 100 can directly obtain the matching degree of the poster image and a certain audio interval from the memory to improve the calculation efficiency. For example, taking the determination of the highest matching degree among various combinations of audio interval X, audio interval 1, audio interval 2 and poster images 1, 2, 3 as an example, the smart TV 100 only needs to recalculate the matching degree between audio interval X and poster images 1, 2, 3, and does not need to recalculate the matching degree between audio interval 1, audio interval 2 and poster images 1, 2, 3.
[0160] Assuming that in the background music to be matched with ID number "000012", the total high matching degree of all combinations of audio intervals and poster images is less than the second preset value, the smart TV 100 will change the background music to be matched, and repeat the above steps 406 to 410 until qualified music is obtained, and use the music as the background music to generate a dynamic poster image card point video. The specific implementation details can be referred to the above related description, which will not be repeated here.
[0161] In some embodiments, when a user clicks on a poster image in a dynamic poster image card video and watches the video corresponding to the poster image, the smart TV 100 will replace the poster image clicked by the user and determine the background music that matches the new poster image and the original dynamic poster image.
[0162] For example, taking the dynamic poster card point video composed of poster images 1, 2, 3 and background music with ID number "000012" shown in Table 2 as an example, assuming that the user clicks on poster image 1 and watches the video corresponding to poster image 1, the smart TV 100 will use the method described in step 402 to re-acquire the new poster image Y, and re-judge the matching degree of the new poster image Y with the audio intervals 1, 2, 3 of the background music with ID number "000012" corresponding to the dynamic poster card point video according to the method of steps 402 to 414. When the new poster image Y and the audio intervals 1, 2, 3 of the background music with ID number "000012", in order to be able to play the new poster image Y and the original poster image 2 and poster image 3 under the same background music, the smart TV 100 will replace the background music again in the method described in step 406, and calculate the highest matching degree of all combinations of the audio intervals of the new background music to be matched with the poster image Y and the original poster image 2 and poster image 3, and determine whether the new background music to be matched meets the above conditions.
[0163] In the above embodiment, in order to play the new poster image Y and the original poster images 2 and 3 under the same background music, the background music needs to be changed. It is understandable that the smart TV 100 can also select a new background music for the new poster image Y and then play the poster image Y under the new background music.
[0164] For example, in other embodiments, taking the dynamic poster card point video composed of poster images 1, 2, 3 and background music with ID number "000012" shown in Table 8 as an example, assuming that the user clicks on poster image 1 and watches the video corresponding to poster image 1, the smart TV 100 uses the method described in step 402 above to re-acquire a new poster image Y, and repeats steps 402 to 414 above to select new background music for the new poster image Y. The specific implementation details can be referred to the relevant description above and will not be repeated here.
[0165] The above describes the implementation process of the method for generating a video of the present application. Through the above video generation method, a card point video of a dynamic poster image can be obtained. Moreover, as mentioned above, since the card point video of the dynamic poster image links the theme of the dynamic poster image with the theme of the audio interval, and the motion range of the foreground object of the dynamic poster image with the motion range of the audio interval, when playing the card point video of the dynamic poster image, it can bring the user the following Figure 3The audio-visual experience shown. The following briefly describes the model training in the above steps and the method of using the trained model to add topic tags to the poster image, perform foreground segmentation on the poster image, and detect audio nodes. It should be understood that the following implementation methods are exemplary and do not constitute a limitation on the specific implementation details of the method for generating the video of the present application. In other embodiments, in order to achieve the technical effect of the method for generating the video of the present application, other alternative means can also be used to implement the following methods, which will not be repeated here.
[0166] In some embodiments, corresponding to step 402, the method for training an image recognition model by the smart TV 100 and the method for adding a theme tag to a poster image include:
[0167] (1) Smart TV 100 trains an image recognition model.
[0168] The smart TV 100 trains the above-mentioned image recognition model based on the images with pixel-level labels in the existing semantic segmentation data set, so that the trained image recognition model can recognize the content in the poster image, for example, the image recognition model can recognize the person, room, car, building, etc. in the poster image. Among them, the image with pixel-level labels means that each pixel in the image has its corresponding type label, for example, some pixels correspond to people, so the labels of these pixels are people, some pixels correspond to rooms, so the labels of these pixels are rooms, some pixels correspond to cars, so the labels of these pixels are cars, etc.
[0169] Specifically, the smart TV 100 takes images in a data set with preset pixel-level category labels as target data; then the smart TV 100 inputs the target data into the image recognition model to be trained, obtains the image recognition result of the target data, and calculates the loss function of the image recognition model based on the image recognition result of the target data.
[0170] In some embodiments, the smart TV 100 can calculate the loss function by using formula (8):
[0171] L Seg = -y(t)logF(x t ) (8)
[0172] Among them, F(x t ) is the result after recognition by the image recognition model, y(t) is the preset pixel-level category label of the target data, L Seg is the loss function of the image recognition model.
[0173] Then, the smart TV 100 adjusts the parameters in the image recognition model according to the result of the loss function, such as the weight value of each layer of the neural network used in the image recognition model, to reduce the result of the loss function so that the output result of the image recognition model is the same as or similar to the input result (that is, the preset pixel-level category label). When the output result of the image recognition model is the same as or similar to the input result, the image recognition model training is considered to be completed.
[0174] In some embodiments, the dataset used to train the image recognition model may be a 2D dataset such as VOC (Pascal Visual Object Classes) and MS COCO (Cityscapes Micro Soft Common Objects in Context); or a 2.5D dataset such as NYU-D V2, SUN-3D, and SUN RGB-D; or a 3D dataset such as Stanford 2D-3D and Shape-Net Core. This application does not impose any restrictions on the dataset used to train the image recognition model.
[0175] In some embodiments, the above-mentioned neural network model architecture for training image recognition can be based on fully convolution networks (FCNs), Seg-Net, U-Net, Deep-Lab V1-V3, etc. This application does not limit the type of neural network model architecture used for image recognition model training.
[0176] (2) The smart TV 100 uses the trained image recognition model to identify the content of the poster image and add a theme tag to the poster image.
[0177] The smart TV 100 uses the trained image recognition model to identify the content of the poster image. When the content matching a certain theme A is identified, the theme tag “A” is added to the poster image.
[0178] For example, if the smart TV 100 uses the trained image recognition model to identify that the poster image 2 includes elements such as people, rooms, sofas, etc., and it is assumed that these elements belong to the theme of "comfort", the smart TV 100 will add the theme label "comfort" to the poster image 2. It should be understood that when a poster image identified by the smart TV 100 includes element contents under multiple themes, the smart TV 100 will classify the poster image according to the proportion of element contents under a certain theme. For example, continuing to take the poster image 2 as an example, the smart TV 100 identifies that the poster image 2 includes elements such as people, rooms, sofas, windows, etc., and it is assumed that the elements such as rooms, sofas, windows, etc. belong to the element contents of the theme of "comfort", while the characters belong to the element contents of the theme of "action". Since the number of element contents of the theme of "comfort" (3) accounts for more than the number of element contents of the theme of "action" (1), the smart TV 100 will add the theme label "comfort" to the poster image 2.
[0179] The above description is merely an exemplary description and does not constitute a limitation on the method of training an image recognition model and adding topic tags to poster images using an image recognition model in the present application. In other embodiments, the smart TV 100 may also train the image recognition model or add topic tags to poster images using an image recognition model in other ways. For example, when the smart TV 100 trains the image recognition model, it uses poster images with preset topic tags as target data. In this way, when the image recognition model training is completed, the smart TV 100 can directly identify the content of the poster image through the image recognition model and add topic tags to the poster image at the same time. That is, the image recognition model trained in this way has the ability to automatically identify the content of the poster image and add topic tags to the poster image.
[0180] In some embodiments, corresponding to step 404, the method for the smart TV 100 to perform foreground segmentation on the poster image is as follows:
[0181] (1) Smart TV 100 trains an image segmentation model.
[0182] The principle of training the image segmentation model of the smart TV 100 is consistent with the principle of training the image recognition model of the smart TV 100 in step 402. For details, please refer to the method of training the image recognition model in step 402, which will not be repeated here.
[0183] (2) The smart TV 100 uses the trained image segmentation model to perform foreground segmentation on the poster image.
[0184] The smart TV 100 uses the trained image segmentation model to perform foreground segmentation on the poster image. For example, assuming that the poster image contains four different target objects, namely, a table, a dog, a cup, and a person, when the smart TV 100 uses the above-mentioned trained image segmentation model to perform foreground segmentation on the poster image, it can obtain four types of regional segmentation images including a table, a cup, a dog, and a person. Then, the smart TV 100 will use a specific regional segmentation image as the foreground object of the poster image according to the pre-set conditions to determine the motion range of the foreground object in the dynamic poster image. In some embodiments, the smart TV 100 can determine that a regional segmentation image is the foreground object of the poster image according to the theme of the poster image. For example, assuming that the theme of the poster image is "emergency", the regional segmentation image obtained after the poster image is processed by the image segmentation model includes people, cars, buildings, windows, etc. In order to match the theme of the poster image "emergency", the smart TV 100 can use the regional segmentation images corresponding to cars and buildings as the foreground objects of the poster image. In other embodiments, the smart TV 100 can determine a certain area segmented image as the foreground object of the poster image based on the information that the poster image intends to convey. For example, assuming that the poster image is a fixed-date poster image, the area segmented image obtained after the poster image is processed by the image segmentation model includes a clock, a person, an animal, etc. Since the information that the poster image intends to convey is the fixed-date, the clock can be used as the foreground object of the poster image.
[0185] In some embodiments, corresponding to step 406, the method for the smart TV 100 to detect background music audio nodes and divide audio intervals for background music according to the audio nodes is as follows:
[0186] In order to understand the implementation process of audio node detection, the principle of audio node detection is first introduced.
[0187] Typically, the frequency of audio used for background music is generally between 70 Hz and 4000 Hz. For some composite audios with multi-frequency signals such as multi-instrument chords, since it is impossible to detect the audio node by simple time domain signal peak extraction, the time domain signal of the audio is first converted into a frequency domain signal, and then the audio signal converted into a frequency domain signal is divided into several windows, and the audio signals of adjacent windows are differentially processed to obtain the difference data between adjacent audio windows, wherein the difference data is used to represent the difference in amplitude between the audio signal of the previous window and the audio signal of the current window. It can be understood that when the difference data exceeds the preset difference, it means that a mutation has occurred between the audio signal of the previous window and the audio signal of the current window, that is, the audio signal of the current window may be the starting node of the next rhythm, and then the difference data between the audio signals of each adjacent window is analyzed in the same way to obtain the audio node of the audio segment.
[0188] Specifically, assume that the duration of the audio f(x) with ID number 0000X is t seconds and the sampling rate is f s =44100 Hz, number of samples k = f s ·t, the method for the smart TV 100 to detect an audio node can be divided into the following steps:
[0189] (1) Use Fourier transform to convert the audio time domain signal into frequency domain signal.
[0190] Divide f(k) into n window fragments fi(k) of equal size, where 1<<i<<n, the length of each window is L=1024, and the audio duration of each window is And the time corresponding to the i-th window This is the moment when the audio starts relative to the midpoint of the window.
[0191] Then for each window f i (k), find its Fourier transform F i (w), forming a Fourier transform window sequence F = {F 1 (k), F 2 (k), …, F n (k)}, where F represents the frequency domain features corresponding to each window of audio, such as amplitude.
[0192] (2) Perform differential processing on the audio signal to obtain a differential window sequence (difference data sequence).
[0193] The smart TV 100 uses equation (9) to perform differential processing on the Fourier window sequence F:
[0194]
[0195] Among them, d i represents the difference data of adjacent Fourier windows, k represents the number of samples, F i (k) represents the frequency domain features corresponding to the audio in the i-th window, 2<<i<<n.
[0196] Then, the smart TV 100 obtains the differential window sequence D={d 1 , d 2 , …, d n}. Among them, when d i When the value is positive and the absolute value is large, it indicates that t i Time F i The amplitude of (k) is greater than that of the previous moment t i-1 F i-1 The amplitude of (k) is stronger, which means that the audio signal is increasing, which means that t iThe moment may be the starting node of the next strong rhythm audio. Accordingly, when d i When the value is negative and the absolute value is large, it indicates that t i Time F i The amplitude of (k) is greater than that of the previous moment t i-1 F i-1 The amplitude of (k) is smaller, which means that the audio signal is weakening, which also indicates that t i The time may be the starting node of the next weak rhythm audio.
[0197] (3) Compare the above differential window sequence with a preset difference value, and take the time nodes of the Fourier window corresponding to the difference data greater than the preset difference value as audio nodes to obtain an audio node sequence.
[0198] In some embodiments, the smart TV 100 compares the difference data with a preset difference. If a certain difference data is greater than the preset difference, the time node of the Fourier window corresponding to the difference data is the audio node. For example, assuming d i If the difference is greater than the preset value, the smart TV 100 will i (k) is regarded as a rhythm point window, that is, t i Using the same method, we can get the audio node sequence T = {t 1 ,t 2 ,…,t m}, and m is less than n.
[0199] Optionally, the preset difference value can be the difference window sequence D={d 1 , d 2 , …, d n The average value of the amplitude difference of each window in}:
[0200]
[0201] Among them, W is the size of the window mean, and the mean r of position i i , is calculated as the mean of its domain W.
[0202] The smart TV 100 can use formula (7) to obtain the average value sequence of amplitude differences corresponding to the differential window sequence R = {r 1 , r 2 ,…,r n}, then for any element d in D i , only when d i >r i When the smart TV 100 i (k) is regarded as a rhythm point window, that is, t iis an audio node, and finally the above audio node sequence T={t 1 ,t 2 ,…,t m ,}.
[0203] Figure 8 A schematic diagram of the hardware structure of a smart TV 100 is shown.
[0204] like Figure 8 As shown, the smart TV 100 may include a processor 10, a memory 20, a display screen 30, a camera 40, and a speaker 50, wherein the processor 10 is used to perform foreground segmentation, depth of field estimation, and the like on the poster image, and the processor 10 is also used to perform audio node detection on the audio, and the memory 20 is used to store computer program instructions. When the computer program instructions are moved by the smart TV 100, the processor 10 of the smart TV 100 will implement the above step 402 In the video generation method to step 418, the display screen 30 is used to display the dynamic poster image card point video obtained by the above-mentioned video generation method, and receive touch operations such as clicking and sliding thereon to form display content that is compatible with the aforementioned touch operation intention. For example, in some embodiments, the user clicks on a dynamic poster image played in the dynamic poster image card point video on the display screen 30, and then the video such as the movie or TV series corresponding to the dynamic poster image will be played on the display screen 30. The camera 40 is used for the user's air gestures. For example, the user waves his hands in the air, and the camera 40 obtains an image of the user's waving hand action in the air, and sends the image containing the air gesture to the processor 10. Identification processing is performed to obtain a touch instruction corresponding to the waving hand, and the smart TV 100 responds to the touch instruction to form corresponding display content to be displayed on the display screen 30. For example, the touch instruction corresponding to the waving hand gesture in the air represents "turning the page". Then when the user makes a waving hand gesture in the air, the content displayed on the display screen 30 will flip from the current page to the next page. The speaker 50 is used to play the audio described in the above embodiment, and the speaker 50 is also used to play the audio content of specific video content such as movies or TV series corresponding to the dynamic poster image.
[0205] It should be understood that the above Figure 8 The hardware structure of the smart TV 100 shown is only exemplary. In other embodiments, the smart TV 100 may further include more structures, such as an antenna for communicating with other electronic devices, a touch sensor for receiving user touch commands, etc. This application does not limit the composition of the hardware structure of the smart TV 100.
[0206] Fig. 9 1 is a software structure block diagram of the smart TV 100 according to an embodiment of the present invention.
[0207] like Fig. 9 As shown, the smart TV 100 can be divided into an application layer, an application framework layer, an Android runtime (Android runtime) and a system library, and a kernel layer.
[0208] Among them, the application layer can include a series of application packages.
[0209] like Fig. 9 As shown, the application package may include applications such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message, etc. In an embodiment of the present application, the application package may include a gallery application, etc.
[0210] The application framework layer may include a view system, a gesture recognition system, and the like.
[0211] In an embodiment of the present application, the gesture recognition system is used to recognize user operations performed by a user on a gallery application on the screen of the smart TV 100 .
[0212] The view system includes visual controls, such as controls for displaying text, controls for displaying pictures, etc. The view system can be used to construct the display interface of the application. The display interface can be composed of one or more display elements, where the display element refers to the element in the display interface of the application on the screen of the electronic device. For example, the display element may include a button, text, picture, pop-up window, menu, title bar, list or search box, etc. The display interface of the application may include at least one display element. In an embodiment of the present application, the view system can be used to implement the layout scheme of the display interface of the application of the present application. For example, when the application is started, the view system can dynamically adjust the position of the display elements in the display interface based on the size of the display area of the display interface of the application in the screen of the smart TV 100; at the same time, the view system can also configure the display style model for the display interface of the application, and when the application is started, the view system uses the display style parameters of the application to calculate the display effect of the display elements in the display interface through the display style model.
[0213] Android Runtime includes core libraries and virtual machines. Android runtime is responsible for scheduling and management of the Android system.
[0214] The core library consists of two parts: one is the function that needs to be called by the Java language, and the other is the Android core library.
[0215] The application layer and the application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.
[0216] The system library may include multiple functional modules, such as surface manager, media library, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.
[0217] The surface manager is used to manage the display subsystem and provide the fusion of 2D and 3D layers for multiple applications.
[0218] The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.
[0219] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.
[0220] A 2D graphics engine is a drawing engine for 2D drawings.
[0221] The kernel layer is the layer between hardware and software. The kernel layer contains at least display driver, camera driver, audio driver, and sensor driver.
[0222] The above describes a flow chart of a method for generating a video on a smart TV 100. It should be understood that the method for generating a video of the present application can also be implemented on a specific video application platform, that is, the method for generating a video of the present application can be implemented on a server. After the smart TV 100 installs the video application, the user only needs to click on the video application to see the above-mentioned dynamic poster image card point video. It should be understood that the principle of implementing the method for generating a video of the present application on the server 200 is consistent with that of implementing the method for generating a video of the present application on the smart TV 100. The following will briefly introduce the process of implementing the method for generating a video of the present application on the server 200, in which the specific implementation method of each step is consistent with the method in the above-mentioned steps 402 to 418, and will not be repeated.
[0223] Fig.10 FIG. 2 is a flow chart showing a method for implementing the above-mentioned video generation method on the server 200. Fig.10 As shown, method 1000 includes:
[0224] Step 1002: Obtain M poster images and determine the theme label of each poster
[0225] Step 1004, determine the motion range of each poster image foreground object
[0226] Step 1006: obtain music as background music to be selected, divide the background music into audio intervals, and determine a theme tag for each audio interval of the background music.
[0227] Step 1008: determine the motion range of each audio interval in the above-mentioned background music to be selected.
[0228] Step 1010, calculating the matching degree between each of the above poster images and each audio interval of the background music to be selected.
[0229] Step 1012, arrange and combine each of the above poster images and each audio interval of the background music to be selected, and calculate the total matching degree between the poster image and the background music to be selected in each combination.
[0230] Step 1014: determine the highest overall matching degree from the above various combinations.
[0231] Determine whether the highest total matching degree is greater than or equal to the second preset value. When the highest total matching degree is greater than or equal to the second preset value, execute step 1016; when the highest total matching degree is less than the second preset value, execute step 1006.
[0232] Step 1016: Generate a card point video of the dynamic poster image according to the combination method with the highest total matching degree.
[0233] Step 1018, sending the card point video of the dynamic poster image to the smart TV 100, and the smart TV 100 plays the card point video of the dynamic poster image.
[0234] Among them, the execution steps in the above method 1000 that are the same as those in method 400 can refer to the relevant description in method 400 and will not be repeated here.
[0235] In some embodiments, the server 200 establishes a communication connection with the smart TV 100. Figure 8 After the video application (or video application) 300 is opened, the display screen 30 of the smart TV 100 will display the following Figure 1 The interface diagram shown in Figure 1 The movie poster image display area 110 shown displays the dynamic poster image card point video.
[0236] Fig.11 A schematic block diagram of the hardware structure of a server 200 is shown.
[0237] like Fig.11As shown, the server 200 may include a processor 210, an external memory interface 220, an internal memory 221, a SIM card interface 295, a universal serial bus (USB) interface 230, a charging management module 240, a power management module 241, a battery 242, an antenna 1, an antenna 2, a mobile communication module 250, a wireless communication module 260, etc.
[0238] It is to be understood that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the server 200. In other embodiments of the present application, the server 200 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.
[0239] In the above structure constituting the server 200, the processor 210 may include one or more processing units, for example: the processor 210 may include an application processor (application processor, AP), a modem processor, a graphics processor (graphics processing unit, GPU), an image signal processor (image signal processor, ISP), a controller, a video codec, a digital signal processor (digital signal processor, DSP), a baseband processor, and / or a neural-network processing unit (neural-network processing unit, NPU), etc. Among them, different processing units may be independent devices or integrated in one or more processors. The controller may generate an operation control signal according to the instruction opcode and the timing signal to complete the control of fetching and executing instructions.
[0240] The processor 210 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 210 is a cache memory. The memory may store instructions or data that the processor 210 has just used or cyclically used. If the processor 210 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 210, and thus improves the efficiency of the system.
[0241] In some embodiments, the processor 210 may include one or more interfaces. The external memory interface 220 may be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the server 200. The external memory card communicates with the processor 210 via the external memory interface 220 to implement a data storage function. For example, files such as music and videos are stored in an external memory card. In an embodiment of the present application, a user can play a video file stored in an external memory card while recording useful information in the video through the input method of the present application.
[0242] The internal memory 221 can be used to store computer executable program codes, which include instructions. The internal memory 221 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the server 200 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 221 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 210 executes various functional applications and data processing of the server 200 by running instructions stored in the internal memory 221, and / or instructions stored in a memory provided in the processor 210. In an embodiment of the present application, the processor 210 may execute the locking function of the input method application of the server 200 by running instructions stored in the internal memory 221.
[0243] The USB interface 230 is an interface that complies with USB standard specifications, and may specifically be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc.
[0244] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present invention is only a schematic illustration and does not constitute a structural limitation on the server 200. In other embodiments of the present application, the server 200 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.
[0245] The charging management module 240 is used to receive charging input from the charger. The power management module 241 is used to connect the battery 242, the charging management module 240 and the processor 210. The power management module 241 receives input from the battery 242 and / or the charging management module 240, and provides power to the processor 210, the internal memory 221, the display screen (not shown in the figure), the camera (not shown in the figure), and the wireless communication module 260.
[0246] The wireless communication function of the server 200 can be implemented through the antenna 1, the antenna 2, the mobile communication module 250, the wireless communication module 260, the modem processor and the baseband processor.
[0247] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals.
[0248] The mobile communication module 250 can provide wireless communication solutions including 2G / 3G / 4G / 5G etc. applied on the server 200 .
[0249] The wireless communication module 260 can provide wireless communication solutions applied on the server 200, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication technology (NFC), infrared technology (IR), etc.
[0250] The buttons (not shown in the figure) include a power button (not shown in the figure), a volume button (not shown in the figure), etc. The server 200 can receive the button input and generate a key signal input related to the user setting and function control of the server 200.
[0251] An embodiment of the present application also provides an electronic device, which includes: at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor, wherein the processor implements the steps of any of the above-mentioned method embodiments when executing the computer program.
[0252] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.
[0253] An embodiment of the present application provides a computer program product. When the computer program product runs on a mobile terminal, the mobile terminal can implement the steps in the above-mentioned method embodiments when executing the computer program product.
[0254] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device that can carry the computer program code to the camera / terminal device, recording medium, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.
[0255] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0256] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.
[0257] In the embodiments provided in the present application, it should be understood that the disclosed devices / network equipment and methods can be implemented in other ways. For example, the device / network equipment embodiments described above are merely schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0258] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0259] In the above description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.
[0260] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.
[0261] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.
[0262] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.
[0263] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.
[0264] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.
[0265] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.
Claims
1. A method for generating a video, applied to an electronic device, It is characterized in that The method comprises: Dividing the background music used to generate the video into N continuous audio intervals, where N is a positive integer; According to the audio rhythm change of each audio interval and the matching degree between the N images, the N images are matched to the corresponding audio interval to generate the video, wherein during the playback of the video, when each audio interval is played, the image matching each audio interval is displayed; The matching degree includes at least one of the following: a first matching degree between the audio rhythm change of the audio interval and the content of the image; a second matching degree between the audio rhythm change of the audio interval and the duration of the audio interval and the dynamic motion range of the foreground object in the image; And, the second matching degree is calculated in the following manner: Calculating a motion range value corresponding to the audio interval according to the audio rhythm change and the duration of the audio interval; The matching degree between the motion range value corresponding to the audio interval and the value corresponding to the dynamic motion range of the foreground object in the image is calculated to obtain the second matching degree.
2. The method according to claim 1, It is characterized in that The dynamic motion range of the foreground object of the image is calculated as follows: The foreground segmentation neural network model is used to perform foreground segmentation on the image to obtain the foreground object and the background object of the image, and the dynamic motion range of the foreground object in the image is obtained according to the positional relationship of the foreground object relative to the background object.
3. The method according to claim 1, It is characterized in that The first matching degree is calculated by: Performing content recognition on the image using a content recognition neural network model, adding a content tag reflecting the image content to the image according to the recognized content, and adding a rhythm tag reflecting the audio rhythm change to the audio interval according to the audio rhythm change of the audio interval; The matching degree between the content tag and the rhythm tag is calculated to obtain the first matching degree.
4. The method according to claim 1, It is characterized in that The matching of the N images to the corresponding audio intervals according to the audio rhythm changes of the audio intervals and the matching degree between the N images includes: assigning the N images to the N consecutive audio intervals respectively according to a first order, and calculating a plurality of first matching degrees and a plurality of second matching degrees between each audio interval and the assigned image; When the plurality of first matching degrees are all greater than the first matching degree threshold, and the plurality of second matching degrees are all greater than the second matching degree threshold, calculating the sum of the plurality of first matching degrees and the plurality of second matching degrees; and, When the sum of the plurality of first matching degrees and the plurality of second matching degrees is greater than a total matching degree threshold, the N images are matched to the corresponding audio intervals in the first order.
5. The method according to claim 4, It is characterized in that When the plurality of first matching degrees are all greater than a first matching degree threshold, and the plurality of second matching degrees are all greater than a second matching degree threshold, calculating the sum of the plurality of first matching degrees and the plurality of second matching degrees comprises: The sum of a first matching degree and a second matching degree between each audio interval in the N audio intervals and the assigned image is calculated respectively, and when the sum of the first matching degree and the second matching degree between each audio interval and the assigned image is greater than a third matching degree threshold, the sum of multiple first matching degrees and the multiple second matching degrees between the N audio intervals and the assigned image is calculated.
6. The method according to claim 1, It is characterized in that The matching of the N images to the corresponding audio intervals according to the audio rhythm changes of the audio intervals and the matching degree between the N images includes: assigning the N images to the N consecutive audio intervals respectively according to a first order, and calculating a plurality of first matching degrees and second matching degrees between each audio interval and the assigned image; The sum of the multiple first matching degrees and the multiple second matching degrees is calculated, and when the sum of the multiple first matching degrees and the multiple second matching degrees is greater than the total matching degree threshold, the N images are matched to the corresponding audio intervals in the first order.
7. The method according to claim 6, It is characterized in that Before calculating the sum of the multiple first matching degrees and the multiple second matching degrees, the sum of the first matching degree and the second matching degree between each audio interval in the N audio intervals and the assigned image is calculated respectively; when the sum of the first matching degree and the second matching degree between each audio interval and the assigned image is greater than a third matching degree threshold, the sum of the multiple first matching degrees and the multiple second matching degrees between the N audio intervals and the assigned image is calculated.
8. The method according to any one of claims 1 to 7, It is characterized in that The step of dividing the background music used to generate the video into N continuous audio intervals includes: According to the rhythm change of the background music, the background music is divided into the N continuous audio intervals.
9. An electronic device, It is characterized in that include: a memory storing computer program instructions; A processor is coupled to a memory, and when the computer program instructions stored in the memory are executed by the processor, the electronic device executes the video generation method according to any one of claims 1 to 8.
10. A computer readable medium, It is characterized in that The computer-readable medium stores instructions, which, when executed on an electronic device, enable the electronic device to execute the video generation method according to any one of claims 1 to 8.
Citation Information
Patent Citations
Music special effect matching method and device
CN108877838A
Video processing method, electronic equipment and storage medium
CN110602550A
Video music matching method and device, electronic equipment and computer readable medium
CN110677711A
Background audio determination method and device, video editing method and device and computer equipment
CN111901626A
Video music matching method and device, electronic equipment and storage medium
CN112153460A
