Video synthesis method, electronic device, and computer-readable storage medium

Abstract

The application relates to the field of image processing, in particular to a video synthesis method, an electronic device and a computer readable storage medium. The method comprises the following steps: receiving a user instruction, wherein the user instruction carries semantic information of materials used for synthesizing a video; in response to the user instruction, acquiring a first material conforming to the semantic information of the user instruction; extracting a second material from the first material according to time; generating a first video according to the second material; and displaying a first interface, wherein the first interface is a playing interface of the first video. In this way, the synthesized video can evenly display materials of each time period, the effect of the synthesized video is effectively improved, and the user experience is improved. The materials in the embodiment of the application include pictures and videos.

Description

Technical Field

[0001] This application relates to the field of image processing, and more particularly to a video synthesis method, an electronic device, and a computer-readable storage medium. Background Technology

[0002] Currently, many electronic devices are equipped with cameras, allowing users to take photos and videos. These devices also have communication capabilities, enabling users to download images or videos from the internet or receive images or videos sent by other electronic devices. Users can edit images and / or videos stored on their electronic devices to create entirely new videos from selected images and / or videos.

[0003] However, in practical applications, the amount of material acquired by electronic devices based on user instructions may exceed the maximum number of materials required for synthesizing a video. In such cases, it is necessary to select from the acquired material to meet the upper limit requirement for the number of materials in the synthesized video. Summary of the Invention

[0004] This application provides a video synthesis method, an electronic device, and a computer-readable storage medium, which effectively improves the quality of synthesized videos and enhances user experience.

[0005] To achieve the above objectives, this application adopts the following technical solution:

[0006] Firstly, a video synthesis method is provided, including:

[0007] Receive user instructions, the user instructions carrying semantic information of materials used for synthesizing video;

[0008] In response to the user instruction, first material conforming to the semantics of the user instruction is obtained;

[0009] The second material is selected from the first material based on the time.

[0010] A first video is generated based on the second material;

[0011] The first interface is displayed, which is the playback interface for the first video.

[0012] In this embodiment, materials are acquired according to user instructions, and then a certain number of materials are evenly selected from the acquired materials based on time. A new video is then synthesized based on the selected materials. This method allows the synthesized video to display materials from various time periods evenly, effectively improving the quality of the synthesized video and enhancing the user experience.

[0013] The materials used in the embodiments of this application include images and videos.

[0014] In this embodiment, the semantic information of the user instruction may include the material type and the material time. Accordingly, the first material that conforms to the semantics of the user instruction may be a material that conforms to the material type and the material time specified in the user instruction.

[0015] In some implementations, the first interface may include a first control. When a user interacts with the first control, the electronic device plays a first video in response to the user's interaction.

[0016] In one implementation of the first aspect, the step of selecting the second material from the first material based on time includes:

[0017] Count the quantity of the first material;

[0018] If the number of the first material exceeds the preset number, then the second material is selected from the first material based on the time.

[0019] The preset quantity can be determined based on the maximum number of clips in the composite video. For example, if the maximum number of clips in the composite video is 30, the preset quantity can be set to 30 or a value less than 30.

[0020] It is understood that in this embodiment, step S303 is performed under the premise that the number of first materials exceeds a preset number. If the number of first materials exceeds the preset number, it means that there are many first materials, and it is necessary to select the first materials; if the number of first materials does not exceed the preset number, it means that there are few first materials, and in this case, there is no need to select, and the video can be generated based on all the first materials.

[0021] In one implementation of the first aspect, the step of selecting the second material from the first material based on time includes:

[0022] Calculate the time span of the first material;

[0023] The time period for clustering is determined based on the aforementioned time span;

[0024] The first material is clustered according to the time period to obtain material groups;

[0025] The second material is selected from each of the material groups.

[0026] In this embodiment of the application, the time span refers to the time interval between the earliest and latest shooting times of the first footage. For example, if three first footage items are acquired, and their respective shooting times are January 1, 2020, February 2, 2021, and March 3, 2023, then the time span of these three first footage items is the time interval between January 1, 2020, and March 3, 2023.

[0027] In some implementations, the time period can be determined based on the largest time unit contained within the time span. In other implementations, if the time span contains a large number of time periods, several time periods can be merged to determine a new time period.

[0028] In one implementation of the first aspect, determining the clustering time period based on the time span includes:

[0029] Obtain a first preset period, wherein the first preset period is determined based on the maximum number of materials in the synthesized video;

[0030] If the time span is greater than the first preset period, then the time period is determined according to the first preset period;

[0031] If the time span is less than the first preset period, then the time period is determined according to the second preset period, wherein the second preset period is less than the first preset period.

[0032] By using the above methods, a more suitable time period can be determined, which effectively reduces the occurrence of unsuitable material selection caused by a time period that is too large or too small, and helps to improve the quality of subsequent video synthesis.

[0033] In one implementation of the first aspect, the step of selecting the second material from each of the material groups includes:

[0034] Calculate the first selection ratio corresponding to each of the material groups;

[0035] The second material is selected from each of the material groups according to the first selection ratio corresponding to each material group.

[0036] In one implementation, the first selection ratio corresponding to each material group is multiplied by a preset quantity to obtain the selection quantity M1 corresponding to each material group; M1 second materials are selected from the material group.

[0037] In this embodiment, the selection ratio for each material group is determined based on the maximum number of materials in the synthesized video, and then the materials are selected proportionally. This method ensures a more even distribution of the selected materials over time, guaranteeing that a certain proportion of materials can be selected within each time period, thus improving the overall quality of the synthesized video.

[0038] In one implementation of the first aspect, the step of selecting the second material from each of the material groups according to a first selection ratio corresponding to each material group includes:

[0039] If the first selection ratio corresponding to the material group is greater than the preset ratio, then the first selection ratio corresponding to the material group is adjusted to obtain the adjusted second selection ratio.

[0040] The second material is selected from the material group according to the adjusted second selection ratio of the material group.

[0041] In this embodiment, by adjusting the larger sampling ratio, it can be ensured that all materials in each material group are sampled, which helps to improve the effect of the synthesized video.

[0042] In one implementation, the first selection ratio of the material group can be adjusted to a preset selection ratio.

[0043] In one implementation of the first aspect, adjusting the first selection ratio corresponding to the material group to obtain the adjusted second selection ratio includes:

[0044] Get the number of the material groups;

[0045] The adjusted second selection ratio of the material group is calculated based on the number of material groups and the preset multiple.

[0046] This method effectively adjusts the selection ratio of material groups with a large selection ratio to a more suitable selection ratio, thereby reducing the chance that materials from other material groups will not be selected.

[0047] In one implementation of selecting the second material according to the second selection ratio, the number of materials contained in the material group is multiplied by the second selection ratio corresponding to the material group to obtain the selection quantity M2 corresponding to the material group; M2 second materials are randomly selected from the material group.

[0048] In one implementation of the first aspect, the step of selecting the second material from the material group according to the adjusted second selection ratio of the material group includes:

[0049] Select n images of the second material from each of the material groups, where n is a positive integer;

[0050] According to the second selection ratio corresponding to each material group, the second material is selected from the remaining materials in each material group.

[0051] This implementation method is equivalent to a two-round selection process. The first round ensures that the same number of clips are selected from each clip group. The second round then selects clips according to a second selection ratio corresponding to each clip group. This method guarantees that each clip group is selected, making the generated video more in line with user expectations.

[0052] In one implementation of the first aspect, the step of selecting the second material from the remaining materials in each material group according to the second selection ratio corresponding to each material group includes:

[0053] Based on the second selection ratio and the number of remaining materials in the material group, the selection ratio corresponding to the material group is recalculated to obtain the third selection ratio;

[0054] The second material is selected from the remaining materials in the material group according to the third selection ratio.

[0055] Specifically, it can be done according to the formula Calculate the remaining selection quantity. Here, p3 represents the remaining selection quantity, L represents the preset quantity, z represents the number of material groups, n0 represents the total selection quantity for one material group, and nt represents the total number of the first material. The third selection ratio is...

[0056] In one implementation of the first aspect, after counting the number of the first materials, the method further includes:

[0057] If the number of the first materials does not exceed the preset number, then a second video is generated based on the first materials;

[0058] The second interface is displayed, which is the playback interface for the second video.

[0059] In a second aspect, a chip system is provided, the chip system being applied to an electronic device, the chip system including one or more processors, the one or more processors being configured to invoke computer instructions to cause the electronic device to perform the method as described in any one of the first aspects.

[0060] Thirdly, an electronic device is provided, the electronic device including one or more processors and a memory; the memory is coupled to the one or more processors, the memory being used to store computer program code, the computer program code including computer instructions, the one or more processors invoking the computer instructions to cause the electronic device to perform the method as described in any one of the first aspects.

[0061] Fourthly, a computer-readable storage medium is provided, the computer-readable storage medium including instructions that, when executed on an electronic device, cause the electronic device to perform the method as described in any one of the first aspects.

[0062] Fifthly, a computer program product is provided that, when run on an electronic device, enables the electronic device to perform the method described in any of the first aspects. Attached Figure Description

[0063] Figure 1 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application;

[0064] Figure 2 This is a schematic diagram illustrating an application scenario of video synthesis provided in the embodiments of this application;

[0065] Figure 3 This is a flowchart illustrating the video synthesis method provided in an embodiment of this application;

[0066] Figure 4 This is a flowchart illustrating the material selection method provided in the embodiments of this application;

[0067] Figure 5 This is a schematic diagram of the video synthesis process provided in the embodiments of this application;

[0068] Figure 6 This is a schematic diagram of the video synthesis process performed by the electronic device provided in the embodiments of this application. Detailed Implementation

[0069] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limiting purposes, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details.

[0070] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0071] It should also be understood that in the embodiments of this application, "one or more" refers to one, two, or more; "and / or" describes the relationship between the associated objects, indicating that three relationships can exist; for example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following associated objects have an "or" relationship.

[0072] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," "fourth," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0073] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0074] The video synthesis method provided in this application can be applied to electronic devices. These electronic devices can be tablets, mobile phones, wearable devices, smart screens, laptops, virtual reality (VR) terminal devices, augmented reality (AR) terminal devices, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), and other electronic devices. This application does not limit the specific type of electronic device.

[0075] See Figure 1This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0076] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0077] Processor 110 may include one or more processing units, such as an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU). Different processing units may be independent devices or integrated into one or more processors. For example, processor 110 is used to execute the video frame playback method in the embodiments of this application.

[0078] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0079] Internal memory 121 can be used to store computer executable program code, including instructions. Processor 110 executes various functional applications and data processing of electronic device 100 by running the instructions stored in internal memory 121. Internal memory 121 may include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a function (such as image playback). Touch sensor 180K, also called a "touch panel," can be disposed on display screen 194. Touch sensor 180K and display screen 194 together form a touch screen, also called a "touch screen." Touch sensor 180K is used to detect touch operations applied to or near it. Touch sensor can transmit the detected touch operation to application processor to determine the type of touch event. Visual output related to the touch operation can be provided through display screen 194. In other embodiments, touch sensor 180K may also be disposed on the surface of electronic device 100, in a different location than display screen 194.

[0080] The electronic device 100 implements display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information. For example, in this embodiment, the process of rendering YUV data can be implemented using a GPU.

[0081] Display screen 194 is used to display images, videos, etc. Display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a minimized display, a microLED, a micro-OLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, electronic device 100 may include one or N displays 194, where N is a positive integer greater than 1. For example, in the embodiments of this application... Figure 2 or Figure 3 All interfaces shown are displayed on the monitor.

[0082] The above is a detailed description of the embodiments of this application using electronic device 100 as an example. It should be understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on electronic device 100. Electronic device 100 may have more or fewer components than shown in the figures, may combine two or more components, or may have different component configurations. The various components shown in the figures can be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application-specific integrated circuits.

[0083] This application does not specifically limit the structure of the execution entity of the video synthesis method. As long as the code recording a video synthesis method according to this application is executed, communication can be performed according to the video synthesis method provided in this application. For example, the execution entity of the video synthesis method provided in this application can be a functional module in an electronic device capable of calling and executing programs, or a communication device applied in an electronic device, such as a chip.

[0084] Currently, many electronic devices are equipped with cameras, allowing users to take photos and videos. These devices also have communication capabilities, enabling users to download images or videos from the internet or receive images or videos sent by other electronic devices. Users can edit images and / or videos stored on their electronic devices to create entirely new videos from selected images and / or videos.

[0085] As a scenario example, a user inputs a user command into an electronic device. In response to the user command, the electronic device retrieves relevant materials from a gallery application and generates a new video from the retrieved materials.

[0086] However, in practical applications, the amount of material acquired by electronic devices based on user instructions may exceed the maximum number of materials required for synthesizing a video. In such cases, it is necessary to select from the acquired material to meet the upper limit requirement for the number of materials in the synthesized video.

[0087] Based on this, this application provides a video synthesis method. In this embodiment, materials are acquired according to user instructions, then a certain number of materials are evenly selected from the acquired materials according to time, and finally, a new video is synthesized based on the selected materials. This method allows the synthesized video to evenly display materials from various time periods, effectively improving the quality of the synthesized video and enhancing the user experience.

[0088] In this embodiment, the user commands can be text commands. For example, this embodiment can utilize a search box provided by an electronic device or other applications with text command recognition capabilities to invoke the video editing application. Specifically, the video editing application is invoked via text commands so that it acquires materials that conform to the semantics of the text commands and synthesizes a video based on the acquired materials.

[0089] In this embodiment, the user commands can be voice commands. For example, this embodiment can utilize a voice assistant provided by an electronic device or other applications with voice functionality to invoke a video editing application. Specifically, a voice command is used to invoke the video editing application, enabling it to acquire materials that conform to the semantics of the voice command and synthesize a video based on the acquired materials.

[0090] The following example, using a voice assistant on an electronic device to call a video editing application, illustrates the application scenario of the video synthesis method provided in this application.

[0091] See Figure 2 This is a schematic diagram of an application scenario for video synthesis provided in the embodiments of this application.

[0092] Reference Figure 2 As shown in (a), the user speaks "yoyo" within the voice detection range of the electronic device to activate its voice assistant. After detecting the "yoyo" voice, the electronic device responds with the voice message "I'm listening, please speak," and simultaneously displays... Figure 2 The interface shown in (a) is shown in the image. Figure 2In the interface shown in (a), a voice assistant window 21 is displayed, which includes the text message "I'm listening, please speak." The purpose of this voice response and text message is to notify the user that the voice assistant has been activated and to prompt the user to speak a voice command. In practical applications, other voice response content and text message content can be set as needed to achieve the same reminder effect.

[0093] The user responds with the voice message "I'm listening, please speak" and / or the text message "I'm listening, please speak," and within the voice detection range of the electronic device, speaks the voice command "Help me make a video of my daughter dancing from childhood to adulthood." After detecting the voice command "Help me make a video of my daughter dancing from childhood to adulthood" (user command), the electronic device displays the user's voice message "Help me make a video of my daughter dancing from childhood to adulthood" in the voice assistant window 21, and generates a video (first video) based on the images and videos of the daughter dancing stored in the electronic device. After successfully generating the video, it issues the voice message "A video of your daughter dancing from childhood to adulthood has been generated for you," and simultaneously displays the following... Figure 2 The interface shown in (b) is as follows. Figure 2 As shown in (b), the voice assistant window displays the text "A video of your daughter dancing from childhood to adulthood has been generated" and the cover of the generated video (first interface). The cover displays a playback control 22 and the video duration "00:30", indicating that the generated video is 30 seconds long. In practical applications, users can click the playback control 22 to trigger the electronic device to play the video; this application will not use illustrations to illustrate this further.

[0094] In a specific implementation, when the electronic device detects the voice message "Help me make a video of my daughter dancing from childhood to adulthood," it can generate a video according to the video synthesis method provided in the embodiments of this application.

[0095] The following describes the video synthesis method provided in the embodiments of this application.

[0096] See Figure 3 This is a flowchart illustrating the video synthesis method provided in an embodiment of this application. It is intended as an example and not a limitation. Figure 3 As shown, the video synthesis method may include the following steps:

[0097] S301, Receive user instructions, the user instructions carrying semantic information of materials used for synthesizing video.

[0098] As mentioned above, user commands can be text commands or voice commands. In some implementations, if the user command is a text command, a text recognition algorithm can be used to recognize the semantic information of the user command. If the user command is a voice command, a voice recognition algorithm can be used to recognize the semantic information of the user command. In this application embodiment, no specific limitation is made on the recognition algorithm for the user command.

[0099] The materials used in the embodiments of this application include images and videos.

[0100] S302, in response to the user instruction, the electronic device acquires first material that conforms to the semantics of the user instruction.

[0101] In this embodiment, the semantic information of the user instruction may include the material type and the material time. Accordingly, the first material that conforms to the semantics of the user instruction may be a material that conforms to the material type and the material time specified in the user instruction.

[0102] In some implementations, electronic devices can identify the search range of materials based on user instructions, and then obtain the first material that matches the semantics of the user instructions based on the search range.

[0103] In one example, such as Figure 2 In the example, the user command is "Help me make a video of my daughter dancing from childhood to adulthood." Based on this user command, the electronic device recognizes that the search scope for materials is the gallery application. Therefore, the electronic device can obtain the first material that matches the semantics of the user command from the gallery application.

[0104] In another example, the user command is "today's news video". Based on this user command, the electronic device recognizes that the search scope of the material is web pages. Then, the electronic device searches the web pages for the first material that matches the semantics of the user command.

[0105] In other implementations, the electronic device can first search for a first image that matches the semantics of the user's command in a gallery application. If the first image is not found in the gallery application, it can then search for the first image on a webpage.

[0106] S303, Select the second material from the first material based on the time.

[0107] In some embodiments, step S303 may include:

[0108] Count the quantity of the first material;

[0109] If the number of the first material exceeds the preset number, then the second material is selected from the first material based on the time.

[0110] The preset quantity can be determined based on the maximum number of clips in the composite video. For example, if the maximum number of clips in the composite video is 30, the preset quantity can be set to 30 or a value less than 30.

[0111] It is understood that in this embodiment, step S303 is performed under the premise that the number of first materials exceeds a preset number. If the number of first materials exceeds the preset number, it means that there are many first materials, and it is necessary to select the first materials; if the number of first materials does not exceed the preset number, it means that there are few first materials, and in this case, there is no need to select, and the video can be generated based on all the first materials.

[0112] In some embodiments, if the number of the first materials does not exceed a preset number, a second video is generated based on the first materials; a second interface is displayed, which is the playback interface of the second video.

[0113] For details on how to extract the second material from the first material based on time, please refer to [link / reference]. Figure 4 Description in the embodiments.

[0114] S304, Generate a first video based on the second material.

[0115] In some implementations, the second source material can be used to generate the first video based on its category. For example, if the second source material includes both images and videos, the first video can be generated by placing the images first, followed by the videos.

[0116] In other implementations, the first video can be generated from the second source material according to the chronological order of the source materials. For example, if the second source material includes image a and video b, and image a was captured before video b, the first video can be generated in the order of image a first, followed by video b.

[0117] In this embodiment of the application, no specific limitation is made on the method of generating video from the material.

[0118] S305, Display the first interface, which is the playback interface of the first video.

[0119] In some implementations, the first interface may include a first control. When a user interacts with the first control, the electronic device plays a first video in response to the user's interaction.

[0120] In this embodiment, materials are acquired according to user instructions, and then a certain number of materials are evenly selected from the acquired materials based on time. A new video is then synthesized based on the selected materials. This method allows the synthesized video to display materials from various time periods evenly, effectively improving the quality of the synthesized video and enhancing the user experience.

[0121] The following describes how to extract the second material from the first material based on time in step S303.

[0122] See Figure 4 This is a flowchart illustrating the material selection method provided in the embodiments of this application. Figure 4 As shown, methods for selecting materials based on time can include:

[0123] S401, calculate the time span of the first material.

[0124] In this embodiment of the application, the time span refers to the time interval between the earliest and latest shooting times of the first footage. For example, if three first footage items are acquired, and their respective shooting times are January 1, 2020, February 2, 2021, and March 3, 2023, then the time span of these three first footage items is the time interval between January 1, 2020, and March 3, 2023.

[0125] S402, determine the time period for clustering based on the time span.

[0126] In some implementations, the time period can be determined based on the largest time unit encompassed by the time span. For example, if the first footage was shot in different years, the largest time unit encompassed by the time span is a year, so the time period is one year. If the first footage was shot in the same year but different months, the largest time unit encompassed by the time span is a month, so the time period is one month. If the first footage was shot in the same month but different days, the largest time unit encompassed by the time span is a day, so the time period is one day. If the first footage was shot on the same day but at different times, the largest time unit encompassed by the time span is an hour, so the time period is one hour.

[0127] In other implementations, if the time span contains a large number of time periods, several time periods can be merged to determine a new time period. For example, if the first set of footage was shot in different years, with a time span of 10 years and a time period of one year, then the time span is equivalent to containing 10 time periods, which is quite a lot. In this case, two time periods can be merged, resulting in a new time period of 2 years.

[0128] In some other implementations, step S402 may include:

[0129] Obtain a first preset period, wherein the first preset period is determined based on the maximum number of materials in the synthesized video;

[0130] If the time span is greater than the first preset period, then the time period is determined according to the first preset period;

[0131] If the time span is less than the first preset period, then the time period is determined according to the second preset period, wherein the second preset period is less than the first preset period.

[0132] Specifically, if the time span is greater than the first preset period, the time period is determined according to the time unit corresponding to the first preset period; if the time span is less than the first preset period, the second preset period is obtained; if the time span is greater than the second preset period and less than the first preset period, the time interval is determined according to the second preset period; if the time span is less than the second preset period, the third preset period is obtained, and so on. The third preset period is less than the second preset period.

[0133] For example, suppose the maximum number of clips for the synthesized video is 30. To ensure that one clip is selected each month, the number of clips for three years is 36 > 30, and the number of clips for two years is 24 < 30. Therefore, the first preset period can be set to 2 years. If the time span is greater than 2 years, clustering is performed on a yearly basis, i.e., the time period is set to 1 year. If the time span is less than 2 years, a second preset period is obtained, such as 1 month. If the time span is greater than 1 month, clustering is performed on a monthly basis, i.e., the time period is set to 1 month. If the time span is less than 1 month, a third preset period is obtained, such as 1 day. If the time span is greater than 1 day, clustering is performed on a daily basis, i.e., the time period is set to 1 day. If the time span is less than 1 day, a fourth preset period is obtained, and so on.

[0134] S403, cluster the first material according to the time period to obtain a material group.

[0135] In the specific implementation, the time span of the first material is divided into multiple time ranges according to the time period; the time range to which each first material belongs is determined; and the first materials belonging to the same time range are grouped into a material group.

[0136] In some implementations, different data storage structures can be used to store the material groups for different time periods. For example, if the time period is one year, the storage structure could be map[year, list]. <mediaitem>For example, [2019, list] or [2022, list]. If the time period is monthly, the storage structure can be map[month, list]. <mediaitem>For example, [201901, list] or [202202, list]. If the time period is daily, the storage structure can be map[day, list]. <mediaitem>], such as [20190108, list], [20220202, list].

[0137] It should be noted that the above is an example of the data storage structure of the material group, and the embodiments of this application do not specifically limit it.

[0138] S404, Select the second material from each of the material groups.

[0139] Figure 4 In this embodiment, the materials are clustered according to their time span, and then materials are extracted from each of the resulting clusters. This method ensures that the synthesized video displays materials from different time periods in a balanced way, effectively improving the overall quality of the synthesized video and enhancing the user experience.

[0140] In some implementations of step S404, a second material can be randomly selected from each material group. However, this method can easily lead to uneven time distribution of the selected materials, making it impossible to guarantee that materials can be selected in every time period.

[0141] To address this issue, in some embodiments, step S404 may include:

[0142] I. Calculate the first sampling ratio corresponding to each of the material groups.

[0143] II. Select the second material from each of the material groups according to the first selection ratio corresponding to each material group.

[0144] In this embodiment, the selection ratio for each material group is determined based on the maximum number of materials in the synthesized video, and then the materials are selected proportionally. This method ensures a more even distribution of the selected materials over time, guaranteeing that a certain proportion of materials can be selected within each time period, thus improving the overall quality of the synthesized video.

[0145] In one implementation of step I, the number of first materials in each material group is counted; the ratio of the number of first materials in each material group to the total number of first materials is calculated to obtain the first selection ratio for each material group. For example, if the total number of first materials obtained is 1000, and they are clustered into 3 material groups, each containing 900, 60, and 40 first materials respectively, the first selection ratio for the first material group is 900 / 1000 = 0.9, the first selection ratio for the second material group is 60 / 1000 = 0.06, and the first selection ratio for the third material group is 40 / 1000 = 0.04.

[0146] Accordingly, in one implementation of step II, the first selection ratio corresponding to each material group is multiplied by a preset quantity to obtain the selection quantity M1 corresponding to each material group; M1 second materials are selected from the material group. Continuing the above example, assume that the maximum number of materials in the synthesized video is 30, that is, the preset quantity is 30. Multiplying the preset quantity by the first selection ratio of the first material group, we get the selection quantity M1 of the first material group = 0.9 × 30 = 27; multiplying the preset quantity by the first selection ratio of the second material group, we get the selection quantity M1 of the second material group = 0.06 × 30 = 1.8; multiplying the preset quantity by the first selection ratio of the third material group, we get the selection quantity M1 of the third material group = 0.04 × 30 = 1.2.

[0147] In some implementations, if the calculated number of selections is not an integer, it can be rounded up or down to obtain an integer number of selections.

[0148] As can be seen from the above examples, if a certain material group contains a large number of materials, there will be a situation where a large number of materials are selected from a certain material group, while the number of materials selected from other material groups is small (or even 0). This results in the selected materials being concentrated in time, which affects the effect of subsequent videos.

[0149] To address this issue, in some embodiments, step II may include:

[0150] If the first selection ratio corresponding to the material group is greater than the preset ratio, then the first selection ratio corresponding to the material group is adjusted to obtain the adjusted second selection ratio; the second material is selected from the material group according to the adjusted second selection ratio of the material group.

[0151] In practical applications, the preset ratio can be set in advance based on experience. It can also be adjusted accordingly during actual calculations. For example, assuming the preset quantity (determined based on the maximum number of materials in the composite video) is 100, and there are 3 material groups, to ensure that at least one material is selected from each material group, the first selection ratio for each material group must be at least 1 / 100 = 0.01. Therefore, the preset ratio can be set to 0.98. If the first selection ratio of the first material group is 0.99, then one of the other two material groups will inevitably fail to reach a selection ratio of 0.01. In this case, the first selection ratio of the first material group is determined to be greater than the preset ratio.

[0152] In one implementation, the first selection ratio of the material group can be adjusted to a preset selection ratio.

[0153] In another implementation, the number of material groups is obtained; the adjusted second selection ratio of the material groups is calculated based on the number of material groups and a preset multiple.

[0154] Specifically, the ratio of the preset multiplier to the number of material groups can be determined as the second selection ratio.

[0155] For example, assuming the preset multiplier is 2, the time span is 5 years, and the time period is 1 year, that is, the number of material groups is 5 / 1 = 5, then the second selection ratio is 2 / 5 = 0.4.

[0156] The preset multiplier can be set according to actual needs. In one example, the preset multiplier can be set to a value less than half the number of media sets.

[0157] Correspondingly, other material groups whose first selection ratio is less than the preset ratio also need to have their selection ratios adjusted accordingly. In one implementation, the remaining ratio is calculated based on the adjusted second selection ratio of the first target group (material groups whose first selection ratio is greater than the preset ratio); the remaining ratio is allocated to the second target group based on the number of materials in the second target group (material groups whose first selection ratio is less than the preset ratio), thus obtaining the adjusted second selection ratio for each second target group.

[0158] For example, the total number of first materials obtained is 1000, clustered into 3 material groups, each containing 900, 60, and 40 first materials respectively. The first selection ratio for the first material group is 900 / 1000 = 0.9, the first selection ratio for the second material group is 60 / 1000 = 0.06, and the first selection ratio for the third material group is 40 / 1000 = 0.04. The selection ratio for the first material group is adjusted from 0.9 to 0.4, resulting in a remaining ratio of 1 - 0.4 = 0.6. The other two material groups (the second target group) contain 60 and 40 materials respectively. Based on the proportional relationship of the material quantities, the remaining ratio is allocated, resulting in an adjusted selection ratio of 60 × 0.6 / (60 + 40) = 0.36 for the second material group and 40 × 0.6 / (60 + 40) = 0.24 for the third material group.

[0159] In this embodiment, by adjusting the larger sampling ratio, it can be ensured that all materials in each material group are sampled, which helps to improve the effect of the synthesized video.

[0160] In one implementation of selecting the second material according to the second selection ratio, the number of materials contained in the material group is multiplied by the second selection ratio corresponding to the material group to obtain the selection quantity M2 corresponding to the material group; M2 second materials are randomly selected from the material group.

[0161] In another implementation of selecting the second material according to the second selection ratio, n images of the second material are selected from each material group, where n is a positive integer; and the second material is selected from the remaining materials in each material group according to the second selection ratio corresponding to each material group.

[0162] This implementation method is equivalent to a two-round selection process. The first round ensures that the same number of clips are selected from each clip group. The second round then selects clips according to a second selection ratio corresponding to each clip group. This method guarantees that each clip group is selected, making the generated video more in line with user expectations.

[0163] In one implementation, the step of selecting the second material from the remaining materials may include:

[0164] Based on the second selection ratio and the number of remaining materials in the material group, the selection ratio corresponding to the material group is recalculated to obtain the third selection ratio; the second material is selected from the remaining materials in the material group according to the third selection ratio.

[0165] Specifically, it can be done according to the formula Calculate the remaining selection quantity. Here, p3 represents the remaining selection quantity, L represents the preset quantity, z represents the number of material groups, n0 represents the total selection quantity for one material group, and nt represents the total number of the first material. The third selection ratio is...

[0166] To more clearly illustrate the video synthesis method of this application's embodiments, the overall flow of the video synthesis method is described below. See also... Figure 5 This is a schematic diagram of the video synthesis process provided in an embodiment of this application. It is intended as an example and not a limitation. Figure 5 As shown, the video compositing process may include the following steps:

[0167] S501, retrieve user instructions.

[0168] S502, obtain the first material according to the semantics of the user's instructions.

[0169] Steps S501-S502 can be found in the description of steps S301-S302 in the embodiment.

[0170] S503, calculate the time span of the first material.

[0171] S504, determine whether the time span is greater than 2 years (first preset period).

[0172] S505: If the time span is greater than 2 years, clustering is performed on a yearly basis, i.e., the time period is 1 year.

[0173] If the time span is less than 2 years, clustering is performed on a monthly basis, meaning the time period is one month. Since the subsequent material selection process is the same as for materials with a one-year time period, Figure 5 This situation is not shown in the text.

[0174] Steps S503-S504 can be found in the description of steps S401-S403 in the embodiment.

[0175] S506, determine whether the quantity of the first material is greater than the preset quantity.

[0176] It should be noted that steps S506 and S503 can be performed in any order, in parallel or in sequence. In the case of serial processing, S503 can be executed first and then S506, or S506 can be executed first and then S503.

[0177] S507: If the number of first materials is not greater than the preset number, then generate the second video based on all the first materials.

[0178] S508, if the quantity of the first material is greater than the preset quantity, then check whether the first sampling ratio of the material group is greater than the preset ratio.

[0179] S509, if the first selection ratio of each material group is less than the preset ratio, then materials are selected from each material group according to the first selection ratio.

[0180] S510 If the first selection ratio of a certain material group is greater than the preset ratio, then adjust the selection ratio of the material group to obtain the second selection ratio.

[0181] S511, Select materials from each material group according to the adjusted second selection ratio.

[0182] Steps S508-S511 can be found in the description of step S404 in the embodiment.

[0183] S512, generate a video based on the selected second source material.

[0184] The material selection method in the above embodiments is illustrated below with specific examples.

[0185] For example, assuming a preset quantity of 30, there are 5 material groups, each containing 900, 30, 30, 20, and 20 first materials respectively, for a total of 1000 first materials. The first selection ratios for each material group are calculated as 900 / 1000 = 0.9, 30 / 1000 = 0.03, 30 / 1000 = 0.03, 20 / 1000 = 0.02, and 20 / 1000 = 0.02. Assuming a preset ratio of 0.5, the selection ratio for the material group with 900 materials exceeds the preset ratio, requiring adjustment of the selection ratio for this material group (the first target group). The adjusted second selection ratio is 2 / 5 = 0.4 (preset multiple is 2). Accordingly, the selection ratios for other material groups (second target group) are adjusted. The adjusted second selection ratios for the five material groups are 0.4, 30×0.6 / (30+30+20+20)=1.8, 30×0.6 / (30+30+20+20)=1.8, 20×0.6 / (30+30+20+20)=1.2, and 20×0.6 / (30+30+20+20)=1.2.

[0186] After determining the second selection ratio for each material group, first select one material from each material group. Then, according to the second selection ratio for each material group, select a second material from the remaining materials in each material group. Specifically, for the first material group, the second selection ratio is 0.4, the total number of materials selected is 900 × 0.4 = 360, and the remaining number of materials selected is... Then, nine second-generation materials are selected from the remaining materials in the first material group. The selection method for other material groups is similar and will not be repeated here.

[0187] To better understand the execution process of the steps in the flowchart above, let's first look at... Figure 6 This describes a sequence diagram depicting how an electronic device generates videos related to its daughter's dancing after detecting a user's request to "make videos of my daughter dancing from childhood to adulthood." This sequence diagram corresponds to... Figure 2 The application scenarios described in the examples.

[0188] S601, the voice assistant detected the user's voice.

[0189] In this embodiment of the application, the user's voice can be "generate a video of the child".

[0190] S602: The voice assistant obtains video generation instructions based on the voice information corresponding to the user's voice.

[0191] In this embodiment, after the voice assistant detects the user's voice, it can first generate text information based on the user's voice, and then extract the keywords "generate", "daughter", "dance", "from childhood to adulthood" and "video" from the text information to obtain the video generation instruction.

[0192] Of course, in practical applications, after the voice assistant generates text information based on the user's voice, it can display the text "Help me make a video of my daughter dancing from childhood to adulthood" in the voice assistant window.

[0193] S603: The voice assistant sends video generation commands to the media processing platform.

[0194] S604: After receiving a video generation instruction, the media processing platform collects media materials related to the instruction from the electronic device.

[0195] As an example, the video generation instruction carries the material types "image" and "video," as well as the content identifier "daughter." The media processing platform determines that the materials to be searched are images and videos based on the material types "image" and "video"; and determines that the search content identifier "daughter" is images and videos with the content identifier "daughter." Then, image detection processing is performed on the images and videos with the content identifier "daughter" to detect the posture of the people in the images and videos, thereby determining whether the people in the images and videos are in a dancing posture.

[0196] Of course, in practical applications, you can set first-level content tags and second-level content tags for videos and images. The first-level content tag can be a person, and the second-level content tags can be attributes such as the person's actions or posture.

[0197] Additionally, you can set a maximum number of media clips (e.g., 30) for each video generation, meaning that the maximum number of media clips collected when generating a video is 30.

[0198] S605, the media processing platform selects from the acquired materials.

[0199] For the material selection method, please refer to the above. Figure 4 Description in the embodiments.

[0200] S606, the media processing platform sends the selected media materials (secondary materials) and themes (content identifier itself or themes determined based on the content identifier) ​​to the light editing service.

[0201] As an example, the topic could be the content tag "daughter," or it could be the topic "daughter dancing," determined by the first-level and second-level content tags. Of course, this is just one example of how to determine the topic.

[0202] Lightweight editing services can be an example of video editing applications.

[0203] S607, after receiving media materials and themes, the light editing service obtains the configuration file.

[0204] In this embodiment, the configuration file defines the nodes in the finished video module that need to be created, and the video can be generated subsequently through the nodes in the finished video module.

[0205] As an example, the nodes in the configuration file include: template retrieval node, theme matching node, template parsing node, music parsing node, music loading node, special effects node, cover generation node, duration adjustment node, etc.

[0206] The template acquisition node is used to acquire template materials; the theme matching node is used to find templates that match the theme from the template materials; the template parsing node is used to parse templates that match the theme; the music parsing node is used to parse the rhythm points in the template; the music loading node is used to match the rhythm points in the music with the rhythm points in the template; the special effects node is used to generate special effects; the cover generation node is used to generate video covers; and the duration adjustment node is used to calculate the duration of the new video and adjust the duration of each segment in the original video during the secondary editing process to obtain the new video.

[0207] It is understandable that these nodes have a specific processing order.

[0208] S608, the lightweight editing service creates block modules (and their individual nodes) based on configuration files.

[0209] In this step, the duration adjustment node is not created yet. It will be created when the video duration needs to be adjusted. Of course, in practical applications, the configuration file can also be configured not to include the duration adjustment node.

[0210] S609, the light editing service sends media materials and themes to the finished product module after it is created.

[0211] S610: After receiving media materials and themes, the finished product module obtains the video model through various nodes.

[0212] Corresponding to each created node: the template retrieval node retrieves template materials; the theme matching node searches for templates that match the theme from the template materials; the template parsing node parses the templates that match the theme; the music parsing node parses the rhythm points in the template; the music loading node matches the rhythm points in the music with the rhythm points in the template; the effects node is used to generate effects; and the cover generation node is used to generate the video cover. After processing by the above nodes, the video model is obtained, which is a video structure.

[0213] This process is just an example. In practical applications, different nodes can be set, or more or fewer nodes. Regardless of the type of nodes used, the video module can obtain the video model by passing through these nodes.

[0214] S611, the finished product module sends the video model to the light editing service.

[0215] S612, after receiving the video model, the light editing service converts the video model into video JSON.

[0216] The lightweight editing service processes videos using a model structure; after processing, it needs to be converted to a JSON structure for transmission.

[0217] S613, the light editing service sends video JSON to the media processing platform.

[0218] S614: After receiving the video JSON, the media processing platform sends the video JSON to the voice assistant.

[0219] After receiving the video JSON, the voice assistant in the S615 displays the text "Videos of your daughter dancing from childhood to adulthood" and a video cover in the voice assistant window.

[0220] In summary, the video synthesis method provided in this application acquires materials according to user instructions, then selects a certain number of materials evenly from the acquired materials based on time, and then synthesizes a new video based on the selected materials. This method ensures that the synthesized video can evenly display materials from various time periods, effectively improving the quality of the synthesized video and enhancing the user experience. Secondly, the materials are clustered according to their time span, and materials are then selected from each clustered material group. This method also ensures that the synthesized video can evenly display materials from various time periods, effectively improving the quality of the synthesized video and enhancing the user experience. Furthermore, by adjusting a larger selection ratio, it is ensured that materials in each material group are selected, further improving the quality of the synthesized video.

[0221] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0222] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, can implement the steps in the above-described method embodiments.

[0223] This application also provides a computer program product that, when run on an electronic device, enables the electronic device to perform the steps described in the various method embodiments above.

[0224] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying the computer program code to the first device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0225] This application also provides a chip system, which includes a processor coupled to a memory. The processor executes a computer program stored in the memory to implement the steps of any method embodiment of this application. The chip system can be a single chip or a chip module composed of multiple chips.

[0226] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0227] Those skilled in the art will recognize that the units and method steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application. Finally, it should be noted that the above descriptions are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions within the technical scope disclosed in this application should be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.< / mediaitem> < / mediaitem> < / mediaitem>

Claims

1. A video synthesis method, characterized in that, include: Receive user instructions, the user instructions carrying semantic information of materials used for synthesizing video; In response to the user instruction, first material conforming to the semantics of the user instruction is obtained; The second material is selected from the first material based on the time. A first video is generated based on the second material; The first interface is displayed, which is the playback interface for the first video; The step of selecting the second material from the first material based on time includes: calculating the time span of the first material; determining the clustering time period based on the time span; clustering the first material according to the time period to obtain material groups; and selecting the second material from each material group.

2. The method according to claim 1, characterized in that, The step of selecting the second material from the first material based on time includes: Count the quantity of the first material; If the number of the first material exceeds the preset number, then the second material is selected from the first material based on the time.

3. The method according to claim 1 or 2, characterized in that, Determining the clustering time period based on the time span includes: Obtain a first preset period, wherein the first preset period is determined based on the maximum number of materials in the synthesized video; If the time span is greater than the first preset period, then the time period is determined according to the first preset period; If the time span is less than the first preset period, then the time period is determined according to the second preset period, wherein the second preset period is less than the first preset period.

4. The method according to claim 1 or 2, characterized in that, The step of selecting the second material from each of the material groups includes: Calculate the first selection ratio corresponding to each of the material groups; The second material is selected from each of the material groups according to the first selection ratio corresponding to each material group.

5. The method according to claim 4, characterized in that, The step of selecting the second material from each of the material groups according to the first selection ratio corresponding to each material group includes: If the first selection ratio corresponding to the material group is greater than the preset ratio, then the first selection ratio corresponding to the material group is adjusted to obtain the adjusted second selection ratio. The second material is selected from the material group according to the adjusted second selection ratio of the material group.

6. The method according to claim 5, characterized in that, The step of adjusting the first selection ratio corresponding to the material group to obtain the adjusted second selection ratio includes: Get the number of the material groups; The adjusted second selection ratio of the material group is calculated based on the number of material groups and the preset multiple.

7. The method according to any one of claims 5 to 6, characterized in that, The step of selecting the second material from the material group according to the adjusted second selection ratio of the material group includes: Select n images of the second material from each of the material groups, where n is a positive integer; According to the second selection ratio corresponding to each material group, the second material is selected from the remaining materials in each material group.

8. The method according to claim 7, characterized in that, The step of selecting the second material from the remaining materials in each material group according to the second selection ratio corresponding to each material group includes: Based on the second selection ratio and the number of remaining materials in the material group, the selection ratio corresponding to the material group is recalculated to obtain the third selection ratio; The second material is selected from the remaining materials in the material group according to the third selection ratio.

9. The method according to claim 2, characterized in that, After counting the quantity of the first material, the method further includes: If the number of the first materials does not exceed the preset number, then a second video is generated based on the first materials; The second interface is displayed, which is the playback interface for the second video.

10. An electronic device, characterized in that, The electronic device includes one or more processors and memory; The memory is coupled to the one or more processors, the memory being used to store computer program code, the computer program code including computer instructions, the one or more processors invoking the computer instructions to cause the electronic device to perform the method as described in any one of claims 1 to 9.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that, when executed on an electronic device, cause the electronic device to perform the method as described in any one of claims 1 to 9.

12. A chip system, characterized in that, The chip system is applied to an electronic device, the chip system including one or more processors, the one or more processors being used to invoke computer instructions to cause the electronic device to perform the method as described in any one of claims 1 to 9.

Images