Visual media data processing method, program product, storage medium and electronic device

Abstract

The present application relates to the field of computer technology, and discloses a method for processing visual media data, a program product, a storage medium, and an electronic device. The method includes: obtaining visual media data to be processed and user prompt information, the visual media data to be processed includes at least one video to be processed, wherein at least one video to be processed includes a first target video segment and a second target video segment with a first arrangement; based on the user's prompt information, generating and displaying a target video after splicing the visual media data to be processed, wherein, in the target video, the first target video segment and the second target video segment that belong to the same category corresponding to the prompt information are arranged continuously, which is different from the first arrangement of the first target video segment and the second target video segment in the video to be processed. In this way, in the process of generating the target video, the user's intention is taken into consideration, and the needs of a large number of users for personalized and diverse videos can be met.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a visual media data processing method, program product, storage medium and electronic device. Background Art

[0002] At present, the gallery of electronic devices usually stores videos and pictures taken or downloaded by users. Based on the videos and pictures selected by the user in the gallery, the electronic device can extract multiple highlights in the video and arrange and merge the multiple highlights and pictures to automatically generate a short new video, wherein the highlights include important information of the video, such as the main characters, scenes or events in the video. The user can publish the new video as a video log (vlog), or keep it as an album as a souvenir, or for other purposes.

[0003] When a user selects a video, the existing visual media data processing method can only arrange and merge the highlights in the video according to the time sequence of the highlights in the video to generate a new video. When a user selects multiple videos, the existing visual media data processing method can only arrange and merge the highlights in each video according to the order selected by the user and the time sequence of the highlights in each video to generate a new video. Therefore, it is difficult for the existing visual media data processing method to generate personalized and diverse videos. Summary of the invention

[0004] In order to solve the above technical problems, the embodiments of the present application provide a visual media data processing method, a program product, a storage medium and an electronic device.

[0005] In a first aspect, an embodiment of the present application provides a visual media data processing method, which is applied to an electronic device, the method comprising: obtaining visual media data to be processed and user prompt information, the visual media data to be processed comprising at least one video to be processed, wherein at least one video to be processed comprises a first target video segment and a second target video segment having a first arrangement; based on the user's prompt information, generating and displaying a target video after splicing the visual media data to be processed, wherein, in the target video, the first target video segment and the second target video segment have a second arrangement different from the first arrangement, and the second arrangement comprises: the first target video segment and the second target video segment are arranged continuously, and the first target video segment and the second target video segment belong to the same category corresponding to the prompt information.

[0006] The visual media data processing method provided in the embodiment of the present application performs splicing processing on the visual media data based on the user's prompt information to generate and display the target video. In this way, in the process of generating the target video, the user's intention is taken into account, and the needs of a large number of users for video personalization and diversity can be met.

[0007] It can be understood that the second arrangement of the target video segments in the target video is different from the first arrangement of the target video segments in the video to be processed. For the target video segments belonging to the same video to be processed, the first arrangement may be to arrange the target video segments in the order of precedence in the video to be processed; for the target video segments belonging to different videos to be processed, the first arrangement may be to arrange the target video segments in the order selected by the user. The second arrangement may be to arrange the target video segments by category, and the target video segments belonging to the same category corresponding to the user's prompt information are arranged continuously.

[0008] In a possible implementation of the first aspect, corresponding to the first target video segment and the second target video segment belonging to the same video to be processed, the first arrangement method includes: there is a third target video segment between the first target video segment and the second target video segment, or the order of the first target video segment and the second target video segment in the video to be processed is different from the order of the first target video segment and the second target video segment in the target video.

[0009] It can be understood that when the first target video segment and the second target video segment belong to the same video to be processed, the first target video segment and the second target video segment corresponding to the user's prompt information belonging to the same category may not be arranged continuously in the video to be processed.

[0010] In a possible implementation of the first aspect, at least one video to be processed includes a first video to be processed and a second video to be processed, the first target video segment belongs to the first video to be processed, the second target video segment belongs to the second video to be processed, and the first arrangement includes the first video to be processed being before the second video to be processed, and the second arrangement includes the first target video segment being after the second target video segment in the target video.

[0011] It can be understood that when the first target video segment and the second target video segment belong to different videos to be processed, the order of the first target video segment and the second target video segment belonging to the same category of prompt information corresponding to the user may be different from the order of the videos to be processed to which they belong.

[0012] In a possible implementation of the first aspect, the first video to be processed and the second video to be processed are arranged in a sequence selected by a user.

[0013] In a possible implementation of the first aspect, based on the user's prompt information, a target video is generated and displayed after the visual media data to be processed are spliced, including: determining multiple first target frames of each visual media data, wherein corresponding to the visual media data being the video to be processed, different first target frames represent different target video segments in the video to be processed, and different target video segments display different video contents, and corresponding to the visual media data being an image, the first target frame is the image; clustering the first target frames based on the user's prompt information, wherein the image features of the first target frames belonging to the same class belong to the same class corresponding to the prompt information; based on the clustering results, splicing the target video segments corresponding to the first target frames to obtain the target video, wherein in the target video, the target video segments belonging to the same class are arranged continuously.

[0014] In the embodiment of the present application, after determining a plurality of first target frames of each visual media data, the first target frames are clustered based on the user's prompt information, and based on the clustering result, the target video segments belonging to the same category are continuously arranged and spliced ​​to obtain the target video. In this way, the user's prompt information is taken into consideration, and a personalized target video can be generated.

[0015] It can be understood that, corresponding to the visual media data being the video to be processed, since the video to be processed is composed of multiple video frames, it is difficult to directly determine the target video segments and perform clustering according to the video content. Therefore, the first target frame representing different target video segments can be determined, and then the first target frame can be clustered, so that the target video segments corresponding to the first target frame can be clustered according to the video content. For images, they can be directly used as the first target frame for clustering analysis.

[0016] In a possible implementation of the first aspect, multiple first target frames of each visual media data are determined, including: corresponding to the visual media data being a video to be processed, based on the video content of the video to be processed, obtaining the semantic boundary of the video to be processed; based on the semantic boundary of the video to be processed, dividing the video to be processed into multiple video segments; extracting the key frames of each video segment, and selecting the first target frame corresponding to the video segment from the key frames of the video segment.

[0017] It can be understood that after segmenting the video to be processed based on the semantic boundary of the video to be processed, video clips showing different video contents can be obtained. By extracting the key frames of each video clip, the first target frame representing each video clip can be determined from the key frames of each video clip.

[0018] In a possible implementation of the first aspect, key frames of each video clip are extracted, and first target frames corresponding to the video clips are selected from the key frames of the video clips, including: extracting key frames of each video clip, and selecting second target frames corresponding to the video clips from the key frames of the video clips; performing image quality evaluation on the second target frames to obtain quality scores of each second target frame; and selecting second target frames whose quality scores are greater than a score threshold from the second target frames as first target frames.

[0019] It can be understood that in the process of determining the first target frame, the second target frame corresponding to each video segment can be selected from the video segments first, and then the second target frames with poor image quality are filtered out to determine the first target frame. In this way, the video quality of each target video segment can be guaranteed to be relatively high, so that the video quality of the target video finally generated is higher and better meets the needs of users.

[0020] In a possible implementation of the first aspect, corresponding to the visual media data being a video to be processed, the target video segment corresponding to the first target frame is determined based on the following method: multiple frame images adjacent to and before the first target frame are obtained, and the multiple frame images and the first target frame are used as the target video segment.

[0021] It can be understood that the video clip composed of the multiple frames of images adjacent to the first target frame before and after the first target frame in the video to be processed and the first target frame can represent the main video content of the video to be processed, and thus can be used as the target video clip. In this way, the generated target video can only include the main video content of the video to be processed, and does not include unimportant video content in the video to be processed, such as unclear video content, transition content, etc., so that the video quality of the target video is higher and more in line with user needs.

[0022] In a possible implementation of the first aspect, multiple first target frames are clustered based on user prompt information, including: determining a text description of each first target frame based on the prompt information; and clustering the multiple first target frames based on the text description of each first target frame and an image feature of each first target frame.

[0023] It can be understood that after determining the text description of each first target frame based on the user's prompt information, clustering the first target frames according to the text description can effectively control the clustering result so that the clustering result is more consistent with the user's assumption. In addition, clustering based on the text description and image features of each first target frame, while taking into account the text information and image feature information, can make the classification of the first target frame more accurate, that is, the clustering result is more accurate.

[0024] In a possible implementation of the first aspect, multiple first target frames are clustered based on text descriptions of each first target frame and image features of each first target frame, including: inputting the text descriptions of each first target frame and each first target frame into a first neural network model to obtain clustering results of the multiple first target frames.

[0025] It can be understood that the first neural network model can utilize text descriptions to perform clustering according to image features of the input image, and the output clustering results are more accurate than the clustering results of general clustering models.

[0026] In a possible implementation of the first aspect, the first target frame corresponds to different prompt information and has different text descriptions.

[0027] It can be understood that when the user's prompt information is different, the text description of the determined first target frame may be different. For example, when the user's prompt information is related to the character's action, the text description of the determined first target frame is a related description of the character's action in the first target frame; when the user's prompt information is related to the human body posture, the text description of the determined first target frame is a related description of the human body posture in the first target frame; when the user's prompt information is related to the position, the text description of the determined first target frame is a related description of the position of the object in the first target frame; when the user's prompt information is related to the character's emotions and style, the text description of the determined first target frame is a related description of the character's emotions and style in the first target frame. In this way, when the user's prompt information is different, the text description of the determined first target frame is also different.

[0028] In a possible implementation of the first aspect, the method further includes: a duration of the target video is less than a sum of durations of at least one video to be processed.

[0029] It can be understood that when the target video segment is the main video segment of the video segment showing different content in the video to be processed, the duration of the target video segment is shorter than the duration of the video to be processed, and thus the duration of the target video is also shorter than the sum of the durations of at least one video to be processed.

[0030] In a second aspect, an embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the visual media data processing method mentioned in the embodiment of the present application when executing the computer program.

[0031] In a third aspect, an embodiment of the present application provides a storage medium having instructions stored thereon, which, when executed on an electronic device, enables the electronic device to execute the visual media data processing method mentioned in the embodiment of the present application.

[0032] In a fourth aspect, an embodiment of the present application provides a program product, which includes computer instructions. When executed by an electronic device, the electronic device executes the visual media data processing method mentioned in the embodiment of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 According to some embodiments of the present application, a schematic diagram of a video processing process is shown;

[0034] Figure 2A According to some embodiments of the present application, another schematic diagram of a video processing process is shown;

[0035] Figure 2B According to some embodiments of the present application, a schematic diagram of a target video segment is shown;

[0036] Figure 3 According to some embodiments of the present application, a schematic diagram of a video semantic boundary is shown;

[0037] Figure 4 According to some embodiments of the present application, a schematic diagram of the hardware structure of an electronic device is shown;

[0038] Figure 5A According to some embodiments of the present application, a schematic diagram of a gallery interface of an electronic device is shown;

[0039] Figure 5B According to some embodiments of the present application, a schematic diagram of a selection interface of an electronic device is shown;

[0040] Figure 5C According to some embodiments of the present application, a schematic diagram of an authoring interface of an electronic device is shown;

[0041] Figure 5D According to some embodiments of the present application, a schematic diagram of a video preference setting interface of an electronic device is shown;

[0042] Figure 6 According to some embodiments of the present application, a flow chart of a visual media data processing method is shown;

[0043] Figure 7 According to some embodiments of the present application, a schematic diagram of a process for determining a target video is shown;

[0044] Figure 8 According to some embodiments of the present application, a schematic diagram of a multimodal fine-grained evaluation model is shown;

[0045] Fig. 9 According to some embodiments of the present application, a schematic diagram of a process for determining a graphic data packet is shown;

[0046] Fig.10 According to some embodiments of the present application, a schematic diagram of a clustering network architecture is shown;

[0047] Fig.11 According to some embodiments of the present application, a schematic diagram of the architecture of a visual media data processing method is shown;

[0048] Fig.12 According to some embodiments of the present application, a schematic diagram of the structure of a backbone network pre-training model of a multimodal fine-grained evaluation model is shown;

[0049] Fig.13 According to some embodiments of the present application, a schematic diagram of a fine-tuning model structure of a multimodal fine-grained evaluation model is shown;

[0050] Fig.14 According to some embodiments of the present application, a schematic diagram of a visual media data processing system is shown. DETAILED DESCRIPTION

[0051] The illustrative embodiments of the present application include, but are not limited to, a visual media data processing method, a program product, a storage medium, and an electronic device.

[0052] In order to more clearly understand the solution of the present application, the relevant field terms involved in the present application are first explained.

[0053] (1) Guided clustering

[0054] Clustering is to classify video frames according to their image content so that video frames of the same category have a high degree of similarity. For example, when the main content of multiple video frames is dogs, the multiple video frames can be determined to be of the same category. Guided clustering is to use external knowledge to assist clustering, such as using text to assist the clustering of video frames. When guiding the clustering of video frames, first match the text description for the video frame, and classify the video frame in combination with the text description and the image content of the video frame, which can improve the accuracy of clustering. For example, when the main content of some video frames in multiple video frames is Shiba Inu and the main content of some video frames is Corgi, since Shiba Inu and Corgi are very similar in appearance and have similar color block pixels, they cannot be distinguished based on the image content alone. When guiding clustering, the text description matched for the video frame with the main content of Shiba Inu may include "longer legs", and the text description matched for the video frame with the main content of Corgi may include "shorter legs", so that the multiple video frames can be divided into two categories.

[0055] (2) Keyframe

[0056] A video is composed of a series of continuous video frames (i.e., images), among which the representative and important video frames that can reflect the significant characteristics of the video can be determined as key frames. In the image coordinate system, if each frame of the video is superimposed together, the feature vectors of each video frame will present a trajectory in space, and the frame corresponding to the feature value in the trajectory can be called a key frame. Key frames can effectively summarize the main content of the video clip, and image feature information can be obtained from key frames, such as the shape, texture, and color of each element, so as to further analyze the video content. For example, when the content of a video is a person dancing, a video frame at the beginning of each dance movement of the person can be determined as a key frame, and the determined key frames can effectively summarize the main content and changes of the video.

[0057] In the embodiment of the present application, the device type of the electronic device can be a mobile phone, a tablet computer, a handheld computer, a desktop computer, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a cellular phone, a personal digital assistant (PDA), as well as smart home devices such as smart large screens and smart speakers, wearable devices such as smart bracelets, smart watches, and smart glasses, extended reality (XR) devices such as augmented reality (AR), virtual reality (VR), and mixed reality (MR), in-vehicle devices or smart city devices, etc.

[0058] As mentioned above, when a user selects videos and pictures in the gallery of an electronic device to generate a new video, the existing visual media data processing method can only extract the highlights in the video and then arrange and merge the highlights in the video according to the time sequence of the appearance of the highlights and the order in which the user selects the videos and pictures to generate a new video. The user's intention cannot be considered in the visual media data processing process. Therefore, the generated video cannot meet the needs of massive users for personalized and diverse videos.

[0059] For example, in some embodiments, reference Figure 1 , Figure 1 A schematic diagram of a video processing process is shown. Figure 1As shown, the electronic device may have a function of automatically generating a new video based on the material (including videos and pictures) selected by the user (also known as a one-click blockbuster function). When the electronic device detects that the user selects the video V1, it determines the wonderful segments V11, V12, V13, V14 and V15 in the video V1 by screening the video segments, and arranges and merges the wonderful segments V11, V12, V13, V14 and V15 in the time sequence in which they appear in the video V1 to generate a new video V2. The user may screen the video segments based on aesthetic principles, such as any one or more of the color continuity of the video frames, the video duration and the integrity of the video shots.

[0060] If the user intends to arrange the video clips with the same main behavior of the characters in the new video in succession, for example Figure 1 The highlight segments V11 and V15 in the video are arranged continuously, and the electronic device still generates a new video V2. Figure 1 It can be seen that between the highlight segments V11 and V15 in the new video V2, there are also highlight segments V12, V13 and V14 with different main behaviors of the characters, which cannot meet the user's needs.

[0061] To solve the above problems, an embodiment of the present application provides a method for processing visual media data. In this method, after the electronic device obtains the visual media data selected by the user, it analyzes the visual media data selected by the user and determines multiple first target frames. Wherein, corresponding to the visual media data being a video to be processed, the first target frame refers to an image representing different target video segments in each video to be processed. For example, for Figure 2A The first video V3 shown in FIG. 1 mainly includes three different video contents, such as "character M3 broadcasts news", "character M1 is interviewed", and "character M2 is interviewed". The first target frame P11, the first target frame P12, and the first target frame P13 can be used to represent the target video segments corresponding to the respective video contents. Figure 2A The second video V4 shown in FIG. 1 mainly includes two different contents, such as "running" and "character M4 broadcasting news". The first target frame P14 and the first target frame P15 can be used to represent the target video segments corresponding to the respective video contents. Corresponding to the visual media data being an image, the first target frame is an image selected by the user.

[0062] Then, the electronic device clusters the first target frames of different target video segments based on the prompt information set by the user, such as a prompt. Based on the clustering result, the target video segments corresponding to the first target frames belonging to the same class are continuously arranged to generate a target video. Figure 2A For the first video V3 and the second video V4 shown, the electronic device can determine the first target frame P11 and the first target frame P15 as the same category, can determine the first target frame P12 and the first target frame P13 as the same category, and can determine the first target frame P14 as a separate category based on the prompt set by the user. Based on the clustering result, the electronic device can continuously arrange the target video segment V31 corresponding to the first target frame P11 and the target video segment V35 corresponding to the first target frame P15, and continuously arrange the target video segment V32 corresponding to the first target frame P12 and the target video segment V33 corresponding to the first target frame P13. For example, the target video segments can be spliced ​​and processed in the order of target video segment V31, target video segment V35, target video segment V32, target video segment V33, and target video segment V34 to generate the target video V5.

[0063] It can be understood that in the embodiment of the present application, each target video segment of the video to be processed can be a main video segment of a video segment with different video content (e.g., different characters, scenes, emotions, etc.), for example, a video segment with unclear or transitional images removed. Figure 2A The first video V3 in the video may include three target video segments V31, target video segment V32 and target video segment V33 showing different video contents, wherein the target video segment V31, target video segment V32 and target video segment V33 respectively represent "character M3 broadcasts news", "character M1 is interviewed" and "character M2 is interviewed". The three display different video contents, and the first target frames representing these three contents can be extracted respectively. It can be understood that the target video segment here and the highlight segment mentioned above can be different. The target video segment refers to the main video segment in the video segment divided based on the content displayed in the video, and the highlight segment is a segment with relatively high aesthetic quality extracted from the video segment based on the aesthetic quality of the video screen (such as composition, color, light and shadow).

[0064] In some embodiments, the target video segment of the video to be processed may also be a video segment with different displayed video content obtained by directly segmenting the video to be processed.

[0065] It can be understood that the video to be processed may include multiple video shots, one of which may include multiple scenes, and one scene may include multiple events, wherein a video shot refers to a group of internally related continuous frames continuously shot by a camera lens. Figure 3 , Figure 3The video V6 shown in (a) may include an interview shot, which includes an outdoor scene. The outdoor scene includes two events. The video segment F1 corresponds to the event of the interview person M1, and the video segment F2 corresponds to the event of the interview person M2. Then the video V6 includes a semantic boundary B1. Figure 3 The video V7 shown in (b) includes two video shots, video segments F3 and video segments F4 correspond to shots taken by the user, and video F5 corresponds to the end shot of application software (APP) X, such as a video segment with a watermark of APP X. The shots taken by the user can be further divided into two scenes, namely, the outdoor scene corresponding to the video segment F3 and the indoor scene corresponding to the video segment F4, and the video V7 includes a semantic boundary B2 and a semantic boundary B3. In some embodiments, the electronic device can segment the video to be processed based on the semantic boundary to obtain video segments showing different video contents.

[0066] In an embodiment of the present application, the electronic device can determine the first target frame of the video to be processed by extracting key frames. The video to be processed may include multiple video contents, each of which may have multiple key frames. In some embodiments, the electronic device can select one of the key frames as the first target frame. For example, for the target video segment V31 in the first video V3, during the process of the character M3 reporting the news, the camera may adjust the shooting angle to make the background of the character M3 larger or smaller. For example, the current Figure 2A The TV behind the person M3 is shown in the figure, but if the focal length of the shooting lens is extended and the shooting angle becomes larger, the background will include not only the TV but also a large screen, an anchor desk or other props. As the background changes, the key frame will also change. That is, the target video segment V11 will have multiple key frames, and the electronic device can select one frame from the multiple key frames as the first target frame.

[0067] For example, the first target frame can be one of multiple key frames. If multiple key frames are adjacent or close to each other in the video to be processed and have similar image contents, the first of the multiple key frames can be used as the first target frame of the video to be processed, or the middle one of the multiple key frames can be used as the first target frame of the video to be processed, or the last one of the multiple key frames can be used as the first target frame of the video to be processed, or one of the multiple key frames can be randomly selected as the first target frame of the video to be processed.

[0068] In some embodiments, corresponding to the visual media data being a video to be processed, the target video segment corresponding to the first target frame may be composed of the first target frame and multiple frames of images adjacent to and before the first target frame. Figure 2BIf the first target frame is the target frame P21, the target video segment corresponding to the target frame P21 may be composed of image frames P22, P23, P24 and P25 that are adjacent to the target frame P21.

[0069] In some embodiments, corresponding to the visual media data being an image, the target video segment corresponding to the first target frame may be a video segment obtained by adding a dynamic effect to the first target frame.

[0070] Let's combine Figure 2A and Figure 2B , a visual media data processing method provided in an embodiment of the present application is briefly described.

[0071] For example, Figure 2A As shown, assuming that the user selects a video from the gallery application to generate a target video, for example, the video to be processed is the first video V3, then in some embodiments of the present application, after analyzing the content of the first video V3, the electronic device can determine the first target frame P11, the first target frame P12, and the first target frame P13. The relationship between the first target frame, the video content represented by the first target frame, and the target video segment corresponding to the first target frame can be referred to in the following Table 1.

[0072] As shown in Table 1 below, if the prompt set by the user is "Describe the main behavior of the character", the video content represented by the first target frame P11 can be "Broadcasting the news", which can be determined as one category. The video content represented by the first target frame P12 and the first target frame P13 can both be "Being interviewed", which can be determined as the same category. Then, the electronic device can arrange the target video segment V32 corresponding to the first target frame P12 and the target video segment V33 corresponding to the first target frame P13 in succession, for example, splicing and processing each target video segment in the order of target video segment V33, target video segment V32, and target video segment V31 to generate a target video. Among them, the first target video segment mentioned in the embodiment of the present application can be the target video segment V32, and the second target video segment can be the target video segment V33.

[0073] If the prompt set by the user is "Describe the emotion of the image", the video content represented by the first target frame P11, the first target frame P12, and the first target frame P13 can all be "calm" and can be determined to be of the same category. The electronic device can then arrange the target video segment V31 corresponding to the first target frame P11, the target video segment V32 corresponding to the first target frame P12, and the target video segment V33 corresponding to the first target frame P13 in succession, for example, splicing and processing each target video segment in the order of target video segment V31, target video segment V33, and target video segment V32 to generate a target video. Among them, the first target video segment mentioned in the embodiment of the present application can be the target video segment V31, and the second target video segment can be the target video segment V33.

[0074] For example, assuming that the user selects multiple videos from the gallery application to generate a target video, continue to refer to Figure 2A Taking the case where the user selects 2 videos as an example, when the user selects the first video to be processed (i.e., the first video V3) and the second video to be processed (i.e., the second video V4) in sequence, in some embodiments of the present application, after the electronic device analyzes the first video to be processed and the second video to be processed, it can be determined that the first target frame of the first video to be processed includes the first target frame P11, the first target frame P12, and the first target frame P13, and the first target frame of the second video to be processed includes the first target frame P14 and the first target frame P15.

[0075] As shown in Table 1 below, if the prompt set by the user is "Describe the main behavior of the character", the video content represented by the first target frame P11 and the first target frame P15 can both be "Broadcasting News", and can be determined as the same category. The video content represented by the first target frame P12 and the first target frame P13 can both be "Accepting an Interview", and can be determined as the same category. The video content represented by the first target frame P14 can be "Running", and can be determined as a separate category. Then, the electronic device can continuously arrange the target video segment V31 corresponding to the first target frame P11 and the target video segment V35 corresponding to the first target frame P15, and continuously arrange the target video segment V32 corresponding to the first target frame P12 and the target video segment V33 corresponding to the first target frame P13, for example, splicing and processing each target video segment in the order of target video segment V31, target video segment V35, target video segment V32, target video segment V33, and target video segment V34 to generate a target video V5.

[0076] If the prompt set by the user is "Describe the location of the person with a brief description", the video content represented by the first target frame P11 and the first target frame P15 can both be "indoor" and can be determined as the same category. The video content represented by the first target frame P12, the first target frame P13, and the first target frame P14 can all be "outdoor" and can be determined as the same category. Then, the electronic device can continuously arrange the target video segment V31 corresponding to the first target frame P11 and the target video segment V35 corresponding to the first target frame P15, and continuously arrange the target video segment V32 corresponding to the first target frame P12, the target video segment V33 corresponding to the first target frame P13, and the target video segment V34 corresponding to the first target frame P14, for example, in the order of target video segment V31, target video segment V35, target video segment V33, target video segment V34, and target video segment V32, each target video segment is spliced ​​and processed to generate a target video. Among them, the first target video segment mentioned in the embodiment of the present application can be the target video segment V32, and the second target video segment can be the target video segment V34.

[0077] If the prompt set by the user is "Describe the emotion of the image", the video contents represented by the first target frame P11, the first target frame P12, the first target frame P13 and the first target frame P15 can all be "calm" and can be determined as the same category. The video content represented by the first target frame P14 can all be "excited" and can be individually determined as a category. Then, the electronic device can continuously arrange the target video segment V31 corresponding to the first target frame P11, the target video segment V32 corresponding to the first target frame P12, the target video segment V33 corresponding to the first target frame P13 and the target video segment V35 corresponding to the first target frame P15, for example, splicing and processing each target video segment in the order of target video segment V31, target video segment V32, target video segment V33, target video segment V35 and target video segment V34 to generate a target video.

[0078] Table 1:

[0079]

[0080] In some embodiments, the first target frame may be determined by the electronic device based on an image selected by the user, or may be determined by the electronic device based on a video selected by the user.

[0081] For example, assuming that a user selects at least one image from a gallery application to generate a target video, in some embodiments of the present application, the electronic device may use the image selected by the user from the gallery application as the first target frame, and cluster multiple first target frames according to the prompt set by the user. Then, according to the clustering result, the target video segments corresponding to the first target frames belonging to the same class are continuously arranged and spliced ​​to generate the target video.

[0082] Exemplarily, assuming that the user selects at least one image and at least one video from the gallery application to generate a target video, in some embodiments of the present application, the electronic device can analyze each video to be processed, determine the first target frame of each video to be processed as part of the first target frame, and use the image selected by the user as another part of the first target frame, thereby determining all the first target frames. Then, according to the prompt set by the user, the multiple first target frames are clustered, and according to the clustering results, the target video segments corresponding to the first target frames belonging to the same category are continuously arranged and spliced ​​to generate the target video.

[0083] In this way, the visual media data processing method provided by the present application can combine user intentions to generate personalized and diverse videos, which can meet the needs of a large number of users.

[0084] The following first introduces the hardware structure of an electronic device provided in an embodiment of the present application.

[0085] Figure 4 FIG. 1 shows a schematic diagram of the hardware structure of the electronic device 100. Figure 4 As shown, the electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker (not shown in the figure), a receiver (not shown in the figure), a microphone (not shown in the figure), an earphone interface (not shown in the figure), a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a touch sensor 180K, etc.

[0086] It is to be understood that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or separate some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0087] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0088] The charging management module 140 is used to receive charging input from a charger, where the charger can be a wireless charger or a wired charger.

[0089] The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140 to power the processor 110, the internal memory 121, the display screen 194, the camera 193, and the wireless communication module 160.

[0090] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0091] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals.

[0092] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to the electronic device 100. In some embodiments, at least some functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some functional modules of the mobile communication module 150 can be set in the same device as at least some modules of the processor 110.

[0093] The modulation and demodulation processor may include a modulator and a demodulator. The modulator is used to modulate the low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing.

[0094] The wireless communication module 160 can provide wireless communication solutions applied to the electronic device 100, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared technology (IR), etc.

[0095] The electronic device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0096] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel, which can be used to display, for example Figure 5A-5D The interface shown and the target video generated by the visual media data processing method mentioned in the embodiment of the present application.

[0097] The electronic device 100 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0098] The ISP is used to process data fed back by the camera 193 .

[0099] The camera 193 is used to capture still images or videos. In some embodiments, the electronic device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0100] Digital signal processors are used to process digital signals. In addition to processing digital image signals, they can also process other digital signals. Video codecs are used to compress or decompress digital videos.

[0101] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. NPU can realize applications such as intelligent cognition of electronic device 100, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0102] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function, such as storing music, video and other files in the external memory card.

[0103] The internal memory 121 may be used to store computer executable program codes, wherein the executable program codes include instructions.

[0104] The electronic device 100 can implement audio functions through the audio module 170 and the application processor, etc. For example, music playing, recording, etc. The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals.

[0105] The touch sensor 180K is also called a "touch control device". The touch sensor 180K can be set on the display screen 194. The touch sensor 180K and the display screen 194 form a touch screen, also called a "touch control screen". The touch sensor 180K is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In other embodiments, the touch sensor 180K can also be set on the surface of the electronic device 100, which is different from the position of the display screen 194.

[0106] In some implementations, the touch sensor 180K may be used to obtain an operation of a user selecting a video or an image.

[0107] The buttons 190 include a power button, a volume button, etc. The motor 191 can generate a vibration prompt. The motor 191 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. The indicator 192 can be an indicator light, which can be used to indicate the charging status. The SIM card interface 195 is used to connect a SIM card.

[0108] The following introduces a visual media data processing method provided by an embodiment of the present application in combination with an application scenario.

[0109] In some application scenarios, the electronic device 100 stores videos and pictures taken or downloaded by the user in the gallery, and the user can select videos and / or pictures in the gallery interface to generate a target video. In addition, in some application scenarios, the user can also set a prompt in the video preference setting, and the electronic device 100 can generate a personalized and diverse target video that meets the user's intention according to the prompt set by the user.

[0110] Reference Figure 5A , Figure 5A FIG. 1 shows a schematic diagram of a gallery interface of an electronic device 100. Figure 5A As shown, the gallery interface 001 includes a selection control A11, and also includes multiple pictures and multiple videos. When the electronic device 100 detects a user operation, such as a single click operation of the user on the selection control A11 or a long press operation of the user on a video or picture in the gallery interface 001, the electronic device 100 can respond to the user operation and display the following Figure 5B Selection interface 002 is shown.

[0111] like Figure 5B As shown in (a) of FIG. 1 , the selection interface 002 includes a close control A21. When the electronic device 100 detects a user operation (such as a single click operation) on the close control A21, the electronic device 100 can respond to the user operation, exit the selection interface 002, and return to FIG. Figure 5A The gallery interface 001 shown. The selection interface 002 also includes a prompt area 0021 below, which includes a video generation control A22 and a prompt T1. The video generation control A22 is in an inoperable state, and the prompt T1 is used to prompt the user to select the material for generating the video, that is, the video and the picture. For example, the video generation control A22 can be displayed as the text "One-click blockbuster", and the prompt T1 can be "Select the material and the film can be automatically generated". After the user selects any picture or video, the video generation control A22 becomes an operable state, and the prompt T1 becomes the prompt T2.

[0112] like Figure 5BAs shown in (b) in the figure, the prompt T2 is used to prompt the user the number of materials that have been selected. The user can select (for example, click) videos and pictures in the selection interface 002, and the electronic device 100 can respond to the user's operation and display the serial number corresponding to the user's selection order in the lower right corner of the selected videos and pictures. For example, if the user selects video S1, video S2, picture S3 and video S4 in sequence, the lower right corners of video S1, video S2, video S3 and video S4 can display serial numbers 1, 2, 3 and 4 in sequence, and correspondingly, the prompt T2 can be displayed as "4 items have been selected". After the user completes the selection, the video generation control A22 can be clicked, and the electronic device 100 can respond to the user's click operation and automatically generate the target video based on video S1, video S2, picture S3 and video S4. In some embodiments, thumbnails of the videos and pictures to be processed can also be displayed below the prompt T2 in the prompt area 0021, and the thumbnails are used to prompt the user of the videos or pictures that have been selected.

[0113] In some application scenarios, the gallery of the electronic device 100 also includes a creation interface 003. Figure 5C , Figure 5C FIG. 1 shows a schematic diagram of a creation interface in a gallery of an electronic device 100. Figure 5C As shown, the creation interface 003 includes a one-key blockbuster control A32, a clip control, a free creation control, a micro-film creation control, a puzzle control, etc. The electronic device 100 can detect the user's operation on the one-key blockbuster control A32 (for example, a single click operation) and display the following: Figure 5B As shown in the selection interface 002 , the user can select videos and / or pictures in the selection interface 002 to generate a target video.

[0114] In some application scenarios, the electronic device 100 further includes a video preference setting interface 004. Figure 5D , Figure 5D FIG. 1 shows a schematic diagram of a video preference setting interface of an electronic device 100. Figure 5DAs shown, the video preference setting interface 004 includes a prompt setting area 0041, a video theme setting area 0042, and a video duration setting area 0043. The prompt setting area 0041 may include a prompt input box A411 and a selection control A412. The prompt input box A411 displays an input prompt for prompting the user how to enter the prompt. For example, the input prompt may be "for example: the instrument played by the character". The user can click the prompt input box A411 to enter the prompt. The user can also click the selection control A412, and the electronic device 100 detects the user's click operation, displays a prompt list, and the user can select a prompt from the prompt list. In some application scenarios, the prompt setting area 0041 may also include only the selection control A412 or only the prompt input box A411, and this application does not limit this. In some application scenarios, the prompt setting area 0041 may also include the prompt set by the electronic device by default, and the user can modify the prompt.

[0115] Continue to refer to Figure 5D The video theme setting area 0042 includes a selection control A42 and a current video theme (e.g., portrait). The user can click on the selection control A42, and the electronic device detects the user's click operation and displays a video theme list. The user can select a video theme from the video theme list. The video duration setting area 0043 includes a selection control A43 and a video preset duration (e.g., 1 minute). The user can click on the selection control A43, and the electronic device 100 detects the user's click operation and displays a list of optional video durations. The user can select a video duration from the video duration list as the preset duration.

[0116] In some embodiments, the electronic device generates a target video in response to the user's click operation on the video generation control A22 or the one-key blockbuster control A32, and can also display the generated target video in the gallery interface. In some embodiments, the user can also edit the target video, such as modifying the video theme, modifying the background music, etc.

[0117] Combine the following Figure 6 , introduces a visual media data method provided by an embodiment of the present application. Figure 6 , Figure 6 FIG. 1 is a flowchart of a method for processing visual media data provided by an embodiment of the present application. Figure 6 As shown, the method comprises the following steps:

[0118] S501: Obtain visual media data to be processed and user prompt information.

[0119] In this step, the electronic device first obtains the visual media data to be processed and the user's prompt information. The visual media data to be processed refers to the video and picture selected by the user; the user's prompt information can be a prompt word preset by the user or a prompt word set by default in the electronic device. For example, the above-mentioned Figure 5D Description.

[0120] It is understood that when a user wants to generate a target video, he can select both the video and the picture, or only the video or only the picture. The video and the picture selected by the user can be collectively referred to as the visual media data to be processed. When the electronic device responds to the user operation of generating the target video, the visual media data to be processed and the user's prompt information can be obtained.

[0121] S502: Based on the user's prompt information, generate and display a target video after splicing the visual media data to be processed.

[0122] In this step, after the electronic device obtains the visual media data to be processed and the user's prompt information, it can analyze the visual media data to be processed and determine multiple first target frames. Then, based on the user's prompt information, the first target frames are clustered, and based on the clustering results, the target video segments corresponding to the first target frames belonging to the same class are continuously arranged, and after splicing processing, the target video is generated and displayed. For example, the above-mentioned Figure 2A and Figure 2B Description.

[0123] In summary, the visual media data processing method provided in the embodiment of the present application can splice the visual media data based on the user's prompt information to generate and display the target video. In this way, in the process of generating the target video, the user's intention is taken into account, and the needs of massive users for video personalization and diversity can be met.

[0124] It is understood that the electronic device can splice the visual media data to be processed based on the user's prompt information to generate a target video. Figure 7 , Figure 7 FIG. 4 shows a flow chart for generating a target video. Figure 7 As shown, the following steps are included:

[0125] S601: Determine a plurality of first target frames of each visual media data.

[0126] In some embodiments, after the electronic device obtains the visual media data to be processed, it first determines a plurality of first target frames of each visual media data. Wherein, corresponding to the visual media data being a video to be processed, different first target frames represent different target video segments in the video to be processed, and different target video segments display different video contents. Corresponding to the visual media data being an image, the first target frame is an image. The plurality of first target frames can be determined by the electronic device based on a video selected by the user, or can be determined based on an image selected by the user.

[0127] It can be understood that, for the video to be processed, since the video to be processed is composed of multiple video frames, it is difficult to directly determine the target video segment and perform clustering according to the video content. Therefore, the first target frame representing different target video segments can be determined, and then the first target frame can be clustered, so that the target video segment corresponding to the first target frame can be clustered according to the video content. For the image, it can be directly used as the first target frame for clustering analysis.

[0128] Corresponding to the visual media data being the video to be processed, specifically, in some embodiments, the electronic device can analyze the video to be processed and determine the semantic boundaries of each video to be processed. Then, based on the semantic boundaries of each video to be processed, the video to be processed is divided into multiple video segments. Among them, the semantic boundary refers to the node where the shot, scene or event in the video is switched, which can be determined according to the semantics of each video frame in the video to be processed (i.e., the meaning of the image content). For example, refer to the aforementioned Figure 3 Description.

[0129] In some embodiments, the visual media data includes a first video to be processed. If the first video to be processed only displays one type of video content, for example, the first video to be processed is Figure 3 The video segment F1 shown in (a) in FIG. 1 is equal to the first video to be processed before segmentation. The method for determining the semantic boundary will be described later and will not be described here.

[0130] In some embodiments, after the electronic device segments the video to be processed, it can extract key frames based on the segmented video segments, and select the first target frame corresponding to each video segment from the extracted key frames.

[0131] In some embodiments, key frame extraction may include extracting key frames of each video segment using a K-means clustering algorithm based on the multiple segmented video segments. In other embodiments, key frame extraction may also use methods based on image content, methods based on motion analysis, methods based on the trajectory curve point density features of feature vectors of video frames, and other clustering algorithms, or use video editing tools (such as fast forward moving picture experts group (ffmpeg)) to extract key frames. This application does not limit the specific method of key frame extraction.

[0132] In some embodiments, after the electronic device extracts the key frames of each video segment, it can first select the target frame corresponding to each video segment (hereinafter referred to as the second target frame) from the key frames of each video segment. In some embodiments, corresponding to the visual media data being an image, the image can be first used as the second target frame.

[0133] Based on this, the electronic device performs an image quality assessment on the second target frame to obtain a quality score for each second target frame. Finally, a second target frame having a quality score greater than a quality score threshold is selected from the second target frames as the first target frame. The quality score of the second target frame represents the image quality of the second target frame. The quality score threshold may be a default empirical value pre-set in the electronic device, or may be a value set by the user. For example, the quality score may be a one-point system, and the corresponding quality score threshold may be 0.8 points. When the quality score is greater than 0.8 points, it indicates that the image quality of the second target frame is high.

[0134] It can be understood that the electronic device selects the second target frame whose quality score is greater than the quality score threshold from the second target frame, that is, filters out the second target frame with poor image quality from the second target frame, so as to ensure the image quality of the first target frame. If the quality score threshold is a default empirical value pre-set in the electronic device, it means that the image quality of the filtered second target frame is poor or the aesthetic evaluation is low; if the quality score threshold is a value set by the user, it means that the image quality of the filtered second target frame does not meet the user's expectations.

[0135] In some embodiments, the electronic device may perform fine-grained evaluation on the second target frame based on multiple attributes of the second target frame, and determine the second target frame with higher image quality as the first target frame. Fine-grained refers to focusing on small features or subtle changes in the image, and fine-grained evaluation refers to evaluating the quality of the image based on detailed features (such as attributes) in the image. For details, please refer to the following Figure 8 The description of is not repeated here.

[0136] In this way, after filtering the second target frame, a first target frame with higher image quality or higher aesthetic evaluation can be obtained, which is more in line with user expectations. Correspondingly, the target video clips corresponding to each first target frame have higher quality or higher aesthetic evaluation, which is more in line with user expectations. That is, the video generated by the visual media data processing method provided in the embodiment of the present application has higher quality or higher aesthetic evaluation, which is more in line with user expectations.

[0137] S602: Clustering the first target frame based on the prompt information of the user.

[0138] In some embodiments, after the electronic device determines multiple first target frames of each visual media data, it can cluster the first target frames based on the user's prompt information.

[0139] In some embodiments, the electronic device first determines a text description of each first target frame based on a user's prompt information, and then guides clustering of the multiple first target frames based on the text description of each first target frame and image features of each first target frame.

[0140] Specifically, in some embodiments, the electronic device inputs the first target frame into a multimodal language model (i.e., a vision-language model (VLM)) based on the user's prompt information, generates a text description (image caption) corresponding to the first target frame, and forms an image-text data pair. The multimodal language model can be pre-trained so that the model has good image-text matching capabilities, and the text description matched for the first target frame can accurately correspond to the semantic information of the first target frame.

[0141] In an embodiment of the present application, the multimodal language model can be an open source model such as a large language model Meta AI (LLMA), a chat generative pre-trained transformer model (ChatGpt), a GPT-V4 model, a mini Chinese pre-trained model (MINI-CPM), a pre-trained model for unified visual language understanding and generation (bootstrapping language-image pre-training, Blip-2) and an end-to-end trained large multimodal model (LLaVA), or it can be a self-customized multimodal language model or title generation model.

[0142] In other embodiments, the electronic device matches a text description for the first target frame based on the user's prompt information by using a pre-trained image dataset and a text encoder.

[0143] Specifically, the electronic device can use the pre-trained image data set and text encoder in advance to extract a series of text (such as noun words) for each image from the text description corresponding to each image in the image data set, form multiple image-text data pairs, and store them as image-text data packets. The process of determining the image-text data packet can refer to the following description of the image-text data packet. Fig. 9 The description of is not repeated here.

[0144] Then, when the electronic device determines the text description of the first target frame, it can directly obtain the image and text data packet, and determine the image and text data packet corresponding to the user's prompt information based on the user's prompt information. For each first target frame, multiple texts can be selected from the image and text data packet as the text description of the first target frame.

[0145] In some embodiments, the image and text data package includes matching score information of the image and text, wherein the matching score represents the degree of matching between the text and the content in the image, and the higher the matching score, the higher the degree of matching between the text and the content in the image. When determining the text description of the first target frame, the first few texts (e.g., 5) with the highest matching scores can be selected from the image and text data package as the text description of the first target frame.

[0146] It can be understood that, since the picture data set usually includes a large number of pictures, the text description in the picture and text data packet can completely cover the semantics of a large number of pictures in the picture data set, and can also accurately distinguish pictures with different semantics. Therefore, it can be ensured that the text description of the determined first target frame can accurately correspond to the semantic information of the first target frame, so that the semantic information provided by the text description can be fully utilized to enhance the image features of the first target frame for clustering distinction ability. When the user's prompt information is different, the text description of the same first target frame determined is also different, that is, when the user's prompt information is different, the same first target frame can be divided into different clusters.

[0147] In some embodiments, after determining the text description of each first target frame, the electronic device may input the text description of each first target frame and each first target frame into a first neural network model to obtain clustering results of multiple first target frames.

[0148] It can be understood that users generally cannot intervene in the clustering process of traditional clustering algorithms, and the clustering results cannot be consistent with the user's assumptions. The visual data processing method provided in the embodiment of the present application can effectively control the clustering results by determining the text description of the first target frame based on the user's prompt information, and then guiding the clustering of the first target frame according to the text description, so as to ensure that the clustering results are more consistent with the user's assumptions. For example, when the user's prompt information is related to the character's action, clustering is performed according to the character's action in the first target frame; when the user's prompt information is related to the human body posture, clustering is performed according to the human body posture of the character in the first target frame; when the user's prompt information is related to the position, clustering is performed according to the position of the object in the first target frame; when the user's prompt information is related to the character's emotion and style, clustering is performed according to the emotion and style of the character in the first target frame. In this way, when the user's prompt information is different, the clustering results are also different.

[0149] In some embodiments, the user's prompt information may relate to the location of objects in the image, the content of the image, the emotions of the characters in the image, etc. Exemplarily, the prompts may include "instruments played by the characters", "describe in detail", "provide a short description", "give a description of the image and label the image", "focus on the emotions of the central character", "describe the emotions of the image", "describe the location of the characters and describe the features of the image with a detailed description", "provide a brief description of the objects in the given image", "provide a short description of the main objects in the given image. Focus on the main objects", "describe the general emotions by observing the background", "the main behavior of the person", "provide a one-word label for the main object in the image", etc.

[0150] In some embodiments, after the electronic device inputs the text description of each first target frame and each first target frame into the first neural network model, the first neural network model can convert each first target frame into a frame vector (i.e., a picture vector) through an image encoder, and convert the text description of each first target frame into a text vector through a text encoder. Then, the frame vector of each first target frame is concatenated with the text vector corresponding to the frame vector, and the concatenated vector can be expressed as , where v i is the image vector, is the same as the image vector v i The corresponding text vector. Finally, guided clustering analysis is performed to determine the category to which each first target frame belongs and obtain the clustering result.

[0151] It can be understood that directly clustering the spliced ​​frame vector only considers the feature information of the first target frame and its corresponding text description, ignoring the neighborhood sample information of the first target frame and the neighborhood sample information of the text, and the clustering effect may not be optimal. Among them, the spliced ​​frame vector can be regarded as a graphic data pair consisting of the first target frame and the text description corresponding to the first target frame, the neighborhood sample of the first target frame can include the video frames adjacent to the first target frame, and the neighborhood sample of the text can include the text description corresponding to the video frames adjacent to the first target frame corresponding to the text description.

[0152] Therefore, in the clustering process, the neighborhood sample information of the first target frame and the neighborhood sample information of the text can be considered, so that the clustering result can be optimized and the clustering result of the first target frame can be more accurate. Fig.10 The description of is not repeated here.

[0153] S603: Based on the clustering result, the target video segments corresponding to the first target frames are spliced ​​to obtain the target video.

[0154] In some embodiments, based on the clustering result, the electronic device can continuously arrange the target video segments corresponding to the first target frames belonging to the same category, and obtain the target video after splicing. For example, when the clustering result output in step S602 includes the first category a and the second category b, the first category a includes the first target frame a1, the first target frame a2, ..., the first target frame an, and the second category b includes the first target frame b1, the first target frame b2, ..., the first target frame bn, then the target video segments corresponding to the first target frames can be arranged in the order of the first target frame a, the first target frame a2, ..., the first target frame an, the first target frame b1, the first target frame b2, ..., the first target frame bn, and the target video can be obtained after splicing.

[0155] Corresponding to the visual media data being a video to be processed, the target video segment corresponding to the first target frame is determined by: obtaining multiple frame images adjacent to and before the first target frame, and using the multiple frame images and the first target frame as the target video segment corresponding to the first target frame. The length of the target video segment, that is, the number of frames of the multiple frame images adjacent to and before the first target frame that need to be selected, can be determined according to the number of the first target frames and the preset duration of the target video.

[0156] Exemplarily, if the visual media data only includes the video to be processed, when the number of the first target frames is small, more adjacent image frames can be selected to make the length of each target video segment longer, so that the target video can reach the preset duration. When the number of the first target frames is large, fewer adjacent image frames can be selected to make the length of each target video segment shorter, so that the target video duration does not exceed the preset duration.

[0157] It can be understood that the preset duration is usually less than the sum of the durations of all videos to be processed, that is, the duration of the target video is usually also less than the sum of the durations of all videos to be processed. When the length of the video to be processed is too short, for example, the length of the video obtained by directly splicing all videos to be processed cannot reach the preset duration, then the target video segment can be equal to the video to be processed, and the target video can be obtained by directly splicing the videos to be processed, and the target video does not need to reach the preset duration. At this time, the duration of the target video can be equal to the sum of the durations of all videos to be processed.

[0158] Corresponding to the visual media data being an image, the target video segment corresponding to the first target frame may be a video segment obtained by adding a dynamic effect to the first target frame. If the visual media data only includes an image, the length of each target video segment may be determined according to a preset duration of the target video. When the preset duration is long, a longer dynamic effect may be added to the first target frame so that the duration of each target video segment is longer, thereby making the target video reach the preset duration. When the preset duration is short, a shorter dynamic effect may be added to the first target frame so that the duration of each target video segment is shorter, thereby making the duration of the target video not exceed the preset duration.

[0159] In some embodiments, if the visual media data includes both a video to be processed and an image, the electronic device can give priority to the first target frame corresponding to the video to be processed and the duration of the corresponding target video segment, and then determine the first target frame corresponding to the image and the duration of the corresponding target video segment, so that the duration of the target video can reach a preset duration.

[0160] In some embodiments, the preset duration of the target video can be determined according to actual needs, for example, the preset duration of the target video can be set to 1 minute, 2 minutes, 5 minutes, etc. The user can also set the preset duration of the target video according to his own preferences.

[0161] In some embodiments, the visual media data to be processed includes at least one video to be processed, and the at least one video to be processed includes a first target video segment and a second target video segment having a first arrangement. Through the visual media data processing method provided in the embodiment of the present application, in the target video obtained by splicing the visual media data to be processed, the first target video segment and the second target video segment have a second arrangement different from the first arrangement, and the second arrangement includes the first target video segment and the second target video segment being arranged continuously, and the first target video segment and the second target video segment corresponding to the user's prompt information belong to the same category.

[0162] If the first target video segment and the second target video segment belong to the same video to be processed, in some embodiments, the first arrangement method may include the presence of a third target video segment between the first target video segment and the second target video segment, or the order of the first target video segment and the second target video segment in the video to be processed is different from the order of the first target video segment and the second target video segment in the target video.

[0163] It can be understood that there may be two target video segments that belong to the same category corresponding to the user's prompt information, which may not be arranged consecutively in the video to be processed, or the order in the target video is different from the order in the video to be processed. This means that in the target video, the target video segments belonging to the same category are not simply arranged according to the order of the target video segments in the video to be processed.

[0164] If the first target video segment and the second target video segment belong to different videos to be processed, in some embodiments, for example, the first target video segment belongs to the first video to be processed and the second target video segment belongs to the second video to be processed, then the first arrangement may include the first video to be processed before the second video to be processed, that is, the first target video segment is before the second target video segment. The second arrangement may include, in the target video, the first target video segment after the second target video segment. The order of the first video to be processed and the second video to be processed is the order selected by the user.

[0165] It can be understood that there may be two target video clips belonging to the same category corresponding to the user's prompt information, and the order of priority in the target video is different from the order of priority of the video to be processed to which they belong, that is, in the target video, the target video clips belonging to the same category are not simply arranged according to the order selected by the user.

[0166] In some embodiments, the target video segments corresponding to the first target frames are spliced ​​together, and then the electronic device matches background music, switching templates, subtitle display effects, etc. for the simply spliced ​​videos based on a preset target video theme (also called a scene) to generate a target video.

[0167] The target video theme may include multiple pre-set themes and a default theme, such as a parent-child theme, a food theme, a portrait theme, a landscape theme, a cheerful theme, a melancholy theme, etc. The user can set the target video theme according to his or her preferences before selecting a video. If the user does not set the target video theme, when generating the target video, the background music, switching template, subtitle display effect, etc. can be matched to the video after simple splicing according to the default target video theme. Among them, the switching template refers to the template of the transition effect for switching between target video clips. Each target video theme can correspond to a switching template, background music and subtitle display effect. The subtitle can be a text description corresponding to the first target frame, and the subtitle display effect may include font, font size, subtitle display position, etc.

[0168] The present application does not limit the arrangement order of the target video segments corresponding to the first target frames belonging to the same category. The target video segments corresponding to the first target frames belonging to the same category can be arranged continuously according to the order selected by the user and the time order of the first target frames appearing in the video to be processed. The arrangement order of the target video segments can also be determined in other ways. For example, after the first target frame is determined in the above step S601, each first target frame is marked with a serial number, and the sequence of the serial numbers is determined as the arrangement order. In addition, the present application does not limit the arrangement order of the categories. After the clustering of step S602 is completed, each category can be marked with a serial number. For example, when the clustering result includes category 1, category 2, ..., category n, each category is arranged in the order of category 1, category 2, ..., category n. The arrangement order of the categories can also be determined in other ways.

[0169] In some embodiments, after the target video is generated, the user can also modify the switching template, background music, subtitle display effects, etc. according to their preferences.

[0170] The visual media data processing method provided in the embodiment of the present application can generate personalized videos based on the user's prompt information, that is, based on the user's preferences. The generated target videos are diverse and can meet the needs of a large number of users.

[0171] The following introduces the aforementioned Figure 7 A method for determining semantic boundaries mentioned in step S601.

[0172] In some embodiments, a method for an electronic device to determine the semantic boundary of a video to be processed may include: extracting and fusing the spatial appearance feature information and the motion information of the time dimension of the video to be processed, determining the boundary score of each video frame, and then screening out the semantic boundary based on the boundary similarity score and feature information of each video frame, combined with the semantic information of each video frame. Among them, the spatial appearance feature information refers to the visual features of the scene and objects in the three-dimensional space contained in the video to be processed, which may include information such as image texture, image color, light and shadow, and the arrangement and shape of objects in the picture. The motion information of the time dimension refers to the dynamic content that changes as the video frame changes, and may include information such as the position, size, shape, and action of the object in the video to be processed. The boundary score can characterize the possibility that the video frame is a semantic boundary.

[0173] It can be understood that by fusing the spatial appearance feature information of the video to be processed and the motion information of the time dimension, the comprehensive feature information of the video to be processed can be obtained. According to the comprehensive feature information of the video to be processed, the content similarity of adjacent video frames can be determined. The higher the content similarity of adjacent video frames, the lower the possibility of switching between shots, scenes and events, and the lower the boundary score of the corresponding video frame, that is, the less likely the video frame is a semantic boundary.

[0174] In some embodiments, the electronic device may use a pre-trained CLIP-vit model to extract spatial appearance feature information of the video to be processed, or may use other pre-trained models, which is not limited in this application.

[0175] In some embodiments, the electronic device may further optimize the semantic boundary according to the video content displayed by the video segments obtained after segmentation based on the semantic boundary.

[0176] Specifically, if the feature vectors of the first and last frames in the segmented video segment are significantly different, it means that the video segment is a transition area or an area where the picture content changes dramatically. For example, the transition segment from one scene to another in the video to be processed should not be segmented into a separate video segment, so the semantic boundary corresponding to the video segment should be filtered out. If there are two adjacent video segments in the segmented video segment and they are semantically coherent, it means that the two video segments can be merged, so the semantic boundary between the two video segments should be filtered out. If there are too short video segments in the segmented video segment, it means that the video content displayed by the video segment may be in a static state, which means that the video segment does not contain much valuable information, and the semantic boundary corresponding to the video segment can be filtered out.

[0177] Combine the following Figure 8 Introduction Figure 7 A fine-grained evaluation method mentioned in step S601.

[0178] In some embodiments, the electronic device may be configured as follows: Figure 8 The multimodal fine-grained evaluation model 701 shown performs a fine-grained evaluation on each second target frame to obtain evaluation scores of different attributes of each second target frame, and then comprehensively evaluates the image quality of each second target frame by combining the evaluation scores of multiple attributes of each second target frame. The multimodal fine-grained evaluation model 701 is a pre-trained model, and the pre-training and fine-tuning of the model and the training data preparation process will be described later and will not be repeated here.

[0179] like Figure 8 As shown, the multimodal fine-grained evaluation model 701 may include a picture encoder and a text encoder, and also include preset attributes. After the second target frame is input into the model, the preset attributes in the model can be converted into a text vector (text_embedding) through the text encoder. When the preset attributes include six attributes, such as overall image attributes, light and shadow attributes, content attributes, background attributes, foreground attributes, and composition attributes, the text encoder can convert the preset attributes into vectors d1, d2, d3, d4, d5, and d6, each of which corresponds to the preset attributes one by one. Among them, the text vector may include information such as the semantics and features of the corresponding preset attributes. For example, when the preset attribute is a light and shadow attribute, the corresponding text vector may include information such as highlights, shadows, light and dark contrast, edges, and contours.

[0180] Then, the second target frame can be converted into an image vector (image_embedding) through an image encoder, wherein the image vector can include image feature information of the corresponding second target frame. Then, the model calculates the similarity between the image vector corresponding to the second target frame and the text vector corresponding to each preset attribute, and obtains the evaluation score corresponding to each preset attribute, namely, the evaluation score g1, evaluation score g2, evaluation score g3, evaluation score g4, evaluation score g5 and evaluation score g6 corresponding to vector d1, vector d2, vector d3, vector d4, vector d5 and vector d6, respectively, for example, the overall score of the picture, the light and shadow score, the content score, the background score, the foreground score and the composition score.

[0181] It can be understood that after each second target frame is evaluated by the multimodal fine-grained evaluation model 701, multiple evaluation scores corresponding to multiple preset attributes can be obtained, and the multiple evaluation scores can characterize the quality of the second target frame in different dimensions. The preset attributes can be set according to actual needs, and this application does not limit the content and quantity of the preset attributes.

[0182] In some embodiments, after the electronic device obtains multiple evaluation scores of each second target frame, for any second target frame, the multiple evaluation scores of the second target frame may be weighted summed to obtain a quality score. It can be considered that the image quality of the second target frame having a quality score greater than the quality score threshold is high.

[0183] In some embodiments, after the electronic device obtains multiple evaluation scores of each second target frame, a corresponding score threshold may be set for each evaluation score. The score threshold may be a default experience value pre-set in the electronic device, and the user may also adjust the score threshold of each evaluation score according to preference.

[0184] For example, when the user pays more attention to the light and shadow of the video screen, the light and shadow score threshold can be set to a higher value; when the user pays more attention to the content of the video screen, the content score threshold can be set to a higher value; when the user pays more attention to the light and shadow, content and composition of the video screen, the light and shadow score threshold, the content score threshold and the composition score threshold can also be set to higher values. If each evaluation score of the second target frame is greater than the score threshold corresponding to each evaluation score, it means that the image quality of the second target frame is higher or the aesthetic evaluation is higher.

[0185] Combine the following Fig. 9 , for the aforementioned Figure 7 A process for determining a graphic data packet mentioned in step S602 is introduced.

[0186] like Fig. 9 As shown, the text space C1 includes text descriptions corresponding to each picture in the picture space C2, and the picture space C2 includes pictures in the pre-trained picture data set. Fig. 9 As shown in (a) in the figure, the images in the image space C2 are clustered, and nouns (which can be called semantic centers) that can represent the categories in the image space C2 are selected in the text space C1 to match the categories with the text. For example, the text corresponding to the semantic center C21 is the valid noun C11, and the semantic center C21 includes the images C211, C212, C213, and C214.

[0187] Specifically, a clustering algorithm, such as a k-means algorithm, may be used to determine the semantic granularity k of the image in the image space C2, wherein the k-means algorithm has the following empirical formula (1):

[0188] k = max{N / 300, K×3} (1)

[0189] Where K is the number of categories and N is the number of images. It can be understood that the above empirical formula (1) means that for a large number of image datasets, 300 images are sufficient to cover the semantics of the selected nouns; while for a smaller number of image datasets, the semantic information of all images in each category can be expressed by 3 words. The above empirical formula (1) can be used to obtain a suitable semantic granularity, which is conducive to distinguishing images of different categories.

[0190] After clustering the images in the image space C2, the semantic center of each image can be obtained by referring to the following formula (2):

[0191] (2)

[0192] in, It is 1 mark, indicating that the image v i Belongs to the lth semantic center.

[0193] Then, if Fig. 9 As shown in (b), a search can be performed in the text space C1 to determine at least one text corresponding to each image in the image space C2, that is, to determine multiple image-text data pairs. Specifically, the CLIP model can be used to divide the text in the text space C1 into corresponding semantic centers, thereby determining multiple image-text data pairs. For example, the division can be performed using the following formula (3):

[0194] (3)

[0195] Among them, t i is the text vector extracted by the CLIP model. It can be understood that for the sake of clarity, Fig. 9 (b) in the figure only shows the text that is most relevant to the picture, and each picture may also correspond to other texts.

[0196] Combine the following Fig.10 , for the aforementioned Figure 7 A cluster analysis process mentioned in step S602 is introduced.

[0197] In some embodiments, the first neural network model may be implemented as follows: Fig.10 The clustering network architecture 901 is shown. Fig.10 As shown, after the electronic device inputs the text description of each first target frame and each first target frame into the clustering network architecture 901, the text description is first encoded by a pre-trained text encoder to obtain a text vector corresponding to the text description, and the first target frame is encoded by a pre-trained picture encoder to obtain a frame vector corresponding to the first target frame. When clustering, a picture clustering function can be introduced to the picture vector v i Perform clustering, such as the image clustering function f: v→p∈R KIn addition, text clustering functions can be introduced to the text vector Perform clustering, such as text clustering function For example, it can be assumed that the image vector v i A random sample in the neighborhood is N(v i ), text vector A random sample in the neighborhood is , then the clustering affiliation of the first target frame and the clustering affiliation of the neighborhood samples of the first target frame can be expressed using the following formula (4):

[0198] (4)

[0199] Furthermore, the clustering of the text description and the clustering of the neighborhood samples of the text can be expressed using the following formula (5):

[0200] (5)

[0201] Continue to refer to Fig.10 , perform text clustering according to the text vector to determine the clustering affiliation of the text vector. Similarly, perform image clustering according to the image vector to determine the clustering affiliation of the image vector, that is, determine the clustering affiliation of the first target frame. In addition, perform neighborhood sample clustering based on the neighborhood samples of the first target frame to determine the clustering affiliation of the neighborhood samples of the first target frame. Combine the clustering affiliation of the text vector and the clustering affiliation of the neighborhood samples of the first target frame to determine the final clustering result of the text description. Similarly, the final clustering result of the first target frame can also be determined. Finally, output the clustering results of each first target frame.

[0202] For example, when clustering through the clustering network architecture 901, the loss function used can refer to the following formula (6):

[0203] (6)

[0204] in,

[0205] ,

[0206] .

[0207] It can be understood that the loss function is used to characterize the degree of coordination between images and texts, that is, it can characterize the quality of the clustering results. For example, if the images input into the clustering network architecture 901 include dog images, and the text descriptions include cats and dogs, when the dog images are classified as cats in the clustering results, the value of the loss function is high, indicating that the clustering results have a large deviation; when the dog images are classified as dogs in the clustering results, the value of the loss function is low, indicating that the clustering results have a small deviation or no deviation. The loss function obtained through each clustering can also adjust the parameters of the clustering network architecture 901, optimize the clustering network architecture 901, and make the clustering results obtained by the next clustering more accurate.

[0208] In an embodiment of the present application, the first neural network model may be a text-aided clustering (TAC) model, a large language model (LLooM), a bagging model, a text clustering method model based on deep learning, etc. The present application does not limit the specific neural network model.

[0209] The visual media data only includes the video to be processed, combined with Fig.11 The following describes a process of a visual media data processing method provided by an embodiment of the present application. Fig.11 , Fig.11 A schematic diagram of the architecture of a visual media data processing method is shown.

[0210] like Fig.11 As shown, the visual media data processing method provided by the embodiment of the present application may include four stages: stage one is the video analysis stage, stage two is the key frame determination stage, stage three is the external guidance clustering stage, and stage four is the video synthesis stage.

[0211] Phase 1: Video analysis phase.

[0212] In some embodiments, the video analysis stage may include: the electronic device analyzes each video to be processed to determine the semantic boundary of the video to be processed. The video to be processed is divided into multiple video segments based on the semantic boundary of the video to be processed. For details, please refer to the relevant description of determining the semantic boundary in step S601 above, which will not be repeated here.

[0213] Phase 2: Key frame determination phase.

[0214] In some embodiments, the key frame determination stage may include: the electronic device extracts key frames based on the segmented video segments to determine the second target frames corresponding to each video segment. Then, the second target frames are evaluated in a fine-grained manner, and the second target frames with poor overall image quality are filtered out, and the second target frames with the best overall image quality are selected as the first target frames. For details, please refer to the relevant description of determining the first target frame in step S601 above, which will not be repeated here.

[0215] Stage 3: External guided clustering stage.

[0216] In some embodiments, the external guided clustering stage may include: the electronic device matches the corresponding text description for the first target frame based on the user's prompt information, and then guides the clustering of the first target frame based on the text description corresponding to each first target frame. For details, please refer to the relevant description of step S602 above, which will not be repeated here.

[0217] Stage 4: Video synthesis stage.

[0218] In some embodiments, the video synthesis stage may include: the electronic device first continuously arranges the video clips corresponding to the key frames belonging to the same cluster for simple splicing. Then, based on the preset target video theme, the background music, switching template, subtitle display effect, etc. are matched for the simply spliced ​​video to generate the target video. For details, please refer to the relevant description of step S603 above, which will not be repeated here.

[0219] Combine the following Fig.12 , introduces the pre-training and fine-tuning process of the multimodal fine-grained evaluation model 701 and the training data preparation process.

[0220] The backbone network pre-training model structure of the multimodal fine-grained evaluation model 701 can be referred to Fig.12 .like Fig.12As shown, the backbone network pre-training model 702 adopts a picture-text contrast learning strategy, and learns the parameters of the picture encoder and the text encoder by inputting picture data and aesthetic review data, aligning the picture and the text, so that the multimodal fine-grained evaluation model 701 learns the corresponding relationship between the picture and the aesthetic review. The picture data and aesthetic review data input into the backbone network pre-training model 702 are data pairs, including pictures and aesthetic reviews corresponding to the pictures, and public aesthetic review datasets can be used, such as a large-scale database for aesthetic visual analysis (AVA) aesthetic evaluation dataset, a large-scale artificial intelligence open network (LAION) aesthetic evaluation dataset, and a reddit community picture critique dataset (RPCD) dataset, etc., or picture aesthetic review data can be captured from a photography website.

[0221] It is understandable that after pre-training, the backbone network only has the ability to match images with aesthetic comments, and cannot be directly used for aesthetic evaluation of images. It is also necessary to add corresponding scoring heads and perform adaptive training to evaluate the different attributes of the images.

[0222] In some embodiments, in order to train the multimodal fine-grained evaluation model 701, it is necessary to collect image-text data triplets containing various attributes (I1 a , I2 a , T1 a ), where I1 a , I2 a T1 is a set of image pairs with similar attribute scores. a The attribute description corresponding to the picture pair may be, for example, "good composition". Then, the multimodal fine-grained evaluation model 701 is fine-tuned by the fine-tuning model 703. The network structure of the fine-tuning model 703 can be referred to Fig.13 .like Fig.13 As shown, the fine-tuning model 703 includes an image encoder 1, an image encoder 2, a text encoder 3, a learnable mapping layer 7031, and a category mapping layer 7032. The loss function of the fine-tuning model 703 is a ternary loss, which can ensure that the loss between a high-scoring image and an attribute description is better than the loss between a low-scoring image and an attribute description. The multimodal fine-grained evaluation model 701 is fine-tuned by the fine-tuning model 703, so that the multimodal fine-grained evaluation model 701 can have the ability to evaluate different attributes of an image.

[0223] In some embodiments, the attribute description text input by the fine-tuning model may include the overall score of the picture, the light and shadow score, the content score, the background score, the foreground score and the composition score, etc. The picture corresponding to the attribute description text may be a self-collected picture, and the attribute score data is annotated by professional editors and verified in multiple rounds to fully ensure the quality and effectiveness of the data annotation. The attribute description text can also use the aesthetic attribute data set of the AVA data set, wherein the aesthetic attribute data set of the AVA data set includes evaluation data of 14 kinds of picture attributes, including complementary colors, two-tone, high dynamic range, image particles, white light, long exposure, Marco, motion blur, negative image, rule of thirds, shallow depth of field, silhouette, soft focus, and vanishing point. According to the metric indicators of the corresponding attribute evaluation, such as composition, color, light, exposure and other factors, the scores are manually annotated, for example, 1 to 10 points. For a picture, the scoring results of multiple annotators are collected, and the average value is used as the final annotation result of the picture.

[0224] It is understandable that when annotating an aesthetic attribute dataset, it is necessary to ensure that the number of images in each score segment is roughly the same to avoid the problem of data category imbalance.

[0225] Combine the following Fig.14 , introducing a visual media data processing system provided in an embodiment of the present application.

[0226] Reference Fig.14 , Fig.14 FIG. 1 shows a schematic diagram of a visual media data processing system provided by an embodiment of the present application. Fig.14 As shown, the visual media data processing system includes a video analysis module 101, a first target frame determination module 102, a guided clustering module 103 and a video synthesis module 104.

[0227] The video analysis module 101 is used to analyze the content of the video to be processed. When the video to be processed includes multiple scenes and / or multiple events and / or multiple video shots, the video to be processed is divided into multiple video clips and output, wherein each video clip corresponds to one scene or event or video shot. When the video to be processed includes only one scene, event and video shot, it is not necessary to divide it, and the output video clip can be the video to be processed.

[0228] The first target frame determination module 102 includes a key frame extraction submodule 1021, an image quality assessment submodule 1022, and a filtering submodule 1023. The key frame extraction submodule 1021 is used to extract key frames in each video segment based on the multiple video segments output by the video analysis module 101, and determine the second target frame from the key frames. The image quality assessment submodule 1022 is used to assess the image quality of the second target frame to obtain an image quality assessment result. The filtering submodule 1023 is used to remove the second target frame with poor image quality according to the image quality assessment result, and output the first target frame.

[0229] The guided clustering module 103 is used to match the text description for the first target frame based on the user's prompt information, and to perform guided clustering on the first target frame according to the text description to obtain a clustering result.

[0230] The video synthesis module 104 is used to continuously arrange the target video segments corresponding to the first target frames belonging to the same class according to the clustering result, and obtain the target video after splicing.

[0231] In some embodiments, the present application also provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the visual media data processing method mentioned in the present application when executing the computer program.

[0232] In some embodiments, the present application further provides a storage medium having instructions stored thereon, which, when executed on an electronic device, enables the electronic device to execute the visual media data processing method mentioned in the present application.

[0233] In some embodiments, the present application also provides a program product, which includes computer instructions. When the program product is executed by an electronic device, the electronic device executes the visual media data processing method mentioned in the present application.

[0234] In some cases, the disclosed embodiments may be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments may also be implemented as instructions carried or stored on one or more temporary or non-temporary machine-readable (e.g., computer-readable) storage media, which may be read and executed by one or more processors. For example, the instructions may be distributed over a network or through other computer-readable media. Therefore, machine-readable media may include any mechanism for storing or transmitting information in a form readable by a machine (e.g., a computer), including, but not limited to, floppy disks, optical disks, optical discs, read-only memories (CD-ROMs), magneto-optical disks, read-only memories (ROMs), random access memories (RAMs), erasable programmable read-only memories (EPROMs), electrically erasable programmable read-only memories (EEPROMs), magnetic or optical cards, flash memory, or a tangible machine-readable memory for transmitting information (e.g., carrier waves, infrared signals, digital signals, etc.) using the Internet in electrical, optical, acoustic, or other forms of propagation signals. Therefore, machine-readable media include any type of machine-readable media suitable for storing or transmitting electronic instructions or information in a form readable by a machine (e.g., a computer).

[0235] In the accompanying drawings, some structural or method features may be shown in a specific arrangement and / or order. However, it should be understood that such a specific arrangement and / or order may not be required. Instead, in some embodiments, these features may be arranged in a manner and / or order different from that shown in the illustrative drawings. In addition, the inclusion of structural or method features in a particular figure does not mean that such features are required in all embodiments, and in some embodiments, these features may not be included or may be combined with other features.

[0236] It should be noted that, in the examples and description of the present application, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or equipment including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or equipment. In the absence of further restrictions, the elements defined by the statement "comprise one" do not exclude the presence of other identical elements in the process, method, article or equipment including the elements.

[0237] Although the present application has been illustrated and described with reference to certain embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the scope of the present application.

Claims

1. A visual media data processing method, applied to electronic equipment, characterized in that: include: Obtaining visual media data to be processed and user prompt information; determining a plurality of first target frames of the visual media data; Clustering the plurality of first target frames based on the prompt information of the user; Based on the clustering results, the visual media data is spliced ​​to obtain a target video, wherein: When the visual media data is at least one video to be processed, the at least one video to be processed includes a first target video segment, a second target video segment and a third target video segment, the first target video segment and the second target video segment have a first arrangement, a third target video segment exists between the first target video segment and the second target video segment, different first target frames represent different target video segments in the video to be processed, the different target video segments display different video contents, the target video segment corresponding to the first target frame is determined based on the first target frame, for image features of first target frames belonging to the same class, the prompt information corresponding to the image features of the first target frames belongs to the same class, and for first target frames belonging to the same class, the target video segments corresponding to the first target frames belong to the same class; In the target video, the first target video segment and the second target video segment have a second arrangement different from the first arrangement, and the second arrangement includes the first target video segment and the second target video segment being arranged continuously, the third target video segment being before or after the first target video segment and the second target video segment being arranged continuously, and, The prompt information corresponding to the first target video segment and the second target video segment belongs to the same category, and the prompt information corresponding to the third target video segment and the first target video segment belongs to different categories.

2. The method according to claim 1, characterized in that Also includes: When the visual media data is an image, the first target frame is the image, wherein, for image features of the first target frame belonging to the same category, the prompt information corresponding to the image features of the first target frame belongs to the same category.

3. The method according to claim 1, characterized in that The step of determining a plurality of first target frames of the visual media data comprises: When the visual media data is a video to be processed, obtaining a semantic boundary of the video to be processed based on the video content of the video to be processed; Based on the semantic boundaries of the video to be processed, the video to be processed is divided into a plurality of video segments; The key frames of each video segment are extracted, and the first target frame corresponding to the video segment is selected from the key frames of the video segment.

4. The method according to claim 3, characterized in that The step of extracting key frames from each video segment and selecting the first target frame corresponding to the video segment from the key frames of the video segment includes: Extracting key frames of each video segment, and selecting a second target frame corresponding to the video segment from the key frames of the video segment; Performing image quality assessment on the second target frames to obtain a quality score for each second target frame; A second target frame whose quality score is greater than a score threshold is selected from the second target frames as the first target frame.

5. The method according to claim 1, characterized in that When the visual media data is a video to be processed, the target video segment corresponding to the first target frame is determined based on the first target frame: A plurality of image frames adjacent to and before the first target frame are acquired, and the plurality of image frames and the first target frame are used as a target video segment corresponding to the first target frame.

6. The method according to claim 1, characterized in that The clustering of the plurality of first target frames based on the prompt information of the user includes: Based on the prompt information, determining a text description of each first target frame; The plurality of first object frames are clustered based on the text descriptions of the first object frames and the image features of the first object frames.

7. The method according to claim 6, characterized in that The clustering of the plurality of first target frames based on the text descriptions of the first target frames and the image features of the first target frames comprises: The text descriptions of the first target frames and the first target frames are input into a first neural network model to obtain clustering results of the multiple first target frames.

8. The method according to claim 6, characterized in that The first target frame corresponds to different prompt information and has different text descriptions.

9. The method according to any one of claims 1 to 8, characterized in that Also includes: The duration of the target video is less than the sum of the durations of the at least one video to be processed.

10. An electronic device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the visual media data processing method according to any one of claims 1 to 9 when executing the computer program.

11. A storage medium, characterized in that: The storage medium stores instructions, which, when executed on an electronic device, enable the electronic device to execute the visual media data processing method according to any one of claims 1 to 9.

12. A computer program product, characterized in that The computer program product comprises computer instructions, and when executed by an electronic device, the electronic device performs the visual media data processing method according to any one of claims 1 to 9.

Images