Video processing methods, readable storage media, and electronic devices

Abstract

The present application relates to the field of terminal technology, and discloses a video processing method, a readable storage medium, and an electronic device. The video processing method of the present application includes: when the electronic device identifies the subject of the video, it extracts the text information and visual information of the video to be processed, and obtains the subject of the video to be processed based on the modalities of the text information and visual information of the video to be processed. Through the present application, even if the video to be processed can only provide visual information, the electronic device can extract relevant text information from the video to be processed, and obtain the subject of the video to be processed according to the multimodal video processing method.

Description

Technical Field

[0001] The present application relates to the field of terminal technology, and in particular to a video processing method, a readable storage medium and an electronic device. Background Art

[0002] The electronic device can classify the videos according to the themes, or implement related functions based on the themes of the videos. For example, when the electronic device edits, clips or produces the video according to the user operation, it can provide the user with a video template that matches the video according to the theme of the video.

[0003] A video is composed of multiple images, each of which can be called an image frame. At present, the method of identifying the theme of a video is generally to determine the theme of the video based on the elements included in each image frame of the video. For example, if the electronic device detects that the image frames including "trees" are the most in the images of multiple frames, the theme of the video can be determined as "nature". However, the accuracy of determining the theme of a video based only on the types of elements in each frame of the video is low.

[0004] For example, in the image frames included in the video taken by the user while traveling, each image frame may include elements such as "people", "food", "trees", etc. If the image frames including "trees" are the most in the video, the electronic device will determine the theme of the video as "nature". However, the actual theme of the video is "travel", resulting in incorrect video theme recognition. Summary of the invention

[0005] Embodiments of the present application provide a video processing method, a readable storage medium, and an electronic device.

[0006] In a first aspect, an embodiment of the present application provides a video processing method for application in an electronic device, the method comprising: obtaining a video to be processed, extracting text information and visual information of the video to be processed; inputting the text information and visual information of the video to be processed into a first model to obtain a theme of the video to be processed.

[0007] It is understood that in some embodiments of the present application, the text information may also be referred to as the description text of the video hereinafter. Through the method of the present application, even if the video to be processed intelligently provides visual information, the electronic device can extract the text information of the video to be processed from the video to be processed, and obtain the theme of the video to be processed based on multiple modes such as text information and visual information, which is conducive to improving the accuracy of the determined theme of the video to be processed.

[0008] In a possible implementation of the first aspect above, the above-mentioned extraction of text information and visual information of the video to be processed includes: obtaining at least one key frame of the video to be processed; inputting at least one key frame into a description text extraction model to obtain a description text of the key frame; and obtaining text information based on the description text of at least one key frame.

[0009] It is understood that in some embodiments of the present application, if the video to be processed has only one key frame, the description text of the key frame is used as the text information of the video to be processed. If the video to be processed has multiple key frames, the description texts of all key frames are merged as the text information of the video to be processed.

[0010] In a possible implementation of the first aspect, the step of obtaining a key frame of the video to be processed includes: obtaining multiple image frames of the video to be processed; dividing the multiple image frames into multiple shot groups, wherein the similarity between the image frames in each shot group exceeds a first similarity threshold; and selecting an image frame in each shot group as a key frame.

[0011] In a possible implementation of the first aspect, the key frame is input into a description text extraction model to obtain the description text of the key frame, including: the description text extraction model includes an image encoding module, an alignment module and a decoding module; the image encoding module extracts the first image feature of the key frame; the alignment module obtains the output text feature based on the first image feature; the decoding module decodes the output text feature to obtain the description text of the key frame.

[0012] It can be understood that the descriptive text extraction model may also be referred to as a label model hereinafter, for example, the first image feature may be, for example, a vector representing a key frame, and the output text feature may be, for example, a vector representing a descriptive text. The alignment module in the descriptive text extraction model also includes a first parameter, based on which the first image feature can be adjusted to an output text feature. When training the descriptive text extraction model, the label text including the sample image and the corresponding sample image may be input into the descriptive text extraction model to train the first parameter of the alignment module. The training process of the descriptive text extraction model is described below.

[0013] Based on the description text extraction model, the description text of the key frames of the video to be processed can be obtained, and the text information of the video to be processed can be generated based on the description text of the key frames. In this way, even if the video to be processed only provides visual information, the present application can obtain text information based on the video to be processed.

[0014] In a possible implementation of the first aspect above, the visual information includes at least one key frame.

[0015] It is understood that in some embodiments of the present application, the visual information may be, for example, a key frame of the video to be processed. If the video to be processed has only one key frame, the visual feature of the video to be processed is the key frame. If the video to be processed includes multiple key frames, the visual feature is multiple key frames.

[0016] In a possible implementation of the first aspect above, the first model includes a text encoding module and a visual encoding module; the text information and visual information of the video to be processed are input into the first model, and obtaining the theme of the video to be processed includes: using the text encoding module to extract features of the text information to obtain a first text feature corresponding to the text information; using the visual encoding module to extract features of the visual information to obtain a visual feature corresponding to the visual information; determining the theme of the video to be processed based on the first text feature and the visual feature.

[0017] It can be understood that in the embodiments of the present application, the first text feature output by the text encoding module can be, for example, a vector representing text information. The visual feature output by the visual encoding module can be, for example, a vector representing visual information, and the vector of the first text feature and the vector of the visual feature are the same type of vector and can be operated on.

[0018] In a possible implementation of the first aspect above, the theme of the video to be processed is determined based on the first text feature and the visual feature, including: obtaining multiple second text features corresponding to the multiple themes based on the first text feature and multiple theme text features corresponding to the preset multiple themes; and taking the theme corresponding to the second text feature with the highest similarity to the visual feature as the theme of the video to be processed.

[0019] It can be understood that in the embodiments of the present application, multiple subject text features can, for example, be vectors referring to multiple preset subjects, and the vectors of multiple subject features and the vectors of the first text features are also the same type of vectors. And multiple second text features are obtained by combining the first text feature with each subject text feature, that is, the number of second text features is the same as the number of subject text features. After obtaining the visual features and multiple second text features, the similarity between the visual features and the multiple second texts can be determined, and the subject corresponding to the second text with the highest similarity to the visual features is selected as the subject of the video to be processed. In this way, when the video to be processed only provides visual information, the present application can also determine the subject of the video to be processed based on the text information and visual information of the video to be processed, thereby improving the accuracy of the subject of the video to be processed determined by the electronic device.

[0020] In a possible implementation of the first aspect above, the method for acquiring multiple topic text features includes: using a text encoding module to extract features of multiple topics to obtain multiple topic text features corresponding to the multiple topics.

[0021] It is understood that in some embodiments of the present application, multiple topics pre-stored in the electronic device can be input into the text encoding module in the first model to obtain multiple topic text features. In other embodiments of the present application, multiple topic text features corresponding to multiple topics can also be directly pre-stored in the electronic device, so that there is no need to extract the features of multiple topics through the text encoding module, thereby reducing the steps of the electronic device identifying the topic of the video to be processed.

[0022] In a second aspect, an embodiment of the present application provides an electronic device, comprising: a memory for storing instructions; and at least one processor for executing instructions so that the electronic device implements the video processing method provided in the first aspect and any possible implementation of the first aspect.

[0023] In a third aspect, an embodiment of the present application provides a readable storage medium having instructions stored thereon. When the instructions are executed on a computer, the computer executes the video processing method provided by the first aspect and any possible implementation of the first aspect.

[0024] In a fourth aspect, an embodiment of the present application provides a program product, which, when running on an electronic device, enables the electronic device to implement the video processing method provided in the first aspect and any possible implementation of the first aspect.

[0025] It should be understood that the beneficial effects of the second to fourth aspects mentioned above can be referred to the description of the first aspect mentioned above and will not be elaborated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 A scenario in which an electronic device identifies a video subject is shown;

[0027] Figure 2 A scene graph for identifying a subject of a video is shown;

[0028] Figure 3 A schematic diagram of identifying video topics based on a multimodal video classification method is shown;

[0029] Figure 4 According to some embodiments of the present application, a flow chart of an implementation of a video processing method is shown;

[0030] Figure 5 According to some embodiments of the present application, a schematic diagram of a video subject recognition method is shown;

[0031] Figure 6a According to some embodiments of the present application, a schematic diagram of a training method for a label model is shown;

[0032] Figure 6bAccording to some embodiments of the present application, a flowchart of an implementation method of a label model training method is shown;

[0033] Figure 7a According to some embodiments of the present application, a schematic diagram of a method for training a visual encoding module and a text encoding module is shown;

[0034] Figure 7b According to some embodiments of the present application, a flowchart of implementing a training video encoding module and a text encoding module is shown;

[0035] Figure 8 According to some embodiments of the present application, a structural schematic diagram of a mobile phone is shown. DETAILED DESCRIPTION

[0036] The illustrative embodiments of the present application include, but are not limited to, a video processing method, a readable storage medium, and an electronic device.

[0037] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0038] The following uses the one-click blockbuster function in the mobile phone gallery application as an example to introduce the application scenarios of video subject recognition.

[0039] Figure 1 A scenario in which an electronic device identifies a video subject is shown.

[0040] like Figure 1 As shown, the creation interface 10 of the gallery of the mobile phone 100 includes icons for users to operate pictures or videos in the gallery, such as a one-click blockbuster icon 11, a clip icon 12, a free creation icon 13, a micro-film creation icon 14, and a puzzle icon 15. After the user clicks the one-click blockbuster icon 11, the electronic device 100 responds to the user's click operation and enters the material selection interface 20.

[0041] The user selects the corresponding video or picture as the video material in the material selection interface 20 to generate the target video 31. For example, the selected videos are marked 1, 2, and 3 on the material selection interface 20 of the mobile phone 100. A one-click blockbuster button 21 and a prompt box "3 items have been selected" prompting the user to select video materials are also displayed on the material selection interface 20. After the user selects the video material, he clicks the one-click blockbuster button 21. The mobile phone 100 responds to the user's click operation and determines the theme of the generated target video 31 according to the content of the video material selected by the user, and selects the template of the target video 31 according to the theme, and then generates a preview video of the target video 31 according to the template and enters the video interface 30. The video interface 30 of the mobile phone 100 displays the video 31 generated by the material selected by the user, the template icon 32, and the export button 33. When the user clicks the export button 33, the mobile phone 100 responds to the user's click operation, exports the target video 31, and completes the production of the target video 31.

[0042] In other embodiments, the user can also click on the template icon 32 to view the template theme of the current target video 31. After the user clicks on the icon of the template 32, the mobile phone 100 responds to the user's click operation and enters the video interface 30'. The video interface 30' displays template types 34, such as recommended templates, popular templates, summer templates, etc. And template themes 35 corresponding to the template types 34, for example, in the recommended template types, including nature 1 theme templates, nature 2 theme templates, nature 3 theme templates, etc. The template theme of the current target video 31 is the recommended nature theme, and the user can also select other templates as the theme of the target video 31. After the user has determined the theme of the target video 31, he can click the export button 33 to export the video 31.

[0043] It can be understood that the mobile phone 100 will first determine the theme of the video selected by the user when generating the target video 31 and generating the recommended template, and generate the target video 31 or recommend the corresponding template based on the determined theme using the template corresponding to the theme.

[0044] In some embodiments, when the mobile phone 100 determines the theme of the video, it is generally determined by the visual information of all the image frames in the video. For example, the mobile phone 100 can determine the type of element corresponding to each image frame (such as people, trees, animals, etc.), and determine the video theme based on the element type with the largest number of frames. Since one type of element can correspond to different video themes, for example, tree elements can correspond to natural themes, travel themes, etc. Therefore, it is not accurate enough to determine the theme of the video only through the visual information of the video. As a result, the template for processing the video recommended by the mobile phone 100 is not accurate enough, and the user needs to find a suitable theme by himself, which reduces the user's experience.

[0045] For example, Figure 2 A scene graph for identifying the subject of a video is shown.

[0046] The following takes the example of using the one-click blockbuster function in the gallery of the mobile phone 100 to create a target video from a video taken by a user during travel to introduce a scene graph of the mobile phone 100 identifying the subject of a video. The target video has a duration of 13 seconds and a frame rate of 60 frames / s.

[0047] like Figure 2 As shown in the figure, the user selected 3 videos as video materials and generated the target video through the one-click blockbuster function. The first video of the target video starts at t0=0s and ends at t1=6s, with a duration of t1-t0=6s. The image frames of the first video are 60 frames / s×6s=360 frames, of which trees appear in 360 frames and animals appear in 60 frames.

[0048] The second segment of the target video starts at t1=6s and ends at t2=10s, with a duration of t2-t1=4s. The image frames of the second segment of the video are 60 frames / s×4s=240 frames, of which animals appear in 240 frames and trees appear in 70 frames.

[0049] The third segment of the target video starts at t2 = 10s and ends at t3 = 13s, with a duration of t3-t2 = 3s. The image frames of the third segment of the video are 60 frames / s×3s=180 frames, of which all 180 frames are of people.

[0050] When determining the theme of the target video, the electronic device detects the number of all elements that appear in the image frames of the target video, and determines the theme of the target video based on the element with the largest number of image frames. For example, the electronic device detects that 360 frames + 70 frames = 430 frames of images contain trees, 60 frames + 240 frames = 300 frames of images contain animals, and 180 frames of images contain people. It can be seen that the image frames containing trees are the most, and the electronic device will determine the theme of the target video as a natural theme. At this time, when the electronic device recommends the template of the target video to the user, it recommends the video template corresponding to the natural theme. However, some of the videos that the user chooses to make the target video are taken when traveling. The above method for determining the theme of the target video determines that the theme of the target video is a natural theme only based on the fact that the tree element appears the most times in the image frames of the target video. Therefore, it can be understood that it is not accurate enough to determine the theme of the target video only through the visual information of the target video.

[0051] In order to solve the above problems, in some embodiments, the electronic device can identify the theme of the video based on a trained multimodal video classification model. The modality indicates the source of the information. For example, in a video, the video provides visual information as a modality, audio information as a modality, and if the video contains text, the text information is also a modality. The multimodal video classification model can determine the theme of the video based on information from at least two modalities of the video.

[0052] For example, Figure 3 A schematic diagram of identifying video topics based on a multimodal video classification method is shown.

[0053] like Figure 3 As shown, Figure 2 The difference between the embodiments is that when the electronic device identifies the subject of a video, it needs multiple modal information of the video, such as visual information of the video, audio information related to the video, and text information. Among them, the audio information can be, for example, the audio recorded by the user while shooting the video. The text information can be, for example, the text inserted by the user in the video, such as the description of the video entered by the user, and the label text inserted in the video.

[0054] It should be understood that Figure 3 The electronic device can input the visual information, audio information and text information of the video into the multimodal video classification model, and identify the higher-order semantic theme of the video based on the multimodal video classification model. The higher-order semantic video theme can be a theme containing multiple elements, for example, the theme of travel can contain elements such as landscape, architecture, people, and food.

[0055] However, since the multimodal video classification model needs to identify the video theme with higher-order semantics based on multiple modalities such as visual information, audio information, and text information of the video. If the user does not record audio when shooting the video, or there is no information in the audio that can be used for identification, and the user does not enter text information related to the video, the video can only provide visual information and cannot provide information in multiple modalities. This will cause the multimodal video classification model in the electronic device to only be able to identify the video theme from a single modality, and the accuracy of the recognition result will be low.

[0056] In order to solve the problem that multimodal information is required to identify the theme of a video through the multimodal video classification method, the present application provides a video processing method. When an electronic device identifies the theme of a video, a description text extraction model can be used to obtain the description text of the video. Then the electronic device can obtain the theme of the video based on the description text and the visual information in the video. In this way, even if the video itself only includes visual information, the electronic device can determine the theme of the video based on information of multiple modalities, which is conducive to improving the accuracy of the electronic device in determining the theme of the video.

[0057] For example, for Figure 3 In the embodiment, for a video that only contains visual information, a description text extraction model can be used to determine the description text of the video, and then the theme of the video can be determined based on multimodal information such as the visual information and the description text of the video.

[0058] In some embodiments, the description text extraction model may be a pre-trained label model, and the electronic device may input the key frames of the video into the label model to obtain the description text of the video. The training process of the label model will be described below and will not be repeated here.

[0059] After obtaining the description text of the video through the label model, the electronic device can input the description text of the video into the text encoder to obtain the first text feature of the description text of the video. Then, the first text feature is added to multiple topic text features corresponding to multiple topics respectively to obtain multiple second text features corresponding to the multiple topics.

[0060] It should be understood that, in some embodiments, the topic text features corresponding to each topic can be obtained by inputting the text description corresponding to each topic into the aforementioned label model.

[0061] Then, the electronic device inputs the key frames of the video into the visual encoder to extract the visual features of the video, and then obtains the similarity between the visual features and multimodal information such as each second text feature, and determines the topic corresponding to the second text feature with the highest similarity to the visual feature as the video topic.

[0062] It should be understood that a key frame represents each image frame in a video whose similarity is lower than a similarity threshold. For example, a video has 60 frames of images, and the similarity between the 29 frames after the first frame and the first frame exceeds the similarity threshold, then the first frame can be taken as a key frame, which represents the content of the first 30 frames. If the similarity between the 31st frame of the video and the first 30 frames is lower than the similarity threshold, but the similarity between the 29 frames after the 31st frame and the 31st frame is higher than the similarity threshold, then the 31st frame can be taken as the second key frame, that is, there are two key frames in the video.

[0063] In other embodiments, any one of the similar image frames can be extracted as a key frame. For example, among the 60 frames of the video, any one of the first 30 similar images can be extracted as a key frame, and any one of the last 30 similar images can be extracted as a key frame for the last 30 frames.

[0064] It should be understood that the description text of a key frame represents the text content of the elements in the key frame. For example, if the image of the key frame includes elements such as people, forests, and ships, the key frame description text may be people, forests, and ships. The description text of a video may include the description texts of all key frames in the video, for example, the description texts of all key frames are spliced ​​together to obtain the description text of the video.

[0065] The first text feature may be, for example, a vector referring to the key frame text expression obtained by a text encoder identifying the key frame text description. The subject text feature may be, for example, a vector referring to the subject text obtained by a text encoder identifying the subject text. The subject text may be, for example, travel, cute pets, parent-child, etc., that is, the subject text is the name of the subject.

[0066] The label model is, for example, a model trained based on training samples that can obtain the description text of the corresponding image based on the input image. The training sample may include, for example, a sample image and the description text corresponding to the sample image. The training process of the label model is described below.

[0067] The second text feature is obtained by combining the first text feature with each subject text feature. The first text feature is combined with a subject text feature corresponding to a subject text to obtain a second text feature, that is, the number of second text features is the same as the number of subject text features.

[0068] The visual feature may be, for example, a vector representing the video content obtained by identifying all key frames based on a visual encoder.

[0069] It can be understood that the multiple models in the embodiments of the present application can be combined with each other to form new models. For example, the output end of the label model is used as the input end of the text encoder to form a model for extracting the first text feature of the key frame. Or multiple models are combined into one model, that is, all the above models together constitute a model for determining the theme of the video.

[0070] Through this solution, even if the video does not contain multimodal information (for example, the video only contains visual information), the electronic device can obtain the key frame of the video, bring the key frame into the first model to obtain the first text feature describing the key frame, and then merge the first text feature with each preset topic text feature to generate multiple second text features corresponding to the topic, and the preset topic text feature includes the features of the high-order semantic video topic. The electronic device then inputs the key frame of the video into the first visual encoder to obtain the visual feature, and then obtains the similarity between the visual feature of the video and the multiple second text features, and selects the topic corresponding to the second text feature with the highest similarity to the visual feature as the video topic, thereby realizing the classification of the video through multimodality and accurately identifying the high-order semantic video topic that matches the video.

[0071] Figure 4 According to an embodiment of the present application, a flow chart of an implementation method of a video processing method is shown.

[0072] It can be understood that the video processing method provided in the embodiment of the present application can be applicable to electronic devices including but not limited to: mobile phone 100, tablet computer, computer, smart watch, car conference terminal, desktop, laptop, handheld computer, netbook, and augmented reality (AR) / virtual reality (VR) devices, smart TVs, smart watches and other wearable devices, servers, portable game consoles, portable music players, reader devices, and other electronic devices embedded or coupled with one or more processors and capable of processing videos.

[0073] It can be understood that the implementation subjects in the following implementation processes can all be electronic devices, and when describing each implementation process, each execution subject will not be repeatedly introduced.

[0074] like Figure 4 As shown, the implementation process includes:

[0075] 401, storyboard the video material and extract key frames.

[0076] Exemplarily, when processing a video through an electronic device, a user first selects a video material, which may include a video or a picture, for example. The storyboard processing of the video may be, for example, dividing the image frames in the video whose similarity is higher than a similarity threshold into a group of shots, and the extracted key frames may be, for example, extracting one image frame from each group of shots as a key frame. It can be understood that a key frame may represent an image frame in the video whose similarity is lower than a similarity threshold.

[0077] If the video material is a picture, the storyboard processing of the picture can be, for example, based on whether the similarity between the pictures exceeds a similarity threshold, determine whether multiple pictures belong to a group of shots, and select a picture from the group of shots as a key frame. Or based on whether the similarity between the picture and the image frame of the video exceeds a similarity threshold, determine to group the picture and the video frame into a group of shots, and select an image frame or a picture from the group of shots as a key frame.

[0078] like Figure 2As shown in the embodiment, there are three video clips in total, and the similarity threshold is 0.9. In the first video, there are 60 frames of images including the same animal and the same tree. Among these 60 frames of images, the similarity between the last 59 frames of images and the first frame of images exceeds 0.9, so these 60 frames of images can be divided into the first group of shots, and the first frame of images can be selected as the key frame. The remaining 300 frames of images are all the same tree, and among the remaining 300 frames of images, the similarity between the last 299 frames of images and the first frame of images exceeds 0.9, then the remaining 300 frames of images are divided into the second group of shots, and the first frame of images is selected as the key frame among these 300 frames of images.

[0079] In the second video, there are 70 frames of images including animals and trees. If the animals and trees in these 70 frames are the same as the trees and animals in the 60 frames of images in the first video, and the similarity between these 70 frames and the key frames of the first group of shots in the first video exceeds 0.9, then these 70 frames are classified as the first group of shots. Similarly, there are 170 frames of images in the second video that are only trees. If the similarity between these 170 frames and the key frames of the second group of shots in the first video exceeds 0.9, then these 170 frames are classified as the second group of shots.

[0080] In the third video, the electronic device recognizes that all 180 frames are of the same person, and the similarity between the key frames of the first and second shots is less than 0.9. Moreover, among these 180 frames, the similarity between the first frame and the next 179 frames exceeds 0.9, so these 180 frames can be divided into the third shot, and the first frame of these 180 frames is used as the key frame.

[0081] 402, input the key frame into the label model to generate a description text of the video.

[0082] Exemplarily, the label model can, for example, identify the text of the elements in the image based on the picture on the image. After obtaining the description text of each key frame, the description text of the video can be generated. If the video has only one key frame, the description text of the key frame is the description text of the video. If the video includes multiple key frames, all the description texts of the multiple key frames are used as the description text of the video. The specific training method of the label model is described below.

[0083] 403 , input the description text of the video into a text encoder to extract a first text feature.

[0084] Exemplarily, after obtaining the description text of the video, the first text feature of the description text of the video can be extracted by a text encoder. The first text feature can be, for example, a vector representing the description text of the video.

[0085] At 404 , the key frame is input into a visual encoder to extract visual features.

[0086] Exemplarily, after obtaining the key frame of the video material, the features of the key frame can be extracted as visual features based on the visual encoder. The visual feature can be, for example, a vector that refers to the video content.

[0087] It is understandable that there is no order between the above 402 process and the 404 process. In other embodiments, the 404 process may be performed first, and then the 402 process. The embodiments of the present application do not limit the order of the 402 process and the 404 process.

[0088] 405, input the topic category of the video into the text encoder to extract the topic text features.

[0089] Exemplarily, the subject category of the video may be data stored in an electronic device, for example, and the subject categories may include travel, cute pets, parent-child, etc. The text of each subject category is input into a text encoder to obtain subject text features referring to each subject category.

[0090] In other embodiments, the electronic device may also pre-store topic text features corresponding to each topic text, thereby eliminating the need to execute the process of inputting the video topic category into a text encoder to extract the topic text features, and directly executing the process of merging the first text feature with multiple topic text features to obtain multiple second text features corresponding to the multiple topic text features.

[0091] It is understandable that there is no sequence between the above processes 401 to 404 and the process 405. In other embodiments, the process 405 may be performed first and then the processes 401 to 404. The embodiments of the present application do not limit the sequence of the above processes.

[0092] 406 , merging the first text feature with the multiple subject text features to obtain multiple second text features corresponding to the multiple subject text features.

[0093] For example, in the embodiment of the present application, the first text feature can be combined with the theme text features corresponding to each theme category to obtain each second text feature. It should be understood that the number of the second text features is the same as the number of the theme text features.

[0094] For example, the theme text feature of the theme text of travel is combined with the first text feature to obtain a second text feature, and the theme text feature of the theme text of cute pets is combined with the first text feature to obtain a second text feature. After the theme text features of all theme texts stored in the electronic device are combined with the first text feature, all second text features can be obtained.

[0095] 407 , determining the similarity between each second text feature and the visual feature, and taking the theme corresponding to the second text feature having the highest similarity with the visual feature as the theme of the video.

[0096] Exemplarily, in some embodiments of the present application, the electronic device can obtain the similarity between the second text feature and the visual feature. The similarity between the second text feature and the visual feature can refer to the similarity between the description text of the video plus the corresponding theme text and the video content. The higher the similarity, the more the corresponding theme text can refer to the content of the video, so the theme text with the highest similarity can be selected as the video theme. The electronic device can recommend the video template required by the user when processing and making the video according to the theme of the video. The process of obtaining the similarity is described in detail below.

[0097] Figure 5 According to an embodiment of the present application, a schematic diagram of a video subject recognition method is shown.

[0098] like Figure 5 As shown, the video material selected by the user includes three videos, which are spliced ​​together in sequence to form one video, and then the subject of the video is identified.

[0099] When an electronic device identifies a video theme, it first extracts the key frames of the video. The key frame extraction method refers to process 401. The key frames are then input into the label model to obtain the description text of the video, such as trees, animals, people, etc., and then the first text feature N1 of the video description text is extracted based on the text encoder. The electronic device stores the text data of the video theme category, and the text data of the video theme category is input into the text encoder to extract multiple theme text features T1, T2, T3, etc. corresponding to multiple theme categories. The electronic device then merges the first text feature N1 with the multiple theme text features T1, T2, T3, etc. to obtain the second text features M1, M2, M3, etc. corresponding to the multiple theme categories.

[0100] In an embodiment of the present application, the electronic device may also input the key frame into the visual encoder to extract the visual feature V1 that refers to the video content. The electronic device then obtains the similarity between the visual feature V1 and the second text features M1, M2, M3, etc., selects the second text feature with the highest similarity to the visual feature V1, obtains the theme category corresponding to the second text feature, and uses the theme category as the video theme.

[0101] Through the above embodiment, when a user processes a video, the electronic device can obtain the key frames of the video, and determine the first text features and visual features describing the video with the key frames. Then, the first text features are combined with the theme text features corresponding to multiple theme texts preset in the electronic device to obtain multiple second text features. The similarity between the second text features and the visual features is determined by using multiple modalities such as multiple second text features and visual features, and the theme corresponding to the second text feature with the highest similarity to the visual feature is selected as the theme of the video. This ensures that the theme of the video can better summarize the content of the video, making video theme identification more accurate.

[0102] The following introduces the label model in the 402 process.

[0103] It can be understood that in other embodiments of the present application, the server for training the label model can also be a laptop computer, a desktop computer, a television with one or more processors embedded or coupled thereto, or other electronic devices capable of accessing the Internet. The electronic device for transplanting the label model to extract the description text of the key frame can also be a tablet computer, a desktop, a laptop, a handheld computer, a netbook, and an augmented reality / virtual reality device, a smart TV, a smart watch, or other electronic devices capable of accessing the Internet; or a camera, and other electronic devices such as a handheld pan / tilt device, which are not limited here.

[0104] It should be understood that the label model can be trained based on text-image pairs. The text-image pairs include multiple sample images and label texts, where the label texts are descriptions of the sample images. The text-image pairs can be obtained from a public database, or developers can add text descriptions to the images.

[0105] Figure 6a A schematic diagram of a label model training method in an embodiment of the present application is shown.

[0106] like Figure 6a As shown, the label model includes an image encoding module 601, a text encoding module 602, an alignment module 603 and a text decoding module 604.

[0107] The image encoding module 601 is used to extract features from the sample image to obtain image features corresponding to the sample image. The image features are, for example, vectors representing the sample image.

[0108] The text encoding module 602 is used to determine a training text feature based on the input label text. The training text feature is, for example, a vector representing the text.

[0109] The alignment module 603 includes a first parameter, which can be used to adjust the image feature based on the training text feature, and output the adjusted image feature to the text decoding module 604. The process of training the label model is to train the first parameter, so that the output text feature after adjusting the image feature by the first parameter can be more similar to the parameter of the training text feature (for example, improve the similarity between the output text feature and the training text feature).

[0110] The text decoding module 604 is used to obtain the description text corresponding to the output text feature based on the output text feature output by the alignment module 603 .

[0111] The alignment module 603 obtains image features and text features from the image encoding module 601 and the text encoding module 602 respectively, and trains the first parameter of the alignment module 603 based on the image features and the text features, so that the similarity between the output text features obtained by the first parameter according to the image features and the training text features reaches the highest level.

[0112] It should be understood that after the first parameter in the alignment module 603 is trained, the label model can infer the input key frame to obtain the descriptive text of the key frame. For example, the key frame is input into the image encoding module 601 to obtain the image features of the key frame. The alignment module 603 obtains the image features of the key frame. The first parameter trained in the alignment module 603 can adjust the image features of the key frame into output text features and then output them. After the text decoding module 604 obtains the output text features of the key frame, it decodes the output text features to obtain the descriptive text of the corresponding key frame.

[0113] It should be understood that the first parameter may be, for example, a plurality of parameter vectors or parameter matrices in the process of adjusting the image features in the alignment module 603 .

[0114] It should be understood that in other embodiments, the label model may also be, for example, a contrastive language-image pre-training model (CLIP), which uses large-scale text-image pairing pre-training. Therefore, in this embodiment, the extracted key frame may be input into CLIP, which stores a large amount of text descriptions, and the description text matching the key frame may be obtained from the large amount of text descriptions based on the trained CLIP model.

[0115] Next, combine Figure 6b Schematic diagram detailing the label model training process.

[0116] Figure 6b According to an embodiment of the present application, a flowchart of an implementation method of a label model training method is shown.

[0117] It can be understood that the training samples for training the label model can be, for example, the above-mentioned text-image pairs, that is, the training samples include sample images and corresponding description texts describing the sample images.

[0118] It can be understood that the parameters in the image encoding module 601 and the text encoding module 602 are fixed. The image encoding module 601 is used to extract the features of the sample image in the training sample. The text encoding module 602 is used to extract the training text features of the description text corresponding to the sample image. The first parameter in the alignment module 603 is used to adjust the image features to obtain the target output features, and the training of the first parameter is determined based on the similarity between the target output features and the training text features. Therefore, when training the label model, it is only necessary to train the first parameter in the alignment module 603.

[0119] In some embodiments of the present application, the image encoding module 601 can be implemented, for example, by a vision transformer (ViT) model or a convolutional neural network (CNN). The text encoding module 602 can be implemented, for example, by a text transformer model, a pre-trained language representation model (bidirectional encoder representation from transformers, BERT) and an optimized BERT (robustly optimized BERT, RoBERTa) model.

[0120] like Figure 6b As shown, the implementation process includes:

[0121] 610, initialize the first parameter in the alignment module 603 and set the first target number of training times.

[0122] Exemplarily, in an embodiment of the present application, when training the alignment module 603, it is necessary to initialize the first parameter in the alignment module 603. The first parameter in the alignment module 603 is trained by training samples. After the training is completed, the alignment module 603 can adjust the input image features based on the first parameter to determine the output text features. The first target training number is the number of times the first parameter in the alignment module 603 is trained. For example, the first target training number can be set to 1000 times. After the first parameter is trained 1000 times, the training can be terminated.

[0123] 620 , input the image features and the training text features corresponding to the image features to the alignment module 603 .

[0124] Exemplarily, when training the first parameter in the alignment module 603 , a training sample may be input into the alignment module 603 , so as to train the first parameter in the alignment module 603 , wherein the training sample includes image features and training text features corresponding to the image features.

[0125] It can be understood that since the image features and training text features input to the alignment module 603 each time are one-to-one corresponding sample pairs, that is, the descriptive text referred to by the training text features input each time describes the image referred to by the input image features (hereinafter referred to as the output text features obtained after the image features are adjusted by the first parameter and the training text features corresponding to the image features are positive label sample pairs), when multiple image features and training text features are input, there will be sample pairs where the image features and the training text features are not one-to-one corresponding (hereinafter referred to as the output text features obtained after the image features are adjusted by the first parameter and the training text features that do not correspond to the image features are negative label sample pairs). For example, the first image feature corresponds to the first training text feature, and the second image feature corresponds to the second training text feature. However, the first image feature does not correspond to the second training text feature, and the second image feature does not correspond to the first text feature.

[0126] 630 , obtaining similarity between the image feature and the training text feature based on the first parameter.

[0127] Exemplarily, when training the first parameter, the image feature is first adjusted based on the first parameter, the output text feature is determined, and the similarity between the output text feature and the training text feature is determined. In this embodiment, the first parameter needs to be trained to increase the similarity of the positive label sample pair so that the similarity of the positive label sample pair reaches the maximum value, and reduce the similarity of the negative label sample pair.

[0128] Exemplarily, Table 1 shows the similarities between four sets of output text features and four sets of training text features obtained after adjusting the four sets of image features based on the first parameter. It can be understood that the similarities along the diagonal are the similarities corresponding to the positive label sample pairs, and the rest are the similarities corresponding to the negative label sample pairs.

[0129] Table 1

[0130]

[0131] It can be understood that the above-mentioned method of calculating similarity can be, for example, cosine similarity, Pearson correlation coefficient, and Jacobi similarity coefficient, etc. In Table 1, the similarities of the positive label sample pairs are all higher than 0.9, and the similarities of the negative label sample pairs are all lower than 0.3, and the training of the label model can be completed.

[0132] 640, determining whether the current number of training times reaches the first target number of training times.

[0133] Exemplarily, if the current number of training times reaches the target number of training times, for example, the current number of training times is the 1000th time, the training of the first parameter is terminated.

[0134] If the current number of training times does not reach the target number of training times, for example, the current number of training times is less than 1000 times, the process 650 is executed to correct the first parameter based on the loss function.

[0135] At 650, the first parameter is optimized.

[0136] Exemplarily, if the current training times do not reach the target training times, the first parameter is optimized, and the process goes to step 630 to obtain the similarity between the image feature and the training text feature based on the first parameter, and the next training of the first parameter is performed.

[0137] It should be understood that after optimizing the first parameter, the similarity between the output text features after adjusting the image features based on the first parameter and the corresponding training text features increases, and the similarity with the unsuccessful training text features decreases, that is, the similarity between the positive label sample pairs increases, and the similarity between the negative label sample pairs decreases.

[0138] The following describes a method for obtaining the similarity between the second text feature and the visual feature in process 407.

[0139] When determining the similarity between the second text feature and the visual feature, the second text feature needs to be aligned with the visual feature, and the alignment method is, for example, to represent the second text feature and the visual feature by the same vector. In an embodiment of the present application, the two text features and the visual feature can be aligned by training a visual encoding module and a text encoding module. Then, the similarity between the second text feature and the visual feature is determined.

[0140] It can be understood that in other embodiments of the present application, the electronic device for training the visual encoding module and the text encoding module can also be a laptop computer, a desktop computer, a television set in which one or more processors are embedded or coupled, or other electronic devices that can access the Internet. The electronic device for transplanting the visual encoding module and the text encoding module to obtain the similarity between the visual feature and the second text feature can also be a tablet computer, a desktop, a laptop, a handheld computer, a netbook, and an augmented reality / virtual reality device, a smart TV, a smart watch, or other electronic devices that can access the Internet; or a camera, and other electronic devices such as a handheld pan-tilt device, which are not limited here.

[0141] It should be understood that the visual encoding module and the text encoding module can be trained based on the theme-video pair. The theme-video pair includes multiple sample videos and theme texts, where the theme text is the theme of the sample video. The theme-video pair can be obtained from a public database, or a developer can add a theme to the video.

[0142] Figure 7a According to an embodiment of the present application, a schematic diagram of a method for training a visual encoding module and a text encoding module is shown.

[0143] like Figure 7a As shown, the method includes a visual encoding module 701 and a text encoding module 702.

[0144] The visual encoding module 701 is used to extract features from the sample video to obtain training visual features corresponding to the sample video. The training visual features are, for example, vectors representing the sample video.

[0145] The text encoding module 702 is used to determine a training topic feature based on the input topic text. The training topic feature is, for example, a vector that refers to the topic text.

[0146] In some embodiments of the present application, the visual encoding module 701 can be implemented, for example, by a ViT model or a convolutional neural network. The text encoding module 702 can be implemented, for example, by a text transformer model, a pre-trained language representation model, and a RoBERTa model.

[0147] The visual encoding module 701 is also provided with a second parameter, and the second parameter is used to adjust the training visual feature to obtain the first adjustment feature. The text encoding module 702 is also provided with a third parameter, and the third parameter is used to adjust the training subject feature to obtain the second adjustment feature. The first adjustment feature can be aligned with the second adjustment feature, that is, the first adjustment feature and the second adjustment feature are the same vector. The training of the visual encoding module 701 and the text encoding module 702 is the training of the second parameter and the third parameter, so that the similarity between the first adjustment feature and the matching second adjustment feature (hereinafter referred to as the positive sample pair) is improved, and the similarity with the unmatched second adjustment feature (hereinafter referred to as the negative sample pair) is reduced.

[0148] It should be understood that the second parameter may be, for example, a plurality of parameter vectors or a parameter matrix in the process of adjusting the training visual features in the visual encoding module 701. The third parameter may be, for example, a plurality of parameter vectors or a parameter matrix in the process of adjusting the training subject features in the text encoding module 702.

[0149] It should be understood that the positive sample pair represents the first adjustment feature and the second adjustment feature corresponding to the main image text of the sample video that matches the sample video, and the negative sample pair represents the first adjustment feature and the second adjustment feature corresponding to the subject text of the sample video that does not match the sample video.

[0150] When training the visual encoding module 701 and the text encoding module 702, first obtain the sample video and the subject text matching the video.

[0151] The sample video in the training sample is then input into the visual encoding module 701 to obtain the training visual features, and then the training video features are adjusted based on the second parameter to generate the first adjusted features.

[0152] The subject text in the training sample, such as travel, cute pets, parent-child and other themes, is input into the text encoding module 702 to obtain the training subject features, and then the training subject features are used to generate the second adjustment features based on the third parameters.

[0153] Then, based on the similarity between the first adjustment feature and the second adjustment feature, the second parameter and the third parameter are trained so that the similarity of the positive sample pair is improved and the similarity of the negative sample pair is reduced. The training process narrows the similarity of the positive sample pair by comparing the loss function.

[0154] Figure 7b According to some embodiments of the present application, an implementation flow chart of training a visual encoding module 701 and a text encoding module 702 is shown.

[0155] like Figure 7b As shown, the implementation process includes:

[0156] 710, initialize the second parameter in the visual encoding module 701 and the third parameter in the text encoding module 702, and set the second target training times.

[0157] Exemplarily, in an embodiment of the present application, when training the visual coding module 701 and the text coding module 702, it is necessary to initialize the second parameter in the visual coding module 701 and the third parameter in the text coding module 702. The second parameter and the third parameter are trained through training samples. After the training is completed, the visual coding module 701 can adjust the training visual features generated by the visual coding module 701 according to the sample video based on the second parameter, thereby determining the first adjustment feature. The text coding module 702 can adjust the training topic features generated by the text coding module 702 according to the topic text based on the third parameter, thereby determining the second adjustment feature. The vectors of the first adjustment feature and the second adjustment feature have the same order. The second target training number is the number of times the second parameter and the third parameter are trained. For example, the second target training number can be set to 1000 times. After the second parameter and the third parameter are trained 1000 times, the training can be ended.

[0158] 720 , input the sample video into the visual encoding module 701 , and input the theme text into the text encoding module 702 .

[0159] For example, when training the second parameter in the visual coding module 701 and the third parameter in the text coding module 702, the visual coding module 701 needs to first determine the training visual features of the sample video, and then adjust the training visual features to the first adjustment features according to the second parameters. Similarly, the text coding module 702 needs to first determine the training topic features of the topic text, and then adjust the training topic features according to the third parameters to determine the second adjustment parameters.

[0160] In other embodiments, the visual encoding module 701 may also determine the first adjustment feature through the second parameter in the process of determining the training video feature (i.e., the visual encoding module 701 directly determines the first adjustment parameter based on the sample video and the second parameter). The text encoding module 702 may also determine the second adjustment feature through the third parameter in the process of determining the training theme feature (i.e., the text encoding module 702 directly determines the second adjustment feature based on the theme text and the third parameter).

[0161] It can be understood that since the sample video and the theme text input to the visual encoding module 701 and the text encoding module 702 each time are one-to-one corresponding sample pairs, that is, the theme text input each time is the theme of the corresponding sample video. Therefore, the first adjustment feature determined according to the input sample video and the second adjustment feature determined according to the corresponding theme text are a positive sample pair.

[0162] 730 , determine the similarity between the first adjustment feature and the second adjustment feature based on the second parameter and the third parameter.

[0163] Exemplarily, when training the second parameter and the third parameter, the similarity between the first adjustment feature and the second adjustment feature is first determined. In this embodiment, the second parameter and the third parameter need to be trained so that the similarity of the positive sample pair exceeds the third similarity threshold, and the similarity of the negative sample pair is lower than the fourth similarity threshold. This ensures that the positive sample pair has a higher similarity, while the negative sample pair has a lower similarity.

[0164] It can be understood that the above-mentioned method of calculating similarity may be, for example, cosine similarity, Pearson correlation coefficient, Jacobi similarity coefficient, etc.

[0165] 740, determining whether the current number of training times reaches the second target number of training times.

[0166] Exemplarily, if the current number of training times reaches the second target number of training times, for example, the current number of training times is the 1000th time, the training of the second parameter and the third parameter is terminated.

[0167] If the current number of training times does not reach the second target number of training times, for example, the current number of training times is less than 1000 times, the process 650 is executed to optimize the second parameter and the third parameter.

[0168] At 750, the second parameter and the third parameter are optimized.

[0169] Exemplarily, if the current number of training times does not reach the second target number of training times, the second parameter and the third parameter are optimized, and the process goes to 720 to input the sample video into the visual encoding module 701 and the theme text into the text encoding module 702. The next training of the second parameter and the third parameter is performed.

[0170] It should be understood that after optimizing the second parameter and the third parameter, the similarity between the first adjusted feature after adjusting the training video feature based on the second parameter and the corresponding second adjusted feature after adjusting the training subject feature based on the third parameter increases, and the similarity with the non-corresponding second adjusted feature decreases, that is, the similarity of the positive sample pair increases, and the similarity of the negative sample pair decreases.

[0171] It can be understood that the second and third parameters after training make the similarity of the positive sample pairs as large as possible, and the similarity of the negative sample pairs as small as possible. The second and third parameters can, for example, shorten the distance between the positive sample pairs and increase the distance between the negative sample pairs. For example, in an embodiment of the present application, the input sample video corresponds to the theme of tourism, and the input main picture text is tourism, parent-child, and cute pets, etc. Therefore, the second and third parameters can be trained based on the contrast learning function, so that the similarity between the first adjustment feature corresponding to the training visual feature obtained based on the second and third parameters and the second adjustment feature corresponding to the training theme feature referring to the theme of tourism increases, while the similarity between the first adjustment feature corresponding to the training visual feature and the second adjustment feature corresponding to the training theme feature referring to themes such as parent-child and cute pets decreases.

[0172] Exemplarily, the contrast loss function can be defined by formula (1).

[0173]

[0174] Wherein, q represents the first adjustment characteristic, and k represents the second adjustment characteristic. + Indicates the similarity of the positive sample pair. n represents the number of sample pairs, that is, the number of pairs of q and k input. i is an integer between 1 and n. q·k i Indicates the similarity between the first adjustment feature and the i-th second adjustment feature. τ is the temperature parameter. The larger τ is, the The smaller the i / τ will be smaller, the difference in similarity between sample pairs will be smaller, and the distribution of sample pairs will be smoother. On the contrary, the smaller τ is, the smaller q·k i The larger the value of / τ, the difference in similarity between sample pairs will change, and the distribution of sample similarity will be more concentrated. q represents the contrastive learning function of the first adjusted feature q.

[0175] From formula (1), we can get that for each first adjustment feature q, there is only one second adjustment feature k + Corresponding to it, a positive sample pair is formed, and the other second adjustment features k form negative sample pairs. For each first adjustment feature q, the similarity of the positive sample pair is q·k + The higher the value, the better the contrast learning function L q The closer it is to 0. For example, in the 750 process, the goal of optimizing the second and third parameters is to make the L of the positive sample pair q Reduce, L of negative sample pairs q Increase.

[0176] After training the second and third parameters, Figure 6a The description text of the key frame obtained in the embodiment is merged into the description text of the video and input into the text encoding module 702 (in Figure 4 In the embodiment of the present invention, the text encoder is used to obtain the first text feature. The text of the video theme is also input into the text encoding module 702 to obtain the theme text feature. The theme text feature is combined with the first text feature to obtain the second text feature. Then the video is input into the video encoding module 701 (in Figure 4 In the embodiment of the present invention, the video encoder is used to obtain the visual features of the video, and then determine the similarity between the visual features and the second text features, and select the topic text corresponding to the second text features with the highest similarity to the visual features as the topic of the video. Determining the topic of the video based on multimodal information in this application is to obtain the topic of the video through the second text features and the visual features.

[0177] In other embodiments, only the second parameter may be set in the visual encoding module 701, the second parameter adjusts the training visual feature to the output theme feature so as to align with the training theme feature, and the second parameter is trained based on the similarity between the output theme feature and the training theme feature. In this way, the training visual feature is aligned with the training theme feature. Alternatively, only the third parameter may be set in the text encoder, the third parameter adjusts the training theme parameter to align the output visual feature with the training visual feature, and the third parameter is trained based on the similarity between the output visual feature and the training visual feature. In this way, the training visual feature is aligned with the training theme feature.

[0178] Through the above-mentioned method of training the visual encoding module 701 and the text encoding module 702, after the second parameter and the third parameter are trained, the similarity between the second text feature and the visual feature can be determined, and the subject category corresponding to the second text feature with the highest similarity to the visual feature can be selected as the video subject. Among them, the second text feature includes the subject text feature and the first text feature indicating the description text of the video. Since the subject text expressing the same subject is diverse, for example, the text expressing the cute pet subject can also be called the subject of pets, cats, dogs, etc. Therefore, it is not accurate enough to obtain the similarity between the subject and the video only by inputting the subject text feature. For example, if cats and dogs are used as the subject text, the similarity between the extracted subject text feature and the visual feature of the video with the content of pet cats or pet dogs can be very high, but the similarity with the video with the content of pet ducks or pet birds will not be very high. Therefore, when determining the video subject, the first text feature corresponding to the description text of the extracted video and the subject text feature can be merged into the second text feature, and the subject of the video can be confirmed by obtaining the similarity between the second text feature and the visual feature. In this way, the similarity between the theme text features and the visual features is shortened through the features of the video description text, thereby avoiding the problem that the difference in the diversity of the theme text leads to a large difference in the similarity between the video and the theme, thereby making the determined theme of the video more accurate.

[0179] Taking a mobile phone as an example, the electronic devices involved in some embodiments of the present invention are described in detail below.

[0180] Figure 8 According to an embodiment of the present application, a structural schematic diagram of a mobile phone is shown.

[0181] like Figure 8As shown, the mobile phone 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0182] It is to be understood that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the mobile phone 100. In other embodiments of the present application, the mobile phone 100 may include more or fewer components than shown in the figure, or combine some components, or separate some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0183] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0184] The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of instruction fetching and execution.

[0185] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0186] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module interface, and / or a universal serial bus interface, etc.

[0187] The charging management module 140 is used to receive charging input from a charger, where the charger can be a wireless charger or a wired charger.

[0188] The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140 to power the processor 110, the internal memory 121, the display screen, the camera, and the wireless communication module 160.

[0189] The wireless communication function of the mobile phone 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0190] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in mobile phone 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization rate of the antennas.

[0191] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to the mobile phone 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, etc. the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1.

[0192] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the mobile phone 100. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and performs filtering, and sends the processed signal to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, modulate the frequency of it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0193] The external memory interface 120 can be used to connect an external memory card, such as a micro SD card, to expand the storage capacity of the mobile phone 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function. For example, files such as music and videos can be stored in the external memory card.

[0194] The internal memory 121 can be used to store computer executable program codes, which include instructions. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the mobile phone 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 110 executes various functional applications and data processing of the mobile phone 100 by running instructions stored in the internal memory 121 and / or instructions stored in a memory provided in the processor.

[0195] The pressure sensor 180A is used to sense pressure signals and convert the pressure signals into electrical signals.

[0196] The touch sensor 180K is also called a "touch control device". The touch sensor 180K can be set on the display screen 194. The touch sensor 180K and the display screen 194 form a touch screen, also called a "touch control screen". The touch sensor 180K is used to detect touch operations acting on or near it.

[0197] The SIM card interface 195 is used to connect a SIM card. The SIM card can be connected to and separated from the mobile phone 100 by inserting it into the SIM card interface 195 or pulling it out from the SIM card interface 195. In some embodiments, the mobile phone 100 uses an eSIM, that is, an embedded SIM card. The eSIM card can be embedded in the mobile phone 100 and cannot be separated from the mobile phone 100. In the accompanying drawings, some structural or method features may be shown in a specific arrangement and / or order. However, it should be understood that such a specific arrangement and / or ordering may not be required. Instead, in some embodiments, these features may be arranged in a manner and / or order different from that shown in the illustrative drawings. In addition, the inclusion of structural or method features in a particular figure does not imply that such features are required in all embodiments, and in some embodiments, these features may not be included or may be combined with other features.

[0198] It should be noted that the units / modules mentioned in the various device embodiments of the present application are all logical units / modules. Physically, a logical unit / module can be a physical unit / module, or a part of a physical unit / module, or can be implemented as a combination of multiple physical units / modules. The physical implementation method of these logical units / modules themselves is not the most important. The combination of functions implemented by these logical units / modules is the key to solving the technical problems proposed by the present application. In addition, in order to highlight the innovative part of the present application, the above-mentioned device embodiments of the present application do not introduce units / modules that are not closely related to solving the technical problems proposed by the present application, which does not mean that there are no other units / modules in the above-mentioned device embodiments.

[0199] It should be noted that in the examples and description of this patent, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "including one" do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0200] Although the present application has been illustrated and described with reference to certain preferred embodiments thereof, it will be apparent to those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the present application.

Claims

1. A video processing method, applied to an electronic device, characterized in that: include: Acquire a video to be processed, and extract text information and visual information of the video to be processed, wherein the text information of the video to be processed is obtained by merging description texts of each key frame in the video to be processed, and the visual information includes at least one key frame; Inputting the text information and visual information of the video to be processed into a first model to obtain the subject of the video to be processed; The step of inputting the text information and visual information of the video to be processed into a first model to obtain the subject of the video to be processed includes: Merging the first text feature corresponding to the text information with a plurality of topic text features corresponding to a plurality of preset topics to obtain a plurality of second text features corresponding to the plurality of topics; The second text feature and the visual feature are represented by the same vector, the similarity between the second text feature and the visual feature is obtained, and the theme corresponding to the second text feature having the highest similarity to the visual feature corresponding to the visual information is used as the theme of the video to be processed.

2. The method according to claim 1, characterized in that The extracting of text information and visual information of the video to be processed includes: Acquire at least one key frame of the video to be processed; Inputting the at least one key frame into a description text extraction model to obtain a description text of the key frame; The description text of the at least one key frame is merged to obtain text information of the video to be processed.

3. The method according to claim 2, characterized in that The obtaining of at least one key frame of the video to be processed includes: Acquire multiple image frames of the video to be processed; Dividing the plurality of image frames into a plurality of shot groups, wherein the similarity between the image frames in each shot group exceeds a first similarity threshold; One image frame is selected in each shot group as the key frame.

4. The method according to claim 2, characterized in that: The step of inputting the at least one key frame into a description text extraction model to obtain the description text of the key frame comprises: The description text extraction model includes an image encoding module, an alignment module and a decoding module; The image encoding module extracts a first image feature of the key frame; The alignment module obtains output text features based on the first image features; The decoding module decodes the output text features into description text of the key frame.

5. The method according to any one of claims 1 to 4, characterized in that: The first model includes a text encoding module and a visual encoding module; The first text feature corresponding to the text information is obtained by extracting the feature of the text information using the text encoding module; The visual features corresponding to the visual information are obtained by extracting features from the visual information using the visual encoding module.

6. The method according to claim 5, characterized in that The multiple subject text features are obtained in the following manner: The text encoding module is used to extract features of the multiple topics to obtain multiple topic text features corresponding to the multiple topics.

7. An electronic device, characterized in that: including a memory for storing instructions; At least one processor is used to execute the instructions so that the electronic device implements the video processing method according to any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that: The readable storage medium stores instructions, and when the instructions are executed on a computer, the computer is caused to execute the video processing method according to any one of claims 1 to 6.

Images