Video processing method and device, equipment and medium

Abstract

The embodiment of the invention relates to a video processing method and device, equipment and a medium. The method comprises the following steps: acquiring a to-be-processed first video; obtaining target content information of the first video; wherein the target content information comprises voice content information and / or picture content information; determining a target clip from the first video based on the target content information of the first video; the number of the target fragments is one or more; and generating a second video based on the target clip. According to the embodiment of the invention, a user does not need to spend time and energy to execute operations such as video segmentation, and a relatively long first video can be automatically and efficiently converted into a relatively short second video based on the content theme contained in the first video, so that user requirements are met.

Description

Technical Field

[0001] This disclosure relates to the field of multimedia processing technology, and in particular to a video processing method, apparatus, device and medium. Background Technology

[0002] With the rise of video sharing platforms, more and more users are posting videos on these platforms. The inventors discovered that, due to the demand for video creation or the video length limitations of video publishing platforms, some users want to convert their existing long videos into shorter ones. Currently, users need to spend time and effort manually performing operations such as video segmentation to convert long videos into short ones, which is cumbersome and requires significant labor costs. Summary of the Invention

[0003] To solve the above-mentioned technical problems, or at least partially solve them, this disclosure provides a video processing method, apparatus, device, and medium.

[0004] This disclosure provides a video processing method, the method comprising: acquiring a first video to be processed; acquiring target content information of the first video; wherein the target content information includes audio content information and / or video content information; determining target segments from the first video based on the target content information of the first video; wherein the number of target segments is one or more; and generating a second video based on the target segments.

[0005] Optionally, obtaining the target content information of the first video includes: performing speech recognition processing on the first video to obtain a speech recognition result; if the speech recognition result indicates that the ratio between the length of the speech in the first video and the duration of the first video is greater than a preset ratio threshold, obtaining the speech content information of the first video based on the speech recognition result; and if the speech recognition result indicates that the ratio between the length of the speech in the first video and the duration of the first video is not greater than a preset ratio threshold, obtaining the screen content information of the first video.

[0006] Optionally, determining the target segment from the first video based on the target content information of the first video includes: identifying the content theme contained in the first video based on the target content information of the first video; wherein the number of the content theme is one or more; and segmenting the first video based on the content theme to obtain the target segment corresponding to the content theme.

[0007] Optionally, the step of segmenting the first video based on the content theme to obtain the target segment corresponding to the content theme includes: when the duration of the first video is greater than the minimum value of the target duration interval, segmenting the first video based on the content theme and the target duration interval to obtain the target segment corresponding to the content theme, and the duration of the target segment is within the target duration interval.

[0008] Optionally, identifying the content theme contained in the first video based on the target content information of the first video includes: identifying key information contained in the first video based on the target content information of the first video; wherein the key information includes key sentences and / or key scenes; and determining the content theme contained in the first video based on the key information and first target information located before the key information and second target information located after the key information in the first video.

[0009] Optionally, identifying key information contained in the first video based on the target content information of the first video includes: determining the target domain to which the first video belongs based on the target content information of the first video; obtaining a key information identification strategy corresponding to the target domain; and identifying key information contained in the first video based on the key information identification strategy.

[0010] Optionally, identifying key information contained in the first video based on the target content information of the first video includes: when the target content information includes voice content information, identifying key sentences contained in the first video based on the semantics of the voice content information; when the target content information includes image content information, performing frame extraction processing on the first video at preset intervals, and identifying key images contained in the first video based on the image content of the extracted video frame images.

[0011] Optionally, generating a second video based on the target segment includes: performing a preset target operation on the target segment to obtain a second video; wherein the target operation includes one or more of the following: screen adjustment operation, split-screen processing operation, and subtitle addition operation.

[0012] Optionally, when the target operation includes a screen adjustment operation, performing a preset target operation on the target segment to obtain a second video includes: identifying and tracking the main object in the target segment, cropping the frame image in the target segment based on the identification and tracking results, wherein the main object in the target segment is located at a specified position in the cropped frame image; and generating a second video based on the cropped frame image.

[0013] Optionally, when the target operation includes a split-screen processing operation, performing a preset target operation on the target segment to obtain a second video includes: if at least two target objects are detected in the target segment, cropping the frame images in the target segment based on the positions of the target objects to obtain cropped frame images corresponding to each of the at least two target objects; wherein the cropped frame image is a frame image containing a single target object; stitching the cropped frame images corresponding to each of the at least two target objects according to a preset layout to obtain a stitched frame image; and generating a second video based on the stitched frame image.

[0014] Optionally, when the target operation includes a subtitle addition operation, performing a preset target operation on the target segment to obtain a second video includes: obtaining subtitles corresponding to the target segment; determining the position of the subtitles based on the position of the main object in the target segment; wherein the position of the subtitles is outside the position of the main object; and adding the subtitles to the target segment based on the position of the subtitles to obtain a second video.

[0015] Optionally, the method further includes: selecting a target frame image from the second video based on specified features of the main object of the frame image in the second video; and adding a cover image to the second video based on the target frame image.

[0016] Optionally, the method further includes: generating a target title corresponding to the second video based on the target content information of the second video.

[0017] Optionally, the method further includes: when there are multiple second videos, obtaining the video evaluation dimension corresponding to the domain to which the first video belongs and the weights corresponding to different video evaluation dimensions, and generating evaluation scores corresponding to each of the multiple second videos based on the video evaluation dimension and the weights; sorting the multiple second videos based on the evaluation scores; and displaying the multiple second videos based on the sorting results.

[0018] This disclosure also provides a video processing apparatus, comprising: a first video acquisition module for acquiring a first video to be processed; a content information acquisition module for acquiring target content information of the first video; wherein the target content information includes audio content information and / or video content information; a target segment determination module for determining target segments from the first video based on the target content information of the first video; wherein the number of target segments is one or more; and a second video generation module for generating a second video based on the target segments.

[0019] This disclosure also provides an electronic device, the electronic device comprising: a processor; a memory for storing executable instructions of the processor; the processor being configured to read the executable instructions from the memory and execute the instructions to implement the video processing method provided in this disclosure.

[0020] This disclosure also provides a computer-readable storage medium storing a computer program for performing the video processing method provided in this disclosure.

[0021] The technical solution provided in this disclosure can acquire target content information (audio content information and / or video content information) of a first video, and determine one or more target segments from the first video based on the target content information, thereby generating a second video based on the target segments. In this way, users do not need to spend time and effort performing video segmentation or other operations; a longer first video can be automatically and efficiently converted into a shorter second video based on its content theme, thus meeting user needs.

[0022] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0023] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0024] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, those skilled in the art can obtain other drawings based on these drawings without creative effort.

[0025] Figure 1 A flowchart illustrating a video processing method provided in an embodiment of this disclosure;

[0026] Figure 2 This is a schematic diagram of a video processing flow provided in an embodiment of the present disclosure;

[0027] Figure 3 This is a schematic diagram illustrating segment determination provided in an embodiment of the present disclosure;

[0028] Figure 4 This is a schematic diagram illustrating segment determination provided in an embodiment of the present disclosure;

[0029] Figure 5 This is a schematic diagram illustrating segment determination provided in an embodiment of the present disclosure;

[0030] Figure 6 This is a schematic diagram of a split-screen processing method provided in an embodiment of the present disclosure;

[0031] Figure 7 This is a schematic diagram of a video processing flow provided in an embodiment of the present disclosure;

[0032] Figure 8 This is a schematic diagram of the structure of a video processing apparatus provided in an embodiment of the present disclosure;

[0033] Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0034] To better understand the above-mentioned objectives, features, and advantages of this disclosure, the solutions disclosed herein will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0035] Numerous specific details are set forth in the following description in order to provide a full understanding of this disclosure, but this disclosure may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some, and not all, of the embodiments of this disclosure.

[0036] Figure 1 This is a flowchart illustrating a video processing method provided in an embodiment of the present disclosure. The method can be executed by a video processing device, which can be implemented using software and / or hardware, and is generally integrated into an electronic device. Figure 1 As shown, the method mainly includes the following steps S102 to S108:

[0037] Step S102: Obtain the first video to be processed. This embodiment of the disclosure does not limit the content of the first video.

[0038] Step S104: Obtain target content information of the first video; wherein, the target content information includes audio content information and / or video content information. In practical applications, the type of target content to be obtained can be determined according to requirements or information such as the length of the audio in the first video. For example, if the audio in the first video is long enough to understand the content, only the audio content information can be obtained. If the audio in the first video is short enough to understand the content, only the video content information can be obtained, or both audio and video content information can be obtained simultaneously to fully understand the content of the first video.

[0039] Step S106: Based on the target content information of the first video, determine the target segments from the first video; wherein the number of target segments is one or more. For example, different target segments correspond to different content themes. In practical applications, the content themes contained in the first video can be identified based on the target content information, thereby determining the target segment corresponding to each content theme.

[0040] Step S108: Generate a second video based on the target segments. In some embodiments, there is a one-to-one correspondence between the target segments and the second videos. The target segments can be directly used as the second videos, or post-processing such as packaging can be applied to the target segments to use them as the second videos. In other embodiments, the relationship between target segments and second videos can be many-to-one, meaning multiple target segments can be merged into one second video. For example, assuming there are M target segments, N second videos are generated; M is greater than or equal to 2, N is greater than or equal to 1, and M is greater than N. Specifically, based on the specific content of the target segments, at least two target segments with related content among the M target segments can be merged and spliced ​​to obtain a second video. If there is a single target segment with independent content among the M target segments, a corresponding second video can be generated separately for that target segment. Furthermore, multiple target segments can be directly merged and spliced ​​into one second video. Since the target segments are all extracted from the first video, even if multiple target segments are merged into one second video, the duration of the second video will be shorter than the duration of the first video. The above methods can be flexibly set and are not limited here. In practical applications, the duration of the second video to be generated can also be obtained. This duration can be determined based on the video duration requirements of the video sharing platform. Different video sharing platforms may have different video duration requirements. Then, based on the duration of the second video to be generated, the method of generating the second video from the target segment can be determined, such as determining whether two or more target segments need to be merged, and determining the number of target segments to be merged.

[0041] Using the methods described above, users can automatically and efficiently convert longer first videos into shorter second videos based on their content themes, without requiring them to spend time and effort on video segmentation or other operations. This meets user needs. Furthermore, these methods also help users efficiently and conveniently convert their own long videos (first videos) into short videos (second videos) that meet the length limits of video sharing platforms, enabling users to easily share their videos across different platforms and further promoting video dissemination and sharing.

[0042] For easier understanding, please refer to Figure 2The diagram illustrates a video processing workflow. It shows how to obtain audio and video content information from a first video, thereby identifying the themes and content contained within the first video. It should be noted that... Figure 2 This is just a simple example using content themes 1 through N. In practical applications, the number of content themes contained in the first video is not fixed; it may be only one or multiple, depending on the recognition results. Then, target segments containing each content theme can be segmented from the first video, and corresponding second videos can be generated based on these target segments. Furthermore... Figure 2 This example illustrates the generation of N second videos from N target segments. As mentioned earlier, in practical applications, M second videos can be generated from N target segments, and this is not a limitation. Using this method, a relatively long first video can be reasonably divided into multiple second videos, efficiently and reliably providing users with the function of generating short videos from long videos, eliminating the need for users to spend time and effort on video segmentation themselves.

[0043] In some implementations, the steps for obtaining the target content information of the first video described above can be performed with reference to steps 1 to 3 below:

[0044] Step 1: Perform speech recognition processing on the first video to obtain the speech recognition result. In practical applications, ASR (Automatic Speech Recognition) technology can be used to perform speech recognition processing on the first video, converting the speech in the first video into text. That is, the speech recognition result can be used to indicate the length of the speech contained in the first video, as well as the text information containing the converted speech.

[0045] Step 2: If the ratio of the audio length to the video duration in the first video, as indicated by the speech recognition result, is greater than a preset ratio threshold, the audio content information of the first video is obtained based on the speech recognition result. The preset ratio threshold can be flexibly set according to needs, such as 15%. That is, it can be determined whether the proportion of audio in the first video is sufficient. Only when the proportion of audio is high can it be concluded that the content theme of the first video can be reliably understood based on the audio content, and therefore the audio content information can be used as the required target content information. The text information in the speech recognition result can be directly used as the audio content information.

[0046] Step 3: If the ratio of the speech length to the duration of the first video in the speech recognition result is not greater than a preset ratio threshold, obtain the video content information from the first video. That is, if the first video has only a low proportion of speech, it indicates that the content theme of the first video cannot be accurately and reliably understood based on speech. To ensure the accuracy of video understanding, the video content information can be used as the required target content information. The above is only one example. In practical applications, if the ratio of the speech length to the duration of the first video in the speech recognition result is not greater than a preset ratio threshold, further obtain the subtitles obtained from the speech conversion and attempt to understand the video based on the subtitles. If the video content can be understood based on the subtitles (e.g., the number of characters in the subtitles is higher than a preset number), or the video content can be accurately understood based on the subtitles using a specific language understanding model, the speech content information can still be used as the target content information. The specific method of obtaining speech content information and / or video content information as target content information can be flexibly set and is not limited here.

[0047] In some specific implementation examples, determining the target segment from the first video based on the target content information can be performed by referring to the following steps A and B:

[0048] Step A: Based on the target content information of the first video, identify the content themes contained in the first video; wherein, the number of content themes is one or more. For example, the content themes can be identified by referring to the following steps A1 to A2.

[0049] Step A1: Based on the target content information of the first video, identify the key information contained in the first video; whereby key information includes key sentences and / or key scenes. The aforementioned key information can also be referred to as highlight information. Specifically, whether each sentence and / or scene in the target content information is key information (highlight information) can be defined according to requirements. In some specific implementation examples, based on the target content information of the first video, determine the target domain to which the first video belongs; obtain the key information identification strategy corresponding to the target domain, and identify the key information contained in the first video based on the key information identification strategy. The target domain can be broadly categorized into core domains and non-core domains. Furthermore, core domains can also be categorized into popular vertical domains such as film and television, e-commerce, and gaming. Understandably, different fields use different standards to measure whether content in a video is critical (highlighted). Therefore, corresponding critical information identification strategies can be set for each field. For example, in the gaming field, critical information identification strategies indicate that sentences or images such as scoring a goal or successfully completing a level are critical information; in the film and television field, critical information identification strategies indicate that sentences containing specific words are critical sentences (highlighted sentences), and images containing facial expressions that meet preset expression constraints are critical images (highlighted images). Specific critical information identification strategies for each field can be flexibly set, and no restrictions are imposed here.

[0050] In some specific implementation examples, when the target content information includes audio content information, key sentences contained in the first video are identified based on the semantics of the audio content information; when the target content information includes video content information, the first video is processed by frame extraction at preset intervals, and key scenes contained in the first video are identified based on the video content of the extracted video frame images. Specifically, the key sentence identification method can be based on the above key information identification strategy, which will not be elaborated here. However, when identifying key scenes, in order to improve processing efficiency, the first video can be processed by frame extraction at preset intervals, and the video content of the extracted video frame images can be analyzed to obtain the key scenes.

[0051] Step A2: Based on the key information and the first target information preceding the key information and the second target information following the key information in the first video, determine the content theme contained in the first video. That is, for the statements before and after the key statements and / or the scenes before and after the key scenes, accurately define the content theme corresponding to the key information, and further, based on the key information and the information before and after it, reasonably and accurately segment the first video to obtain the target segments corresponding to the content theme.

[0052] Step B involves segmenting the first video based on its content theme to obtain the target segment corresponding to that theme. Based on the content theme, the aforementioned key information, and the information preceding and following it, a target segment meeting the requirements can be obtained. To ensure the target segment's duration meets the requirements, in some implementations, if the first video's duration exceeds the minimum value of the target duration range, the first video can be segmented based on the content theme and the target duration range to obtain the target segment corresponding to the content theme, and the target segment's duration will fall within the target duration range. The target duration range indicates the length of the target segment. In practical applications, the target duration range can be set by the user, such as less than 60 seconds, 60 seconds to 90 seconds, 90 seconds to 3 minutes, etc. Specifically, multiple selectable duration ranges can be provided for the user to choose from. It should be noted that if the first video's duration is less than the minimum value of the target duration range, then the first video need not be segmented. In addition, to ensure the video segmentation effect, a minimum duration threshold (such as 60 seconds) can be set in practical applications. After obtaining the first video, the duration of the first video is compared with the minimum duration threshold. Only if the duration of the first video is greater than the minimum duration threshold will subsequent processing be performed. If the duration of the first video is not greater than the minimum duration threshold, then only optimization processing such as removing filler words, adding subtitles, and cropping the first video is required.

[0053] For ease of understanding, this disclosure provides reference examples for steps A to B described above, and provides examples such as... Figures 3-5 The diagrams shown are schematic representations of segment determination. Figure 3 The diagram illustrates that the identified key phrases (also known as highlighted phrases) can be expanded before and after to form corresponding target segments. Figure 4 The diagram illustrates that target segments can be directly determined based on topics. Long topics such as Topic 1 or Topic 3 can be further divided into multiple shorter topics, and target segments can be formed based on each short topic to meet users' short video length requirements. Figure 5 The diagram illustrates that key scenes (also known as highlight scenes) can be identified based on extracted frame images from a video, and the target segment can be obtained by further performing operations such as deduplication and aggregation on the video frame images. In practical applications, when the target content information includes audio content information, it can be referred to... Figure 3 or Figure 4 When the target content information includes screen content information, it can be referred to Figure 5 Achieved. (The above is correct.) Figures 3-5 This is merely an illustrative example; in practical applications, other methods may be used to determine the target segment, and no limitations are imposed here.

[0054] In some embodiments, the above-mentioned generation of a second video based on a target segment includes: performing a preset target operation on the target segment to obtain a second video; wherein the target operation includes one or more of the following: screen adjustment operation, split-screen processing operation, and subtitle addition operation. For ease of understanding, these will be described in detail below.

[0055] For example, when the target operation includes a screen adjustment operation, a preset target operation is performed on the target segment to obtain a second video, including: identifying and tracking the main object in the target segment; cropping the frame image in the target segment based on the identification and tracking results, wherein the main object in the target segment is located at a specified position in the cropped frame image; and generating a second video based on the cropped frame image. The aforementioned specified position in the frame can be flexibly set according to needs, such as the center position of the screen. Specifically, neural network models such as video saliency detection models, face and body detection models, and video object tracking models can be used to implement the above steps of identifying and tracking the main object in the target segment, so that the main object is always located in the center of the screen, presenting the user with a clearer and more stable visual perception of the main object.

[0056] For example, when the target operation includes a split-screen processing operation, performing a preset target operation on the target segment to obtain a second video includes: if at least two target objects are detected in the target segment, cropping the frame images in the target segment based on the positions of the target objects to obtain cropped frame images corresponding to each of the at least two target objects; wherein, the cropped frame image is a frame image containing a single target object; stitching the cropped frame images corresponding to each of the at least two target objects according to a preset layout to obtain a stitched frame image; and generating a second video based on the stitched frame image. The preset layout could be a top-to-bottom stitched layout, thereby achieving the effect of switching from landscape to portrait mode. For ease of understanding, refer to... Figure 6 The diagram illustrates a split-screen processing approach. In a dialogue scenario, the original video presents two people talking side-by-side. Through the split-screen processing described above, the visual effect of the two people talking vertically can be achieved. In practical applications, the layout can be flexibly changed based on the number of target objects; no restrictions are imposed here.

[0057] For example, when the target operation includes adding subtitles, a preset target operation is performed on the target segment to obtain a second video, including: obtaining subtitles corresponding to the target segment; determining the position of the subtitles based on the position of the main object in the target segment; wherein the position of the subtitles is outside the position of the main object; and adding subtitles to the target segment based on the position of the subtitles to obtain the second video. In practical applications, the speech recognition result of the target segment can be obtained, and the subtitles corresponding to the target segment can be obtained based on the speech recognition result of the target segment. To avoid the subtitles obscuring the main object, the position of the subtitles can be determined based on the position of the main object in the target segment. For example, taking a video canvas size of 9:16 vertical screen as an example, and taking a face as an example, assuming the face is concentrated in 70% of the video screen and there is no face in the lower 30%, the subtitles are centered in the lower 30% position; assuming a face appears in the lower 30%, it is further determined whether there is also a face in the upper 30%; if there is a face, the subtitles are displayed in the center of the canvas; assuming a face appears in the lower 30%, it is determined whether there is also a face in the upper 30%; if there is no face, the subtitles are in the upper 30% position. In addition, you can set the distance threshold between the subtitles and the edge of the video screen, and set the display of long subtitles in multiple lines to ensure the visual effect of the added subtitles.

[0058] In some embodiments, the method further includes: selecting a target frame image from the second video based on specified features of the main object in the frame images of the second video; and adding a cover image to the second video based on the target frame image. When the main object has a face, the specified features may include facial expression features; furthermore, the specified features may also include the complete outline features of the main object, etc. Based on the specified features, a frame image with a complete outline and optimal expression can be selected as the target frame image, thereby serving as an attractive cover image.

[0059] In some implementations, the method further includes generating a target title corresponding to the second video based on the target content information of the second video. In practical applications, a neural network model can be used to extract the content theme from the second video to generate a target title that meets preset word count and / or preset sentence count conditions. In addition, the target title can be marked on the target cover, and the position of the target title is determined based on the position of the main object in the target cover.

[0060] In some implementations, the method further includes: when there are multiple second videos, obtaining the video evaluation dimensions corresponding to the domain of the first video and the weights corresponding to different video evaluation dimensions, and generating evaluation scores for each of the multiple second videos based on the video evaluation dimensions and weights; sorting the multiple second videos based on the evaluation scores; and displaying the multiple second videos based on the sorting results. Specifically, for each second video, it can be scored according to each video evaluation dimension, and the scores corresponding to each video evaluation dimension can be weighted based on the weights corresponding to each video evaluation dimension to obtain the evaluation score of the second video. Then, the multiple second videos are sorted in descending order of evaluation scores. In the specific method of displaying multiple second videos based on the sorting results, all second videos can be displayed in chronological order, or only the first M second videos can be displayed, which can be flexibly set.

[0061] In practical applications, the same or different video evaluation dimensions can be set for different fields, and the weights of the video evaluation dimensions corresponding to different fields can also be set according to the characteristics of the field. For example, a division can be made according to core fields and non-core fields. Core fields include film and television, e-commerce, and games. The video evaluation dimensions included in these fields are structure and content dimensions. Specifically, the weight of the structure dimension in film and television is 0.5, and the weight of the content dimension is 0.5; the weight of the structure dimension in e-commerce is 0.3, and the weight of the content dimension is 0.7; the weight of the structure dimension in games is 0.2, and the weight of the content dimension is 0.8. Furthermore, each dimension can be divided into sub-dimensions. For example, for the structure dimension, it can be divided into a sub-dimension of structural clarity and a dimension of judging whether the beginning of the video has content that attracts users to stay and watch; for the content dimension, it can be divided into sub-dimensions of trending topics, audience resonance, and vertical stylized highlights, etc. The weights of the sub-dimensions corresponding to different fields can be the same or different. For non-core areas, video evaluation dimensions can include structural dimensions, content dimensions, and video theme relevance dimensions. For example, the structural dimension has a weight of 0.4, the content dimension has a weight of 0.2, and the video theme relevance dimension has a weight of 0.4. Similarly, each dimension can be further divided into sub-dimensions. For example, the structural dimension can be divided into the structural integrity sub-dimension and the dimension of judging whether the beginning of the video has content that attracts users to stay and watch; the content dimension can be divided into the audience resonance sub-dimension and the hot topic trend sub-dimension, etc.

[0062] In some implementation methods, reference may also be made to Figure 7The diagram illustrates a video processing flow. In this embodiment, a first generation model can be used to process speech content information, and a second generation model can be used to process screen content information. In practical applications, the generation model suitable for processing the first video can be selected based on the speech recognition results. For example, if the speech content of the first video is relatively high and sufficient for the model to understand, only the first generation model can be selected; if the speech content of the first video is relatively low and insufficient for the model to understand, only the second generation model can be selected. Of course, both can be used, and this is not a limitation. Prompt words can be determined based on key information recognition strategies, the information expected to be output by the generation model, and the corresponding format. These prompts are then input into the generation model. Based on the first generation model and / or the second generation model, content theme information, target segment information corresponding to each content theme, titles corresponding to each target segment, rating and ranking results, etc., can be directly generated. Based on the information generated by the model, N target segments can be obtained from the first video. Target operations such as screen adjustment, split-screen processing, and subtitle addition are then performed on each of the N target segments to obtain the second video. Furthermore, a cover can be added to the second video, and this is not a limitation. Furthermore, a second video can be displayed based on the rating ranking results for users to choose from. In practical applications, users can further edit the second video according to their needs, without any restrictions.

[0063] In summary, the video processing method provided in this disclosure can automatically and efficiently convert a longer first video into a shorter second video based on its content theme without requiring users to spend time and effort performing operations such as video segmentation, thereby meeting user needs.

[0064] Corresponding to the aforementioned video processing method, this disclosure further provides a video processing apparatus. Figure 8 This is a schematic diagram of a video processing apparatus provided in an embodiment of the present disclosure. The apparatus can be implemented by software and / or hardware, and is generally integrated into an electronic device, such as... Figure 8 As shown, the video processing apparatus includes:

[0065] The first video acquisition module 802 is used to acquire the first video to be processed.

[0066] The content information acquisition module 804 is used to acquire the target content information of the first video; wherein, the target content information includes audio content information and / or video content information;

[0067] The target segment determination module 806 is used to determine target segments from the first video based on the target content information of the first video; wherein the number of target segments is one or more.

[0068] The second video generation module 808 is used to generate a second video based on the target segment.

[0069] With the above-mentioned device, users can automatically and efficiently convert a longer first video into a shorter second video based on its content theme without having to spend time and effort performing operations such as video segmentation, thereby meeting user needs.

[0070] In some embodiments, the content information acquisition module 804 is specifically used to: perform speech recognition processing on the first video to obtain a speech recognition result; when the speech recognition result indicates that the ratio between the speech length in the first video and the duration of the first video is greater than a preset ratio threshold, obtain the speech content information of the first video based on the speech recognition result; and when the speech recognition result indicates that the ratio between the speech length in the first video and the duration of the first video is not greater than a preset ratio threshold, obtain the screen content information of the first video.

[0071] In some embodiments, the target segment determination module 806 is specifically used to: identify the content theme contained in the first video based on the target content information of the first video; wherein the number of the content theme is one or more; and perform segmentation processing on the first video based on the content theme to obtain the target segment corresponding to the content theme.

[0072] In some implementations, the target segment determination module 806 is specifically used to: when the duration of the first video is greater than the minimum value of the target duration interval, to perform segmentation processing on the first video based on the content theme and the target duration interval to obtain the target segment corresponding to the content theme, and the duration of the target segment is within the target duration interval.

[0073] In some embodiments, the target segment determination module 806 is specifically used to: identify key information contained in the first video based on the target content information of the first video; wherein the key information includes key sentences and / or key scenes; and determine the content theme contained in the first video based on the key information and first target information located before the key information and second target information located after the key information in the first video.

[0074] In some implementations, the target segment determination module 806 is specifically used to: determine the target domain to which the first video belongs based on the target content information of the first video; obtain the key information recognition strategy corresponding to the target domain, and identify the key information contained in the first video based on the key information recognition strategy.

[0075] In some embodiments, the target segment determination module 806 is specifically used to: when the target content information includes voice content information, identify key sentences contained in the first video based on the semantics of the voice content information; when the target content information includes image content information, perform frame extraction processing on the first video at preset intervals, and identify key images contained in the first video based on the image content of the extracted video frame images.

[0076] In some embodiments, the second video generation module 808 is specifically used to: perform a preset target operation on the target segment to obtain a second video; wherein the target operation includes one or more of the following: screen adjustment operation, split-screen processing operation, and subtitle addition operation.

[0077] In some embodiments, where the target operation includes a screen adjustment operation, the second video generation module 808 is specifically used to: identify and track the main object in the target segment, and crop the frame image in the target segment based on the identification and tracking results, wherein the main object in the target segment is located at a specified position in the screen of the cropped frame image; and generate a second video based on the cropped frame image.

[0078] In some embodiments, when the target operation includes a split-screen processing operation, the second video generation module 808 is specifically configured to: if at least two target objects are detected in the target segment, crop the frame images in the target segment based on the positions of the target objects to obtain cropped frame images corresponding to each of the at least two target objects; wherein the cropped frame image is a frame image containing a single target object; stitch the cropped frame images corresponding to each of the at least two target objects according to a preset layout to obtain a stitched frame image; and generate a second video based on the stitched frame image.

[0079] In some implementations, where the target operation includes a subtitle addition operation, the second video generation module 808 is specifically configured to: obtain subtitles corresponding to the target segment; determine the position of the subtitles based on the position of the main object in the target segment; wherein the position of the subtitles is outside the position of the main object; and add the subtitles to the target segment based on the position of the subtitles to obtain a second video.

[0080] In some embodiments, the apparatus further includes a cover-adding module for selecting a target frame image from the second video based on specified features of the main object of a frame image in the second video; and adding a cover image to the second video based on the target frame image.

[0081] In some embodiments, the apparatus further includes a title generation module for generating a target title corresponding to the second video based on the target content information of the second video.

[0082] In some embodiments, the apparatus further includes a video sorting module, configured to, when there are multiple second videos, obtain video evaluation dimensions corresponding to the domain to which the first video belongs and weights corresponding to different video evaluation dimensions, and generate evaluation scores corresponding to each of the multiple second videos based on the video evaluation dimensions and the weights; sort the multiple second videos based on the evaluation scores; and display the multiple second videos based on the sorting results.

[0083] The video processing apparatus provided in this disclosure can execute the video processing method provided in any embodiment of this disclosure, and has the corresponding functional modules and beneficial effects for executing the method.

[0084] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the above-described device embodiments can be referred to the corresponding process in the method embodiments, and will not be repeated here.

[0085] This disclosure provides an electronic device, which includes: a storage device storing a computer program thereon; and a processing device for executing the computer program in the storage device to implement the steps of any method of this disclosure.

[0086] The following is for reference. Figure 9 This document illustrates a structural schematic diagram of an electronic device 900 suitable for implementing embodiments of the present disclosure. The terminal devices in these embodiments may include, but are not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 9 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0087] like Figure 9As shown, electronic device 900 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 901, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 902 or a program loaded from storage device 908 into random access memory (RAM) 903. RAM 903 also stores various programs and data required for the operation of electronic device 900. Processing device 901, ROM 902, and RAM 903 are interconnected via bus 904. Input / output (I / O) interface 905 is also connected to bus 904.

[0088] Typically, the following devices can be connected to I / O interface 905: input devices 906 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 907 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 908 including, for example, magnetic tapes, hard disks, etc.; and communication devices 909. Communication device 909 allows electronic device 900 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 9 An electronic device 900 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0089] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 909, or installed from a storage device 908, or installed from a ROM 902. When the computer program is executed by a processing device 901, it performs the functions defined in the methods of embodiments of this disclosure.

[0090] In addition to the methods and devices described above, embodiments of this disclosure can also be computer program products, comprising computer program instructions that, when executed by a processor, cause the processor to perform the image processing methods provided in the embodiments of this disclosure. The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this disclosure. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0091] Furthermore, embodiments of this disclosure may also be computer-readable storage media storing computer program instructions that, when executed by a processor, cause the processor to perform the video processing method provided in embodiments of this disclosure.

[0092] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0093] This disclosure also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the video processing method of this disclosure.

[0094] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0095] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.

[0096] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0097] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0098] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0099] The above description is merely a specific embodiment of this disclosure, enabling those skilled in the art to understand or implement it. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method of video processing, the method comprising: The method comprises: obtaining a first video to be processed; obtaining target content information of the first video; wherein the target content information comprises voice content information and / or picture content information; determining target segments from the first video based on the target content information of the first video; wherein the number of the target segments is one or more; generating a second video based on the target segments.

2. The method of claim 1, wherein, The obtaining of the target content information of the first video comprises: performing voice recognition processing on the first video to obtain a voice recognition result; in a case where the voice recognition result indicates that a ratio between a voice length in the first video and a time length of the first video is greater than a preset proportion threshold, obtaining voice content information of the first video based on the voice recognition result; in a case where the voice recognition result indicates that the ratio between the voice length in the first video and the time length of the first video is not greater than the preset proportion threshold, obtaining picture content information in the first video.

3. The method of claim 1, wherein, The determining of the target segments from the first video based on the target content information of the first video comprises: identifying content themes contained in the first video based on the target content information of the first video; wherein the number of the content themes is one or more; performing segmentation processing on the first video based on the content themes to obtain target segments corresponding to the content themes.

4. The method of claim 3, wherein, The performing of the segmentation processing on the first video based on the content themes to obtain the target segments corresponding to the content themes comprises: in a case where the time length of the first video is greater than a minimum value of a target time length interval, performing segmentation processing on the first video based on the content themes and the target time length interval to obtain the target segments corresponding to the content themes, and the time lengths of the target segments are located within the target time length interval.

5. The method of claim 3, wherein, The identifying of the content themes contained in the first video based on the target content information of the first video comprises: identifying key information contained in the first video based on the target content information of the first video; wherein the key information comprises key sentences and / or key pictures; determining content themes contained in the first video based on the key information and first target information located before the key information and second target information located after the key information in the first video.

6. The method of claim 5, wherein, The identifying of the key information contained in the first video based on the target content information of the first video comprises: determining a target field to which the first video belongs based on the target content information of the first video; obtaining a key information identification strategy corresponding to the target field, and identifying the key information contained in the first video based on the key information identification strategy.

7. The method of claim 5, wherein, The identifying of the key information contained in the first video based on the target content information of the first video comprises: in a case where the target content information comprises voice content information, identifying key sentences contained in the first video based on semantics of the voice content information; in a case where the target content information comprises picture content information, identifying key pictures contained in the first video based on semantics of the picture content information. In a case where the target content information comprises picture content information, the first video is frame-extracted at preset intervals, and key pictures contained in the first video are identified based on picture content of the extracted video frames.

8. The method of claim 1, wherein, The generating of the second video based on the target segment comprises: performing a preset target operation on the target segment to obtain the second video, wherein the target operation comprises one or more of a picture adjustment operation, a split-screen processing operation, and a subtitle adding operation.

9. The method of claim 8, wherein, In a case where the target operation comprises the picture adjustment operation, the performing of the preset target operation on the target segment to obtain the second video comprises: identifying and tracking a subject object in the target segment, and cropping a frame image in the target segment based on a result of the identifying and tracking, wherein the subject object is located at a specified position of the cropped frame image; generating the second video based on the cropped frame image.

10. The method of claim 8, wherein, In a case where the target operation comprises the split-screen processing operation, the performing of the preset target operation on the target segment to obtain the second video comprises: if at least two target objects are detected in the target segment, cropping the frame image in the target segment based on positions of the target objects to obtain cropped frame images corresponding to the at least two target objects respectively, wherein each of the cropped frame images comprises a single target object; splicing the cropped frame images corresponding to the at least two target objects according to a preset layout to obtain a spliced frame image; and generating the second video based on the spliced frame image.

11. The method of claim 8, wherein, In a case where the target operation comprises the subtitle adding operation, the performing of the preset target operation on the target segment to obtain the second video comprises: obtaining a subtitle corresponding to the target segment; determining a position of the subtitle based on a position of a subject object in the target segment, wherein the position of the subtitle is outside the position of the subject object; and adding the subtitle to the target segment based on the position of the subtitle to obtain the second video.

12. The method of claim 1, wherein, The method further comprises: selecting a target frame image from the second video based on a specified feature of a subject object in the frame image of the second video; and adding a cover to the second video based on the target frame image.

13. The method of claim 1, wherein, The method further comprises: generating a target title corresponding to the second video based on target content information of the second video.

14. The method of claim 1, wherein, The method further comprises: in a case where the second videos are multiple, obtaining a video evaluation dimension corresponding to a field to which the first video belongs and weights corresponding to different video evaluation dimensions, and generating evaluation scores corresponding to the second videos respectively based on the video evaluation dimension and the weights; sorting the second videos based on the evaluation scores; and displaying the second videos based on a sorting result.

15. A video processing apparatus, comprising: The method comprises: a first video obtaining module configured to obtain a first video to be processed; a content information obtaining module configured to obtain target content information of the first video, wherein the target content information comprises voice content information and / or picture content information; and a second video generating module configured to generate a second video based on the target segment. A target segment determination module is configured to determine target segments from the first video based on target content information of the first video, wherein the number of the target segments is one or more. A second video generation module is configured to generate a second video based on the target segments.

16. An electronic device, comprising: The electronic device comprises: A storage device having a computer program stored thereon; A processing device configured to execute the computer program in the storage device to implement the steps of the video processing method according to any one of claims 1-14.

17. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, and the computer program is configured to execute the video processing method according to any one of claims 1-14.

18. A computer program product, characterised in that, The computer program is configured to implement the video processing method according to any one of claims 1-14 when executed by a processor.

Images