Multimedia data processing method and device, electronic equipment, chip and medium
By processing multimedia clips on the terminal side and uploading text description information, and obtaining editing scripts generated in the cloud, the problems of user privacy leakage and low editing efficiency are solved, and efficient multimedia video editing is achieved.
Patent Information
- Application Number
- CN202410773906.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-14
- Publication Date
- 2024-11-08
AI Technical Summary
Existing multimedia processing technology requires users to upload the original video to the cloud for processing, resulting in user privacy leakage, and manual editing is cumbersome and inefficient.
By obtaining multimedia clips on the terminal side for processing, extracting text description information and uploading them to the cloud, and then obtaining editing scripts to perform video editing on the terminal side to avoid uploading original videos, protecting user privacy and simplifying the process.
It realizes the generation of edited multimedia clips without uploading the original video, protects user privacy, improves processing efficiency, and simplifies the video editing process.
Smart Images

Figure CN118921522A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of multimedia processing, and in particular to a multimedia data processing method, device, electronic device, chip and medium. Background Art
[0002] Traditional multimedia processing methods rely on manual screening of clips, fusion splicing, and narration dubbing, which requires a lot of manpower and material resources. With the development of large language models, multimodal information can be comprehensively analyzed and processed in the cloud, reducing manual operations to a certain extent. However, cloud-based implementation solutions require users to upload multimedia videos for processing. Summary of the invention
[0003] The present disclosure provides a multimedia data processing method, device, electronic device, chip and medium. By deploying the processing of multimedia clips on the terminal side, users only need to select multimedia clips for editing, and do not need to upload the original multimedia clips to the cloud to obtain the edited multimedia clips, thereby realizing terminal-side deployment and generation.
[0004] The first aspect of the present disclosure proposes a multimedia data processing method, which includes: obtaining multiple multimedia segments to be processed; performing a first processing on the multiple multimedia segments to be processed respectively to obtain first information of the multiple multimedia segments to be processed, and the first information is used to provide a text description of the multiple multimedia segments to be processed; uploading the first information to the cloud, and obtaining a editing script of the multiple multimedia segments to be processed generated by the cloud based on the first information; performing a second processing on the multiple multimedia segments to be processed according to the editing script to obtain target multimedia data.
[0005] In some embodiments of the present disclosure, obtaining multiple multimedia segments to be processed includes: obtaining multiple original multimedia segments; adaptively cropping the multiple original multimedia segments respectively to obtain multiple first multimedia segments; segmenting the multiple first multimedia segments according to content categories of the multiple first multimedia segments respectively to obtain multiple second multimedia segments; extracting content that meets preset conditions in each second multimedia segment to obtain multiple multimedia segments to be processed.
[0006] In some embodiments of the present disclosure, performing first processing on multiple multimedia segments to be processed respectively includes: performing feature extraction on the multiple multimedia segments to be processed respectively to obtain feature vectors of the multiple multimedia segments to be processed; uploading the feature vectors of the multiple multimedia segments to be processed to the cloud to obtain content description information of the multiple multimedia segments to be processed generated in the cloud; using multiple large language models for identifying different categories of information to perform second processing on the multiple multimedia segments to be processed respectively to obtain second information of the multiple multimedia segments to be processed; integrating the content description information and the second information of the multiple multimedia segments to be processed to obtain the first information of the multiple multimedia segments to be processed.
[0007] In some embodiments of the present disclosure, performing a second processing on multiple multimedia segments to be processed respectively includes at least one of the following: performing scene recognition on the multiple multimedia segments to be processed using a scene recognition model to obtain scene information of the multiple multimedia segments to be processed; performing speech recognition on the multiple multimedia segments to be processed using a speech recognition model to obtain speech information of the multiple multimedia segments to be processed.
[0008] In some embodiments of the present disclosure, the first information includes: at least one of content description information, scene information, voice information, and user-preset style information; the editing script includes sequence information of multiple multimedia clips to be processed, image modification information, music audio, voice audio, and conversion information between clips.
[0009] The second aspect of the present disclosure proposes a multimedia data processing method, which is executed by the cloud, and includes: receiving first information of multiple multimedia segments to be processed sent by a terminal, the first information being obtained by the terminal performing a first processing on the multiple multimedia segments to be processed; based on the first information, generating a clipping script of the multiple multimedia segments to be processed; sending the clipping script to the terminal, and the clipping script is used by the terminal to process the multiple multimedia segments to be processed.
[0010] In some embodiments of the present disclosure, generating a editing script for multiple multimedia segments to be processed based on the first information includes: determining sequence information of the multiple multimedia segments to be processed based on the first information, the sequence information including sorting numbers of the multiple multimedia segments to be processed; filtering out conversion information that meets the conversion rules from a script library based on the first information of two multimedia segments to be processed with adjacent sorting numbers, and determining image modification information, music audio, and / or generating voice audio based on the conversion information and the first information.
[0011] In some embodiments of the present disclosure, the method also includes: receiving feature vectors of multiple multimedia segments to be processed sent by a terminal, the feature vectors being obtained by the terminal performing feature extraction on the multiple multimedia segments to be processed; based on the feature vectors, respectively determining content description information of the multiple multimedia segments to be processed, the content description information being used by the terminal to determine the first information; and sending the content description information to the terminal.
[0012] The third aspect embodiment of the present disclosure proposes a multimedia data processing device, including: an acquisition module, used to acquire multiple multimedia segments to be processed; a first processing module, used to perform a first processing on the multiple multimedia segments to be processed respectively, and obtain first information of the multiple multimedia segments to be processed, and the first information is used to provide a text description of the multiple multimedia segments to be processed; a transceiver module, used to upload the first information to the cloud, and obtain a editing script of the multiple multimedia segments to be processed generated by the cloud based on the first information; a second processing module, used to perform a second processing on the multiple multimedia segments to be processed according to the editing script, and obtain target multimedia data.
[0013] The fourth aspect embodiment of the present disclosure proposes a multimedia data processing device, including a transceiver module for receiving first information of multiple multimedia fragments to be processed sent by a terminal, the first information being obtained by the terminal performing a first processing on the multiple multimedia fragments to be processed; a processing module for generating a clipping script of the multiple multimedia fragments to be processed based on the first information; and a transceiver module for sending the clipping script to the terminal, the clipping script being used by the terminal to perform a second processing on the multiple multimedia fragments to be processed.
[0014] The fifth aspect embodiment of the present disclosure proposes an electronic device, comprising: a processor and a memory for storing a computer program that can be run on the processor, wherein when the processor is used to run the computer program, it executes the method described in any one of the embodiments of the first aspect or the second aspect of the present disclosure.
[0015] The sixth aspect embodiment of the present disclosure proposes a non-transitory computer-readable storage medium storing computer instructions, and the computer instructions are used to enable a computer to execute the method described in any one of the embodiments of the first aspect or the second aspect of the present disclosure.
[0016] The seventh aspect embodiment of the present disclosure proposes a chip, including at least one processor and a communication interface, the communication interface is used to receive signals input into the chip or output signals from the chip, the processor communicates with the communication interface and implements the method described in any one of the first aspect or the second aspect through logic circuits or execution code instructions.
[0017] In summary, the multimedia data processing method, device, electronic device, chip and medium proposed in the present disclosure include: obtaining multiple multimedia segments to be processed; performing first processing on the multiple multimedia segments to be processed respectively to obtain first information of the multiple multimedia segments to be processed, the first information being used to describe the multiple multimedia segments to be processed in words; uploading the first information to the cloud, and obtaining editing scripts of the multiple multimedia segments to be processed generated by the cloud based on the first information; processing the multiple multimedia segments to be processed according to the editing script to obtain target multimedia data. By obtaining the original video of the user at the terminal, performing a second processing on the original video, uploading the description information of the original video to the cloud, obtaining the editing script generated by the cloud for video processing, and obtaining the edited video, the terminal-side editing video can be realized with one click.
[0018] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.
[0020] Figure 1 A flowchart of a multimedia data processing method proposed in an embodiment of the present disclosure;
[0021] Figure 2 A flow chart of a method for acquiring multiple multimedia segments to be processed proposed in an embodiment of the present disclosure;
[0022] Figure 3 A flow chart of a method for performing first processing on a plurality of multimedia segments to be processed respectively proposed in an embodiment of the present disclosure;
[0023] Figure 4 A method flow chart of a multimedia data processing method proposed in an embodiment of the present disclosure;
[0024] Figure 5 A flow chart of a method for generating a clipping script proposed in an embodiment of the present disclosure;
[0025] Figure 6 A flowchart of a multimedia data processing method proposed in an embodiment of the present disclosure;
[0026] Figure 7 A flowchart of a specific implementation of a multimedia data processing method provided by the present disclosure;
[0027] Figure 8 A schematic diagram of the structure of a multimedia data processing device proposed in an embodiment of the present disclosure;
[0028] Fig. 9 A schematic diagram of the structure of a multimedia data processing device proposed in an embodiment of the present disclosure;
[0029] Fig.10 A schematic diagram of the structure of an electronic device proposed in an embodiment of the present disclosure;
[0030] Fig.11 A schematic diagram of the structure of a chip proposed in an embodiment of the present disclosure. DETAILED DESCRIPTION
[0031] Embodiments of the present disclosure are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present disclosure, and should not be construed as limiting the present disclosure.
[0032] At present, large language models are deployed in the cloud and applied in the multimedia field to generate editing scripts. For example, the corresponding visual description information can be obtained using the visual language large model, and the voice information of the video can be generated based on audio recognition. In addition, the prompt information sent by the user can be obtained, and the text information is semantically parsed based on the large language model to identify the target text information corresponding to the prompt information for editing. However, the above methods all require users to upload the original video to the cloud for processing, which is not conducive to the protection of user privacy.
[0033] In summary, in order to solve the technical problems in the related art, the embodiment of the present disclosure provides a multimedia data processing method, which is executed by a terminal, including obtaining multiple multimedia segments to be processed; performing a first processing on the multiple multimedia segments to be processed respectively to obtain first information of the multiple multimedia segments to be processed, and the first information is used to describe the multiple multimedia segments to be processed in words; uploading the first information to the cloud, and obtaining the editing scripts of the multiple multimedia segments to be processed generated by the cloud based on the first information; performing a second processing on the multiple multimedia segments to be processed according to the editing script to obtain the target multimedia data. Through the terminal-side deployment, uploading the user's original video to the cloud is avoided, and the user's privacy is protected. The user does not need to enter additional prompt information. The selected video segment can realize the editing and generation of the video, which improves the speed of the editing process and simplifies the video editing process.
[0034] The multimedia data processing method provided by the present application will be described in detail below with reference to the accompanying drawings.
[0035] Figure 1 FIG. 1 is a flow chart of a multimedia data processing method proposed in an embodiment of the present disclosure. The method is executed by a terminal. Figure 1As shown, the method includes the following steps.
[0036] Step 101: Acquire multiple multimedia segments to be processed.
[0037] In some embodiments, obtaining a plurality of to-be-processed multimedia segments may be obtaining a plurality of original multimedia segments of a user.
[0038] In some embodiments, obtaining multiple multimedia segments to be processed may be obtaining multimedia segments after preprocessing multiple original multimedia segments of the user, wherein the preprocessing may include unifying the multimedia segments into horizontal and vertical screens, segmenting the multimedia segments, screening and eliminating the multimedia segments, classifying the multimedia segments, etc., which is not limited by the present disclosure.
[0039] In some embodiments, obtaining a plurality of multimedia segments to be processed may be original video segments selected by a user for editing on the terminal side, or obtaining segments after pre-processing the original video segments selected by a user for editing on the terminal side.
[0040] For example, a user selects n videos, and the videos are uniformly preprocessed to obtain video clips for editing by multiple users.
[0041] Step 102 : performing first processing on a plurality of multimedia segments to be processed respectively to obtain first information of the plurality of multimedia segments to be processed.
[0042] In an embodiment of the present disclosure, the first information is used to provide a text description for a plurality of multimedia segments to be processed.
[0043] In some embodiments, the first processing is performed on a plurality of multimedia segments to be processed respectively, and the first processing may be performed on each multimedia segment to be processed to obtain a text description of each multimedia segment to be processed.
[0044] In some embodiments, the first processing performed on multiple multimedia segments to be processed may include performing visual information recognition, speech recognition, scene recognition, key information recognition, timeline clustering, etc. on each multimedia segment to be processed, and may also include other processing methods, which are not limited by the present disclosure.
[0045] For example, information description is generated for the video clips used for editing by the user to obtain complete description information of each video segment.
[0046] Step 103, uploading the first information to the cloud, and obtaining editing scripts of multiple multimedia segments to be processed generated by the cloud based on the first information.
[0047] In some embodiments, uploading the first information to the cloud may be uploading text descriptions of multiple multimedia segments to be processed to the cloud, and the cloud recognizes and processes the text descriptions to generate editing scripts of the multiple multimedia segments to be processed.
[0048] In some embodiments, the editing script includes sequence information of multiple multimedia segments to be processed, image modification information, music audio, voice audio, and transition information between segments. The sequence information includes the order numbers of the multiple multimedia segments to be processed, the image modification information is information such as filters or special effects, the music audio can be background music, the voice audio can be narration, and the transition information between segments can be transition animation or transition effect.
[0049] In some embodiments, the first information may include text description information of multiple multimedia segments to be processed and style information preset by the user, wherein the text description information may include content description information, audio information, scene information, etc.
[0050] In some embodiments, the editing script may be an editing script that conforms to preset style information.
[0051] In some embodiments, the first information is uploaded to the cloud, and the cloud can generate a editing script based on the first information, for example, according to the text description information, extract key information therein as a title, and / or extract the information obtained by voice recognition as subtitles, and / or sort the timeline and sort multiple multimedia clips to be processed, and / or generate narration according to the text description information, and / or filter audio and / or special effects that match the content, and / or determine the animation effect of transition according to the text description information of two adjacent clips, etc.
[0052] In some embodiments, the editing script generated by the cloud based on the first information can be used on the terminal side to edit and splice multiple multimedia clips to be processed.
[0053] For example, the complete description information of multiple videos used by the user for editing is uploaded to the cloud, and the editing script generated in the cloud is obtained.
[0054] In some embodiments, uploading the first information to the cloud may be uploading a text description of multiple multimedia segments to be processed to the cloud, or may be uploading user-preset style information and a text description of multiple multimedia segments to be processed to the cloud, and obtaining an editing script generated by the cloud based on the first information and / or the style information.
[0055] For example, the complete description information of multiple video clips used for editing by the user and the user-preset style information are uploaded to the cloud, and the editing script generated in the cloud is obtained, where the user-preset style information is, for example, a video in the style of a certain director, and the editing script generated in the cloud is an editing script in the style of the certain director, including the order of editing, screen filters, transition animations, background music, narration, etc.
[0056] In the above embodiment, uploading the first information to the cloud and obtaining the editing script generated by the cloud can avoid uploading the user's original multimedia clips or pre-processed multimedia clips, or multimedia clips to be processed to the cloud, thereby protecting the user's privacy.
[0057] Step 104 , processing the plurality of multimedia segments to be processed according to the editing script to obtain target multimedia data.
[0058] In some embodiments, multiple multimedia segments to be processed are processed according to the editing script, and the multiple multimedia segments to be processed can be spliced according to sequence information; the multiple multimedia segments to be processed are modified according to image modification information, such as adding filters or increasing contrast, increasing brightness, etc.; music audio is added to the spliced multimedia segments; voice audio is added to the corresponding multimedia segments, and conversion information between segments is added between corresponding segments.
[0059] In the above embodiment, the editing script includes sequence information of multiple multimedia segments to be processed, image modification information, music audio, voice audio, and conversion information between segments.
[0060] In some embodiments, the plurality of multimedia segments to be processed are processed according to a clipping script, and the plurality of multimedia segments to be processed may be spliced and clipped according to a clipping script that conforms to preset style information to obtain target multimedia data.
[0061] In the above embodiment, the editing script is generated in the cloud based on text description information and style information preset by the user, wherein the text description information includes content description information, audio information, scene information, etc.
[0062] In some embodiments, multiple multimedia clips to be processed are processed according to the editing script, and the multiple multimedia clips to be processed can be processed according to the order of the multimedia clips in the editing script and one or more of the guidance information such as the picture description, audio guidance, transition method, time stamp, special effects use, etc. in the editing script.
[0063] In some embodiments, the editing script may include information such as splicing order, transition effects, playback speed, color and filter, background music, text description, title, subtitle, etc. Among them, the transition effect may be fade in and fade out, dissolve, slide, etc., and the transition effect may enhance the coherence and viewing experience between multimedia clips; the color and filter may be to adjust parameters such as brightness, contrast, saturation, etc. to improve the visual effect of the picture, and the filter may be various existing filter effects; the background music may be audio that matches the content of the multimedia clip; the text description, title, and subtitle may be the refinement and summary of the content in the multimedia clip, used to emphasize the key information in the clip.
[0064] For example, multiple multimedia segments to be processed may be spliced together in the order of the multimedia segments in the editing script, and picture descriptions, audio, transition animations, special effects, etc. may be added to the spliced multimedia segments to obtain edited multimedia segments, where the picture descriptions include textual information such as text descriptions, titles, and subtitles.
[0065] For example, the generated editing script is transmitted back to the terminal side, where the videos are spliced and fused, and effects such as filters, background music, narration, and transition animations are added to obtain the final edited video as a return result.
[0066] In summary, according to the multimedia data processing method proposed in the present invention, by processing the multimedia clips, text description information is obtained, the text description information is uploaded to the cloud, and the editing script generated by the cloud based on the text description information is obtained. According to the editing script, the multimedia clips are edited, which can avoid uploading the multimedia clips to the cloud and protect user privacy. After obtaining the multimedia clips, the edited multimedia clips can be directly generated without the user inputting prompt information, thereby improving the processing efficiency, solving the cumbersome operation process of manual editing, and simplifying the processing method.
[0067] based on Figure 1 The embodiment shown, Figure 2 A flowchart of a method for obtaining multiple multimedia segments to be processed. Figure 2 right Figure 1 Step 101 in is further described as follows: Figure 2 As shown, the following steps are included.
[0068] Step 201: Acquire multiple original multimedia segments.
[0069] In the embodiment of the present disclosure, the multiple original multimedia segments obtained may be multiple original video segments selected by the user at the terminal.
[0070] In the embodiments of the present disclosure, the original multimedia clip may be a video clip of a close-up of a person, or a video clip of a close-up of a scene, or a video clip of both a person and a scene. The original multimedia clip may be a clip in multiple states including static and dynamic.
[0071] In some embodiments, the original video clip may be a video containing wonderful clips, such as a slam dunk clip during a basketball game, a surprise clip during a dinner party, or a thrilling clip during a mountain climbing.
[0072] In some embodiments, a plurality of original multimedia segments are obtained for editing to generate target multimedia data.
[0073] Step 202: adaptively crop the multiple original multimedia segments respectively to obtain multiple first multimedia segments.
[0074] In some embodiments, adaptive cropping can be to unify the resolution of the original multimedia clip in the horizontal or vertical screen, for example, to crop the original multimedia clip in the horizontal screen and adjust the resolution to the resolution of the original multimedia clip in the vertical screen, or to crop the original multimedia clip in the vertical screen and adjust the resolution to the resolution of the original multimedia clip in the horizontal screen.
[0075] In some embodiments, adaptive cropping of original multimedia segments can unify the resolution of the multimedia segments used for editing to achieve consistent display effects and avoid interlaced playback of horizontal and vertical screens in the edited multimedia data, which affects the playback effect.
[0076] For example, an adaptive cropping strategy is adopted for horizontal screen videos to unify the resolutions of horizontal and vertical screen videos.
[0077] Step 203: segment the plurality of first multimedia segments respectively according to the content categories of the plurality of first multimedia segments to obtain a plurality of second multimedia segments.
[0078] In some embodiments, the content category may be the specific content of the multimedia clip, such as a close-up of a person, a display of a landscape, different types of sports, etc., which is not limited in the present disclosure.
[0079] For example, a first multimedia segment includes a segment of two people playing basketball and a segment of one person dancing. Then, the basketball segment and the dancing segment are segmented according to the content category to obtain two second multimedia segments.
[0080] In some embodiments, segmenting the plurality of first multimedia segments separately according to content categories may be performing content recognition on the first multimedia segments. When segments of different content categories are recognized, the segments of different content categories are segmented; if segments of different content categories are not recognized, no segmentation is performed.
[0081] Step 204: extract the content that meets the preset conditions in each second multimedia segment to obtain a plurality of multimedia segments to be processed.
[0082] In some embodiments, extracting content that meets preset conditions in each second multimedia segment can be performed by using a trained model to score each frame of the multimedia segment, and selecting a video segment with a score exceeding a first threshold to extract as the multimedia segment to be processed.
[0083] For example, for a video clip of multiple people playing football, use the trained scoring model to score each frame in this video. The score of the goal moment can reach more than 90 points. Finally, multiple frames of video with scores exceeding 80 are extracted, that is, the video clip of the goal moment can be extracted. Similarly, the wonderful clips from multiple video clips can be extracted to obtain multiple wonderful video clips.
[0084] In the above embodiment, by adaptively cropping, segmenting and extracting the original multimedia segments, segments of different resolutions in the original multimedia segments are adjusted, segments of different content categories are classified, and wonderful segments are extracted, so that the obtained multiple multimedia segments to be processed can reflect the content of the original multimedia segments only through shorter segments.
[0085] based on Figure 1-2 The embodiment shown, Figure 3 is a flow chart of a method for performing a first processing on a plurality of multimedia segments to be processed respectively. Figure 3 right Figure 1 Step 102 in is further described as follows: Figure 3 As shown, the following steps are included:
[0086] Step 301 : extract features from a plurality of multimedia segments to be processed respectively to obtain feature vectors of the plurality of multimedia segments to be processed.
[0087] In some embodiments, feature extraction is performed on multiple multimedia segments to be processed respectively by using a trained large visual language model to extract frames for each multimedia segment to be processed, which may be 16 frames or 8 frames. For example, 16 frames of multimedia segments are extracted and input into a video encoder to obtain a feature vector of the video.
[0088] For example, the terminal side uses the visual language large model to perform video feature extraction operations on multiple wonderful video clips to obtain video features of each wonderful video clip.
[0089] Step 302 : Upload feature vectors of the plurality of multimedia segments to be processed to the cloud, and obtain content description information of the plurality of multimedia segments to be processed generated in the cloud.
[0090] In some embodiments, feature vectors of multiple multimedia segments to be processed are uploaded to the cloud, and content description information of the multiple multimedia segments to be processed generated in the cloud is obtained. The feature vectors can be uploaded to the cloud, and the feature vectors are recognized by a large visual language model in the cloud to generate content description information for each multimedia segment to be processed.
[0091] In some embodiments, the content description information may comprehensively reflect the specific content of the multimedia segment to be processed, including characters, scenes, story content, etc.
[0092] For example, the video features are uploaded to the cloud, and a large language model in the cloud is used to generate a description text of the video.
[0093] Step 303 : Use a plurality of large language models for identifying different types of information to perform second processing on the plurality of multimedia segments to be processed respectively, so as to obtain second information of the plurality of multimedia segments to be processed.
[0094] In some embodiments, a large language model for recognizing different categories of information can be obtained by training the large language model using labels of different categories in the same training data, such as a scene recognition model for recognizing scenes, a speech recognition model for recognizing speech, an emotion recognition model for recognizing emotions, etc., which is not limited by the present disclosure.
[0095] In some embodiments, the scene recognition model for identifying scene can identify scene information including: wide panorama, close-up of person, panorama of person, close-up of person, close-up of object, panorama of object, close-up of object. In different embodiments, the scene information that can be identified by the scene recognition model can be different and can also include other scene information, which is not limited by the present disclosure.
[0096] In some embodiments, the speech recognition model for recognizing speech can recognize speech such as conversations between characters or confessions of characters in a multimedia clip and convert it into text-based speech information.
[0097] In some embodiments, performing the second processing on the plurality of multimedia segments to be processed includes performing scene recognition on the plurality of multimedia segments to be processed using a scene recognition model to obtain scene information of the plurality of multimedia segments to be processed.
[0098] In some embodiments, performing the second processing on the plurality of multimedia segments to be processed includes performing speech recognition on the plurality of multimedia segments to be processed using a speech recognition model to obtain speech information of the plurality of multimedia segments to be processed.
[0099] In some embodiments, the second processing of multiple multimedia segments to be processed includes using a speech recognition model to perform speech recognition on the multiple multimedia segments to be processed, and using a scene recognition model to perform scene recognition on the multiple multimedia segments to be processed, to obtain speech information and scene information of the multiple multimedia segments to be processed.
[0100] In some embodiments, the second information may include voice information and / or scene information of multiple multimedia segments to be processed.
[0101] For example, a scene classification model is used to obtain scene information of each wonderful video clip as a supplementary description of the video; the audio in the video is recognized to generate corresponding text information as a supplementary description of the video.
[0102] Step 304: Integrate the content description information and the second information of the plurality of multimedia segments to be processed to obtain the first information of the plurality of multimedia segments to be processed.
[0103] In some embodiments, the content description information and the second information of the plurality of multimedia segments to be processed are integrated, and the content description information, the voice information, and the scene information of each multimedia segment to be processed can be integrated as the first information of each multimedia segment to be processed.
[0104] In some embodiments, the first information includes at least one of content description information, scene information, and voice information of a plurality of multimedia segments to be processed.
[0105] For example, the description text of the video generated in the cloud, the scene information and voice information generated by the terminal are used as the complete description information of each wonderful video clip, wherein the scene information and voice information can be used as supplementary description information.
[0106] In some embodiments, the first information includes at least one of content description information, scene information, voice information, and user-preset style information of the plurality of multimedia segments to be processed.
[0107] For example, the user's preset requirements for editing style are obtained, such as the style of director A or the style of director B, and the preset requirements are combined with the video's description text, scene information, and voice information as the first information of multiple multimedia clips to be processed.
[0108] In the above embodiment, features of multimedia clips are extracted at the terminal, and the clip features are uploaded to the cloud to generate text description information. At the same time, a large language model for identifying different categories deployed on the terminal side is used to identify multiple multimedia clips to be processed, and text information of multiple categories is obtained. This avoids uploading the original multimedia clips to the cloud and protects the privacy of users. At the same time, multiple recognition models of different categories are used to identify multimedia clips, so that information of multimedia clips in multiple dimensions can be obtained, and multimedia clips can be described more accurately. These description information are used as input for generating editing scripts in the cloud, so that the generated editing scripts can be more consistent with real manual editing.
[0109] In summary, the multimedia data processing method proposed in the present disclosure obtains multiple multimedia clips to be processed; performs first processing on the multiple multimedia clips to be processed respectively to obtain first information of the multiple multimedia clips to be processed; uploads the first information to the cloud, and obtains the editing scripts of the multiple multimedia clips to be processed generated by the cloud based on the first information; processes the multiple multimedia clips to be processed according to the editing script to obtain target multimedia data. The method provided by the present disclosure processes the original video on the terminal side without uploading the original video to the cloud, thus protecting the privacy of the user, and can directly obtain the edited video after obtaining the video, without the user having to input prompt information, thus improving the processing efficiency, solving the cumbersome operation process of manual editing, and simplifying the processing method.
[0110] Figure 4 This is a flowchart of the multimedia data processing method proposed in the embodiment of the present disclosure. The method is executed by the cloud, such as Figure 4 As shown, the following steps are included:
[0111] Step 401: receiving first information of a plurality of multimedia segments to be processed sent by a terminal.
[0112] In some embodiments, the first information is obtained by the terminal performing a first process on a plurality of multimedia segments to be processed.
[0113] In some embodiments, the first information includes at least one of content description information, scene information, voice information, and user-preset style information of the plurality of multimedia segments to be processed.
[0114] In some embodiments, the first information is that the terminal extracts features from multiple multimedia segments to be processed, obtains feature vectors of multiple multimedia segments to be processed, uploads the feature vectors to the cloud, and obtains content description information of the multiple multimedia segments to be processed generated by the cloud based on the feature vectors; the terminal uses a large language model that recognizes different categories of information, performs second processing on the multiple multimedia segments to be processed respectively, obtains second information of the multiple multimedia segments to be processed, integrates the content description information and the second information, and obtains the first information of the multiple multimedia segments to be processed.
[0115] In some embodiments, the first information may be Figure 3 The method and steps of performing the first processing on a plurality of multimedia segments to be processed by the terminal shown in the figure are obtained.
[0116] In some embodiments, the first information may reflect the content, scene, and audio information of the plurality of multimedia segments to be processed.
[0117] Step 402: Generate a plurality of editing scripts of the multimedia segments to be processed based on the first information.
[0118] In some embodiments, the editing script may include sequence information of multiple multimedia segments to be processed, image modification information, music audio, voice audio, and transition information between segments. The sequence information includes the order numbers of the multiple multimedia segments to be processed, the image modification information is information such as filters or special effects, the music audio may be background music, the voice audio may be narration, and the transition information between segments may be transition animation or transition effect.
[0119] In some embodiments, the editing script may be sequence information of a plurality of multimedia segments to be processed in a certain director's style, image modification information, music audio, and transition information between segments.
[0120] In some embodiments, the editing script may include information such as splicing order, transition effects, playback speed, color and filter, background music, text description, title, subtitle, etc. Among them, the transition effect may be fade in and fade out, dissolve, slide, etc., and the transition effect may enhance the coherence and viewing experience between multimedia clips; the color and filter may be to adjust parameters such as brightness, contrast, saturation, etc. to improve the visual effect of the picture, the filter may be various existing filter effects, and the background music may be audio that matches the content of the multimedia clip; the text description, title, and subtitle may be the refinement and summary of the content in the multimedia clip, used to emphasize the key information in the clip.
[0121] For example, the terminal uploads the complete description information of multiple video clips to the cloud, and the cloud uses the complete description information as the input of the large language model, and uses the large language model on the cloud to give recommended video sorting order, transition animation recommendations, filter / background music recommendations, and generate narration.
[0122] For example, the terminal uploads complete description information of multiple video clips and the director's style preset by the user to the cloud, and the large language model in the cloud gives an editing script with the preset director's style.
[0123] Step 403: Send the editing script to the terminal, where the editing script is used by the terminal to perform a second process on the plurality of multimedia segments to be processed.
[0124] In some embodiments, the editing script is sent to the terminal, and the terminal can be used to edit and splice multiple multimedia segments to be processed according to the editing script to generate target multimedia data.
[0125] In some embodiments, the editing script is sent to the terminal, and the terminal can be used to edit and splice multiple multimedia segments to be processed according to the editing script that meets the preset style information to generate target multimedia data that meets the preset style.
[0126] In the above embodiment, the cloud can generate editing scripts for multiple multimedia clips to be processed by receiving complete description information of multiple multimedia clips to be processed uploaded by the terminal, so as to be used for the terminal side to edit and splice the multiple multimedia clips to be processed. By utilizing multi-dimensional information, the generated editing script can be more in line with expectations, and the target multimedia data obtained according to the editing script can be more story-telling and have better effects.
[0127] Figure 5 A flowchart of a method for generating a clipping script proposed in an embodiment of the present disclosure, based on Figure 4 The embodiment shown, Figure 5 right Figure 4 Step 402 in is further described as follows: Figure 5 As shown, the following steps are included:
[0128] Step 501: Determine sequence information of a plurality of multimedia segments to be processed based on first information.
[0129] In some embodiments, based on the first information, the sequence information of multiple multimedia segments to be processed is determined. The large language model in the cloud can sort the multiple multimedia segments to be processed based on at least one of the content description information, voice information, and scene information of the multiple multimedia segments to be processed to obtain the sorting number of each multimedia segment to be processed.
[0130] Step 502: According to the first information of two to-be-processed multimedia segments with adjacent sequence numbers, the conversion information that meets the conversion rule is screened out from the script library.
[0131] In some embodiments, based on the first information of two multimedia segments to be processed with adjacent sorting numbers, a large language model in the cloud can filter conversion information that meets the conversion rules from a pre-provided script library based on at least one of the content description information, scene information, and voice information of the two multimedia segments to be processed with adjacent sorting numbers, wherein the conversion rules can be to make the scene transition between the two adjacent multimedia segments to be processed slow or fast, or to make the connection between the two adjacent multimedia segments to be processed more natural.
[0132] In some embodiments, the script library may be a collection of transition materials provided in advance to the large language model. The transition materials may include fade-in, fade-out, dissolve, slide, etc., which is not limited by the present disclosure.
[0133] Step 503: determine image modification information, music audio, and / or generate speech audio based on the conversion information and the first information.
[0134] In some embodiments, based on the conversion information and the first information, determining the image modification information, music audio, and / or generating the voice audio can be a large language model in the cloud. Based on the conversion information, content description information, voice information, and scene information, the image modification information and music audio are selected from the script library and the voice audio is generated.
[0135] For example, the cloud-based large language model selects filters, background music, and / or generates narration based on transition animations and text description information of multiple multimedia clips to be processed.
[0136] In the above embodiment, the cloud uses a trained large language model to generate a clipping script that conforms to the multiple multimedia segments to be processed based on the first information, and the generated clipping script is more consistent with the content and theme of the multimedia segments.
[0137] Figure 6 A flowchart of a multimedia data processing method proposed in an embodiment of the present disclosure, based on Figure 4 , Figure 5 In the embodiment shown, the method further comprises the following steps:
[0138] Step 601: receiving feature vectors of a plurality of multimedia segments to be processed sent by a terminal.
[0139] In some embodiments, the feature vector sent by the cloud receiving terminal may be feature vectors of multiple multimedia segments to be processed obtained by the terminal extracting frames from multiple multimedia segments to be processed and using a video encoder.
[0140] Step 602: Determine content description information of a plurality of multimedia segments to be processed based on the feature vectors.
[0141] In some embodiments, based on the feature vector, respectively determining the content description information of the plurality of multimedia segments to be processed may be by using a cloud-based visual language large model, taking the feature vector as the input of the model, and generating the content description information of the plurality of multimedia segments to be processed.
[0142] Step 603: Send the content description information to the terminal.
[0143] In some embodiments, the content description information is sent to the terminal so that the terminal can determine the first information.
[0144] In the above embodiment, based on the feature vectors of multiple multimedia segments to be processed sent by the terminal, the model deployed in the cloud can generate corresponding content description information based on the feature vectors, thereby realizing the generation of description text of the multimedia segments without uploading the original multimedia segments, thereby protecting the user's privacy.
[0145] In summary, the multimedia data processing method proposed in the present invention processes the first information uploaded by the terminal for textual description of multiple multimedia segments to be processed to generate a editing script, which is used by the terminal to edit the multiple multimedia segments to be processed according to the editing script to obtain the edited target multimedia data. It realizes the generation of editing scripts based on textual description information for video editing and generation of the terminal, avoids obtaining the user's original video, protects the user's privacy, and can generate editing scripts without the user providing prompt information, thereby improving processing efficiency and simplifying the processing flow.
[0146] Figure 7 is a flowchart of a specific implementation of a multimedia data processing method provided by the present disclosure, such as Figure 7 Said, comprising the following steps:
[0147] Step 1: Video preprocessing.
[0148] 1.1 Adopt an adaptive editing strategy for horizontal screen videos and unify the resolution of vertical and horizontal screen videos.
[0149] 1.2 Use a segmentation algorithm to segment the video, divide each video into multiple short videos according to the specific content category, and then use a highlight segment extraction algorithm to extract the highlight segments in each short video for subsequent editing.
[0150] Optional, the optional implementation of step 1 can be found in Figure 2 An optional implementation method for obtaining multiple multimedia segments to be processed is shown.
[0151] Step 2: Video information description stage.
[0152] 2.1 Video main information description generation. Use the visual language model to generate the description text of each short video segment. Use the end-side to extract video features, upload the video features to the cloud, and use the cloud-based language model to generate the final video description text.
[0153] 2.2 Use the scene classification model to obtain the scene information of each video as a supplementary description of the video.
[0154] 2.3 Identify the audio in the video and generate corresponding text information, which also serves as a supplementary description of the video.
[0155] The description information of the above three parts is combined as the complete description information of each short video segment.
[0156] Optional implementations of step 2 can be found in Figure 3 An optional implementation manner of performing a first processing on a plurality of multimedia segments to be processed is shown.
[0157] Step 3: Video editing script generation stage.
[0158] The cloud uses each highlight video as input to the big language model, and uses the cloud-based big language model to give recommended video sorting order, transition animation recommendations, filter / background music recommendations, and narration generation, etc. You can also input user needs and use the big language model to generate script recommendations with a director's style.
[0159] Optional implementations of step 3 can be found in Figure 4 and Figure 5 An alternative embodiment of the multimedia data processing method shown.
[0160] Step 4: Video editing and synthesis stage.
[0161] The client obtains the generated editing script, splices and merges the videos, adds filters, background music, narration, transition animations and other effects, and obtains the final edited video as the return result.
[0162] Optional implementations of step 4 can be found in Figure 1 Alternative implementations of the illustrated embodiments.
[0163] In summary, the multimedia data processing method provided by the present invention does not need to upload the original video to the cloud, thus protecting the privacy of the user, and utilizes the multi-dimensional information of the video and uses a large language model to generate editing scripts, so that the final generated editing video is more story-telling and has better effects.
[0164] Figure 8FIG. 8 is a schematic diagram of the structure of a multimedia data processing device 800 according to an embodiment of the present disclosure. Figure 8 As shown, the device includes: an acquisition module 810, a first processing module 820, a transceiver module 830, and a second processing module 840.
[0165] The acquisition module 810 is used to acquire a plurality of multimedia segments to be processed.
[0166] The first processing module 820 is used to perform first processing on the multiple multimedia segments to be processed respectively to obtain first information of the multiple multimedia segments to be processed, and the first information is used to perform text description on the multiple multimedia segments to be processed.
[0167] The transceiver module 830 is used to upload the first information to the cloud, and obtain the editing scripts of multiple multimedia segments to be processed generated by the cloud based on the first information.
[0168] The second processing module 840 is used to process the multiple multimedia segments to be processed according to the editing script to obtain target multimedia data.
[0169] In some embodiments, the acquisition module is also used to acquire multiple original multimedia segments; adaptively crop the multiple original multimedia segments respectively to obtain multiple first multimedia segments; segment the multiple first multimedia segments according to the content categories of the multiple first multimedia segments respectively to obtain multiple second multimedia segments; extract the content that meets preset conditions in each second multimedia segment to obtain multiple multimedia segments to be processed.
[0170] In some embodiments, the second processing module is also used to perform feature extraction on multiple multimedia segments to be processed respectively to obtain feature vectors of multiple multimedia segments to be processed; upload the feature vectors of the multiple multimedia segments to be processed to the cloud to obtain content description information of the multiple multimedia segments to be processed generated in the cloud; use multiple large language models for identifying different categories of information to perform second processing on the multiple multimedia segments to be processed respectively to obtain second information of the multiple multimedia segments to be processed; integrate the content description information and the second information of the multiple multimedia segments to be processed to obtain first information of the multiple multimedia segments to be processed.
[0171] In some embodiments, the second processing module is also used to use a scene recognition model to perform scene recognition on multiple multimedia clips to be processed to obtain scene information of multiple multimedia clips to be processed; and use a speech recognition model to perform speech recognition on multiple multimedia clips to be processed to obtain speech information of multiple multimedia clips to be processed.
[0172] In some embodiments, the first information includes: at least one of content description information, scene information, voice information, and user-preset style information; the editing script includes sequence information of multiple multimedia clips to be processed, image modification information, music audio, voice audio, and conversion information between clips.
[0173] In summary, the multimedia data processing device proposed in the present disclosure obtains multiple multimedia segments to be processed, extracts text description information of the multiple multimedia segments to be processed, uploads the text description information to the cloud, and obtains the editing script generated by the cloud based on the text description information, which is used to edit and splice the multiple multimedia segments to be processed. It can be achieved that there is no need to upload the original multimedia segments to the cloud, and an expected editing script can be generated, which reduces complexity and improves efficiency. The target multimedia data obtained according to the editing script is more story-telling, and has better coherence and effect.
[0174] Fig. 9 FIG. 9 is a schematic diagram of the structure of a multimedia data processing device 900 according to an embodiment of the present disclosure. Fig. 9 As shown, the device includes: a transceiver module 910 and a generation module 920.
[0175] The transceiver module 910 is used to receive first information of a plurality of multimedia segments to be processed sent by a terminal, where the first information is obtained by the terminal performing a first process on the plurality of multimedia segments to be processed.
[0176] The processing module 920 is used to generate a plurality of editing scripts of the multimedia segments to be processed based on the first information.
[0177] The transceiver module 910 is further used to send a clipping script to the terminal, and the clipping script is used by the terminal to perform a second process on the multiple multimedia segments to be processed.
[0178] In some embodiments, the processing module is also used to determine the sequence information of multiple multimedia segments to be processed based on the first information, and the sequence information includes the sorting numbers of the multiple multimedia segments to be processed; according to the first information of two multimedia segments with adjacent sorting numbers, the conversion information that meets the conversion rules is filtered out from the script library, and based on the conversion information and the first information, the image modification information, music audio, and / or voice audio is determined.
[0179] In some embodiments, the transceiver module is further used to receive feature vectors of multiple multimedia segments to be processed sent by the terminal, where the feature vectors are obtained by the terminal performing feature extraction on the multiple multimedia segments to be processed.
[0180] In some embodiments, the processing module is further used to determine content description information of a plurality of multimedia segments to be processed respectively based on the feature vector, and the content description information is used by the terminal to determine the first information.
[0181] In some embodiments, the transceiver module is further configured to send content description information to the terminal.
[0182] In summary, the cloud-based multimedia data processing device proposed in the present disclosure receives first information of multiple multimedia segments to be processed sent by a terminal; based on the first information, generates editing scripts of multiple multimedia segments to be processed; and sends the editing scripts to the terminal, where the editing scripts are used by the terminal to process the multiple multimedia segments to be processed. This can avoid the need for the terminal to send the original multimedia segments to the cloud, and only requires uploading the text description information of the multimedia segments to the cloud, which can be used to generate the editing scripts for the large language model in the cloud, thereby simplifying the editing process and protecting the privacy of users.
[0183] Fig.10 is a schematic structural diagram of an electronic device 1000 for implementing the above-mentioned multimedia data processing method according to an exemplary embodiment.
[0184] Reference Fig.10 , the electronic device 1000 may include one or more of the following components: a processing component 1002 , a memory 1004 , a power component 1006 , a multimedia component 1008 , an audio component 1010 , an input / output (I / O) interface 1012 , a sensor component 1014 , and a communication component 1016 .
[0185] The processing component 1002 generally controls the overall operation of the electronic device 1000, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 1002 may include one or more processors 1020 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 1002 may include one or more modules to facilitate the interaction between the processing component 1002 and other components. For example, the processing component 1002 may include a multimedia module to facilitate the interaction between the multimedia component 1008 and the processing component 1002.
[0186] The memory 1004 is configured to store various types of data to support operations on the electronic device 1000. Examples of such data include instructions for any application or method operating on the electronic device 1000, contact data, phone book data, messages, pictures, videos, etc. The memory 1004 may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.
[0187] The power supply component 1006 provides power to various components of the electronic device 1000. The power supply component 1006 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 1000.
[0188] The multimedia component 1008 includes a screen that provides an output interface between the electronic device 1000 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 1008 includes a front camera and / or a rear camera. When the electronic device 1000 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.
[0189] The audio component 1010 is configured to output and / or input audio signals. For example, the audio component 1010 includes a microphone (MIC), and when the electronic device 1000 is in an operation mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 1004 or sent via the communication component 1016. In some embodiments, the audio component 1010 also includes a speaker for outputting audio signals.
[0190] I / O interface 1012 provides an interface between processing component 1002 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: home button, volume button, start button, and lock button.
[0191] The sensor assembly 1014 includes one or more sensors for providing various aspects of status assessment for the electronic device 1000. For example, the sensor assembly 1014 can detect the open / closed state of the electronic device 1000, the relative positioning of components, such as the display and keypad of the electronic device 1000, and the sensor assembly 1014 can also detect the position change of the electronic device 1000 or a component of the electronic device 1000, the presence or absence of user contact with the electronic device 1000, the orientation or acceleration / deceleration of the electronic device 1000, and the temperature change of the electronic device 1000. The sensor assembly 1014 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 1014 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 1014 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.
[0192] The communication component 1016 is configured to facilitate wired or wireless communication between the electronic device 1000 and other devices. The electronic device 1000 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, 4G LTE, 5G NR (NewRadio) or a combination thereof. In an exemplary embodiment, the communication component 1016 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 1016 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.
[0193] In an exemplary embodiment, the electronic device 1000 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.
[0194] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 1004 including instructions, and the instructions can be executed by the processor 1020 of the electronic device 1000 to complete the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.
[0195] The embodiments of the present disclosure further provide a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute the multimedia data processing method described in the above embodiments of the present disclosure.
[0196] The embodiments of the present disclosure further provide a computer program product, including a computer program, which executes the multimedia data processing method described in the above embodiments of the present disclosure when a processor is used to execute the computer program.
[0197] A chip is also proposed in an embodiment of the present disclosure, including at least one processor and a communication interface, the communication interface is used to receive signals input into the chip or output signals from the chip, the processor communicates with the communication interface and implements the multimedia data processing method described in the above embodiment of the present disclosure through logic circuits or execution code instructions.
[0198] Fig.11 FIG. 1 is a schematic diagram of a chip 1100 for implementing the multimedia data processing method according to an exemplary embodiment. Fig.11 The chip 1100 includes at least one communication interface 1101 and a processor 1102. The communication interface 1101 is used to receive signals input to the chip 1100 or signals output from the above chip 1100. The processor 1102 communicates with the communication interface 1101 and implements the multimedia data processing method described in the above embodiments of the present disclosure through logic circuits or execution code instructions.
[0199] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.
[0200] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "illustrative embodiments", "examples", "specific examples" or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiments or examples are included in at least one embodiment or example of the present disclosure. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.
[0201] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present disclosure includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present disclosure belong.
[0202] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processing module, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (control method), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or otherwise processing in a suitable manner if necessary, and then stored in a computer memory.
[0203] It should be understood that the various parts of the embodiments of the present disclosure can be implemented by hardware, software, firmware or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.
[0204] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.
[0205] In addition, each functional unit in each embodiment of the present disclosure may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium. The above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.
[0206] Although the embodiments of the present disclosure have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present disclosure. A person skilled in the art may change, modify, substitute and vary the above embodiments within the scope of the present disclosure.
Claims
1. A multimedia data processing method, characterized in that: The method is executed by a terminal, and includes: Obtain multiple multimedia segments to be processed; Performing first processing on the plurality of multimedia segments to be processed respectively to obtain first information of the plurality of multimedia segments to be processed, wherein the first information describes the plurality of multimedia segments to be processed in text; Uploading the first information to a cloud, and obtaining a clipping script of the plurality of multimedia segments to be processed generated by the cloud based on the first information; According to the editing script, the multiple multimedia segments to be processed are processed to obtain target multimedia data.
2. The method according to claim 1, characterized in that The obtaining of a plurality of multimedia segments to be processed comprises: Acquire multiple original multimedia segments; Adaptively cutting the multiple original multimedia segments respectively to obtain multiple first multimedia segments; Segmenting the plurality of first multimedia segments respectively according to content categories of the plurality of first multimedia segments to obtain a plurality of second multimedia segments; Content that meets a preset condition is extracted from each second multimedia segment to obtain the plurality of multimedia segments to be processed.
3. The method according to claim 1 or 2, characterized in that: The performing first processing on the plurality of multimedia segments to be processed respectively comprises: Extracting features from the plurality of multimedia segments to be processed respectively to obtain feature vectors of the plurality of multimedia segments to be processed; Uploading the feature vectors of the plurality of multimedia segments to be processed to the cloud, and obtaining content description information of the plurality of multimedia segments to be processed generated by the cloud; Using a plurality of large language models for identifying different types of information, respectively performing second processing on the plurality of multimedia segments to be processed to obtain second information of the plurality of multimedia segments to be processed; The content description information and the second information of the plurality of multimedia segments to be processed are integrated to obtain the first information of the plurality of multimedia segments to be processed.
4. The method according to claim 3, characterized in that The performing second processing on the plurality of multimedia segments to be processed respectively comprises at least one of the following: Using a scene recognition model to perform scene recognition on the plurality of multimedia clips to be processed, to obtain scene information of the plurality of multimedia clips to be processed; The speech recognition model is used to perform speech recognition on the multiple multimedia segments to be processed to obtain speech information of the multiple multimedia segments to be processed.
5. The method according to claim 4, characterized in that The first information includes: at least one of the content description information, the scene information, the voice information, and user-preset style information; The editing script includes: sequence information of the plurality of multimedia segments to be processed, image modification information, music audio, voice audio, and conversion information between segments.
6. A multimedia data processing method, characterized in that: The method is executed by the cloud, and the method includes: receiving first information of a plurality of multimedia segments to be processed sent by a terminal, where the first information is obtained by the terminal performing a first process on the plurality of multimedia segments to be processed; Based on the first information, generating a clipping script of the plurality of multimedia segments to be processed; The clipping script is sent to the terminal, and the clipping script is used by the terminal to perform a second process on the multiple multimedia segments to be processed.
7. The method according to claim 6, characterized in that The step of generating a clipping script for the plurality of multimedia segments to be processed based on the first information comprises: Based on the first information, determining sequence information of the plurality of multimedia segments to be processed, the sequence information comprising sequence numbers of the plurality of multimedia segments to be processed; According to the first information of the two to-be-processed multimedia segments with adjacent sequence numbers, the conversion information that meets the conversion rule is screened out from the script library; Based on the conversion information and the first information, image modification information, music audio, and / or speech audio are determined.
8. The method according to claim 6, characterized in that The method further comprises: receiving feature vectors of the plurality of multimedia segments to be processed sent by the terminal, where the feature vectors are obtained by the terminal through feature extraction of the plurality of multimedia segments to be processed; Based on the feature vectors, respectively determine content description information of the multiple multimedia segments to be processed, where the content description information is used by the terminal to determine the first information; The content description information is sent to the terminal.
9. A multimedia data processing device, characterized in that: include: An acquisition module, used for acquiring a plurality of multimedia segments to be processed; A first processing module, configured to perform first processing on the plurality of multimedia segments to be processed respectively to obtain first information of the plurality of multimedia segments to be processed, wherein the first information describes the plurality of multimedia segments to be processed in text; A transceiver module, configured to upload the first information to a cloud, and obtain a clipping script of the plurality of multimedia segments to be processed generated by the cloud based on the first information; The second processing module is used to process the multiple multimedia segments to be processed according to the editing script to obtain target multimedia data.
10. A multimedia data processing device, characterized in that: include: a transceiver module, configured to receive first information of a plurality of multimedia segments to be processed sent by a terminal, wherein the first information is obtained by the terminal performing a first processing on the plurality of multimedia segments to be processed; A generating module, configured to generate a clipping script of the plurality of multimedia segments to be processed based on the first information; The transceiver module is used to send the clipping script to the terminal, and the clipping script is used by the terminal to perform a second process on the multiple multimedia segments to be processed.
11. An electronic device, characterized in that: include: a processor and a memory for storing a computer program capable of being executed on the processor, Wherein, when the processor is used to run the computer program, it executes the method described in any one of claims 1-5 or 6-8.
12. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-5 or 6-8.
13. A computer program product, characterized in that The invention comprises a computer program which, after being executed, implements the method according to any one of claims 1 to 5 or 6 to 8.
14. A chip, characterized in that: It includes at least one processor and a communication interface; the communication interface is used to receive a signal input to the chip or a signal output from the chip, and the processor communicates with the communication interface and implements the method as described in any one of claims 1-5 or 6-8 through a logic circuit or executing code instructions.
Citation Information
Cited By
Video generation method and device, and electronic equipment
CN120812344A
