Video processing method and related equipment

Abstract

The invention provides a video processing method and related equipment. The method comprises the following steps: acquiring attribute information of a target object and an initial video, wherein the initial video comprises audio data about the target object; obtaining a video category and a video effect list of the initial video based on a target text corresponding to the audio data and the attribute information; determining a target material corresponding to the effect object based on the video category and the effect material label; and adding the target material into the original video based on a target timestamp of the text segment in the audio data to obtain a target video.

Description

Technical Field

[0001] The present disclosure relates to the technical field of video processing, and in particular to a video processing method and related equipment. Background Art

[0002] Video processing can enrich the visual and auditory effects of the video through the combination of text, music, sound effects, stickers, special effects and other materials, making the transmission of information and emotions more prominent. However, although existing videos can achieve automatic addition of effects, the resulting video effects are not good. Summary of the invention

[0003] The present disclosure proposes a video processing method and related devices to solve the technical problem of poor video processing effect to a certain extent.

[0004] In a first aspect, the present disclosure provides a video processing method, comprising:

[0005] Acquire attribute information of a target object and an initial video, wherein the initial video includes audio data about the target object;

[0006] Based on the target text corresponding to the audio data and the attribute information, a video category and a video effect list of the initial video are obtained; wherein the video effect list includes at least one effect object, and the effect object includes an effect attribute label about a preset effect attribute and a text segment corresponding to at least part of the audio data;

[0007] Determine the target material corresponding to the effect object based on the video category and the effect material label;

[0008] The target material is added to the original video based on the target timestamp of the text segment in the audio data to obtain a target video.

[0009] In a second aspect of the present disclosure, a video processing device is provided, comprising:

[0010] An acquisition module, used to acquire attribute information of a target object and an initial video, wherein the initial video includes audio data about the target object;

[0011] an effect list module, which uses language to obtain a video category and a video effect list of the initial video based on the target text corresponding to the audio data and the attribute information; wherein the video effect list includes at least one effect object, and the effect object includes an effect attribute label about a preset effect attribute and a text segment corresponding to at least part of the audio data;

[0012] A material determination module, used to determine a target material corresponding to the effect object based on the video category and the effect material tag;

[0013] The material adding module is used to add the target material to the original video based on the target timestamp of the text segment in the audio data to obtain the target video.

[0014] In a third aspect of the present disclosure, an electronic device is provided, comprising one or more processors, a memory; and one or more programs, wherein the one or more programs are stored in the memory and executed by the one or more processors, and the programs include instructions for executing the method described in the first aspect.

[0015] According to a fourth aspect of the present disclosure, a non-volatile computer-readable storage medium containing a computer program is provided. When the computer program is executed by one or more processors, the processors execute the method described in the first aspect.

[0016] In a fifth aspect of the present disclosure, a computer program product is provided, comprising computer program instructions, which, when executed on a computer, enable the computer to execute the method described in the first aspect.

[0017] From the above, it can be seen that the video processing method and related equipment provided by the present disclosure automatically generate video categories and effect objects including at least one with effect attribute tags through the attribute information of the target object and the target text corresponding to the audio data to form a video effect list. Then, according to the video category and the effect attribute tag, suitable effect materials are selected and embedded into the original video of the target object to obtain the target video. The content quality, viewing experience and dissemination efficiency of the video can be significantly improved, making the video more attractive and the information transmission more effective. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the present disclosure or related technologies, the drawings required for use in the embodiments or related technical descriptions are briefly introduced below. Obviously, the drawings described below are only embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0019] Figure 1 A schematic diagram of a video processing architecture according to an embodiment of the present disclosure.

[0020] Figure 2 The figure is a schematic diagram of the hardware structure of an exemplary electronic device according to an embodiment of the present disclosure.

[0021] Figure 3 The figure is a schematic flow chart of a video processing method according to an embodiment of the present disclosure.

[0022] Figure 4Schematic diagram of a video processing method according to an embodiment of the present disclosure.

[0023] Figure 5 Schematic diagram of a video processing device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0024] In order to make the objectives, technical solutions and advantages of the present disclosure more clearly understood, the present disclosure is further described in detail below in combination with specific embodiments and with reference to the accompanying drawings.

[0025] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present disclosure should be understood by people with ordinary skills in the field to which the present disclosure belongs. The "first", "second" and similar words used in the embodiments of the present disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. "Including" or "comprising" and similar words mean that the elements or objects appearing before the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0026] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0027] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0028] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0029] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet the relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0030] Figure 1 FIG. 1 is a schematic diagram showing a video processing architecture of an embodiment of the present disclosure. Figure 1 The video processing architecture 100 may include a server 110, a terminal 120, and a network 130 that provides a communication link. The server 110 and the terminal 120 may be connected via a wired or wireless network 130. The server 110 may be an independent physical server, or a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, security services, and CDN.

[0031] The terminal 120 may be implemented in hardware or software. For example, when the terminal 120 is implemented in hardware, it may be various electronic devices having a display screen and supporting page display, including but not limited to smart phones, tablet computers, e-book readers, laptop portable computers, and desktop computers, etc. When the terminal 120 device is implemented in software, it may be installed in the electronic devices listed above; it may be implemented as multiple software or software modules (such as software or software modules used to provide distributed services), or it may be implemented as a single software or software module, which is not specifically limited here.

[0032] It should be noted that the video processing method provided in the embodiment of the present application can be executed by the terminal 120 or by the server 110. It should be understood that Figure 1 The number of terminals, networks and servers in the embodiment is only for illustration and is not intended to limit the number of terminals, networks and servers. Any number of terminals, networks and servers may be provided as required.

[0033] Figure 2 FIG. 2 shows a schematic diagram of the hardware structure of an exemplary electronic device 200 provided in an embodiment of the present disclosure. Figure 2 As shown, the electronic device 200 may include: a processor 202, a memory 204, a network module 206, a peripheral interface 208 and a bus 210. The processor 202, the memory 204, the network module 206 and the peripheral interface 208 are connected to each other in communication within the electronic device 200 through the bus 210.

[0034] Processor 202 may be a central processing unit (CPU), a video processor, a neural network processor (NPU), a microcontroller (MCU), a programmable logic device, a digital signal processor (DSP), an application specific integrated circuit (ASIC), or one or more integrated circuits. Processor 202 may be used to perform functions related to the technology described in this disclosure. In some embodiments, processor 202 may also include multiple processors integrated into a single logical component. For example, Figure 2 As shown, processor 202 may include a plurality of processors 202a, 202b, and 202c.

[0035] The memory 204 may be configured to store data (eg, instructions, computer code, etc.). Figure 2 As shown, the data stored in the memory 204 may include program instructions (e.g., program instructions for implementing the video processing method of the embodiment of the present disclosure) and data to be processed (e.g., the memory may store configuration files of other modules, etc.). The processor 202 may also access the program instructions and data stored in the memory 204, and execute the program instructions to operate on the data to be processed. The memory 204 may include a volatile storage device or a non-volatile storage device. In some embodiments, the memory 204 may include a random access memory (RAM), a read-only memory (ROM), an optical disk, a magnetic disk, a hard disk, a solid-state drive (SSD), a flash memory, a memory stick, etc.

[0036] The network module 206 can be configured to provide communication with other external devices to the electronic device 200 via a network. The network can be any wired or wireless network capable of transmitting and receiving data. For example, the network can be a wired network, a local wireless network (e.g., Bluetooth, WiFi, near field communication (NFC) etc.), a cellular network, the Internet or a combination thereof. It is understood that the type of network is not limited to the above specific examples. In some embodiments, the network module 306 can include any number of network interface controllers (NICs), radio frequency modules, transceivers, modems, routers, gateways, adapters, cellular network chips, etc., in any combination.

[0037] The peripheral interface 208 can be configured to connect the electronic device 200 to one or more peripheral devices to achieve information input and output. For example, the peripheral devices can include input devices such as a keyboard, a mouse, a touch pad, a touch screen, a microphone, and various sensors, and output devices such as a display, a speaker, a vibrator, and an indicator light.

[0038] The bus 210 can be configured to transmit information between various components of the electronic device 200 (e.g., the processor 202, the memory 204, the network module 206, and the peripheral interface 208), such as an internal bus (e.g., a processor-memory bus), an external bus (USB port, PCI-E bus), etc.

[0039] It should be noted that, although the architecture of the electronic device 200 only shows the processor 202, the memory 204, the network module 206, the peripheral interface 208 and the bus 210, in the specific implementation process, the architecture of the electronic device 200 may also include other components necessary for normal execution. In addition, it can be understood by those skilled in the art that the architecture of the electronic device 200 may also only include the components necessary for implementing the embodiments of the present disclosure, and does not necessarily include all the components shown in the figure.

[0040] Video processing can enrich the visual and auditory effects of the video through the combination of text, music, sound effects, stickers, special effects and other materials, making the transmission of information and emotions more prominent. Existing video processing algorithms can automatically mark and classify the content and text of the video first, and then match the corresponding text templates and sound effects and other materials according to the preset mapping rule strategy. These methods have high training costs. Video content classification and keyword classification rely on the determination of the classification system, data annotation and model training process. Each link has high time, annotation and training costs. In addition, these methods have poor scalability. The mapping rules need to be designed manually. For new materials, a series of rule configurations are required to ensure their effects, and they cannot be expanded on a large scale. Therefore, how to improve the quality of video processing, make videos more attractive, and expand the application scope of materials have become technical problems that need to be solved urgently.

[0041] In view of this, the embodiments of the present disclosure provide a video processing method and related equipment. A video category and an effect object including at least one effect attribute tag are automatically generated through the attribute information of the target object and the target text corresponding to the audio data to form a video effect list. Then, according to the video category and the effect attribute tag, suitable effect materials are selected and embedded into the original video of the target object to obtain the target video. The content quality, viewing experience and dissemination efficiency of the video can be significantly improved, making the video more attractive and the information transmission more effective.

[0042] See also Figure 3 , Figure 3 The schematic flow chart of the video processing method according to the embodiment of the present disclosure is shown. The video processing method according to the embodiment of the present disclosure can be deployed on a terminal or a server. Figure 3 In the example, the video processing method 300 may further include the following steps.

[0043] In step S310, attribute information of a target object and an initial video are acquired, wherein the initial video includes audio data about the target object.

[0044] The target object may refer to the subject of the initial video to be processed, and all the contents of the initial video are centered around the target object. The attribute information may refer to various features of the object, such as color, size, function, material, etc. The audio data may refer to the sound information accompanying the video screen, such as dialogue, background music, ambient sound effects, etc. It should be understood that in the initial video, these attribute information may be presented to the audience through screen display and audio data (such as narration).

[0045] Specifically, the target object may be a product to be recommended to the user, and the corresponding attribute information may be product information. For example, the initial video video_1 may be an introduction to the target object object, and the attribute information Product Info of the target object may include: product name ProductName, product brand brand, product selling point SellingPoint, price price, and promotion information Promotion. The attribute information may be data in JSON format. The audio data studio may be the narration or spoken text in the initial video video_1. The target text corresponding to the initial video video_1 may be obtained based on speech recognition, such as Figure 4 As shown, Figure 4 A schematic diagram of a video processing method according to an embodiment of the present disclosure is shown.

[0046] In step S320, the video category and video effect list of the initial video are obtained based on the target text corresponding to the audio data and the attribute information; wherein the video effect list includes at least one effect object, and the effect object includes an effect attribute label about a preset effect attribute and a text segment corresponding to at least part of the audio data.

[0047] Among them, a video category (video_category) may refer to a tag or category that describes the type of video content. A video effect list (effect_list) may contain a list of at least one effect object, each of which is designed to enhance the expressiveness of a specific part of a video. An effect object (effect_object) may refer to an element in a video effect list, used to indicate a specific part of the enhanced video. An effect attribute tag (effect_attribute) may indicate the attributes of an effect object. A text segment (text_segment) may refer to text content in audio data associated with an effect object, used to indicate the time when the effect is applied.

[0048] In some embodiments, the effect object further includes an effect weight, which is used to indicate the importance of the effect object. The effect weight (effect_weight) may refer to the display duration and / or display intensity of the effect object. For example, the effect weight effect_weight may be a value between 1 and 5, indicating the importance of the effect object, and 5 may represent the highest importance.

[0049] Specifically, the target object may be furniture, and the initial video is a video of the furniture, and the characteristics of the furniture, such as load-bearing capacity and reinforced edges, are introduced in the video. According to the target text corresponding to the audio data in the initial video, a video category ["Furniture", "Home Improvement"] can be generated, indicating that the video content is related to furniture and home improvement. A video effect list is generated based on the target text and attribute information, including: effect object 1 and its effect attributes; effect object 2 and its effect attributes. Effect object 1 and effect object 2 can describe two important characteristics in the original video, namely, the load-bearing capacity and reinforced edges of the furniture. By analyzing the audio data, the relevant text segments of these two characteristics can be determined, and a series of effect attributes are designed for each characteristic to highlight these characteristics in the target video. For example, for the characteristic of "load-bearing capacity", warm tones and bright brightness can be used to highlight it, and a soft and simple font style can be selected. For the characteristic of "reinforced edges", neutral tones and bright brightness can be used, and a soft font style can also be selected, but with different brightness. In this way, specific visual effects can be added to each important part of the video to enhance the expressiveness and attractiveness of the video.

[0050] In some embodiments, obtaining the video category and video effect list of the initial video based on the target text corresponding to the audio data and the attribute information includes:

[0051] Generate an effect generation instruction for the initial video based on the target text, the attribute information and the task prompt text;

[0052] In response to the effect generation instruction, classifying the target text and the attribute information based on the task prompt text to obtain the video category;

[0053] Dividing the target text into at least one text segment, wherein each text segment corresponds to each effect object;

[0054] Classifying the preset effect attribute of the effect object based on the text segment and the attribute information to determine the effect attribute label of the preset effect attribute of the effect object in the preset classification label;

[0055] The video effect list is formed based on the effect attribute tags of the effect objects and the corresponding text segments.

[0056] Among them, the target text, attribute information and task prompt text are used to automatically create effect generation instructions for editing video effects. For example, if the task prompt text requires adding a certain special effect, then it will be decided when to add the special effect based on the content and attribute information of the target text. The target text is divided into multiple text segments, each of which can correspond to an effect object in the video. For example, one video segment may focus on introducing the appearance of a product, and another segment may focus on the function of the product. The preset effect attributes of the effect object are classified according to the content and attribute information of the text segment, so as to determine the effect attribute label applicable to each segment. For example, if a text segment mentions the unique design of a product, a visually prominent design effect may be selected. According to the effect attribute label of the effect object and the corresponding text segment, a video effect list is formed, which includes various effects to be implemented in the video. For example, for a text segment that emphasizes the innovation of the product, a futuristic animation effect may be added to the list. In this way, the entire video can be automatically classified, and appropriate effects are added according to the content and attribute information, thereby improving the quality and efficiency of video processing.

[0057] In some embodiments, the task prompt text includes at least one of the following: an input data format, an output data format, an input data example, an output data example, or a preset rule regarding the effect attribute label.

[0058] Specifically, the task prompt text can be input into the language processing model to prompt the language processing model how to generate the video effect list. For example, the task prompt text can include "In this task, you play the role of an experienced video editor who is good at video script creation and video post-production. Your ultimate goal is to design a post-production strategy for each shot to make the entire video more attractive. Task description: You will be given a video narration script that introduces a product, as well as detailed product information (PRODUCT_INFO). Based on the specified preset effect attributes (EFFECT_ATTRIBUTE), you need to create a JSON object for the packaging element, which contains two fields: the video category suitable for the video; and a list of video effects including one or more effect objects. Each effect object can contain the following fields: a JSON object containing effect attributes, and a text snippet from the narration script that appears with the effect object."

[0059] Among them, the preset effect attributes can have six characteristic dimensions: Effect text, a string displayed in the effect object, the display style depends on the effect name. Effect name, which can be four different types, such as price, which may contain currency type and numbers. Discount number, which may contain a percent sign "%". Highlight, emphasize the key selling points or related details in the current shot. Subscribe, prompt the user to take actions such as follow or buy. Color style, the main color of the effect object, such as warm tones, cool tones, neutral colors, or colorful styles. Color brightness, the brightness of the effect object, such as bright or dim. Color hue: the color hue of the effect object, including red, orange, yellow, green, blue, purple, pink, black, brown, gray, white, gold, silver, etc. Font style: the font style of the effect object, including minimalism, magazine style, business style, graffiti, street style, high fashion, luxury, soft and gentle, retro, cute style, innocent and fresh, technology style, fashionable and cool, 3D, realistic style, bright, etc.

[0060] The input data is a JSON object with the following fields:VOICEOVER: a punctuated string representing the entire voiceover script of the video.Product information: a JSON object containing various attributes of the product.The output data can contain two fields:video_category: a list of video categories suitable for the video.Video effect list: a list of effect objects, each with fields such as effect attributes, text snippets, and effect weights.You need to create a suitable video category and effect list based on the input voiceover script and product information, combined with the effect attributes, to meet the needs of video production.

[0061] The preset rules for the effect attribute tag may include at least one of the following:

[0062] A unified visual style means that all visual effects (such as color, tone, style) are consistent to ensure the overall beauty and coordination of the video. For example, in the output example, different effects use different color styles and brightness, but the overall sense of harmony is maintained.

[0063] Effect list order,The entries in the effect list can be arranged in the order they appear in the voice description,,ensuring that no text paragraphs overlap between different effects.

[0064] Follow the output format, that is, the output must strictly follow the given JSON format so that it can be correctly parsed by subsequent code; this includes all necessary keys and values, as well as correct data types and nested structures.

[0065] Effect text specification, the effect_text attribute can be no more than a preset number of characters, for example no more than 18 characters, and can accurately reflect the expected effect. It may not be exactly the same as the text fragment in the audio data, but should convey the same core information.

[0066] Text segment specification, text_segment attribute needs to exactly match the target text corresponding to the corresponding audio data. For example, do not convert the text into numbers or symbols, keep its original form.

[0067] Price type effect, that is, if the price is mentioned in the audio data, a price type effect object can be included in the video effect list, so as to highlight the price information and attract the audience's attention.

[0068] At least one highlight effect, that is, the "Highlight" effect can be used at least once in the effect object to emphasize the key features or selling points of the product.

[0069] In step S330, the target material corresponding to the effect object is determined based on the video category and the effect material tag.

[0070] In some embodiments, determining the target material corresponding to the effect object based on the video category and the effect material tag includes:

[0071] Determine candidate materials in a material library based on the video category;

[0072] Matching the effect material tag with the material tag of the candidate material to obtain a matching score;

[0073] The candidate material with the highest matching score is determined as the target material.

[0074] Among them, materials that meet the video category are screened out from the material library as candidate materials. For example, if the video category is "travel", materials related to travel, scenery, etc. are selected from the material library. The similarity between the candidate material and the effect object is reflected based on the degree of matching between the material label of the candidate material and the effect material label. For example, if the effect material label contains "dynamic" and "modern", the candidate material that matches these labels will obtain a higher matching score. The candidate material with the highest matching score is determined as the target material to be added to the original video. Specifically, the identifier of the target material (such as the material ID) can be inserted into the corresponding effect object of the video effect list.

[0075] In some embodiments, matching the effect material tag with the material tag of the candidate material in the material library to obtain a matching score includes:

[0076] The matching score of the candidate material is obtained based on the material label in the candidate material that is consistent with the effect material label and the corresponding label weight.

[0077] Among them, considering the importance of different tags, the tag weight can be used to adjust the matching score. For example, for the material s of the video category, the tag weight of the effect material tag S1 is q1, the tag weight of the effect material tag S2 is q2, the tag weight of the effect material tag S3 is q3, ... If the material tag of a candidate material is consistent with the effect material tag S1 and the effect material tag S2, its matching score can be q1+q2; if the material tag of a candidate material is consistent with the effect material tag S1, its matching score can be q1.

[0078] In some embodiments, matching the effect material tag with the material tag of the candidate material in the material library to obtain a matching score further includes:

[0079] In response to the candidate material being consistent with the styles of other effect objects, the matching score of the candidate material is increased by a preset value.

[0080] If the candidate material has a similar style to other effect objects, the matching score of the candidate material will be increased by a certain value. For example, if for effect object o1, candidate material s and other effect objects o2 in the video effect list are bright and colorful, then the matching score of candidate material s can be increased by a preset value y. If the matching score of candidate material s is q1+q2 before, the matching score of candidate material s is q1+q2+y. In this way, materials with the same style can be used to keep the style of the overall video processing effect consistent, thereby improving the quality and effect of video processing.

[0081] In some embodiments, matching the effect material tag with the material tag of the candidate material in the material library to obtain a matching score further includes:

[0082] Determine a recommendation weight of the candidate material based on the historical usage; wherein the greater the historical usage, the smaller the corresponding recommendation weight;

[0083] The matching score is updated based on a product of the recommendation weight and the matching score.

[0084] Among them, the recommendation weight of the candidate material can be determined according to the frequency with which the candidate material was used in the past. If a material has been used frequently in the past, its recommendation weight is low; conversely, if it is rarely used, its recommendation weight is high. Use the recommendation weight to adjust the matching score to avoid using certain materials too frequently. For example, if the matching score of a candidate material s is q1+q2 points, but its recommendation weight is x (because it has been used many times), then the adjusted matching score is (q1+q2)*x. In order to ensure the diversity and novelty of the material, the system will also adjust the recommendation weight of the candidate material based on the historical usage. If a material has been used frequently in the past, its recommendation weight will be reduced, which means that its matching score will be lowered, thereby reducing the possibility of it being selected again. In this way, the most suitable material for the video can be automatically selected while ensuring the novelty and diversity of the material, improving the overall quality and viewing experience of the video.

[0085] In some embodiments, method 300 further includes:

[0086] Get the original material;

[0087] Parsing at least a portion of the images in the original material to obtain at least one material attribute tag related to a preset material attribute;

[0088] The original material and the corresponding material attribute tag are stored in a material library.

[0089] Among them, the original material may refer to the unprocessed data or resources initially collected. For example, the original material may be a preview animated GIF image of a text template in a video. The material attribute label may refer to a label that describes the characteristics or attributes of the material. For example, the material attribute label may include the category of the original material (such as price category, discount category, etc.) and other labels related to visual effects. The material library may refer to a database or system that stores and manages the material and its attribute labels. For example, the material library may refer to an online database system used to store and maintain materials and their corresponding labels.

[0090] The user uses the original material as the input of the visual speech model, analyzes the image, and outputs the results in the specified format, which includes different types of effect names and descriptions. The type can be judged based on the description, and the reason for the judgment can be output to the text effect type field. The output results can be presented based on the JSON format, including fields such as language, color style, color brightness, main color extraction, font style, video category, etc. It should be understood that the above-mentioned fields of preset material attributes are only examples and are not intended to limit the preset material attributes. More preset material attributes and corresponding preset tags can also be set as needed, again without limitation.

[0091] Specifically, a series of classification and labeling tasks can be completed based on the image data and text effects of the original material itself. Classification and labeling can include: language identification, determining the language of the text effect, and outputting the result as a string to the language field, including the language number; optional languages ​​include English, Chinese, Korean, Japanese and other languages. Color style classification, classifying the color style of the text effect, and outputting the result as a string to the color style field; optional color styles include warm tones, cool tones, neutral colors and colorful styles. Brightness classification, classifying the brightness of the text effect, and outputting the result as a string to the color brightness field; optional brightness classification includes bright and dim. Color hue, extracting the main color from the text effect, and outputting the color with a proportion exceeding 30% to the color hue field, arranged in descending order by proportion; optional colors include red, orange, yellow, green, blue, purple, pink, black, brown, gray, white, gold, silver, etc. Visual style determination: determine the visual style of the text effect, and output the matching style to the visual style field, sorted by the degree of matching; the optional visual styles include minimalism, magazine style, business style, graffiti, street style, high fashion, luxury, soft and warm, retro, cute style, innocent and fresh, technology style, trendy and cool, three-dimensional, real style, bright, etc. Video category determination: determine the video category suitable for the text effect, and output the result to the video category field, sorted by the degree of applicability; the optional product themes include e-commerce platform, clothing accessories, beauty care, food and beverage, electronic products and home, sports and entertainment, maternal and childbirth, pets, real estate leasing, furniture decoration, tourism, education and training, games, finance, automobiles, practical software, professional services, primary industries, machinery industry, general product themes, etc. Text effect name matching: determine the effect type that best matches the text effect, and output the name to the text effect name field.

[0092] In some embodiments, parsing at least part of the image in the original material to obtain at least one material attribute tag of a preset material attribute includes:

[0093] Selecting the image frame with the largest image area in the original material;

[0094] Classifying the image frames based on the preset material attributes to obtain corresponding classification results;

[0095] The classification result is determined as a material attribute label of the preset material attribute.

[0096] Specifically, a label system for classifying and describing material attributes can be determined to define a label system, which includes several categories: price category, discount category, subscription click-through conversion category, product feature highlight category, user question category, and user comment category. In addition, labels related to visual effects can also be defined. The trained labeling model can be used to automatically label the material. For example, for an animated GIF, you can capture the image_max that has the largest screen share in the text template preview animated GIF. Use the prompt information and the image_max that has the largest screen share to input into the trained labeling model for labeling based on multiple dimensions (i.e., preset material attributes). Store the labeled label results in the system and ensure that these labels can be easily accessed and updated. When there is new original material, the label can be pulled and maintained in real time through the labeling model. This can effectively organize and utilize the material and improve the efficiency and quality of video processing.

[0097] In step S340, the target material is added to the original video based on the target timestamp of the text segment in the audio data to obtain a target video.

[0098] Among them, the timestamp of the text segment can be located based on the target text, and used for post-production, such as the synthesis of special effects scripts, and the target material is added to the original video to obtain the target video.

[0099] In some embodiments, method 300 further includes:

[0100] Preprocessing the text segment to obtain an intermediate segment in text format;

[0101] Perform similarity matching in the target text based on the intermediate segment to obtain a matching text segment;

[0102] The target timestamp of the text segment is determined based on the matching text segment and the text-timestamp mapping relationship; wherein the text-timestamp mapping relationship is obtained based on the words in the target text and the time when the words appear in the audio data.

[0103] Specifically, the target text corresponding to the audio data is merged into a continuous text string, with letters, numbers or Chinese characters separated by spaces. A text-timestamp mapping relationship is established for each character in the merged text, mapped to its word index in the original list. In this way, the specific word corresponding to each character can be tracked, and the time position of this word in the original audio data can be known. For the text segment to be queried (text_segment), there may be some inconsistencies with the original speech-to-text text (such as differences in numbers and text forms). The numbers in the text segment can be converted into text form to maintain consistency with the original text. An edit distance algorithm is used to find the closest matching text segment to the preprocessed text segment to tolerate a certain degree of difference (such as extra or missing characters) and find the start and end positions of the matching text segment. Using the previously created text-timestamp mapping relationship, the original word index corresponding to each character in the matching text segment is found. Based on these indexes, the start and end timestamps of each word are extracted from the original audio data. In this way, the time position of a specific text segment in the original audio data can be effectively located and used for subsequent video editing work, such as adding special effects scripts or synthesizing packaging elements. This helps automate the video production process and improve work efficiency.

[0104] It should be noted that the method of the embodiment of the present disclosure can be performed by a single device, such as a computer or a server. The method of the present embodiment can also be applied in a distributed scenario and completed by multiple devices cooperating with each other. In the case of such a distributed scenario, one of the multiple devices can only perform one or more steps in the method of the embodiment of the present disclosure, and the multiple devices will interact with each other to complete the described method.

[0105] It should be noted that the above describes some embodiments of the present disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the above embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0106] Based on the same technical concept, corresponding to any of the above-mentioned embodiments and methods, the present disclosure also provides a video processing device, see Figure 5 , the video processing device comprises:

[0107] An acquisition module, used to acquire attribute information of a target object and an initial video, wherein the initial video includes audio data about the target object;

[0108] an effect list module, which uses language to obtain a video category and a video effect list of the initial video based on the target text corresponding to the audio data and the attribute information; wherein the video effect list includes at least one effect object, and the effect object includes an effect attribute label about a preset effect attribute and a text segment corresponding to at least part of the audio data;

[0109] A material determination module, used to determine a target material corresponding to the effect object based on the video category and the effect material tag;

[0110] The material adding module is used to add the target material to the original video based on the target timestamp of the text segment in the audio data to obtain the target video.

[0111] For the convenience of description, the above device is described by dividing it into various modules according to its functions. Of course, when implementing the present disclosure, the functions of each module can be implemented in the same or multiple software and / or hardware.

[0112] The device of the above embodiment is used to implement the corresponding video processing method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be described in detail here.

[0113] Based on the same technical concept, corresponding to any of the above-mentioned embodiment methods, the present disclosure also provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the video processing method described in any of the above embodiments.

[0114] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0115] The computer instructions stored in the storage medium of the above embodiments are used to enable the computer to execute the video processing method described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0116] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present disclosure (including the claims) is limited to these examples. Based on the concept of the present disclosure, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the embodiments of the present disclosure as described above, which are not provided in detail for the sake of simplicity.

[0117] In addition, to simplify the description and discussion, and in order not to make the embodiments of the present disclosure difficult to understand, the known power / ground connections to the integrated circuit (IC) chips and other components may or may not be shown in the provided figures. In addition, the device can be shown in the form of a block diagram to avoid making the embodiments of the present disclosure difficult to understand, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform on which the embodiments of the present disclosure will be implemented (that is, these details should be fully within the scope of understanding of those skilled in the art). Where specific details (e.g., circuits) are set forth to describe exemplary embodiments of the present disclosure, it is apparent to those skilled in the art that the embodiments of the present disclosure can be implemented without these specific details or with changes in these specific details. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0118] Although the present disclosure has been described in conjunction with specific embodiments of the present disclosure, many replacements, modifications and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may use the embodiments discussed.

[0119] The embodiments of the present disclosure are intended to cover all such substitutions, modifications and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present disclosure should be included in the scope of protection of the present disclosure.

Claims

1. A video processing method, comprising: Acquire attribute information of a target object and an initial video, wherein the initial video includes audio data about the target object; Based on the target text corresponding to the audio data and the attribute information, a video category and a video effect list of the initial video are obtained; wherein the video effect list includes at least one effect object, and the effect object includes an effect attribute label about a preset effect attribute and a text segment corresponding to at least part of the audio data; Determine the target material corresponding to the effect object based on the video category and the effect material label; The target material is added to the original video based on the target timestamp of the text segment in the audio data to obtain a target video.

2. The method according to claim 1, wherein: Determining a target material corresponding to the effect object based on the video category and the effect material tag includes: Determine candidate materials in a material library based on the video category; Matching the effect material tag with the material tag of the candidate material to obtain a matching score; The candidate material with the highest matching score is determined as the target material.

3. The method according to claim 2, wherein: Based on the matching of the effect material label with the material label of the candidate material in the material library, a matching score is obtained, including: The matching score of the candidate material is obtained based on the material label in the candidate material that is consistent with the effect material label and the corresponding label weight.

4. The method according to claim 3, wherein: Based on the matching of the effect material label with the material label of the candidate material in the material library, a matching score is obtained, which further includes: In response to at least two of the candidate materials having the same style, increasing the matching score of the candidate materials by a preset value; or, Determine a recommendation weight of the candidate material based on the historical usage; wherein the greater the historical usage, the smaller the corresponding recommendation weight; The matching score is updated based on a product of the recommendation weight and the matching score.

5. The method according to claim 1, wherein: Obtaining a video category and a video effect list of the initial video based on the target text corresponding to the audio data and the attribute information includes: Generate an effect generation instruction for the initial video based on the target text, the attribute information and the task prompt text; In response to the effect generation instruction, classifying the target text and the attribute information based on the task prompt text to obtain the video category; Dividing the target text into at least one text segment, wherein each text segment corresponds to each effect object; Classifying the preset effect attribute of the effect object based on the text segment and the attribute information to determine the effect attribute label of the preset effect attribute of the effect object in the preset classification label; The video effect list is formed based on the effect attribute tags of the effect objects and the corresponding text segments.

6. The method according to claim 1, further comprising: Preprocessing the text segment to obtain an intermediate segment in text format; Perform similarity matching in the target text based on the intermediate segment to obtain a matching text segment; The target timestamp of the text segment is determined based on the matching text segment and the text-timestamp mapping relationship; wherein the text-timestamp mapping relationship is obtained based on the words in the target text and the time when the words appear in the audio data.

7. The method according to claim 1, further comprising: Get the original material; Parsing at least a portion of the images in the original material to obtain at least one material attribute tag related to a preset material attribute; The original material and the corresponding material attribute tag are stored in a material library.

8. The method according to claim 7, wherein: Parsing at least part of the image in the original material to obtain at least one material attribute tag of a preset material attribute includes: Selecting the image frame with the largest image area in the original material; Classifying the image frames based on the preset material attributes to obtain corresponding classification results; The classification result is determined as a material attribute label of the preset material attribute.

9. The method according to claim 1, wherein the effect object further comprises an effect weight, which is used to indicate the display duration and / or display intensity of the effect object; Alternatively, the task prompt text includes at least one of the following: an input data format, an output data format, an input data example, an output data example, or a preset rule regarding the effect attribute label.

10. A video processing device, comprising: An acquisition module, used to acquire attribute information of a target object and an initial video, wherein the initial video includes audio data about the target object; an effect list module, which uses language to obtain a video category and a video effect list of the initial video based on the target text corresponding to the audio data and the attribute information; wherein the video effect list includes at least one effect object, and the effect object includes an effect attribute label about a preset effect attribute and a text segment corresponding to at least part of the audio data; A material determination module, used to determine a target material corresponding to the effect object based on the video category and the effect material tag; The material adding module is used to add the target material to the original video based on the target timestamp of the text segment in the audio data to obtain the target video.

11. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 9 when executing the program.

12. A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to execute the method according to any one of claims 1 to 9.

Images