Method and device for intelligently explaining audio played by short video port

Abstract

The embodiment of the invention provides an intelligent explanation method and device for audio played by a short video port, and the method comprises the steps: receiving product multi-source heterogeneous data sent by a developer, extracting the feature representation of a bottom-layer product in the product multi-source heterogeneous data, inputting the extracted feature representation of the bottom-layer product into a set deep learning multilayer perceptron, and carrying out the deep learning of the bottom-layer product. Corresponding high-order product feature representations are obtained, feature weighted fusion is carried out on the high-order product feature representations, and product multi-modal features are obtained; and constructing a corresponding playing speed adjustment function according to the audio playing duration of the sentence corresponding to each video slice, updating the playing speed adjustment function according to the user characteristics in the user group portrait sent by the developer, and adjusting the playing speed of each video slice according to the playing speed adjustment function. Adding the product playing audio as a video background audio into the product video to obtain an adjusted product video; the quality and the propagation effect of the video content can be effectively improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and specifically to a method and device for intelligent audio explanation of short video oral broadcast. Background Art

[0002] With the rapid development of mobile Internet and short video technology, short videos have become one of the most popular forms of information dissemination and marketing. Compared with traditional static text or image advertisements, short videos can more vividly display product features, attract consumers' attention, and enhance product experience and memory. Therefore, more and more companies are beginning to use short videos in product marketing and promotion.

[0003] However, it is not easy to produce high-quality product marketing short videos. First, it is necessary to generate appealing video content and spoken audio based on the characteristics and selling points of the product. This requires marketers to have certain creativity and professional skills, and invest a lot of manpower and time costs. Secondly, for different target user groups, it is also necessary to personalize and optimize the video content to improve the dissemination effect of the video among various audiences. This personalized adjustment usually relies on in-depth analysis of user portraits and requires a lot of data support. Summary of the invention

[0004] In response to the problems in the prior art, the present application provides a method and device for intelligent audio explanation of short video oral broadcast, which can effectively improve the quality and dissemination effect of video content.

[0005] In order to solve at least one of the above problems, the present application provides the following technical solutions:

[0006] In a first aspect, the present application provides a short video oral audio intelligent explanation method, comprising:

[0007] Receive product multi-source heterogeneous data sent by developers, extract the underlying product feature representations from the product multi-source heterogeneous data, input the extracted underlying product feature representations into a set deep learning multi-layer perceptron to obtain corresponding high-order product feature representations, perform feature weighted fusion on each of the high-order product feature representations to obtain product multimodal features, perform semantic recognition on the product multimodal features through a preset knowledge graph reasoning algorithm, and determine the corresponding product selling points;

[0008] According to the product selling point and the preset product video generation model, a product oral audio and a product video are obtained, wherein the sentences in the product oral audio correspond to the product selling point, and the product video is sliced ​​according to each sentence in the product oral audio to obtain corresponding video slices;

[0009] A corresponding playback speed adjustment function is constructed according to the audio playback duration of the sentence corresponding to each of the video slices, and the playback speed adjustment function is updated according to the user characteristics in the user group portrait sent by the developer. The playback speed of each of the video slices is adjusted according to the playback speed adjustment function, and the product spoken audio is added to the product video as video background audio to obtain an adjusted product video.

[0010] Furthermore, the extracting of the underlying product feature representations from the multi-source heterogeneous data of the product, inputting the extracted underlying product feature representations into a deep learning multi-layer perceptron to obtain the corresponding high-order product feature representations, and performing feature weighted fusion on each of the high-order product feature representations to obtain product multimodal features, including:

[0011] Performing a data preprocessing operation including at least one of text segmentation, image enhancement, and video frame segmentation on the multi-source heterogeneous product data, and extracting underlying product feature representations from the multi-source heterogeneous product data;

[0012] The extracted underlying product feature representations are unified into a multimodal feature set and input into a set deep learning multi-layer perceptron to obtain the corresponding high-order product feature representations. The high-order product feature representations are weighted and fused through an attention mechanism to obtain product multimodal features.

[0013] Furthermore, the extracting of the underlying product feature representations from the multi-source heterogeneous data of the product, unifying the extracted underlying product feature representations into a multimodal feature set and inputting the extracted features into a set deep learning multi-layer perceptron to obtain the corresponding high-order product feature representations, includes:

[0014] The text data in the multi-source heterogeneous data of the product is converted into a mosquito vector representation by a word embedding algorithm, the visual features of the image data in the multi-source heterogeneous data of the product are extracted by a preset convolutional neural network, the spatiotemporal features of the video data in the multi-source heterogeneous data of the product are extracted by a frame division operation, and the underlying product feature representation is obtained according to the text vector representation and / or the visual features and / or the spatiotemporal features;

[0015] The underlying product feature representations are unified into a multimodal feature set through weighted fusion and input into a set deep learning multi-layer perceptron for complex interactive relationship learning to obtain the corresponding high-order product feature representation.

[0016] Furthermore, the high-order product feature representations are weighted and fused through the attention mechanism to obtain product multimodal features, including:

[0017] Calculate the correlation between each of the high-order product feature representations and the feature set consisting of all high-order product feature representations through a self-attention mechanism to obtain an attention weight for each of the high-order product feature representations;

[0018] The high-order product feature representations are weightedly fused using the attention weights to obtain product multimodal features.

[0019] Furthermore, the semantic recognition of the multimodal features of the product by a preset knowledge graph reasoning algorithm to determine the corresponding product selling points includes:

[0020] Performing semantic matching and association reasoning on the multimodal features of the product through a preset knowledge graph reasoning algorithm to obtain a concept entity in the preset knowledge graph that is most relevant to the multimodal features of the product, wherein the concept entity includes at least one of product function, product performance, and product value;

[0021] The concept entity is determined as a corresponding product selling point.

[0022] Furthermore, the step of obtaining the product oral audio and product video according to the product selling point and the preset product video generation model includes:

[0023] Inputting historical product selling points as a model training set into a preset autoregressive generation model for model training, thereby obtaining a product video generation model after the model training and a product oral audio output by the product video generation model;

[0024] The historical product selling points and the corresponding historical product spoken audio are input as a model training set into a preset generative adversarial network model for model training, so as to obtain a product video generation model after the model training and a product video output by the product video generation model.

[0025] Further, the slicing operation of the product video according to each sentence in the product oral audio to obtain corresponding video slices includes:

[0026] Determine the corresponding slice label contained in the product video according to each sentence in the product oral audio;

[0027] The product video is sliced ​​according to the slice tags to obtain corresponding video slices.

[0028] In a second aspect, the present application provides a short video oral audio intelligent explanation device, comprising:

[0029] A product selling point determination module is used to receive multi-source heterogeneous product data sent by developers, perform data preprocessing operations on the multi-source heterogeneous product data, including at least one of text segmentation, image enhancement, and video frame segmentation, and extract the underlying product feature representations in the multi-source heterogeneous product data, unify the extracted underlying product feature representations into a multimodal feature set and input them into a set deep learning multi-layer perceptron to obtain corresponding high-order product feature representations, perform feature weighted fusion on each of the high-order product feature representations through an attention mechanism to obtain product multimodal features, perform semantic recognition on the product multimodal features through a preset knowledge graph inference algorithm, and determine the corresponding product selling points;

[0030] A spoken audio determination module, used to obtain a product spoken audio and a product video according to the product selling point and a preset product video generation model, wherein the sentences in the product spoken audio correspond to the product selling point, and the product video is sliced ​​according to each sentence in the product spoken audio to obtain corresponding video slices;

[0031] The video playback module is used to adjust the playback speed of each video slice according to the audio playback duration of the sentence corresponding to each video slice and the user group portrait sent by the developer by setting a user portrait personalization model, and add the product spoken audio as video background audio to the product video to obtain an adjusted product video.

[0032] In a third aspect, the present application provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the short video oral audio intelligent explanation method when executing the program.

[0033] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the short video oral audio intelligent explanation method.

[0034] In a fifth aspect, the present application provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the short video oral audio intelligent explanation method.

[0035] It can be seen from the above technical scheme that the present application provides a short video oral audio intelligent explanation method and device, which receives multi-source heterogeneous product data sent by developers, extracts the underlying product feature representation in the multi-source heterogeneous product data, inputs the extracted underlying product feature representation into a set deep learning multi-layer perceptron, obtains the corresponding high-order product feature representation, and performs feature weighted fusion on each high-order product feature representation to obtain product multimodal features; constructs a corresponding playback speed adjustment function according to the audio playback time of the sentence corresponding to each video slice, updates the playback speed adjustment function according to the user characteristics in the user group portrait sent by the developer, adjusts the playback speed of each video slice according to the playback speed adjustment function, and adds the product oral audio as the video background audio to the product video to obtain the adjusted product video, thereby effectively improving the quality of the video content and the dissemination effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0037] Figure 1 This is one of the flow charts of the short video oral audio intelligent explanation method in the embodiment of the present application;

[0038] Figure 2 This is the second flow chart of the short video oral audio intelligent explanation method in the embodiment of the present application;

[0039] Figure 3 This is the third flow chart of the short video oral audio intelligent explanation method in the embodiment of the present application;

[0040] Figure 4 This is a fourth flow chart of the short video oral audio intelligent explanation method in the embodiment of the present application;

[0041] Figure 5 This is the fifth flow chart of the short video oral audio intelligent explanation method in the embodiment of the present application;

[0042] Figure 6 This is the sixth flow chart of the short video oral audio intelligent explanation method in the embodiment of the present application;

[0043] Figure 7 This is the seventh flow chart of the short video oral audio intelligent explanation method in the embodiment of the present application;

[0044] Figure 8This is a structural diagram of a short video oral audio intelligent explanation device in an embodiment of the present application;

[0045] Fig. 9 It is a schematic diagram of the structure of an electronic device in an embodiment of the present application.

[0046] Reference numerals:

[0047] Electronic device 9600, central processing unit 9100, memory 9140, communication module 9110, input unit 9120, audio processor 9130, display 9160, power supply 9170, buffer memory 9141, application / function storage unit 9142, data storage unit 9143, driver program storage unit 9144, antenna 9111, speaker 9131, microphone 9132. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0049] The acquisition, storage, use, and processing of data in the technical solution of this application comply with the relevant provisions of national laws and regulations.

[0050] In view of the problems existing in the prior art, the present application provides a short video oral audio intelligent explanation method and device, which receives multi-source heterogeneous product data sent by developers, extracts the underlying product feature representation in the multi-source heterogeneous product data, inputs the extracted underlying product feature representation into a set deep learning multi-layer perceptron, obtains the corresponding high-order product feature representation, and performs feature weighted fusion on each high-order product feature representation to obtain product multimodal features; constructs a corresponding playback speed adjustment function according to the audio playback time of the sentence corresponding to each video slice, updates the playback speed adjustment function according to the user characteristics in the user group portrait sent by the developer, adjusts the playback speed of each video slice according to the playback speed adjustment function, and adds the product oral audio as the video background audio to the product video to obtain the adjusted product video, thereby effectively improving the quality of the video content.

[0051] In order to effectively improve the quality and dissemination effect of video content, this application provides an embodiment of a short video oral audio intelligent explanation method, see Figure 1 The short video oral audio intelligent explanation method specifically includes the following contents:

[0052] Step S101: receiving product multi-source heterogeneous data sent by the developer, extracting the underlying product feature representation in the product multi-source heterogeneous data, inputting the extracted underlying product feature representation into a set deep learning multi-layer perceptron to obtain the corresponding high-order product feature representation, and performing feature weighted fusion on each of the high-order product feature representations to obtain product multimodal features, performing semantic recognition on the product multimodal features through a preset knowledge graph reasoning algorithm, and determining the corresponding product selling points;

[0053] Optionally, in this embodiment, in step S101, the system first receives multi-source heterogeneous data sent by the developer. These data may include text, images, and videos, representing different dimensional features of the product. In order to effectively process these diverse data, the system first performs underlying feature extraction. For text data, the system uses natural language processing technology to extract keywords and sentence features. Image data obtains visual features through convolutional neural networks, while video data extracts dynamic information through frame decomposition.

[0054] Next, these low-level features are input into a deep learning multi-layer perceptron (MLP). The multi-layer perceptron can handle nonlinear relationships and map low-level features into high-level product feature representations through the operation of multiple layers of neurons. These high-level features are more abstract than single-dimensional features and can better capture the overall picture of the product.

[0055] After obtaining the high-level product feature representation, the system integrates it using feature weighted fusion technology. Weighted fusion highlights key features by assigning different weights. This process ensures the comprehensive display of multimodal features, making important information more prominent in the overall features.

[0056] The system then applies a preset knowledge graph reasoning algorithm for semantic recognition. The knowledge graph provides a semantic network that associates multimodal features with known concepts. This process can identify the unique selling points of a product, such as features, benefits, or market positioning.

[0057] The technical problem solved by this solution is how to efficiently integrate and understand multi-source heterogeneous data. Traditional methods find it difficult to process text, images, and videos at the same time, but through the combination of deep learning and knowledge graphs, the system can comprehensively analyze product features and improve recognition accuracy.

[0058] In terms of technical effects, this step significantly improves the depth and breadth of product feature recognition. The multi-layer perceptron ensures efficient data integration, while the knowledge graph enhances the intelligence of product selling point recognition. This not only helps product development and optimization, but also provides strong support for marketing.

[0059] For example, suppose a technology company launches a new smartwatch. The system receives the product manual (text), product photos (images), and usage videos (videos). Through the above steps, the system extracts and fuses the multimodal features of the watch, and ultimately identifies its selling points such as "health monitoring function" and "long battery life". This information can be used for advertising and customer communication to improve the market competitiveness of the product.

[0060] In summary, step S101 realizes intelligent identification and presentation of product selling points through feature extraction, fusion and semantic recognition of multi-source data. This solution not only improves the automation level of data processing, but also provides important support for the market competitiveness of products, and has wide practical application value.

[0061] Step S102: obtaining a product audio and a product video according to the product selling point and a preset product video generation model, wherein the sentences in the product audio correspond to the product selling point, and slicing the product video according to each sentence in the product audio to obtain corresponding video slices;

[0062] Optionally, in this embodiment, in step S102, the system generates product audio and product video through product selling points and a preset product video generation model. First, the system analyzes the product selling points and converts them into oral copy that is easy to understand and attract users. The sentence structure of these copies is closely related to the product selling points to ensure that each key point can be clearly communicated.

[0063] Next, the system generates spoken audio using text-to-audio conversion technology. This process may use natural language processing and speech synthesis technology to convert the copy into natural and fluent audio output. The generated audio needs to be consistent with the product's brand image and target audience, so adjustments may be made to the tone, speed, and sound quality.

[0064] Based on the generated spoken audio, the system slices the product video according to each sentence in the audio. This step involves breaking down the complete video material into multiple segments, each of which corresponds to a specific sentence in the spoken audio. In this way, the video can more effectively showcase the product selling points and enhance the audience's understanding and memory.

[0065] Slicing typically relies on video editing techniques to ensure that each video clip is precisely aligned in time with the corresponding audio sentence. This may require the use of automated tools to detect keywords in the audio and find corresponding visual content in the video to match.

[0066] The main technical problem solved by this solution is how to effectively integrate audio and video materials so that the final product video is both visually attractive and can clearly convey product information. Traditional video production often requires manual editing, while this automated generation method improves efficiency and reduces human errors.

[0067] In terms of technical effects, through the above steps, the system significantly improves the efficiency and quality of video content production. Automated audio generation and video slicing ensure accurate communication of product information, while also providing viewers with a more attractive viewing experience. This is of great significance in marketing and can effectively enhance users' willingness to buy and brand loyalty.

[0068] For example, a technology company wants to promote a new type of headphones. The system first generates spoken audio related to the selling points of the headphones, such as "noise reduction function" and "long battery life". Then, the product video is sliced ​​according to the audio content, so that each video can intuitively show the characteristics of the headphones, such as the scene of using them in a noisy environment. The final video not only shows the product functions, but also enhances the audience's understanding through audio.

[0069] In summary, step S102 realizes automated and efficient product video generation by converting product selling points into audio and combining it with video slicing technology.

[0070] Step S103: construct a corresponding playback speed adjustment function according to the audio playback duration of the sentence corresponding to each of the video slices, update the playback speed adjustment function according to the user characteristics in the user group portrait sent by the developer, adjust the playback speed of each of the video slices according to the playback speed adjustment function, and add the product spoken audio as video background audio to the product video to obtain an adjusted product video.

[0071] Optionally, in this embodiment, the first step is to adjust the playback speed of the video slice based on the audio playback duration. In the above steps, we have obtained a series of video slices that closely correspond to the product's spoken audio. In order to further optimize the viewing experience of the video content, we can automatically adjust the playback speed of each video slice according to the audio playback duration corresponding to it.

[0072] Specifically, we can design an audio duration analysis module to calculate the playback duration of each spoken audio sentence. Then, we build a playback speed adjustment strategy to determine the optimal playback speed by analyzing the audio duration. For sentences with longer audio, we can appropriately reduce the playback speed of its video slice to ensure that users can fully listen to the audio content; for sentences with shorter audio, we can appropriately increase the playback speed to avoid excessive blank time when watching.

[0073] This automatic playback speed adjustment based on audio duration can effectively alleviate the problem of video content being presented too fast or too slow, maximizing the user's viewing experience. At the same time, it also lays the foundation for subsequent personalized optimization and provides targeted video presentation for different user groups.

[0074] On the other hand, we also need to combine the user group portraits provided by developers to further optimize the personalized performance of video content. User portraits can describe the age, gender, interests and hobbies of the target audience, providing a basis for personalized adjustment of video content.

[0075] Specifically, we can design a personalized adjustment module based on user portraits. First, we convert the user group data provided by developers into structured user feature vectors, including multiple dimensions such as age, gender, and interests. Then, we build a user portrait-video feature association model and preset the optimal video playback parameters for different user groups.

[0076] In actual applications, we associate and match the video slice playback speed calculated above with the user portrait features. Based on the matching results, we dynamically adjust the playback speed of each video slice. For example, for young user groups, we can appropriately increase the video playback speed to cater to their short attention span; while for elderly users, we can reduce the playback speed to ensure that they can better understand and appreciate the video content.

[0077] Finally, we reassembled the adjusted video slices and added the product audio as background audio to obtain the personalized and optimized final product video. This personalized video adjustment method that combines audio duration and user portrait can effectively improve the viewing experience and dissemination effect of product marketing videos among different audiences.

[0078] In general, the two core technologies involved in step S103 are video slice playback speed adjustment based on audio duration and personalized video optimization combined with user portraits. The former can automatically optimize the presentation rhythm of video content, while the latter further enhances the appeal of the video among various user groups based on the characteristics of the target audience. The integration of these two technologies lays an important foundation for the personalized optimization of product marketing short videos, which can significantly improve the conversion effect of video content.

[0079] From the above description, it can be seen that the short video oral audio intelligent explanation method provided in the embodiment of the present application can extract the underlying product feature representation in the multi-source heterogeneous data of the product by receiving the multi-source heterogeneous data of the product sent by the developer, and input the extracted underlying product feature representation into the set deep learning multi-layer perceptron to obtain the corresponding high-order product feature representation, and perform feature weighted fusion on each high-order product feature representation to obtain the product multimodal feature; construct a corresponding playback speed adjustment function according to the audio playback time of the sentence corresponding to each video slice, update the playback speed adjustment function according to the user characteristics in the user group portrait sent by the developer, adjust the playback speed of each video slice according to the playback speed adjustment function, and add the product oral audio as the video background audio to the product video to obtain the adjusted product video, thereby effectively improving the quality of the video content.

[0080] In one embodiment of the short video oral audio intelligent explanation method of the present application, see Figure 2 , and can also include the following:

[0081] Step S201: performing a data preprocessing operation including at least one of text segmentation, image enhancement and video frame segmentation on the product multi-source heterogeneous data, and extracting underlying product feature representations from the product multi-source heterogeneous data;

[0082] Step S202: unify the extracted underlying product feature representations into a multimodal feature set and input them into a set deep learning multi-layer perceptron to obtain the corresponding high-order product feature representations, perform feature weighted fusion on each of the high-order product feature representations through an attention mechanism to obtain product multimodal features.

[0083] Optionally, in this embodiment, the first step is step S201, that is, preprocessing the multi-source heterogeneous product data and extracting the underlying product feature representation. This step aims to extract the underlying feature representation that can fully describe the product characteristics from multiple data sources such as product text, images, and videos.

[0084] Specifically, for product text data, we can use text segmentation technology to segment the original text content into independent vocabulary units. This can remove some stop words and punctuation marks and highlight the keyword information in the text. Then, we can apply word embedding algorithms, such as Word2Vec or GloVe, to convert these segmented words into dense numerical vector representations. This text feature extraction method based on word embedding can capture the semantic associations between words and lay the foundation for subsequent feature fusion.

[0085] For product image data, we can use image enhancement techniques, such as flipping, scaling, and noise addition, to preprocess the original image. This data enhancement method can improve the diversity and robustness of the image and enhance the effect of subsequent visual feature extraction. Then, we can apply pre-trained convolutional neural networks (CNNs), such as VGG, ResNet, etc., to extract features from the processed image and obtain image feature vectors rich in visual semantic information.

[0086] For product video data, we can use frame operations to convert the video into a series of continuous image frames. In this way, we can draw on the aforementioned image feature extraction technology to extract visual features for each video frame, and combine the frame sequence information to extract video representations containing temporal dynamic features.

[0087] Through the above preprocessing and feature extraction methods for different data sources, we can obtain the underlying feature representations of product text, images, and videos, respectively, laying the foundation for subsequent multimodal fusion. This parallel processing of multi-source data can comprehensively capture the multi-faceted features of the product and overcome the limitations of a single data source.

[0088] Next is step S202, which is to perform multimodal fusion on the extracted underlying features through deep learning to obtain a high-order multimodal feature representation of the product.

[0089] This step aims to unify the various underlying features (text vectors, visual features, spatiotemporal features, etc.) extracted above into a product feature representation rich in multi-dimensional semantic information. To this end, we can design a deep learning fusion network based on a multi-layer perceptron (MLP) or Transformer.

[0090] First, we perform weighted fusion of different types of underlying features (text vectors, visual features, spatiotemporal features, etc.) to construct a unified multimodal feature set. This weighted fusion can dynamically adjust the contribution of each feature in the final feature vector according to its importance.

[0091] Then, we input this multimodal feature set into a preset deep learning network, allowing the network to automatically learn the complex interactive relationships between the various features. Through multiple layers of feature transformation and combination, the network can extract high-level product feature representations that contain rich semantic information.

[0092] In this process, we can also introduce the attention mechanism to allow the network to adaptively assign relevance weights to different features. This can further enhance the expressiveness of key features and improve the quality and pertinence of product multimodal features.

[0093] Compared with traditional manual feature engineering, the deep learning technology used in these two steps has several key advantages:

[0094] Automatic feature extraction. Without relying on the experience and knowledge of domain experts, the network can automatically learn to extract high-level features from raw data.

[0095] Cross-modal association learning. It can model the complex interaction between different types of data sources and extract feature representations containing multi-dimensional semantic information.

[0096] End-to-end optimization. Integrate feature extraction and fusion into a unified deep learning framework to achieve end-to-end automated learning.

[0097] In general, these two steps construct a product multimodal feature representation learning framework based on deep learning. It can automatically extract underlying features from multi-source heterogeneous product data and generate high-order product feature vectors through multimodal fusion. This feature representation is not only rich in multidimensional semantic information of the product, but also provides high-quality input for subsequent semantic analysis and video generation.

[0098] In one embodiment of the short video oral audio intelligent explanation method of the present application, see Figure 3 , and can also include the following:

[0099] Step S301: converting text data in the product multi-source heterogeneous data into mosquito vector representation through a word embedding algorithm, extracting visual features of image data in the product multi-source heterogeneous data through a preset convolutional neural network, extracting spatiotemporal features of video data in the product multi-source heterogeneous data through a frame division operation, and obtaining underlying product feature representation according to the text vector representation and / or visual features and / or spatiotemporal features;

[0100] Step S302: unify the underlying product feature representation into a multimodal feature set through weighted fusion and input it into a set deep learning multi-layer perceptron to perform complex interactive relationship learning to obtain the corresponding high-order product feature representation.

[0101] Optionally, in this embodiment, the first step is step S301, that is, extracting the underlying feature representation from the multi-source heterogeneous product data. This step aims to extract the underlying feature representation that can fully describe the product characteristics from multiple data sources such as product text, images, and videos.

[0102] Specifically, for product text data, we can use word embedding technology to convert text into numerical vector representation. Commonly used word embedding algorithms include Word2Vec, GloVe, etc., which can map text words into dense semantic vector space and capture the implicit relationship between words. In this way, we can convert product text data into computable text vector features.

[0103] For product image data, we can use a pre-trained convolutional neural network (CNN) model to extract visual features. Convolutional neural networks are good at discovering low-level visual features in images, such as lines, textures, etc., and combining them layer by layer into higher-level semantic features. By fine-tuning the pre-trained CNN model, we can obtain visual feature representations specifically for product images.

[0104] For product video data, we can convert the video into a series of image frames through frame segmentation, and then extract the spatiotemporal features of each frame. These spatiotemporal features can include motion information, scene changes, etc., which can reflect the presentation of the product in the video.

[0105] Through the above feature extraction methods, we can obtain the underlying feature representations of product text, image, and video data, including text vectors, visual features, and spatiotemporal features. These underlying features lay the foundation for subsequent multimodal fusion.

[0106] Next is step S302, which is to perform multimodal fusion on the extracted underlying features through deep learning to obtain a high-order multimodal feature representation of the product.

[0107] This step aims to automatically learn and fuse the various underlying features extracted above into a high-level multimodal feature representation of the product. Specifically, we can design a deep learning fusion network based on a multi-layer perceptron (MLP) or Transformer.

[0108] First, we perform weighted fusion of different types of underlying features (text vectors, visual features, spatiotemporal features, etc.) to construct a unified multimodal feature set. This weighted fusion can dynamically adjust the contribution of each feature in the final feature vector according to its importance.

[0109] Then, we input this multimodal feature set into a preset deep learning network, allowing the network to automatically learn the complex interactive relationships between the various features. Through multiple layers of feature transformation and combination, the network can extract high-level product feature representations that contain rich semantic information.

[0110] Compared with traditional manual feature engineering, the deep learning technology used in these two steps has several key advantages:

[0111] Automatic feature extraction. Without relying on the experience and knowledge of domain experts, the network can automatically learn to extract high-level features from raw data.

[0112] Cross-modal association learning. It can model the complex interaction between different types of data sources and extract feature representations containing multi-dimensional semantic information.

[0113] End-to-end optimization. Integrate feature extraction and fusion into a unified deep learning framework to achieve end-to-end automated learning.

[0114] In general, these two steps construct a product multimodal feature representation learning framework based on deep learning. It can automatically extract underlying features from multi-source heterogeneous product data and generate high-order product feature vectors through multimodal fusion. This feature representation is not only rich in multidimensional semantic information of the product, but also provides high-quality input for subsequent semantic analysis and video generation.

[0115] In one embodiment of the short video oral audio intelligent explanation method of the present application, see Figure 4 , and can also include the following:

[0116] Step S401: Calculate the correlation between each high-order product feature representation and a feature set consisting of all high-order product feature representations through a self-attention mechanism to obtain an attention weight of each high-order product feature representation;

[0117] Step S402: performing feature weighted fusion on each of the high-order product feature representations using the attention weights to obtain product multimodal features.

[0118] Optionally, in this embodiment, the first step is step S401, that is, the correlation between each high-order product feature and the entire feature set is calculated through the self-attention mechanism to obtain the attention weight of each feature. This step aims to adaptively highlight those components that are more important to the shaping of product features.

[0119] Specifically, we can use the self-attention mechanism to achieve this goal. The self-attention mechanism is a core component of deep learning models such as Transformer, which can capture the interdependencies between elements in the input sequence. In our scenario, we take the above-generated set of high-order product feature representations as input, and calculate the correlation between each feature vector and the entire feature set through self-attention.

[0120] The calculation process of self-attention is as follows: First, we transform the high-order product feature representation X through three linear transformations to obtain the Query (Q), Key (K), and Value (V) matrices. Then, we calculate the dot product of Query and Key and divide it by the scaling factor to obtain the attention weight matrix. Finally, we multiply the attention weight matrix with the Value matrix to obtain the final self-attention output, that is, the attention weight of each high-order feature.

[0121] This self-attention-based feature importance modeling can adaptively enhance those components that are more important to product feature shaping and improve the expressiveness of key features. Compared with simple weighted average or concatenation, the self-attention mechanism can capture the interaction between features more finely, thereby generating more representative feature representations.

[0122] Next is step S402, which is to perform weighted fusion on the high-order product features through the obtained attention weights to obtain the final product multimodal features.

[0123] Specifically, we perform weighted summation of the attention weights calculated in step S401 and the original high-order product feature vector. Such a weighted fusion process can adaptively enhance those feature components that are more important for shaping product features and generate more representative product multimodal features.

[0124] In order to further enhance the expressive power of multimodal features, we can also introduce some technical means, such as residual connection or gating mechanism.

[0125] Residual connections can effectively alleviate the gradient vanishing problem of deep networks and improve the quality of feature fusion. By adding the original features to the transformed features, residual connections can better preserve the original useful information.

[0126] The gating mechanism can dynamically adjust the contribution of each modal feature in the final fusion vector. We can design a gating-based fusion unit to dynamically adjust the fusion weights of each modal feature according to its importance, thereby obtaining a more optimized multimodal feature representation of the product.

[0127] In general, these two steps realize an adaptive fusion method of product multimodal features based on the attention mechanism. First, the importance weights of each high-order feature are calculated through self-attention; then, these weights are used to perform weighted fusion on the features to generate the final product multimodal features. This feature fusion mechanism can effectively give play to the advantages of each feature source, enhance the expressiveness of key features, and provide high-quality input for subsequent semantic analysis and video generation.

[0128] Compared with simple feature concatenation or average fusion, the attention mechanism can adaptively highlight those components that are more critical to the description of product features, greatly improving the quality and pertinence of feature representation. At the same time, the introduction of technical means such as residual connection and gating mechanism also further enhances the expression ability of product multimodal features. In general, these two steps reflect the innovation and technical advantages of this invention patent in feature fusion.

[0129] In one embodiment of the short video oral audio intelligent explanation method of the present application, see Figure 5 , and can also include the following:

[0130] Step S501: performing semantic matching and association reasoning on the multimodal features of the product through a preset knowledge graph reasoning algorithm to obtain the most relevant concept entity to the multimodal features of the product in the preset knowledge graph, wherein the concept entity includes at least one of product function, product performance and product value;

[0131] Step S502: Determine the concept entity as the corresponding product selling point.

[0132] Optionally, in this embodiment, the first step is step S401, that is, semantic matching and association reasoning are performed on the multimodal features of the product through a preset knowledge graph reasoning algorithm. This step aims to mine the most relevant conceptual entities with the multimodal features of the product from the existing knowledge graph.

[0133] Specifically, we first need to build a preset knowledge graph covering product knowledge, industry background, marketing strategy and other fields. The knowledge graph organizes various related knowledge in the form of entity-relationship and establishes rich semantic connections. The knowledge graph can be automatically constructed through knowledge extraction, ontology learning, etc., or manually modeled by domain experts.

[0134] Then, we inject the product multimodal features generated above as input information into the knowledge graph reasoning module. Through semantic matching and association reasoning, we can retrieve the concept entities most relevant to the product multimodal features from the knowledge graph. These highly relevant concept entities often represent the core selling points of the product, such as function, performance, value, etc.

[0135] The specific algorithms of knowledge graph reasoning can include rule-based reasoning and semantic-based reasoning. Rule-based reasoning mainly relies on preset ontology rules and mines implicit knowledge through logical deduction. Semantic reasoning focuses more on the similarity and correlation between concepts, and uses indicators such as semantic distance to identify semantically related entities.

[0136] This semantic analysis method based on knowledge graph can effectively make up for the limitations of single natural language processing and improve the depth and comprehensiveness of understanding of product connotations. Compared with semantic recognition that only relies on text features, knowledge graph reasoning can introduce rich domain knowledge and more accurately extract the core selling features of products.

[0137] Next is step S402, which is to determine the conceptual entity obtained through knowledge graph reasoning as the corresponding product selling point.

[0138] This step is very simple and direct. We only need to directly identify the concept entities identified in step S401 as the core selling features of the product. These concept entities include the functional advantages, performance parameters, use value and other aspects of the product, which can comprehensively describe the main highlights of the product.

[0139] Identifying these concept entities as product selling points not only provides an important basis for subsequent video generation, but also lays the foundation for subsequent personalized adjustments. Because each video slice will correspond to a specific product selling point feature, it can more accurately meet the needs of different user groups.

[0140] In general, these two steps reflect the innovation of using knowledge graph technology to conduct in-depth semantic analysis and selling point extraction of product multimodal features. First, we built a preset knowledge graph covering product domain knowledge; then, through the knowledge graph reasoning algorithm, we mined the most relevant conceptual entities to the product multimodal features and determined them as the core selling points of the product.

[0141] This knowledge-driven semantic analysis method can make up for the limitations of single natural language processing and enhance the depth of understanding of product connotations. Compared with methods that rely solely on text features, knowledge graph reasoning can introduce rich background knowledge and more accurately extract the key selling features of products. This lays a solid technical foundation for subsequent video generation and personalized optimization.

[0142] In one embodiment of the short video oral audio intelligent explanation method of the present application, see Figure 6 , and can also include the following:

[0143] Step S601: inputting historical product selling points as a model training set into a preset autoregressive generation model for model training, thereby obtaining a product video generation model after the model training and a product oral audio output by the product video generation model;

[0144] Step S602: Input the historical product selling points and the corresponding historical product oral audio as a model training set into a preset generative adversarial network model for model training, and obtain a product video generation model after the model training and a product video output by the product video generation model.

[0145] Optionally, in this embodiment, the first step is step S601, that is, using the autoregressive generation model to generate product oral audio. The goal of this step is to automatically generate the corresponding oral audio content according to the aforementioned determined product selling point features.

[0146] To this end, we can design a Transformer-based autoregressive generative network. This generative language model is good at automatically inferring and generating coherent and fluent text based on context. We use historical product selling points as the training set input of the model to let the model learn the language style and expression skills of product descriptions.

[0147] During the training process, we can further introduce knowledge graph enhancement. That is, we can integrate the background knowledge of the product field, such as functional characteristics, performance parameters, etc., into the training of the language model. This can enhance the professionalism and pertinence of the generated audio, making it closer to the actual product situation.

[0148] After the training is completed, we only need to input the aforementioned product selling point features into the trained generative language model to automatically generate contagious product audio. These audios can not only fully reflect the core highlights of the product, but also follow the voice style of high-quality marketing content, providing high-quality audio materials for subsequent video generation.

[0149] Next is step S602, which is to generate a product promotion video using a generative adversarial network (GAN). The goal of this step is to automatically generate a product promotion video that meets user needs based on the product selling points and spoken audio.

[0150] We can design a video generation model based on the GAN architecture. The generator is responsible for generating a preliminary product promotion video based on the product selling point features and spoken audio, while the discriminator provides feedback optimization for the generator by evaluating the authenticity, attractiveness and other indicators of the video.

[0151] Through the adversarial training mechanism of GAN, the generator can continuously optimize the video content and generate product promotion videos that are more in line with user needs. During the training process, we can also introduce historical high-quality product audio and corresponding videos as supervision signals to further guide the learning of the generator.

[0152] After the training is completed, we can input the product selling point features and spoken audio determined above into the trained video generation model to automatically generate appealing product promotion videos. These videos can not only fully display the core highlights of the product, but also follow the visual style and content rhythm preferred by users.

[0153] In general, these two steps reflect the use of generative modeling technology to achieve automatic creation of product marketing content. Specifically include:

[0154] Product audio generation based on the autoregressive generative model can automatically generate appealing product description content.

[0155] Product video generation based on generative adversarial networks can automatically generate product promotion videos that fit user aesthetics and needs through adversarial training mechanisms.

[0156] This generative modeling method has greatly improved the ability to automatically create product marketing content, saving companies a lot of labor costs. At the same time, it can also ensure that the final output of audio and video content is more in line with user preferences and marketing demands, improving marketing effectiveness.

[0157] In general, these two steps build an automatic creation framework for product marketing content based on generative modeling, which fully reflects the innovation and practicality of this invention patent in intelligent marketing creation.

[0158] In one embodiment of the short video oral audio intelligent explanation method of the present application, see Figure 7 , and can also include the following:

[0159] Step S701: determining the corresponding slice label contained in the product video according to each sentence in the product oral audio;

[0160] Step S702: Slice the product video according to the slice tags to obtain corresponding video slices.

[0161] Optionally, in this embodiment, the first step is step S701, that is, according to each sentence in the product broadcast audio, the slice label contained in the corresponding product video is determined. This step is intended to establish an accurate mapping relationship between text content and video clips.

[0162] Specifically, we can adopt a multimodal association learning method based on the attention mechanism. First, we obtain the aforementioned generated product audio and the corresponding product promotion video. Then, we design an attention-based text-video alignment network module.

[0163] The input of this module includes audio sequences (product audio) and video frame sequences. Through the attention mechanism, the network can adaptively assign relevance weights to various elements of audio and video and establish a fine audio-video alignment relationship.

[0164] Finally, we can accurately match each sentence in the spoken audio with the corresponding segment in the video and determine the label information of each video slice. This attention-based text-video alignment method can effectively capture the fine correlation between the two modal data and lay the foundation for subsequent video slicing operations.

[0165] Next is step S702, which is to slice the product video according to the slice tags determined above to obtain corresponding video slices.

[0166] This step is very straightforward. We only need to apply each video slice label determined in step S701 to the original product promotion video and accurately cut it into a series of short video slices.

[0167] The benefit of this is that each video slice corresponds to a specific sentence in the product's spoken audio, reflecting the structured expression of the video content. This not only helps with subsequent personalized video adjustments, but also provides an important basis for the intelligent generation and optimization of video content.

[0168] For example, in personalized optimization, we can adjust the playback speed of different video slices according to the characteristics of target users. Or, when generating new product videos, we can organize and edit these video slices in a targeted manner according to the product selling points to meet the marketing needs of different scenarios.

[0169] In general, these two steps reflect the technical innovation of this invention patent in accurately aligning the product audio and product video. First, we use the attention mechanism to build a multimodal learning model of audio-video association and achieve a fine mapping between the two types of data. Then, based on this alignment relationship, we perform a structured slicing operation on the original product video.

[0170] This text-video structured expression method not only helps to improve the comprehensibility and editability of video content, but also creates a good technical foundation for subsequent personalized optimization and intelligent generation. Compared with the traditional single video presentation method, it can better meet the needs of different usage scenarios and user groups, and enhance the flexibility and pertinence of product marketing videos.

[0171] In general, these two steps have made important contributions to the structured expression of product marketing videos, further demonstrating the technical innovation and practicality of this invention patent.

[0172] In order to effectively improve the quality and dissemination effect of video content, the present application provides an embodiment of a short video oral audio intelligent explanation device for realizing all or part of the content of the short video oral audio intelligent explanation method, see Figure 8 The short video oral audio intelligent explanation device specifically includes the following contents:

[0173] The product selling point determination module 10 is used to receive the product multi-source heterogeneous data sent by the developer, perform data preprocessing operations on the product multi-source heterogeneous data, including at least one of text segmentation, image enhancement and video frame segmentation, and extract the underlying product feature representation in the product multi-source heterogeneous data, unify the extracted underlying product feature representation into a multimodal feature set and input it into a set deep learning multi-layer perceptron to obtain the corresponding high-order product feature representation, perform feature weighted fusion on each of the high-order product feature representations through an attention mechanism to obtain product multimodal features, perform semantic recognition on the product multimodal features through a preset knowledge graph inference algorithm, and determine the corresponding product selling point;

[0174] The oral audio determination module 20 is used to obtain the product oral audio and product video according to the product selling point and the preset product video generation model, wherein the sentences in the product oral audio correspond to the product selling point, and the product video is sliced ​​according to each sentence in the product oral audio to obtain corresponding video slices;

[0175] The video playback module 30 is used to adjust the playback speed of each video slice according to the audio playback duration of the sentence corresponding to each video slice and the user group portrait sent by the developer by setting a user portrait personalization model, and add the product spoken audio as video background audio to the product video to obtain an adjusted product video.

[0176] From the above description, it can be seen that the short video oral audio intelligent explanation device provided in the embodiment of the present application can extract the underlying product feature representation in the multi-source heterogeneous data of the product by receiving the multi-source heterogeneous data of the product sent by the developer, and input the extracted underlying product feature representation into the set deep learning multi-layer perceptron to obtain the corresponding high-order product feature representation, and perform feature weighted fusion on each high-order product feature representation to obtain the product multimodal feature; construct a corresponding playback speed adjustment function according to the audio playback time of the sentence corresponding to each video slice, update the playback speed adjustment function according to the user characteristics in the user group portrait sent by the developer, adjust the playback speed of each video slice according to the playback speed adjustment function, and add the product oral audio as the video background audio to the product video to obtain the adjusted product video, thereby effectively improving the quality of the video content.

[0177] From the hardware level, in order to effectively improve the quality and dissemination effect of video content, the present application provides an embodiment of an electronic device for implementing all or part of the content of the short video oral audio intelligent explanation method, and the electronic device specifically includes the following content:

[0178] Processor, memory, communication interface and bus; wherein the processor, memory and communication interface communicate with each other through the bus; the communication interface is used to realize information transmission between the short video oral audio intelligent explanation device and the core business system, user terminal and related databases and other related equipment; the logic controller can be a desktop computer, a tablet computer and a mobile terminal, etc., and the present embodiment is not limited thereto. In the present embodiment, the logic controller can be implemented with reference to the embodiment of the short video oral audio intelligent explanation method and the embodiment of the short video oral audio intelligent explanation device in the embodiment, and the contents thereof are incorporated herein, and the repeated parts are not repeated.

[0179] It is understandable that the user terminal may include a smart phone, a tablet electronic device, a network set-top box, a portable computer, a desktop computer, a personal digital assistant (PDA), a vehicle-mounted device, a smart wearable device, etc. Among them, the smart wearable device may include smart glasses, a smart watch, a smart bracelet, etc.

[0180] In practical applications, part of the short video oral audio intelligent explanation method can be executed on the electronic device side as described above, or all operations can be completed in the client device. The specific selection can be based on the processing capability of the client device and the limitations of the user's usage scenario. This application does not limit this. If all operations are completed in the client device, the client device may also include a processor.

[0181] The client device may have a communication module (i.e., a communication unit) that can communicate with a remote server to achieve data transmission with the server. The server may include a server on the task scheduling center side, and other implementation scenarios may also include a server on an intermediate platform, such as a server on a third-party server platform that has a communication link with the task scheduling center server. The server may include a single computer device, or a server cluster consisting of multiple servers, or a server structure of a distributed device.

[0182] Fig. 9 FIG. 9 is a schematic block diagram of the system structure of the electronic device 9600 according to an embodiment of the present application. Fig. 9As shown, the electronic device 9600 may include a central processor 9100 and a memory 9140; the memory 9140 is coupled to the central processor 9100. It is worth noting that Fig. 9 is exemplary; other types of structures may also be used to supplement or replace this structure to implement telecommunication functions or other functions.

[0183] In one embodiment, the function of the short video oral audio intelligent explanation method can be integrated into the central processing unit 9100. The central processing unit 9100 can be configured to perform the following control:

[0184] Step S101: receiving product multi-source heterogeneous data sent by the developer, extracting the underlying product feature representation in the product multi-source heterogeneous data, inputting the extracted underlying product feature representation into a set deep learning multi-layer perceptron to obtain the corresponding high-order product feature representation, and performing feature weighted fusion on each of the high-order product feature representations to obtain product multimodal features, performing semantic recognition on the product multimodal features through a preset knowledge graph reasoning algorithm, and determining the corresponding product selling points;

[0185] Step S102: obtaining a product audio and a product video according to the product selling point and a preset product video generation model, wherein the sentences in the product audio correspond to the product selling point, and slicing the product video according to each sentence in the product audio to obtain corresponding video slices;

[0186] Step S103: construct a corresponding playback speed adjustment function according to the audio playback duration of the sentence corresponding to each of the video slices, update the playback speed adjustment function according to the user characteristics in the user group portrait sent by the developer, adjust the playback speed of each of the video slices according to the playback speed adjustment function, and add the product spoken audio as video background audio to the product video to obtain an adjusted product video.

[0187] From the above description, it can be seen that the electronic device provided in the embodiment of the present application receives multi-source heterogeneous product data sent by the developer, extracts the underlying product feature representation in the multi-source heterogeneous product data, inputs the extracted underlying product feature representation into the set deep learning multi-layer perceptron, obtains the corresponding high-order product feature representation, and performs feature weighted fusion on each high-order product feature representation to obtain the product multimodal feature; constructs a corresponding playback speed adjustment function according to the audio playback time of the sentence corresponding to each video slice, updates the playback speed adjustment function according to the user characteristics in the user group portrait sent by the developer, adjusts the playback speed of each video slice according to the playback speed adjustment function, and adds the product oral audio as the video background audio to the product video to obtain the adjusted product video, thereby effectively improving the quality of the video content.

[0188] In another embodiment, the short video oral audio intelligent explanation device can be configured separately from the central processing unit 9100. For example, the short video oral audio intelligent explanation device can be configured as a chip connected to the central processing unit 9100, and the function of the short video oral audio intelligent explanation method can be realized through the control of the central processing unit.

[0189] like Fig. 9 As shown, the electronic device 9600 may also include: a communication module 9110, an input unit 9120, an audio processor 9130, a display 9160, and a power supply 9170. It is worth noting that the electronic device 9600 does not necessarily have to include Fig. 9 In addition, the electronic device 9600 may also include Fig. 9 For components not shown, reference may be made to the prior art.

[0190] like Fig. 9 As shown, the central processing unit 9100 is sometimes also referred to as a controller or an operation control, and may include a microprocessor or other processor device and / or logic device. The central processing unit 9100 receives input and controls the operation of various components of the electronic device 9600.

[0191] The memory 9140 may be, for example, one or more of a cache, a flash memory, a hard drive, a removable medium, a volatile memory, a non-volatile memory or other suitable devices. The above-mentioned information related to the failure may be stored, and a program for executing the relevant information may also be stored. The CPU 9100 may execute the program stored in the memory 9140 to implement information storage or processing, etc.

[0192] The input unit 9120 provides input to the central processing unit 9100. The input unit 9120 is, for example, a key or a touch input device. The power supply 9170 is used to provide power to the electronic device 9600. The display 9160 is used to display display objects such as images and texts. The display may be, for example, an LCD display, but is not limited thereto.

[0193] The memory 9140 may be a solid-state memory, such as a read-only memory (ROM), a random access memory (RAM), a SIM card, etc. It may also be a memory that saves information even when the power is off, can be selectively erased, and is provided with more data, examples of which are sometimes referred to as EPROMs, etc. The memory 9140 may also be some other type of device. The memory 9140 includes a buffer memory 9141 (sometimes referred to as a buffer). The memory 9140 may include an application / function storage unit 9142, which is used to store application programs and function programs or processes for executing the operation of the electronic device 9600 through the central processor 9100.

[0194] The memory 9140 may also include a data storage unit 9143 for storing data, such as contacts, digital data, pictures, sounds, and / or any other data used by the electronic device. The driver storage unit 9144 of the memory 9140 may include various drivers for communication functions of the electronic device and / or for executing other functions of the electronic device (such as messaging applications, address book applications, etc.).

[0195] The communication module 9110 is a transmitter / receiver that sends and receives signals via the antenna 9111. The communication module 9110 (transmitter / receiver) is coupled to the central processor 9100 to provide input signals and receive output signals, which may be the same as the case of a conventional mobile communication terminal.

[0196] Based on different communication technologies, multiple communication modules 9110 may be provided in the same electronic device, such as a cellular network module, a Bluetooth module and / or a wireless LAN module, etc. The communication module 9110 (transmitter / receiver) is also coupled to a speaker 9131 and a microphone 9132 via an audio processor 9130 to provide an audio output via the speaker 9131 and receive an audio input from the microphone 9132, thereby realizing a common telecommunication function. The audio processor 9130 may include any suitable buffer, decoder, amplifier, etc. In addition, the audio processor 9130 is also coupled to the central processor 9100, so that recording can be performed on the local machine through the microphone 9132, and the sound stored on the local machine can be played through the speaker 9131.

[0197] The embodiments of the present application also provide a computer-readable storage medium capable of implementing all the steps of the short video oral audio intelligent explanation method in the above-mentioned embodiment, where the execution subject is a server or a client. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, all the steps of the short video oral audio intelligent explanation method in the above-mentioned embodiment are implemented. For example, when the processor executes the computer program, the following steps are implemented:

[0198] Step S101: receiving product multi-source heterogeneous data sent by the developer, extracting the underlying product feature representation in the product multi-source heterogeneous data, inputting the extracted underlying product feature representation into a set deep learning multi-layer perceptron to obtain the corresponding high-order product feature representation, and performing feature weighted fusion on each of the high-order product feature representations to obtain product multimodal features, performing semantic recognition on the product multimodal features through a preset knowledge graph reasoning algorithm, and determining the corresponding product selling points;

[0199] Step S102: obtaining a product audio and a product video according to the product selling point and a preset product video generation model, wherein the sentences in the product audio correspond to the product selling point, and slicing the product video according to each sentence in the product audio to obtain corresponding video slices;

[0200] Step S103: construct a corresponding playback speed adjustment function according to the audio playback duration of the sentence corresponding to each of the video slices, update the playback speed adjustment function according to the user characteristics in the user group portrait sent by the developer, adjust the playback speed of each of the video slices according to the playback speed adjustment function, and add the product spoken audio as video background audio to the product video to obtain an adjusted product video.

[0201] From the above description, it can be seen that the computer-readable storage medium provided in the embodiment of the present application receives multi-source heterogeneous product data sent by developers, extracts the underlying product feature representation in the multi-source heterogeneous product data, inputs the extracted underlying product feature representation into the set deep learning multi-layer perceptron, obtains the corresponding high-order product feature representation, and performs feature weighted fusion on each high-order product feature representation to obtain product multimodal features; constructs a corresponding playback speed adjustment function according to the audio playback time of the sentence corresponding to each video slice, updates the playback speed adjustment function according to the user characteristics in the user group portrait sent by the developer, adjusts the playback speed of each video slice according to the playback speed adjustment function, and adds the product oral audio as the video background audio to the product video to obtain the adjusted product video, thereby effectively improving the quality of the video content.

[0202] The embodiments of the present application also provide a computer program product capable of implementing all the steps of the short video oral audio intelligent explanation method in the above embodiments, where the execution subject is a server or a client. When the computer program / instruction is executed by a processor, the steps of the short video oral audio intelligent explanation method are implemented. For example, the computer program / instruction implements the following steps:

[0203] Step S101: receiving product multi-source heterogeneous data sent by the developer, extracting the underlying product feature representation in the product multi-source heterogeneous data, inputting the extracted underlying product feature representation into a set deep learning multi-layer perceptron to obtain the corresponding high-order product feature representation, and performing feature weighted fusion on each of the high-order product feature representations to obtain product multimodal features, performing semantic recognition on the product multimodal features through a preset knowledge graph reasoning algorithm, and determining the corresponding product selling points;

[0204] Step S102: obtaining a product audio and a product video according to the product selling point and a preset product video generation model, wherein the sentences in the product audio correspond to the product selling point, and slicing the product video according to each sentence in the product audio to obtain corresponding video slices;

[0205] Step S103: construct a corresponding playback speed adjustment function according to the audio playback duration of the sentence corresponding to each of the video slices, update the playback speed adjustment function according to the user characteristics in the user group portrait sent by the developer, adjust the playback speed of each of the video slices according to the playback speed adjustment function, and add the product spoken audio as video background audio to the product video to obtain an adjusted product video.

[0206] From the above description, it can be seen that the computer program product provided in the embodiment of the present application receives multi-source heterogeneous data of products sent by developers, extracts the underlying product feature representation in the multi-source heterogeneous data of products, inputs the extracted underlying product feature representation into the set deep learning multi-layer perceptron, obtains the corresponding high-order product feature representation, and performs feature weighted fusion on each high-order product feature representation to obtain product multimodal features; constructs a corresponding playback speed adjustment function according to the audio playback time of the sentence corresponding to each video slice, updates the playback speed adjustment function according to the user characteristics in the user group portrait sent by the developer, adjusts the playback speed of each video slice according to the playback speed adjustment function, and adds the product oral audio as the video background audio to the product video to obtain the adjusted product video, thereby effectively improving the quality of the video content.

[0207] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0208] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (apparatus), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0209] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0210] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0211] The present invention uses specific embodiments to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A short video oral audio intelligent explanation method, characterized in that: The method comprises: Receive product multi-source heterogeneous data sent by developers, extract the underlying product feature representations from the product multi-source heterogeneous data, input the extracted underlying product feature representations into a set deep learning multi-layer perceptron to obtain corresponding high-order product feature representations, perform feature weighted fusion on each of the high-order product feature representations to obtain product multimodal features, perform semantic recognition on the product multimodal features through a preset knowledge graph reasoning algorithm, and determine the corresponding product selling points; According to the product selling point and the preset product video generation model, a product oral audio and a product video are obtained, wherein the sentences in the product oral audio correspond to the product selling point, and the product video is sliced ​​according to each sentence in the product oral audio to obtain corresponding video slices; A corresponding playback speed adjustment function is constructed according to the audio playback duration of the sentence corresponding to each of the video slices, and the playback speed adjustment function is updated according to the user characteristics in the user group portrait sent by the developer. The playback speed of each of the video slices is adjusted according to the playback speed adjustment function, and the product spoken audio is added to the product video as video background audio to obtain an adjusted product video.

2. The short video oral audio intelligent explanation method according to claim 1 is characterized in that: The step of extracting the underlying product feature representation from the multi-source heterogeneous data of the product, inputting the extracted underlying product feature representation into a deep learning multi-layer perceptron to obtain the corresponding high-order product feature representation, and performing feature weighted fusion on each of the high-order product feature representations to obtain product multimodal features includes: Performing a data preprocessing operation including at least one of text segmentation, image enhancement, and video frame segmentation on the multi-source heterogeneous product data, and extracting underlying product feature representations from the multi-source heterogeneous product data; The extracted underlying product feature representations are unified into a multimodal feature set and input into a set deep learning multi-layer perceptron to obtain the corresponding high-order product feature representations. The high-order product feature representations are weighted and fused through an attention mechanism to obtain product multimodal features.

3. The short video oral audio intelligent explanation method according to claim 2 is characterized in that: The step of extracting the underlying product feature representation from the multi-source heterogeneous data of the product, unifying the extracted underlying product feature representation into a multimodal feature set and inputting the extracted feature representation into a set deep learning multi-layer perceptron to obtain the corresponding high-order product feature representation includes: The text data in the multi-source heterogeneous data of the product is converted into a mosquito vector representation by a word embedding algorithm, the visual features of the image data in the multi-source heterogeneous data of the product are extracted by a preset convolutional neural network, the spatiotemporal features of the video data in the multi-source heterogeneous data of the product are extracted by a frame division operation, and the underlying product feature representation is obtained according to the text vector representation and / or the visual features and / or the spatiotemporal features; The underlying product feature representations are unified into a multimodal feature set through weighted fusion and input into a set deep learning multi-layer perceptron for complex interactive relationship learning to obtain the corresponding high-order product feature representation.

4. The short video oral audio intelligent explanation method according to claim 2 is characterized in that: The weighted fusion of the high-level product feature representations is performed through the attention mechanism to obtain product multimodal features, including: Calculate the correlation between each of the high-order product feature representations and the feature set consisting of all high-order product feature representations through a self-attention mechanism to obtain an attention weight for each of the high-order product feature representations; The high-order product feature representations are weightedly fused using the attention weights to obtain product multimodal features.

5. The short video oral audio intelligent explanation method according to claim 1 is characterized in that: The method of performing semantic recognition on the multimodal features of the product by using a preset knowledge graph inference algorithm to determine the corresponding product selling points includes: Performing semantic matching and association reasoning on the multimodal features of the product through a preset knowledge graph reasoning algorithm to obtain a concept entity in the preset knowledge graph that is most relevant to the multimodal features of the product, wherein the concept entity includes at least one of product function, product performance, and product value; The concept entity is determined as a corresponding product selling point.

6. The short video oral audio intelligent explanation method according to claim 1 is characterized in that: The step of obtaining the product oral audio and product video according to the product selling point and the preset product video generation model includes: Inputting historical product selling points as a model training set into a preset autoregressive generation model for model training, thereby obtaining a product video generation model after the model training and a product oral audio output by the product video generation model; The historical product selling points and the corresponding historical product spoken audio are input as a model training set into a preset generative adversarial network model for model training, so as to obtain a product video generation model after the model training and a product video output by the product video generation model.

7. The short video oral audio intelligent explanation method according to claim 1 is characterized in that: The slicing operation of the product video according to each sentence in the product oral audio to obtain corresponding video slices includes: Determine the corresponding slice label contained in the product video according to each sentence in the product oral audio; The product video is sliced ​​according to the slice tags to obtain corresponding video slices.

8. A short video oral audio intelligent explanation device, characterized in that: The device comprises: A product selling point determination module is used to receive multi-source heterogeneous product data sent by developers, perform data preprocessing operations on the multi-source heterogeneous product data, including at least one of text segmentation, image enhancement, and video frame segmentation, and extract the underlying product feature representations in the multi-source heterogeneous product data, unify the extracted underlying product feature representations into a multimodal feature set and input them into a set deep learning multi-layer perceptron to obtain corresponding high-order product feature representations, perform feature weighted fusion on each of the high-order product feature representations through an attention mechanism to obtain product multimodal features, perform semantic recognition on the product multimodal features through a preset knowledge graph inference algorithm, and determine the corresponding product selling points; A spoken audio determination module, used to obtain a product spoken audio and a product video according to the product selling point and a preset product video generation model, wherein the sentences in the product spoken audio correspond to the product selling point, and the product video is sliced ​​according to each sentence in the product spoken audio to obtain corresponding video slices; The video playback module is used to adjust the playback speed of each video slice according to the audio playback duration of the sentence corresponding to each video slice and the user group portrait sent by the developer by setting a user portrait personalization model, and add the product spoken audio as video background audio to the product video to obtain an adjusted product video.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the short video oral audio intelligent explanation method described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the short video oral audio intelligent explanation method described in any one of claims 1 to 7 are implemented.

Images