APP auxiliary video generation method and device, equipment and storage medium

Abstract

The invention discloses an APP auxiliary video generation method and device, equipment and a storage medium, and the method comprises the steps: obtaining the multi-modal input information of a user, converting the multi-modal input information into a vector in a unified target format, facilitating the input of a video generation model, and carrying out the fusion, enabling the video generation model to achieve the video generation according to the multi-modal input information of the user, according to the method and the device, the technical problems that the video production cost is high and the user experience is poor due to the lack of the auxiliary video because the video shooting cost is high and the number of required videos is large for different application scenes and user requirements are solved.

Description

Technical Field

[0001] The present application relates to the field of app application technology, and in particular to an APP-assisted video generation method, device, equipment and storage medium. Background Art

[0002] Current apps often integrate many functions, with increasingly complex interfaces and more powerful functions. While they meet the needs of a large number of user groups, they greatly reduce the experience of individual users. In particular, for a single user, during the entire period of using the app, they often only use a small function, but the complex operation interface often discourages users and makes them unable to start, thus reducing the user experience, especially in insurance apps. With the rapid development of modern society, people's patience has also decreased. How to increase users' favorability when using the app in the first place is strongly related to the user retention rate, especially in modern society where the interface and product functions of insurance apps are becoming increasingly homogenized.

[0003] The current mainstream method is to provide a question-and-answer robot in the interface, which guides users to operate in the interface according to the needs of users' questions, and most of them are text-based operation methods. However, this method has a big disadvantage, that is, users need to actively read the text to operate. From a psychological point of view, human individuals like to passively accept knowledge, so it leaves a bad impression on users subconsciously. How to make users passively accept knowledge and thus greatly improve the user's operating comfort is an urgent problem that technicians in this field need to solve.

[0004] Since the cost of shooting videos is relatively high and a large number of videos are required for different application scenarios and user needs, how to provide users with auxiliary videos to guide user operations and improve user experience while reducing the cost of video production is an issue that technical personnel in this field urgently need to solve. Summary of the invention

[0005] The present application provides an APP-assisted video generation method, device, equipment and storage medium, which solves the technical problems that the video shooting cost is high and the number of videos required for different application scenarios and user needs is also large, resulting in high video production cost, while the lack of auxiliary videos leads to poor user experience.

[0006] In view of this, the first aspect of the present application provides an APP-assisted video generation method, the method comprising:

[0007] Step S101: obtaining user input information, wherein the input information includes text information, picture information and / or audio information;

[0008] Step S102: performing vector conversion processing on the input information, converting it into an embedding of a target format and performing weighted fusion to form a fusion vector;

[0009] Step S103: using the fusion vector as an input of a video generation model to generate an APP auxiliary video, wherein the video generation model is a transformer structure.

[0010] Optionally, the step S101 specifically includes:

[0011] Acquire demand information input by a user, wherein the demand information includes demand text information, demand audio information and / or demand audio information;

[0012] The demand information is parsed through the RAG system to generate corresponding application process text information, application process image information and / or application process audio information.

[0013] Optionally, the parsing the demand information through the RAG system to generate corresponding application process text information, application process image information and / or application process audio information specifically includes:

[0014] The demand information is parsed through the RAG system to determine the application process of the user and the application process image information involved in the application process;

[0015] Application process text information and application process audio information are generated according to the application process.

[0016] Optionally, the step S102 specifically includes:

[0017] If the input information is text information, convert the text information into a first embedding of a target format by using NPL technology;

[0018] If the input information is picture information, extracting picture features of the picture information by image recognition technology, and converting the picture features into a second embedding in a target format;

[0019] If the input information is audio information, the audio information is converted into text information by using speech recognition technology, and then the text information is converted into a third embedding of a target format by using NPL technology;

[0020] When the input information includes two or more different types of information, the first embedding, the second embedding and / or the third embedding are weighted fused to form a fusion vector.

[0021] Optionally, when the input information includes two or more different types of information, weighted fusion of the first embedding, the second embedding and / or the third embedding to form a fusion vector specifically includes:

[0022] When the input information includes two or more different types of information, the first embedding, the second embedding and / or the third embedding are weighted fused according to preset weights corresponding to the information types to form a fusion vector.

[0023] Optionally, it also includes:

[0024] Step S201: Obtain user information of a user, and generate a user profile of the user based on analysis of the user information;

[0025] Step S202: determining a number of recommendation solutions according to the user portrait;

[0026] Step S203: Generate text information containing the plurality of recommended solutions based on the large language model;

[0027] Step S204, performing vector conversion processing on the text information to convert it into a fourth embedding of a target format;

[0028] Step S205: Use the fourth embedding as an input of a video generation model to generate a user recommended video.

[0029] Optionally, the step S203 specifically includes:

[0030] The solution information of the plurality of recommended solutions is input into the large language model, and the large language model generates text information containing the plurality of recommended solutions by calling a preset highlight extraction template.

[0031] A second aspect of the present application provides an APP-assisted video generation device, the device comprising:

[0032] A first acquisition unit, used to acquire input information of a user, wherein the input information includes text information, picture information and / or audio information;

[0033] A first processing unit is used to perform vector conversion processing on the input information, convert it into an embedding of a target format and perform weighted fusion to form a fusion vector;

[0034] The first generating unit is used to use the fusion vector as an input of a video generating model to generate an APP auxiliary video, and the video generating model is a transformer structure.

[0035] Optionally, the first acquiring unit is specifically configured to:

[0036] Acquire demand information input by a user, wherein the demand information includes demand text information, demand audio information and / or demand audio information;

[0037] The demand information is parsed through the RAG system to generate corresponding application process text information, application process image information and / or application process audio information.

[0038] Optionally, the parsing the demand information through the RAG system to generate corresponding application process text information, application process image information and / or application process audio information specifically includes:

[0039] The demand information is parsed through the RAG system to determine the application process of the user and the application process image information involved in the application process;

[0040] Application process text information and application process audio information are generated according to the application process.

[0041] Optionally, the first processing unit is specifically configured to:

[0042] If the input information is text information, convert the text information into a first embedding of a target format by using NPL technology;

[0043] If the input information is picture information, extracting picture features of the picture information by image recognition technology, and converting the picture features into a second embedding in a target format;

[0044] If the input information is audio information, the audio information is converted into text information by using speech recognition technology, and then the text information is converted into a third embedding of a target format by using NPL technology;

[0045] When the input information includes two or more different types of information, the first embedding, the second embedding and / or the third embedding are weighted fused to form a fusion vector.

[0046] Optionally, when the input information includes two or more different types of information, weighted fusion of the first embedding, the second embedding and / or the third embedding to form a fusion vector specifically includes:

[0047] When the input information includes two or more different types of information, the first embedding, the second embedding and / or the third embedding are weighted fused according to preset weights corresponding to the information types to form a fusion vector.

[0048] Optionally, it also includes:

[0049] A second acquisition unit is used to acquire user information of a user, and generate a user portrait of the user according to the analysis of the user information;

[0050] A determination unit, configured to determine a number of recommendation solutions according to the user portrait;

[0051] A copywriting generation unit, used to generate copywriting information including the plurality of recommendation solutions based on the large language model;

[0052] A second processing unit, configured to perform vector conversion processing on the text information to convert it into a fourth embedding in a target format;

[0053] The second generating unit is used to use the fourth embedding as an input of the video generation model to generate a user recommended video.

[0054] Optionally, the second processing unit is specifically configured to:

[0055] The solution information of the plurality of recommended solutions is input into the large language model, and the large language model generates text information containing the plurality of recommended solutions by calling a preset highlight extraction template.

[0056] A third aspect of the present application provides an APP-assisted video generation device, the device comprising a processor and a memory:

[0057] The memory is used to store program code and transmit the program code to the processor;

[0058] The processor is used to execute the steps of the method for APP-assisted video generation as described in the first aspect above according to the instructions in the program code.

[0059] A fourth aspect of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store program code, and the program code is used to execute the method described in the first aspect above.

[0060] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:

[0061] In the present application, a method, apparatus, device and storage medium for generating APP-assisted videos are provided. By obtaining multimodal input information of the user, the multimodal input information is converted into vectors in a unified target format that is convenient for input into a video generation model and fused, so that the video generation model can generate corresponding APP-assisted videos that meet the conditions according to the multimodal input information of the user, thereby solving the technical problems that the cost of video shooting is high, a large number of videos are required for different application scenarios and user needs, resulting in high video production costs, and the lack of auxiliary videos leads to poor user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 This is a method flow chart of the APP-assisted video generation method in an embodiment of the present application;

[0063] Figure 2 This is a schematic diagram of the structure of an APP-assisted video generation device in an embodiment of the present application;

[0064] Figure 3 It is a structural schematic diagram of the APP-assisted video generation device in an embodiment of the present application. DETAILED DESCRIPTION

[0065] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0066] The present application designs an APP-assisted video generation method, device, equipment and storage medium, which solves the technical problems that the video shooting cost is high and a large number of videos are required for different application scenarios and user needs, resulting in high video production costs, and the lack of auxiliary videos leads to poor user experience.

[0067] For easier understanding, see Figure 1 , Figure 1 : is a method flow chart of the APP-assisted video generation method in the embodiment of the present application, such as Figure 1 As shown, specifically:

[0068] Step S101: obtaining user input information, where the input information includes text information, picture information and / or audio information;

[0069] It should be noted that obtaining user input information may include text information, image information and / or audio information. For example, the user inputs or types “I want to view the specific terms of XX insurance”, or imports “XX insurance promotional picture”, etc.

[0070] Step S102: Perform vector conversion processing on the input information, convert it into an embedding of a target format and perform weighted fusion to form a fusion vector;

[0071] It should be noted that since the formats of text information, image information and audio information are different, a unified representation space, namely a vector, is required. The input information is then vectorized and converted into an embedding of the target format.

[0072] After forming a unified vector, the input needs to be weightedly fused with ATTENTION to form a fusion vector, so as to proceed to the next step of model generation. The fusion vector includes the extraction of input information.

[0073] Step S103: Use the fusion vector as the input of the video generation model to generate an APP auxiliary video. The video generation model is a transformer structure.

[0074] It should be noted that by using the front-end Transformer module, the three different dimensions of input information are weighted and fused to form a single vector, which enters the video generation model to control the generation of APP-assisted videos. The video generation model is the current mainstream Transformer structure, and its advantage is that it can better utilize the input information.

[0075] Furthermore, step S101 specifically includes:

[0076] Acquire demand information input by a user, the demand information including demand text information, demand audio information and / or demand audio information;

[0077] The demand information is parsed through the RAG system to generate corresponding application process text information, application process image information and / or application process audio information.

[0078] It should be noted that the user's demand information is received, which may include text, picture or audio format.

[0079] Clean and standardize the demand information, such as text segmentation, image format conversion, audio transcription, etc.

[0080] The RAG system can be used to analyze demand information, including:

[0081] Text parsing: Use natural language processing technology to parse text information and identify keywords, intent, and demand details;

[0082] Image parsing: Using computer vision techniques to extract information from images and identify objects, scenes, and possible process instructions in the images;

[0083] Audio parsing: transcribe audio information into text and further parse it using natural language processing technology.

[0084] Based on the parsing results, relevant information and data are retrieved from the knowledge base to enhance the understanding of the requirements.

[0085] Integrate the retrieved information with the original input information to form a comprehensive understanding of the requirements.

[0086] Generate corresponding application process text information, application process image information and / or application process audio information, which can be:

[0087] Text generation: Use the text generation model to generate a text description of the application process based on the integrated information.

[0088] Image generation: Use image generation models to generate flowcharts or schematics based on text descriptions to show the various steps of the application process.

[0089] Audio generation: Use text-to-speech (TTS) technology to convert text descriptions into audio information so that users can obtain process information through hearing.

[0090] Further, the requirement information is parsed by the RAG system to generate corresponding application process text information, application process image information and / or application process audio information, specifically including:

[0091] Analyze the demand information through the RAG system to determine the user's application process and the application process image information involved in the application process;

[0092] Generate application process text information and application process audio information according to the application process.

[0093] It should be noted that text generation: uses a text generation model to generate a text description of the application process based on the integrated information.

[0094] Image generation: Use image generation models to generate flowcharts or schematics based on text descriptions to show the various steps of the application process.

[0095] Audio generation: Use text-to-speech (TTS) technology to convert text descriptions into audio information so that users can obtain process information through hearing.

[0096] Furthermore, step S102 specifically includes:

[0097] If the input information is text information, the text information is converted into the first embedding of the target format through NPL technology;

[0098] If the input information is image information, the image features of the image information are extracted through image recognition technology, and the image features are converted into a second embedding of the target format;

[0099] If the input information is audio information, the audio information is converted into text information through speech recognition technology, and then the text information is converted into a third embedding of the target format through NPL technology;

[0100] When the input information includes two or more different types of information, the first embedding, the second embedding and / or the third embedding are weighted fused to form a fusion vector.

[0101] Furthermore, when the input information includes two or more different types of information, weighted fusion of the first embedding, the second embedding and / or the third embedding to form a fusion vector specifically includes:

[0102] When the input information includes two or more different types of information, the first embedding, the second embedding and / or the third embedding are weighted fused according to preset weights corresponding to the information types to form a fusion vector.

[0103] It should be noted that a preset weight is assigned to each embedding based on the importance of the information type and its contribution to the task. For example, if the text information is crucial to understanding user needs, it can be given a higher weight.

[0104] Multiply each embedding by its corresponding weight, and then perform summation or averaging to form a fusion vector.

[0105] If the fusion vector dimension is too high, dimensionality reduction may be required to reduce the computational complexity.

[0106] Furthermore, it also includes:

[0107] Step S201: Obtain user information of the user, and generate a user profile of the user based on the user information analysis;

[0108] Step S202: Determine a number of recommendation solutions based on the user portrait;

[0109] Step S203: Generate copywriting information including several recommended solutions based on the large language model;

[0110] Step S204, performing vector conversion processing on the text information to convert it into a fourth embedding of the target format;

[0111] Step S205: Use the fourth embedding as the input of the video generation model to generate a user recommended video.

[0112] It should be noted that when users use an app for the first time, they often need the most appropriate recommendations. Therefore, a shopping guide video is exactly what users need. By analyzing the user's profile information, we can recommend shopping guide videos of the target solution type and provide users with suggestions for choosing a solution.

[0113] Further, step S203 specifically includes:

[0114] The solution information of several recommended solutions is input into the large language model, and the large language model generates copywriting information containing several recommended solutions by calling a preset highlight extraction template.

[0115] See also Figure 2 , Figure 2 : is a schematic diagram of the structure of the APP-assisted video generation device in the embodiment of the present application, such as Figure 2 As shown, specifically:

[0116] The first acquisition unit 301 is used to acquire user input information, where the input information includes text information, picture information and / or audio information;

[0117] It should be noted that obtaining user input information may include text information, image information and / or audio information. For example, the user inputs or types “I want to view the specific terms of XX insurance”, or imports “XX insurance promotional picture”, etc.

[0118] The first processing unit 302 is used to perform vector conversion processing on the input information, convert it into an embedding of a target format and perform weighted fusion to form a fusion vector;

[0119] It should be noted that since the formats of text information, image information and audio information are different, a unified representation space, namely a vector, is required. The input information is then vectorized and converted into an embedding of the target format.

[0120] After forming a unified vector, the input needs to be weightedly fused with ATTENTION to form a fusion vector, so as to proceed to the next step of model generation. The fusion vector includes the extraction of input information.

[0121] The first generating unit 303 is used to use the fusion vector as the input of the video generation model to generate an APP auxiliary video, and the video generation model is a transformer structure.

[0122] It should be noted that by using the front-end Transformer module, the three different dimensions of input information are weighted and fused to form a single vector, which enters the video generation model to control the generation of APP-assisted videos. The video generation model is the current mainstream Transformer structure, and its advantage is that it can better utilize the input information.

[0123] Furthermore, the first acquisition unit 301 is specifically configured to:

[0124] Acquire demand information input by a user, the demand information including demand text information, demand audio information and / or demand audio information;

[0125] The demand information is parsed through the RAG system to generate corresponding application process text information, application process image information and / or application process audio information.

[0126] It should be noted that the user's demand information is received, which may include text, picture or audio format.

[0127] Clean and standardize the demand information, such as text segmentation, image format conversion, audio transcription, etc.

[0128] The RAG system can be used to analyze demand information, including:

[0129] Text parsing: Use natural language processing technology to parse text information and identify keywords, intent, and demand details;

[0130] Image parsing: Using computer vision techniques to extract information from images and identify objects, scenes, and possible process instructions in the images;

[0131] Audio parsing: transcribe audio information into text and further parse it using natural language processing technology.

[0132] Based on the parsing results, relevant information and data are retrieved from the knowledge base to enhance the understanding of the requirements.

[0133] Integrate the retrieved information with the original input information to form a comprehensive understanding of the requirements.

[0134] Generate corresponding application process text information, application process image information and / or application process audio information, which can be:

[0135] Text generation: Use the text generation model to generate a text description of the application process based on the integrated information.

[0136] Image generation: Use image generation models to generate flowcharts or schematics based on text descriptions to show the various steps of the application process.

[0137] Audio generation: Use text-to-speech (TTS) technology to convert text descriptions into audio information so that users can obtain process information through hearing.

[0138] Further, the requirement information is parsed by the RAG system to generate corresponding application process text information, application process image information and / or application process audio information, specifically including:

[0139] Analyze the demand information through the RAG system to determine the user's application process and the application process image information involved in the application process;

[0140] Generate application process text information and application process audio information according to the application process.

[0141] It should be noted that text generation: uses a text generation model to generate a text description of the application process based on the integrated information.

[0142] Image generation: Use image generation models to generate flowcharts or schematics based on text descriptions to show the various steps of the application process.

[0143] Audio generation: Use text-to-speech (TTS) technology to convert text descriptions into audio information so that users can obtain process information through hearing.

[0144] Furthermore, the first processing unit 302 is specifically configured to:

[0145] If the input information is text information, the text information is converted into the first embedding of the target format through NPL technology;

[0146] If the input information is image information, the image features of the image information are extracted through image recognition technology, and the image features are converted into a second embedding of the target format;

[0147] If the input information is audio information, the audio information is converted into text information through speech recognition technology, and then the text information is converted into a third embedding of the target format through NPL technology;

[0148] When the input information includes two or more different types of information, the first embedding, the second embedding and / or the third embedding are weighted fused to form a fusion vector.

[0149] Furthermore, when the input information includes two or more different types of information, weighted fusion of the first embedding, the second embedding and / or the third embedding to form a fusion vector specifically includes:

[0150] When the input information includes two or more different types of information, the first embedding, the second embedding and / or the third embedding are weighted fused according to preset weights corresponding to the information types to form a fusion vector.

[0151] It should be noted that a preset weight is assigned to each embedding based on the importance of the information type and its contribution to the task. For example, if the text information is crucial to understanding user needs, it can be given a higher weight.

[0152] Multiply each embedding by its corresponding weight, and then perform summation or averaging to form a fusion vector.

[0153] If the fusion vector dimension is too high, dimensionality reduction may be required to reduce the computational complexity.

[0154] Furthermore, it also includes:

[0155] A second acquisition unit is used to acquire user information of the user and generate a user portrait of the user based on the user information analysis;

[0156] A determination unit, used to determine a number of recommendation solutions based on the user profile;

[0157] A copywriting generation unit, used to generate copywriting information containing several recommendation solutions based on a large language model;

[0158] The second processing unit is used to perform vector conversion processing on the text information to convert it into a fourth embedding of a target format;

[0159] The second generating unit is used to use the fourth embedding as an input of the video generation model to generate a user recommended video.

[0160] It should be noted that when users use an app for the first time, they often need the most appropriate recommendations. Therefore, a shopping guide video is exactly what users need. By analyzing the user's profile information, we can recommend shopping guide videos of the target solution type and provide users with suggestions on choosing a solution.

[0161] Furthermore, the second processing unit is specifically configured to:

[0162] The solution information of several recommended solutions is input into the large language model, and the large language model generates copywriting information containing several recommended solutions by calling a preset highlight extraction template.

[0163] The present application also provides another APP-assisted video generation device, such as Figure 3 For the sake of convenience, only the parts related to the embodiments of the present application are shown. For specific technical details not disclosed, please refer to the method part of the embodiments of the present application. The terminal can be any terminal device including a mobile phone, a tablet computer, a personal digital assistant (English full name: Personal Digital Assistant, English abbreviation: PDA), a sales terminal (English full name: Point of Sales, English abbreviation: POS), a car computer, etc., taking the terminal as a mobile phone as an example:

[0164] Figure 3 FIG. 1 is a block diagram showing a partial structure of a mobile phone related to a terminal provided in an embodiment of the present application. Figure 3 The mobile phone includes: a radio frequency (RF) circuit 1010, a memory 1020, an input unit 1030, a display unit 1040, a sensor 1050, an audio circuit 1060, a wireless fidelity (WiFi) module 1070, a processor 1080, and a power supply 1090. Those skilled in the art can understand that Figure 3 The mobile phone structure shown in the figure does not constitute a limitation on the mobile phone, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0165] Combine the following Figure 3 A detailed introduction to the various components of the mobile phone:

[0166] The RF circuit 1010 can be used for receiving and sending signals during information transmission or calls. In particular, after receiving the downlink information of the base station, it is sent to the processor 1080 for processing; in addition, the designed uplink data is sent to the base station. Generally, the RF circuit 1010 includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (full name: Low Noise Amplifier, English abbreviation: LNA), a duplexer, etc. In addition, the RF circuit 1010 can also communicate with the network and other devices through wireless communication. The above-mentioned wireless communications can use any communication standard or protocol, including but not limited to Global System of Mobile communications (Global System of Mobile communication, English abbreviation: GSM), General Packet Radio Service (General Packet Radio Service, GPRS), Code Division Multiple Access (Code Division Multiple Access, English abbreviation: CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (Long Term Evolution, English abbreviation: LTE), e-mail, Short Messaging Service (SMS), etc.

[0167] The memory 1020 can be used to store software programs and modules. The processor 1080 executes various functional applications and data processing of the mobile phone by running the software programs and modules stored in the memory 1020. The memory 1020 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory 1020 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0168] The input unit 1030 can be used to receive input digital or character information, and to generate key signal input related to the user settings and function control of the mobile phone. Specifically, the input unit 1030 may include a touch panel 1031 and other input devices 1032. The touch panel 1031, also known as a touch screen, can collect the user's touch operation on or near it (such as the user's operation on the touch panel 1031 or near the touch panel 1031 using any suitable object or accessory such as a finger, stylus, etc.), and drive the corresponding connection device according to a pre-set program. Optionally, the touch panel 1031 may include two parts: a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch orientation, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into contact coordinates, and then sends it to the processor 1080, and can receive and execute commands sent by the processor 1080. In addition, the touch panel 1031 can be implemented in various types such as resistive, capacitive, infrared, and surface acoustic waves. In addition to the touch panel 1031, the input unit 1030 may further include other input devices 1032. Specifically, the other input devices 1032 may include but are not limited to one or more of a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, a joystick, and the like.

[0169] The display unit 1040 can be used to display information input by the user or information provided to the user and various menus of the mobile phone. The display unit 1040 may include a display panel 1041. Optionally, the display panel 1041 may be configured in the form of a liquid crystal display (full name in English: Liquid Crystal Display, English abbreviation: LCD), an organic light-emitting diode (full name in English: Organic Light-Emitting Diode, English abbreviation: OLED), etc. Further, the touch panel 1031 may cover the display panel 1041. When the touch panel 1031 detects a touch operation on or near it, it is transmitted to the processor 1080 to determine the type of touch event. Subsequently, the processor 1080 provides a corresponding visual output on the display panel 1041 according to the type of touch event. Although in Figure 3 In the embodiment, the touch panel 1031 and the display panel 1041 are used as two independent components to realize the input and output functions of the mobile phone, but in some embodiments, the touch panel 1031 and the display panel 1041 can be integrated to realize the input and output functions of the mobile phone.

[0170] The mobile phone may also include at least one sensor 1050, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor may adjust the brightness of the display panel 1041 according to the brightness of the ambient light, and the proximity sensor may turn off the display panel 1041 and / or the backlight when the mobile phone is moved to the ear. As a type of motion sensor, the accelerometer sensor can detect the magnitude of acceleration in all directions (generally three axes), and can detect the magnitude and direction of gravity when stationary. It can be used for applications that identify the posture of the mobile phone (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc.; as for other sensors that can be configured in the mobile phone, such as gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc., they will not be repeated here.

[0171] The audio circuit 1060, the speaker 1061, and the microphone 1062 can provide an audio interface between the user and the mobile phone. The audio circuit 1060 can transmit the received audio data to the speaker 1061 after converting the received audio data into an electrical signal, which is converted into a sound signal for output; on the other hand, the microphone 1062 converts the collected sound signal into an electrical signal, which is received by the audio circuit 1060 and converted into audio data, and then the audio data is output to the processor 1080 for processing, and then sent to another mobile phone through the RF circuit 1010, or the audio data is output to the memory 1020 for further processing.

[0172] WiFi is a short-range wireless transmission technology. The mobile phone can help users send and receive emails, browse web pages and access streaming media through the WiFi module 1070. It provides users with wireless broadband Internet access. Figure 3 A WiFi module 1070 is shown, but it is understandable that it is not an essential component of the mobile phone and can be omitted as needed without changing the essence of the invention.

[0173] The processor 1080 is the control center of the mobile phone. It uses various interfaces and lines to connect various parts of the entire mobile phone. By running or executing software programs and / or modules stored in the memory 1020 and calling data stored in the memory 1020, it executes various functions of the mobile phone and processes data, thereby monitoring the mobile phone as a whole. Optionally, the processor 1080 may include one or more processing units; preferably, the processor 1080 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface and application programs, etc., and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 1080.

[0174] The mobile phone also includes a power supply 1090 (such as a battery) for supplying power to various components. Preferably, the power supply can be logically connected to the processor 1080 through a power management system, so that the power management system can manage functions such as charging, discharging, and power consumption.

[0175] Although not shown, the mobile phone may also include a camera, a Bluetooth module, etc., which will not be described in detail here.

[0176] In the embodiment of the present application, the processor 1080 included in the terminal also has the following functions:

[0177] Step S101: obtaining user input information, where the input information includes text information, picture information and / or audio information;

[0178] It should be noted that obtaining user input information may include text information, image information and / or audio information. For example, the user inputs or types “I want to view the specific terms of XX insurance”, or imports “XX insurance promotional picture”, etc.

[0179] Step S102: Perform vector conversion processing on the input information, convert it into an embedding of a target format and perform weighted fusion to form a fusion vector;

[0180] It should be noted that since the formats of text information, image information and audio information are different, a unified representation space, namely a vector, is required. The input information is then vectorized and converted into an embedding of the target format.

[0181] After forming a unified vector, the input needs to be weightedly fused with ATTENTION to form a fusion vector, so as to proceed to the next step of model generation. The fusion vector includes the extraction of input information.

[0182] Step S103: Use the fusion vector as the input of the video generation model to generate an APP auxiliary video. The video generation model is a transformer structure.

[0183] It should be noted that by using the front-end Transformer module, the three different dimensions of input information are weighted and fused to form a single vector, which enters the video generation model to control the generation of APP-assisted videos. The video generation model is the current mainstream Transformer structure, and its advantage is that it can better utilize the input information.

[0184] Furthermore, step S101 specifically includes:

[0185] Acquire demand information input by a user, the demand information including demand text information, demand audio information and / or demand audio information;

[0186] The demand information is parsed through the RAG system to generate corresponding application process text information, application process image information and / or application process audio information.

[0187] It should be noted that the user's demand information is received, which may include text, picture or audio format.

[0188] Clean and standardize the demand information, such as text segmentation, image format conversion, audio transcription, etc.

[0189] The RAG system can be used to analyze demand information, including:

[0190] Text parsing: Use natural language processing technology to parse text information and identify keywords, intent, and demand details;

[0191] Image parsing: Using computer vision techniques to extract information from images and identify objects, scenes, and possible process instructions in the images;

[0192] Audio parsing: transcribe audio information into text and further parse it using natural language processing technology.

[0193] Based on the parsing results, relevant information and data are retrieved from the knowledge base to enhance the understanding of the requirements.

[0194] Integrate the retrieved information with the original input information to form a comprehensive understanding of the requirements.

[0195] Generate corresponding application process text information, application process image information and / or application process audio information, which can be:

[0196] Text generation: Use the text generation model to generate a text description of the application process based on the integrated information.

[0197] Image generation: Use image generation models to generate flowcharts or schematics based on text descriptions to show the various steps of the application process.

[0198] Audio generation: Use text-to-speech (TTS) technology to convert text descriptions into audio information so that users can obtain process information through hearing.

[0199] Further, the requirement information is parsed by the RAG system to generate corresponding application process text information, application process image information and / or application process audio information, specifically including:

[0200] Analyze the demand information through the RAG system to determine the user's application process and the application process image information involved in the application process;

[0201] Generate application process text information and application process audio information according to the application process.

[0202] It should be noted that text generation: uses a text generation model to generate a text description of the application process based on the integrated information.

[0203] Image generation: Use image generation models to generate flowcharts or schematics based on text descriptions to show the various steps of the application process.

[0204] Audio generation: Use text-to-speech (TTS) technology to convert text descriptions into audio information so that users can obtain process information through hearing.

[0205] Furthermore, step S102 specifically includes:

[0206] If the input information is text information, the text information is converted into the first embedding of the target format through NPL technology;

[0207] If the input information is image information, the image features of the image information are extracted through image recognition technology, and the image features are converted into a second embedding of the target format;

[0208] If the input information is audio information, the audio information is converted into text information through speech recognition technology, and then the text information is converted into a third embedding of the target format through NPL technology;

[0209] When the input information includes two or more different types of information, the first embedding, the second embedding and / or the third embedding are weighted fused to form a fusion vector.

[0210] Furthermore, when the input information includes two or more different types of information, weighted fusion of the first embedding, the second embedding and / or the third embedding to form a fusion vector specifically includes:

[0211] When the input information includes two or more different types of information, the first embedding, the second embedding and / or the third embedding are weighted fused according to preset weights corresponding to the information types to form a fusion vector.

[0212] It should be noted that a preset weight is assigned to each embedding based on the importance of the information type and its contribution to the task. For example, if the text information is crucial to understanding user needs, it can be given a higher weight.

[0213] Multiply each embedding by its corresponding weight, and then perform summation or averaging to form a fusion vector.

[0214] If the fusion vector dimension is too high, dimensionality reduction may be required to reduce the computational complexity.

[0215] Step S201: Obtain user information of the user, and generate a user profile of the user based on the user information analysis;

[0216] Step S202: Determine a number of recommendation solutions based on the user portrait;

[0217] Step S203: Generate copywriting information including several recommended solutions based on the large language model;

[0218] Step S204, performing vector conversion processing on the text information to convert it into a fourth embedding of the target format;

[0219] Step S205: Use the fourth embedding as the input of the video generation model to generate a user recommended video.

[0220] It should be noted that when users use an app for the first time, they often need the most appropriate recommendations. Therefore, a shopping guide video is exactly what users need. By analyzing the user's profile information, we can recommend shopping guide videos of the target solution type and provide users with suggestions for choosing a solution.

[0221] Further, step S203 specifically includes:

[0222] The solution information of several recommended solutions is input into the large language model, and the large language model generates copywriting information containing several recommended solutions by calling a preset highlight extraction template.

[0223] An embodiment of the present application also provides a computer-readable storage medium for storing program code, which is used to execute any implementation of an APP-assisted video generation method described in the aforementioned embodiments.

[0224] In an embodiment of the present application, an APP-assisted video generation method, apparatus, device and storage medium are provided. By obtaining multimodal input information of the user, the multimodal input information is converted into a vector in a unified target format that is convenient for input into a video generation model and fused, so that the video generation model can generate a corresponding APP-assisted video that meets the conditions according to the multimodal input information of the user, thereby solving the technical problems that the video production cost is high due to the high cost of video shooting and the large number of videos required for different application scenarios and user needs, while the lack of auxiliary videos leads to a poor user experience.

[0225] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0226] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein, for example. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0227] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0228] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0229] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0230] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0231] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (full name in English: Read-Only Memory, English abbreviation: ROM), random access memory (full name in English: Random Access Memory, English abbreviation: RAM), disk or optical disk and other media that can store program codes.

[0232] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. An APP-assisted video generation method, characterized in that: include: Step S101: obtaining user input information, wherein the input information includes text information, picture information and / or audio information; Step S102: performing vector conversion processing on the input information, converting it into an embedding of a target format and performing weighted fusion to form a fusion vector; Step S103: using the fusion vector as an input of a video generation model to generate an APP auxiliary video, wherein the video generation model is a transformer structure.

2. The APP-assisted video generation method according to claim 1, characterized in that: The step S101 specifically includes: Acquire demand information input by a user, wherein the demand information includes demand text information, demand audio information and / or demand audio information; The demand information is parsed through the RAG system to generate corresponding application process text information, application process image information and / or application process audio information.

3. The APP-assisted video generation method according to claim 2, characterized in that: The parsing of the demand information by the RAG system to generate corresponding application process text information, application process image information and / or application process audio information specifically includes: The demand information is parsed through the RAG system to determine the application process of the user and the application process image information involved in the application process; Application process text information and application process audio information are generated according to the application process.

4. The APP-assisted video generation method according to claim 1, characterized in that: The step S102 specifically includes: If the input information is text information, convert the text information into a first embedding of a target format by using NPL technology; If the input information is picture information, extracting picture features of the picture information by image recognition technology, and converting the picture features into a second embedding in a target format; If the input information is audio information, the audio information is converted into text information by using speech recognition technology, and then the text information is converted into a third embedding of a target format by using NPL technology; When the input information includes two or more different types of information, the first embedding, the second embedding and / or the third embedding are weighted fused to form a fusion vector.

5. The APP-assisted video generation method according to claim 4, characterized in that: When the input information includes two or more different types of information, weighted fusion of the first embedding, the second embedding and / or the third embedding to form a fusion vector specifically includes: When the input information includes two or more different types of information, the first embedding, the second embedding and / or the third embedding are weighted fused according to preset weights corresponding to the information types to form a fusion vector.

6. The APP-assisted video generation method according to claim 1, characterized in that: Also includes: Step S201: Obtain user information of a user, and generate a user profile of the user based on analysis of the user information; Step S202: determining a number of recommendation solutions according to the user portrait; Step S203: Generate text information containing the plurality of recommended solutions based on the large language model; Step S204, performing vector conversion processing on the text information to convert it into a fourth embedding of a target format; Step S205: Use the fourth embedding as an input of a video generation model to generate a user recommended video.

7. The APP-assisted video generation method according to claim 6, characterized in that: The step S203 specifically includes: The solution information of the plurality of recommended solutions is input into the large language model, and the large language model generates text information containing the plurality of recommended solutions by calling a preset highlight extraction template.

8. An APP-assisted video generation device, characterized in that: include: A first acquisition unit, used to acquire input information of a user, wherein the input information includes text information, picture information and / or audio information; A first processing unit is used to perform vector conversion processing on the input information, convert it into an embedding of a target format and perform weighted fusion to form a fusion vector; The first generating unit is used to use the fusion vector as an input of a video generating model to generate an APP auxiliary video, and the video generating model is a transformer structure.

9. An APP-assisted video generation device, characterized in that: The device comprises a processor and a memory: The memory is used to store program code and transmit the program code to the processor; The processor is used to execute the APP-assisted video generation method described in any one of claims 1-7 according to the instructions in the program code.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store program code, and the program code is used to execute the APP-assisted video generation method described in any one of claims 1-7.

Images