Virtual image video generation method and device, equipment and medium

Abstract

The invention relates to the technical field of artificial intelligence and the field of financial science and technology, and discloses a virtual image video generation method, which comprises the following steps of: receiving a video generation instruction of a user, judging whether a virtual image is generated or not, determining the gender of the virtual image when the virtual image is not generated, and receiving audio data and a virtual character agent picture; generating a virtual image based on the information, starting a video generation process by using the virtual image, polling to obtain a generation progress and a generation result, judging whether the virtual image video is successfully generated or not, and displaying the preview of the virtual image video after the generation is successful. According to the method, the personalized virtual image is generated according to the gender input by the user, the audio data and the virtual character agent picture, so that the generated virtual image can accurately reflect the characteristics of the user. Meanwhile, the video generation process is started after the virtual image is generated, and the generation progress is obtained by using a polling mechanism, so that the video generation efficiency is improved, long-time waiting is avoided, and the operation experience and the video generation speed are improved.

Description

Technical Field

[0001] The present invention relates to the fields of artificial intelligence technology and financial technology, and in particular to a method, device, equipment and storage medium for generating a virtual image video. Background Art

[0002] With the rapid development of the financial industry, the demand for digitalization and remote services is increasing, and financial institutions are increasingly relying on digital means to provide efficient customer service. Especially in the field of customer service and marketing, virtual image and video generation technology has gradually become an innovative communication tool that can effectively improve customer experience. In the existing technology, the main application scenarios of virtual image video generation include customer online identity verification, personalized customer service, and intelligent financial assistants.

[0003] In existing technologies, financial institutions use virtual images to conduct remote identity verification of customers, simulating face-to-face communication scenarios by making video calls with customers or recording videos of virtual images. This method is usually used to improve customer trust and simplify the identity verification process. In addition, virtual images are also used in customer service platforms to provide personalized service suggestions, product recommendations, and business consultations by simulating human customer service.

[0004] Although virtual image and video generation technologies have played an initial role in financial services, existing technologies have the following shortcomings:

[0005] Most existing avatar generation systems use preset templates and lack a high degree of customization of user personal characteristics. Especially in the synthesis of audio and image, the generated avatar is difficult to truly reflect the customer's personality and appearance characteristics, reducing the immersion and realism of the interaction.

[0006] In the process of video generation and rendering, existing technologies often rely on single-threaded operations, and users need to wait for a long time to see the final result. This not only reduces the user experience, but may also lead to customer loss. For financial institutions, it is crucial to improve the speed of virtual image generation and interactive responsiveness. Summary of the invention

[0007] The main purpose of the present invention is to provide a virtual image video generation method, device, equipment and storage medium, aiming to solve the technical problem of insufficient personalization in the virtual image generation and video generation process in the prior art.

[0008] To achieve the above object, the present invention provides a method for generating a virtual image video, comprising:

[0009] receiving a video generation instruction from a user, and determining whether a virtual image has been generated;

[0010] If the avatar is not generated, determining the gender of the avatar based on the video generation instruction, and receiving audio data and an avatar proxy image;

[0011] generating an avatar based on the gender of the avatar, the audio data, and the avatar proxy picture;

[0012] Using the virtual image to start the video generation process, and polling to obtain the generation progress until the generation result is obtained;

[0013] Determining whether the virtual image video is successfully generated based on the generation result;

[0014] If the virtual image video is generated successfully, a preview of the virtual image video is displayed.

[0015] Furthermore, in order to achieve the above-mentioned object, the present invention provides a virtual image video generating device, comprising:

[0016] A generation instruction receiving module is used to receive a user's video generation instruction and determine whether a virtual image has been generated;

[0017] A virtual image generation preparation module is used to determine the gender of the virtual image based on the video generation instruction if the virtual image has not been generated, and receive audio data and a virtual character agent picture;

[0018] A virtual image generation module, used to generate a virtual image based on the gender of the virtual image, audio data and a virtual character proxy picture;

[0019] A video generation module, used to start the video generation process using the virtual image, and poll to obtain the generation progress until the generation result is obtained;

[0020] A generation result judgment module, used to judge whether the virtual image video is successfully generated based on the generation result;

[0021] The video preview module is used to display a preview of the virtual image video if the virtual image video is successfully generated.

[0022] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a computer device, which includes a memory, a processor, and a virtual image video generation program stored in the memory and executable on the processor, and the virtual image video generation program, when executed by the processor, implements the steps of the virtual image video generation method as described above.

[0023] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, on which a virtual image video generation program is stored, and when the virtual image video generation program is executed by a processor, the steps of the virtual image video generation method as described above are implemented.

[0024] Beneficial effects: The present invention relates to the fields of artificial intelligence technology and financial technology, and discloses a method for generating a virtual image video. By receiving a video generation instruction from a user, it is determined whether a virtual image has been generated. If not, the gender of the virtual image is determined, and audio data and a virtual character proxy picture are received. A virtual image is generated based on the gender, audio data, and virtual character proxy picture. The video generation process is started using the virtual image, and the generation progress and generation results are obtained by polling. It is determined whether the virtual image video is successfully generated, and a preview of the virtual image video is displayed after success. The present invention generates a personalized virtual image according to the gender, audio data, and virtual character proxy picture input by the user, so that the generated virtual image can accurately reflect the characteristics of the user. At the same time, after the virtual image is generated, the video generation process is started, and the generation progress is obtained by using a polling mechanism, which improves the efficiency of video generation, avoids long waiting, improves the operating experience and video generation speed, and solves the problems of insufficient personalization and slow generation process in the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which:

[0026] Figure 1 A schematic diagram of an application environment of a method for generating a virtual image video in an embodiment of the present invention;

[0027] Figure 2 It is a flowchart of an embodiment of a method for generating a virtual image video according to the present invention;

[0028] Figure 3 A schematic diagram of functional modules of a preferred embodiment of a virtual image video generating device of the present invention;

[0029] Figure 4 A schematic diagram of the structure of a computer device in one embodiment of the present invention;

[0030] Figure 5 FIG. 4 is another schematic diagram of the structure of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION

[0031] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0032] The virtual image video generation method provided by the embodiment of the present invention can be applied in the following aspects: Figure 1In an application environment, the user terminal communicates with the server terminal through a network. The server terminal can receive the user's video generation instruction through the user terminal, determine whether the virtual image has been generated, determine the gender of the virtual image if it has not been generated, and receive audio data and a virtual character proxy picture, generate a virtual image based on the gender, audio data and the virtual character proxy picture, use the virtual image to start the video generation process, poll to obtain the generation progress and obtain the generation result, determine whether the virtual image video is successfully generated, and display a preview of the virtual image video after success. The present invention generates a personalized virtual image according to the gender, audio data and virtual character proxy picture input by the user, so that the generated virtual image can accurately reflect the characteristics of the user. At the same time, after the virtual image is generated, the video generation process is started, and the generation progress is obtained by the polling mechanism, which improves the efficiency of video generation, avoids long waiting, improves the operation experience and video generation speed, and solves the problems of insufficient personalization and slow generation process in the prior art. Among them, the user terminal can be but not limited to various personal computers, laptops, smart phones, tablet computers and portable wearable devices. The server terminal can be implemented by an independent server or a server cluster composed of multiple servers. The present invention is described in detail below through specific embodiments.

[0033] See also Figure 2 , Figure 2 This is a flow chart of an embodiment of a method for generating a virtual image video provided by the present invention. It should be noted that although a logical sequence is shown in the flow chart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0034] like Figure 2 As shown, the virtual image video generation method proposed by the present invention includes the following steps:

[0035] S10, receiving a video generation instruction from a user, and determining whether a virtual image has been generated;

[0036] In this embodiment, receiving a video generation instruction input by a user means that the system is able to capture the user's request from a user interface or an API, and this request may include multiple parameters, such as the theme, background, audio selection, etc. of the video.

[0037] It is usually implemented through a user interface (UI), such as a mobile app, web page, or desktop client. The user selects the relevant parameters for video generation (such as virtual image, background, action) through the interface and clicks the confirmation button. The system communicates with the API of the front-end and back-end, receives the data submitted by the user, and triggers the processing flow of the back-end. When implementing it, interface technologies such as RESTfulAPI and GraphQL can be used.

[0038] After receiving the generation instruction, the system needs to determine whether the current user has generated an avatar. This step involves querying whether the avatar data is stored under the user account. The system accesses the avatar record bound to the user account in the database or cache system to determine whether there is a valid avatar file or data.

[0039] The system first searches the user database to retrieve the avatar records in the user account. Usually, this data includes the avatar's unique identifier, generation time, data integrity, and other information. Database queries can be performed through SQL statements or NoSQL database queries. If a valid avatar is found, the system will proceed to the next step; if no relevant records are found, the system will prompt the user to create a new avatar. Caching systems such as Redis can also be used to improve query efficiency.

[0040] In one specific implementation, the system receives the user's video generation instructions through a voice interface, and the user directly inputs the video generation related parameters by voice. The system uses voice recognition technology (such as Google Speech API or Azure Speech Services) to convert the voice instructions into text and submit it to the background for processing. It can be further combined with natural language processing (NLP) technology to make intelligent parameter configuration suggestions based on the context of the user's voice content to improve the user experience. The user does not need to express all the details explicitly, and the system can automatically infer the parameters based on habits.

[0041] Example description: In the financial field, virtual image technology can be used for personalized customer service. For example, users can generate a virtual assistant based on their own image through a financial service app to help with personalized asset management recommendations. When a user makes a request to create a virtual image assistant, the system first determines whether the user has already generated a virtual assistant. If the user already has a ready-made virtual image, the system directly calls and displays the virtual assistant. If the user has not generated one, the user is guided to personalize options such as gender and voice to generate a financial service virtual assistant dedicated to the user. Through caching technology, the system can quickly respond to user requests and improve the user experience.

[0042] By determining whether a virtual image has been generated after receiving a user instruction, repeated generation of the virtual image can be avoided, unnecessary consumption of computing resources can be reduced, and the overall efficiency of the system can be improved.

[0043] S20, if the avatar is not generated, determining the gender of the avatar based on the video generation instruction, and receiving audio data and an avatar agent image;

[0044] In this embodiment, if it is detected that the avatar has not been generated before, the system needs to parse the user's input data from the video generation instruction to determine the gender of the avatar. The video generation instruction can include a clear gender selection (such as "male" or "female"), or the system can infer the gender based on the user's historical data and preferences. This is an important step in the personalized generation of the avatar.

[0045] The system can provide a gender selection option in the user interface for users to manually select the gender of the avatar. If the user does not explicitly specify the gender, the system can generate the gender of the avatar through preference analysis based on the user's past data or through default values. This process can be implemented on the front end through JavaScript or on the back end through analysis of user preference data.

[0046] The system receives the user's audio data and uses it to generate the avatar's voice content. This can be done by the user uploading an audio file or directly recording the audio through a microphone. The audio data will be used for subsequent speech synthesis or synchronization.

[0047] The system can use the Web Audio API in the browser or the audio recording interface in the mobile application (such as Android's MediaRecorder or iOS's AVAudioRecorder) to receive the audio recorded by the user. The received audio data can be pre-processed, such as noise reduction, format conversion, etc., to ensure that the sound quality meets the requirements for generating the virtual image. The system can also accept audio files uploaded by users, supporting multiple formats such as MP3, WAV, etc.

[0048] The system receives the virtual character proxy pictures uploaded by users as the basic material for generating the virtual image, and the system will generate the corresponding virtual image based on this picture.

[0049] Avatar proxy images refer to the basic images used by users to generate avatars or avatars. They are usually a personal photo uploaded by the user, or an image representing the user's appearance. The system uses this image to generate a avatar image that is similar to the user or meets the specified requirements.

[0050] Avatar proxy images are used to provide a basic visual reference for the avatar. This image usually contains the user's facial features, and the system will generate the corresponding avatar based on this image. By extracting feature information from the image, such as facial structure, skin color, hairstyle, etc., avatar proxy images can help generate a highly personalized avatar that is similar to the user's real appearance.

[0051] The image upload interface can be implemented through the HTML5 file upload control or the image selector in the mobile application. Users can choose to upload photos from the album or take photos with the camera. The uploaded pictures will be cropped, resized and converted to meet the needs of subsequent avatar generation through image processing algorithms. To improve processing efficiency, image preprocessing can be performed using image processing libraries such as OpenCV.

[0052] In one specific implementation, if the user does not select a gender, the system can infer the gender of the avatar by analyzing the user's personal data, such as historical preferences, profile pictures, etc. The inference mechanism can be based on simple rules or machine learning models. Through a pre-trained gender classification model, the system can automatically identify and infer the gender based on the uploaded user profile pictures, thereby improving the personalized experience.

[0053] By combining the gender, audio data and virtual character images input by the user, a highly personalized virtual image can be generated to achieve a more realistic visual and auditory presentation that meets user needs. At the same time, the combination of audio data and virtual images enables the generated virtual image to have a more natural interactive ability. The system automatically infers gender, which improves the intelligence of the generation process and user experience. Automatic preprocessing of audio and images also improves data quality to ensure that the generation effect meets expectations.

[0054] S30, generating a virtual image based on the gender of the virtual image, the audio data and the virtual character agent picture;

[0055] In this embodiment, gender is one of the important parameters for generating a virtual image, which directly affects the appearance and performance of the virtual image. For example, virtual images of different genders may have different presentation effects in terms of facial features, body shape, hairstyle, voice, etc. When generating a virtual image, the system will adjust the generation parameters according to the gender selected by the user to ensure that the virtual image meets the gender characteristics.

[0056] The system selects a suitable avatar template based on the gender parameters entered by the user. The generation process may include preset image models or basic frameworks of different genders. For male avatars, the system will select facial contours, body structures, etc. that are more masculine; for female avatars, a template with more feminine characteristics will be used.

[0057] Audio data is used to generate speech content for the avatar that matches the user's voice. Audio data not only determines the avatar's voice characteristics (such as pitch and timbre), but is also used to generate the avatar's lip sync when speaking. By analyzing the user's audio data, the system can adjust the avatar's mouth movement so that the avatar's mouth shape is synchronized with the speech content.

[0058] The system uses audio processing technology (such as spectrum analysis, phoneme segmentation, etc.) to analyze audio data. Based on the results of the analysis, the system adjusts the avatar's mouth movement to synchronize it with the voice. At the same time, the system can generate a personalized voice model based on the characteristics of the user's audio, so that the voice of the avatar is similar to that of the user.

[0059] The avatar proxy pictures provide the visual basis of the avatar. By analyzing the proxy pictures, the system can extract facial feature information (such as facial features, hairstyle, skin color, etc.). These features will be mapped into the avatar generation model, making the avatar similar in appearance to the user's proxy pictures.

[0060] The system uses convolutional neural networks (CNN) or other image processing algorithms to extract facial features from the avatar proxy images. These feature data will be used to generate a 3D model or 2D image of the avatar to ensure that the appearance of the avatar is consistent with the user's photo. In actual implementation, image processing libraries (such as OpenCV) can be used to pre-process the images, such as cropping, filtering, and other operations.

[0061] Combining information about gender, audio data, and proxy images, the system builds the final avatar through a generation engine. This process requires fusing data from different sources to generate a avatar with personalized appearance and voice characteristics. The avatar can also dynamically present the user's voice expressions or interactive reactions.

[0062] The avatar generation process involves 3D modeling or 2D graphics generation engines. Through the generation engine (such as Unity, Unreal Engine, etc.), the system can combine the user's facial features, gender characteristics and voice data to create an avatar that meets the user's personalized needs. After the avatar is generated, it can be used in various application scenarios, such as virtual assistants, video generation, etc.

[0063] In one embodiment, combined with AR technology, the system can overlay the generated avatar on the user's real-time video stream. The avatar will be dynamically presented based on the information of gender, audio data and proxy image, and synchronized with the user's actions and voice. It is suitable for real-time interactive scenarios, such as virtual meetings or online live broadcasts. Users can control the avatar's actions and language in real time to interact with people in the real environment.

[0064] By combining the avatar's gender, audio data, and avatar proxy images, a highly personalized avatar can be generated that matches the user's visual appearance and voice characteristics. At the same time, the synchronous generation of audio data and avatars enables natural voice interaction and dynamic lip control, greatly improving the user experience. This ensures that the avatar is highly matched to the user's personalized needs and is suitable for a variety of application scenarios, such as virtual assistants, virtual meetings, online education, etc.

[0065] S40, starting a video generation process using the virtual image, and polling to obtain generation progress until a generation result is obtained;

[0066] In this embodiment, once the avatar is generated or selected, the system imports it into the video generation engine to start the video generation process. The video generation process involves synthesizing the avatar with other video content elements (such as background, action sequence, sound effects, etc.) to generate a user-specified video. As the core character of the video, the appearance and actions of the avatar will be displayed in the video.

[0067] The system first imports the model data of the avatar (such as 3D models, textures, animations, etc.) into the video generation engine. The video generation engine can be Unity, Unreal Engine or other dedicated video rendering engines. After the avatar is generated, it will be combined with preset scenes or dynamic elements, such as the background, environmental effects and action sequences of the avatar in the video. This process can be achieved by calling the interface of the rendering engine through the API to start the complete rendering task.

[0068] Polling means that the system periodically checks the progress of video generation. During the video generation process, the system needs to constantly query the status of the rendering engine or video generation module to obtain the progress information of the current generation task, such as completion percentage, current frame number, etc. This process ensures that users can understand the progress of video generation in real time and make adjustments or cancel operations in a timely manner.

[0069] When implementing polling progress, the system can query the video generation progress at regular intervals through a timer or event-driven method. The rendering engine usually provides a status query API, through which the system obtains the progress status of the current task and feeds the information back to the user interface for display. The progress can be presented to the user in real time in the form of a progress bar, percentage, or other form. The typical implementation method is to record the unique identifier (Task ID) of the task when the rendering task is started, and the system queries the task status based on the identifier.

[0070] After the video is generated, the system needs to obtain the final result and display it to the user. The result is usually a complete video file, which may include detailed information such as the exported video format, resolution, and number of frames. The system needs to ensure that the generated avatar and video content meet the user's expectations and provide the option to preview or download.

[0071] When the polling progress shows that the generation task has been completed, the system will obtain the generated final video file through the rendering engine API. This file can be stored in the local file system, cloud storage, or provided to the user for download. After the generation is completed, the system can provide a video preview function to display the final generated content through the HTML5 video player or the in-app video player. The user can choose to save or publish the video.

[0072] In one specific implementation, on a cloud server, the user-generated avatar will be uploaded to a cloud video generation engine (such as a cloud computing-based rendering service) through an API. The system regularly obtains the progress of video generation by polling the API of the cloud server. After the generation is completed, the video will be automatically stored in the cloud and a download link will be provided. During the cloud generation process, the system can process video generation requests for multiple users at the same time and provide real-time progress updates. At the same time, users do not need to wait for the generation process to end to perform other operations. After the video generation is completed, the system will notify the user through a push notification or email.

[0073] Example description: In the financial field, users can generate personalized avatars for asset management advice or interactive videos through the financial service platform. When the user submits a generation request, the system starts the video generation process and obtains the generation progress through a polling mechanism. Users can view the progress of video generation in real time. After the generation is completed, the system automatically stores the video in the user's cloud account and provides a video preview function.

[0074] The polling mechanism is used to obtain the video generation progress in real time, so that users can clearly understand the video generation process, thereby improving the user experience. The video generation process is started using a virtual image, and the generation status is obtained regularly through polling to ensure that the efficiency and progress of video generation are controllable. Especially in cloud or distributed generation scenarios, the system can handle a large number of concurrent generation requests, significantly improving the generation speed and efficiency.

[0075] S50, judging whether the virtual image video is successfully generated based on the generation result;

[0076] In this embodiment, the video generation result refers to the feedback information obtained by the system from the video generation engine, including generation progress, generated file status, generated final video file, error information, etc. The system needs to determine whether the video is successfully generated based on this feedback information. The generation result includes parameters such as the path of the video file, file size, generation status (success or failure), and generation time.

[0077] The system obtains the status code of the build task or the build result data from the build engine. Usually, the build engine returns a response in JSON format, including the build status (such as "success", "fail"), the address of the generated video file, and error information during the build process. After the system parses these build results, it determines whether the build is successful. If the build is successful, the build result will contain complete video file information; if the build fails, the system will receive an error message and prompt the user with the possible cause of the error.

[0078] The system determines whether the video generation is successful by checking the status information in the generation result and the integrity of the generated file. If the generation result returns a status code indicating success and the generated file is complete (such as the file size and duration meet expectations), the generation is considered successful; if an error message is returned or the generated file is incomplete, it is considered a failure and a relevant prompt is given.

[0079] The generation engine will return key information including file path, file size, generation time, etc., and the system will verify these parameters. For example, if the size of the generated video file is 0 or the file path is invalid, the system will prompt the user that the generation failed and record the log for subsequent troubleshooting. If the video is generated successfully, the system will display the generated avatar video for the user to preview or download.

[0080] In one specific implementation, in a distributed rendering scenario, the generation results may be distributed on multiple nodes. The system will check the return results of each node and merge them into a complete video file to determine whether all nodes have successfully generated and synthesized the final file. During the synthesis process, the system will check the generation status of each segment to ensure that the generation of each segment is successful. If a node fails to generate, the system will start re-rendering or obtain a backup from other nodes.

[0081] Example description: In financial applications, users generate personalized avatars for asset management advice or financial services. After the system completes the generation, it will determine whether the video is successfully generated based on the generation results returned by the cloud. If the generation is successful, the system will automatically push the generated video file to the user's financial service platform for the user to view or download at any time. If the generation fails, the system will prompt the user to resubmit the generation request or contact customer service to resolve the problem.

[0082] By judging the generation results, the system can ensure the integrity and quality of the generated avatar video. If the generation is successful, the user can preview or download the video immediately, which improves the user experience and satisfaction. When the generation fails, the system can prompt the user with possible solutions based on the specific error message, such as regeneration or parameter adjustment. This process ensures the reliability of the generation process and the timeliness of feedback.

[0083] S60: If the virtual image video is generated successfully, display a preview of the virtual image video.

[0084] In this embodiment, after the video generation is completed, the system first determines whether the generation is successful. The system will confirm the generation of the video based on the status code in the generation result, the integrity of the video file and other parameters. If the video file is successfully generated and meets the expected conditions (such as the correct file format, reasonable size, and valid path), the system will proceed to the next step to display the video preview.

[0085] The system will obtain the results returned by the video generation engine through polling or callback mechanism, and parse the generated file path and status code. If the status code is "success", and the generated file exists and passes the integrity check (such as video length, resolution, etc.), the system will determine that the generation is successful.

[0086] When the avatar video is successfully generated, the system will display a preview of the video on the user interface. The video preview function allows users to quickly view the generated video content without downloading the file to confirm whether the video meets expectations. The preview usually includes basic functions such as video play, pause, fast forward, volume control, etc.

[0087] The system can use built-in video players (such as HTML5 <video>The generated video file can be directly played by native players in tags or mobile applications. The preview function can provide basic playback controls such as play, pause, progress bar dragging, etc.

[0088] If the video file is stored in the cloud, the system can obtain the video's streaming URL (such as a CDN address) and play it through a streaming player (such as Video.js) on the front end. The preview function can also compress the video file or adjust the resolution in the background to increase loading speed.

[0089] In one specific implementation, after the video is generated in the cloud, the system displays the generated video preview through the front-end web player. The video stream is transmitted through the CDN and played in the player embedded in the browser. The front-end player (such as Video.js or JWPlayer) supports streaming media playback, and users can preview the generated video content in real time. In order to improve the user's viewing experience, the system can use adaptive bit rate (ABR) technology during the preview process to dynamically adjust the video quality according to the user's network conditions to ensure smooth playback. Watermarks or copyright logos can also be added to protect the video content.

[0090] Example: In the financial sector, users can generate avatars for personalized asset management videos through online financial platforms. When the generation is complete, the system will automatically display a preview of the video, and users can directly view the generated video content on the web page or mobile application to ensure that the avatar is consistent with expectations. If the user is satisfied with the video effect, they can choose to download or share the video content for further display or promotion of financial products. The preview function provides users with an efficient feedback mechanism, which improves the personalization and interactivity of financial services.

[0091] By showing a preview of the avatar video, users can view the generated results immediately after the video is successfully generated to confirm whether the video content meets expectations. The preview function greatly shortens the user's feedback cycle, and users can view the effect without waiting to download the entire video file. Especially in the cloud or streaming preview mode, the preview function can quickly load video content, improving the user experience while also saving bandwidth resources.

[0092] The present invention relates to the fields of artificial intelligence technology and financial technology, and discloses a method for generating a virtual image video. By receiving a video generation instruction from a user, it is determined whether a virtual image has been generated, and if not, the gender of the virtual image is determined, and audio data and a virtual character proxy picture are received, a virtual image is generated based on the gender, audio data and the virtual character proxy picture, and a video generation process is started using the virtual image, and the generation progress and generation result are obtained by polling, and it is determined whether the virtual image video is successfully generated, and a preview of the virtual image video is displayed after success. The present invention generates a personalized virtual image according to the gender, audio data and virtual character proxy picture input by the user, so that the generated virtual image can accurately reflect the characteristics of the user. At the same time, after the virtual image is generated, the video generation process is started, and the generation progress is obtained by using a polling mechanism, so that the efficiency of video generation is improved, long waiting is avoided, the operation experience and the video generation speed are improved, and the problems of insufficient personalization and slow generation process in the prior art are solved.

[0093] In one embodiment, in the above S10, determining whether the virtual image has been generated includes:

[0094] S101, querying whether there is generated virtual image data in the user account;

[0095] S102, if there is no generated virtual image data, it is determined that the virtual image has not been generated;

[0096] S103, if there is generated virtual image data, analyzing the validity and integrity of the virtual image data;

[0097] S104, judging whether the virtual image data meets the requirements of current video generation based on the validity and completeness;

[0098] S105, if the virtual image data meets the requirements for video generation, it is determined that the virtual image has been generated;

[0099] S106: If the virtual image data does not meet the requirements for video generation, it is deemed that the virtual image is not generated.

[0100] In this embodiment, the system first needs to query whether the generated virtual image data has been created in the user account. Each user account can be bound to corresponding virtual image information, such as the virtual image ID, generation time, storage path of the virtual image, etc. By querying the relevant records of the user account, the system can determine the existence of the virtual image data.

[0101] The system retrieves the avatar records in the user account through database query. This can be implemented using SQL query (such as SELECT statement) or document retrieval method of NoSQL database (such as MongoDB). The key fields of the query include user ID, avatar ID and its status (whether it has been generated). If there is a relevant record in the database, the system returns the storage path or ID of the avatar.

[0102] If no generated avatar data is found in the query results, the system will directly determine that the user has not yet generated an avatar, which means that the user needs to generate a new avatar through subsequent steps for use in video generation.

[0103] When the query result is empty, the system automatically enters the avatar generation process and prompts the user to start inputting the required image, audio, and gender information. The system can use a conditional judgment structure (such as an IF statement) to determine whether the query result is empty, thereby deciding to enter the avatar generation process.

[0104] If there is avatar data in the user account, the system needs to further check the validity and integrity of this data. The validity of the avatar data refers to whether the data can be used normally, such as whether the file is damaged, whether the data format is correct, etc. The integrity refers to whether the data contains all the elements required to generate the avatar, such as images, audio and other content.

[0105] The system can check whether the virtual image file is damaged through file verification algorithms (such as hash verification), or verify whether the file format is correct through metadata checks (such as file header information). The integrity of the data can also be confirmed through field flags in the database (such as data flags, whether the file path is valid).

[0106] The system determines whether the avatar data meets the current video generation requirements based on its validity and integrity. Different generation requirements may require different data features, such as high-definition avatar images, specific audio formats, etc. If the avatar data meets the current video generation requirements, the system will directly use the data for video generation; otherwise, the user is prompted to regenerate the avatar.

[0107] The system can check whether the avatar data meets certain conditions based on the parameter requirements when the video is generated. For example, for high-definition generation requirements, the system will check whether the image resolution meets the requirements, whether the format of the audio file matches the video generation standard, etc.

[0108] If the validity and integrity of the avatar data meet the requirements of video generation, the system determines that the user has generated an avatar and can directly use this data for video generation. At this time, there is no need to generate the avatar again, and the existing data can be directly called to proceed to the next step. The system marks this avatar data as available and records it in the task execution process. This status information will be passed to the subsequent video generation module to ensure that the generation process can proceed smoothly.

[0109] If the avatar data has problems with validity and integrity, or does not meet the specific requirements of video generation, the system will consider the data invalid and prompt the user to regenerate the avatar. At this time, the system will re-enter the avatar generation process to collect new data. Based on the inspection results, the system will determine that the current avatar data is unavailable and prompt the user to re-provide image, audio and other data. This can be achieved through interface prompts or automatically enter the avatar generation operation steps.

[0110] In one specific implementation, the system queries the avatar data records in the user account by accessing a cloud database or storage service. If relevant data exists, the system obtains and checks the validity of the data through an API (such as whether the file is lost or whether the storage time is too long). The data check includes the availability and timestamp of the avatar data file in the cloud. The system can dynamically check whether the stored avatar data meets the current requirements according to different video generation requirements (such as high resolution or low resolution), and select different versions of avatar data according to the requirements.

[0111] By checking the avatar data in the user account, the system can avoid repeated generation of avatars, save computing resources and time, and improve the efficiency of video generation. The validity and integrity of the avatar data are verified to ensure that the data can meet the video generation requirements, thereby improving the reliability and quality of the generation process. The combination of local storage, cloud storage or cache system can adapt to the application requirements of different scenarios and further improve the system response speed.

[0112] In one embodiment, the above S30 includes:

[0113] S301, using a convolutional neural network to extract features from the virtual character proxy image to obtain image features;

[0114] S302, extracting features from the audio data using a recurrent neural network to obtain audio features;

[0115] S303, combining the image feature and the audio feature through a feature fusion module to generate a fusion feature;

[0116] S304, adjusting the fusion feature to match the gender feature according to the gender of the avatar;

[0117] S305, generating a virtual image with natural expression and synchronized lip movements based on the adjusted fusion features.

[0118] In this embodiment, a convolutional neural network (CNN) is a deep learning model that is good at processing image data. The system uses CNN to extract features from the virtual character proxy image and extract key features related to the face, such as eyes, nose, mouth, contour, etc. These features will serve as the basis for subsequent virtual image generation to ensure that the generated virtual image is highly similar in appearance to the proxy image provided by the user.

[0119] The system uses pre-trained CNN models (such as VGG, ResNet, etc.) to forward propagate the proxy image and extract multi-layer feature maps. The convolution operation of each layer extracts different details from the image, such as edges and textures at low levels and complex structures and semantic information at high levels. The extracted features can be stored as vectors or feature maps for subsequent processing.

[0120] Recurrent Neural Networks (RNNs) are good at processing sequential data, and are particularly suitable for feature extraction of audio data. By analyzing audio data with RNNs, the system can extract temporal features in the audio, such as pitch, rhythm, and pitch changes, thereby obtaining the characteristics of the user's voice. These audio features will be used to generate the voice and lip sync of the avatar.

[0121] The audio data is first converted into a sequence data format suitable for RNN processing through preprocessing, such as Mel-frequency spectrogram or MFCC (Mel-frequency cepstral coefficients). Then, the system extracts audio features through a recurrent neural network (RNN, LSTM or GRU) to capture the time correlation information in the audio. Through the multi-layer RNN structure, the system can obtain a deep understanding of the audio data.

[0122] Image features and audio features reflect the appearance and sound of the avatar, respectively. In order to generate a coherent avatar, the system needs to fuse these features together. The feature fusion module is used to combine features from images and audio, so that the generated avatar is not only visually similar to the user, but also able to generate lip movements and expressions synchronously based on audio features.

[0123] Feature fusion is usually achieved through simple concatenation or more complex fusion mechanisms such as Attention. The concatenation operation combines image features and audio features into a vector and inputs it into the fusion module. More advanced fusion methods can ensure the coordination of image and audio features by weighting or dynamically adjusting the proportion of the two features.

[0124] The gender characteristics of the avatar will affect its appearance, movements, expressions, etc. Therefore, the system adjusts the fusion features according to the gender parameters so that the generated avatar matches the user's input in terms of gender characteristics. This step ensures that the avatar is personalized and matches gender requirements.

[0125] When generating an avatar, the system weights or adjusts some dimensions in the feature vector in combination with the gender parameter. For example, a male image may have thicker facial lines, while a female image may have softer facial features. By adjusting the gender-related dimensions of image and audio features, the system is able to generate an avatar with significant gender characteristics.

[0126] The final step of generating an avatar is to generate an avatar that can be synchronized with the audio based on the adjusted fusion features. The avatar not only has the visual appearance of the user, but also can present natural lip movements and expression synchronization according to the audio content.

[0127] The system uses a 3D modeling engine or a 2D animation engine to generate a visual representation of the avatar. Based on the timing characteristics of the audio, the system synchronously generates the avatar's lip shape changes to ensure that the avatar's lip shape and voice are synchronized when speaking. Expression generation analyzes the emotional information in the audio so that the avatar's facial expressions change with the changes in the voice.

[0128] In one specific implementation, an Attention mechanism is used to dynamically select important image and audio features during the feature fusion process. By adaptively weighting the feature vectors, the system can generate a virtual image that is more consistent with the user's appearance and voice characteristics. It is suitable for high-precision virtual image generation needs. Through the attention mechanism, the system can capture subtle facial expressions and voice changes, and achieve highly personalized generation.

[0129] In this embodiment, by using convolutional neural networks (CNN) and recurrent neural networks (RNN) to extract image and audio features, the system can generate a highly personalized virtual image with natural expressions and dynamic performance of synchronized lip movements. The feature fusion module ensures the consistency of the virtual image in vision and sound. Combined with the adjustment of gender characteristics, the system can generate a virtual image that meets the user's expectations and is suitable for a variety of application scenarios, such as virtual assistants, video generation, etc.

[0130] In one embodiment, the above S40 includes:

[0131] S401, importing the generated virtual image into a real-time rendering engine based on a graphics processing unit;

[0132] S402, configuring the background image, preset action sequence and scene parameters required for video generation, and transmitting them to the rendering engine through the graphic interface;

[0133] S403, starting the video rendering process and recording the unique identifier of the rendering task;

[0134] S404, periodically obtain the current progress information of the rendering task through the status query interface of the rendering engine until the rendering is completed and the generated result is obtained.

[0135] In this embodiment, during the video generation process, the system first imports the generated virtual image (including its 3D model or 2D image, audio data and motion features) into a real-time rendering engine driven by a graphics processing unit (GPU). The GPU is designed for efficient parallel computing and is suitable for processing a large number of graphics rendering tasks, such as animation, special effects, model operations, etc.

[0136] The system calls the API interface of the rendering engine (such as Unity, Unreal Engine or FFmpeg) to load the model, texture, skeleton animation and other data of the virtual image into the GPU memory for rendering. GPU rendering can speed up the real-time generation and processing of the virtual image and ensure high-quality video output. The graphics processing unit will generate animation effects in real time based on the action and expression characteristics of the virtual image.

[0137] After the avatar is imported into the rendering engine, the system needs to configure the video generation environment, including the background image, the action sequence of the avatar, and other scene parameters (such as lighting, camera angle, etc.). These configurations ensure that the avatar performs the specified actions in the appropriate scene and completes the video generation task.

[0138] The system configures the rendering environment through the graphics API (such as OpenGL, DirectX), and passes the background image, scene model, action animation and other data to the rendering engine. The preset action sequence can include the virtual character's walking, talking, expression changes, etc., and the scene parameters include lighting, light and shadow effects, scene objects, etc. The system needs to load this data into the rendering engine through the graphics interface and prepare to start the rendering process.

[0139] After a rendering task is started, the system assigns a unique identifier (ID) to each rendering task to distinguish different rendering tasks. These identifiers can be used to query the rendering progress, locate the rendering results, and handle subsequent operations of the rendering task.

[0140] The system starts video rendering by calling the rendering engine's task management interface. The rendering engine returns a unique task ID, which the system records in the task management module for subsequent task monitoring and status query. The task ID is usually a unique identifier that can be used to track task status and rendering results.

[0141] During the rendering process, the system needs to periodically query the current progress of the rendering task. By calling the status query interface of the rendering engine, the system can obtain real-time progress information of the rendering task (such as the current completion percentage, remaining time, etc.), and update the user interface or perform other operations based on this information. After the rendering is completed, the system will obtain the final generated result, that is, the rendered video file.

[0142] The system uses a scheduled task (such as background polling or callback mechanism) to periodically query the task status of the rendering engine and obtain rendering progress information. The rendering engine usually returns the task progress (such as percentage, current rendering frame number, task status, etc.) through the status interface, and the system displays a progress bar or other progress feedback information based on the returned information. When the rendering task is completed, the system will receive the rendering result file and perform subsequent processing.

[0143] In one embodiment, using distributed rendering technology, the avatar and scene data are distributed to multiple nodes for parallel rendering. Each rendering node processes a part of the scene, summarizes the rendering progress information through the scheduling system, and finally synthesizes the complete video. Distributed rendering can significantly improve the rendering speed, especially for complex scenes and large-scale video generation. The system periodically queries the rendering progress of each node, and after all nodes have completed the task, the data is merged to generate the final video.

[0144] In this embodiment, by importing the virtual image into the real-time rendering engine and configuring the scene and action sequence, the system can dynamically generate high-quality video content. The polling rendering progress function enables users to track the progress of video generation in real time, avoid unnecessary waiting, and ensure the transparency of task status.

[0145] In one embodiment, after the above S60, the method further includes:

[0146] S701, when an upload instruction is detected, executing an asynchronous download task to download the virtual image video;

[0147] S702, when the download is complete, save the virtual image video to the user's photo album, and perform an asynchronous upload task to upload the virtual image video to the target platform;

[0148] S703, during the task execution process, providing task progress feedback and completion prompts.

[0149] In this embodiment, after the user previews the avatar video, the system waits for the user to issue an upload instruction. When the upload instruction is detected, the system needs to perform an asynchronous download task, first downloading the generated avatar video from the server or cloud to the local device. Asynchronous download allows the user to continue to perform other operations during the download process, avoiding interface blocking or freezing.

[0150] The system schedules asynchronous download tasks through the background task manager or task queue, and uses HTTP or FTP protocol to download the generated avatar video from the cloud server. Asynchronous tasks can be implemented through multi-threading or multi-process technology, such as using Promise or async / await in JavaScript, and using NSURLSession (iOS) or AsyncTask (Android) in mobile applications. The system continuously monitors the task status during the download process to provide progress feedback to the user.

[0151] After the download is complete, the system will save the avatar video to local storage or the user's photo album for offline access. At the same time, the system will start another asynchronous task to upload the video to the specified target platform, such as social media, video sharing platforms, etc. This step is also performed asynchronously to ensure that the upload task is carried out in the background and the user can continue to use the application during this period.

[0152] The system first saves the downloaded video file to the user's photo album through the mobile device's file system API (such as Android's MediaStore or iOS's PhotoKit). After that, the system starts an asynchronous upload task and uses the target platform's API (such as RESTAPI, AWS S3, Google Drive, etc.) to perform the upload operation. The upload task usually includes a progress bar or other feedback mechanism to allow users to understand the progress and completion of the task.

[0153] During the execution of download and upload tasks, the system needs to provide users with real-time task progress feedback. For example, when downloading, display the percentage of downloaded data; when uploading, display the progress bar of the file transfer. When the task is completed, the system should inform the user of the completion status of the task through prompts (such as pop-up notifications or display prompt information).

[0154] Through the callback mechanism or polling mechanism, the system continuously obtains the progress of the task during the download and upload process and updates the progress bar or status indicator in the user interface. The progress of download and upload can be implemented through the onProgress callback function, and the progress information can be displayed to the user in the form of percentage or amount of data transferred. When the task is successfully completed, the system feedbacks the completion status of the task through the notification center, pop-up window or simple prompt information.

[0155] In one specific implementation, in a mobile application, after a user previews a video and clicks the upload button, the system executes an asynchronous task to download the video file to the device album, and then asynchronously uploads it to the target platform. The download and upload tasks are performed through a multi-threaded framework such as AsyncTask or WorkManager. To improve the user experience, the system can provide the function of executing multiple tasks simultaneously, such as allowing the user to continue to operate the application during the download process, and uploading videos to multiple platforms (such as social media and cloud storage) at the same time.

[0156] In another specific implementation, on the web page, after the user views the generated video through the preview page and clicks the upload button, the system executes the video download and upload tasks in the background, uses JavaScript's async / await and Promise to handle asynchronous tasks, and downloads and uploads files through HTTP requests. The web page can combine the WebSocket technology on the server side to provide users with real-time task progress feedback, such as sending progress information to users through WebSocket during the download or upload process to ensure that users can understand the file transfer status in real time.

[0157] In this embodiment, by asynchronously executing download and upload tasks, the system can quickly and efficiently process file transfer operations after the user previews the video without affecting the user's normal operating experience. The combination of asynchronous download and upload tasks avoids the problem that the user has to wait for the upload to complete before using other functions. Task progress feedback ensures that the user has real-time understanding of the task execution process, greatly improving the user experience. At the same time, background processing ensures the efficiency of the task and the optimization of system performance.

[0158] In one embodiment, in the above S10, receiving a user's video generation instruction includes:

[0159] S107, providing a variety of video generation options through a user interface, wherein the video generation options include different background themes, action styles, and audio types;

[0160] S108, receiving video generation parameters from the video generation option, and converting the video generation parameters into an instruction format to obtain a video generation instruction;

[0161] S109, performing format verification and parameter integrity check on the video generation instruction to verify the validity and integrity of the video generation instruction.

[0162] In this embodiment, the system needs to provide users with a variety of video generation options through a user interface (UI) so that users can customize the generated video content. These options may include:

[0163] Background theme: the background picture of the video, such as scenery, indoor scenes, virtual world, etc.

[0164] Action style: the behavior and movements of the virtual character, such as dancing, speaking, gestures, etc.

[0165] Audio Type: Voiceover or background audio of the avatar, including music, user-provided speech, or system-generated speech.

[0166] By selecting these options, users can personalize the generated video content.

[0167] Through the front-end user interface (such as mobile applications, web applications), the system can use graphical controls (such as drop-down menus, button groups, graphical previews, etc.) to allow users to select different video generation options. Front-end technology can be implemented using HTML5, CSS, JavaScript, or on the mobile side, native development frameworks (such as iOS UIKit, Android XML layout) can be used to display the interface. Background, motion, and audio options can be loaded from the back-end database or static resources and dynamically displayed to the user.

[0168] After the user selects the video generation option, the system needs to receive these generation parameters and convert them into a standardized instruction format. These parameters may include: selected background theme; type of action style; audio file or background sound type.

[0169] The system combines these user-input parameters into a complete video generation instruction to ensure that it can be understood and executed by subsequent video generation modules.

[0170] The system saves each video generation option selected by the user in the form of a key-value pair, and then serializes it into an instruction through JSON, XML or other standardized formats. This instruction may contain multiple fields, such as:

[0171] background:"office_scene",action:"walking",audio:"user_uploaded_voice.mp3". These data are passed to the backend system through the API communication between the frontend and the backend to trigger the video generation process.

[0172] After receiving the user's video generation instruction, the system needs to verify the format and parameters of the instruction. The verification content includes:

[0173] Format verification: Check whether the generated instructions conform to the predetermined format specifications (such as whether the JSON format structure is complete and whether the field name is correct).

[0174] Parameter integrity check: Ensure that the user provides all required parameters, such as background, motion, and audio. If necessary parameters are missing, the system will prompt the user to complete the input.

[0175] The system can verify the generated instructions through regular expressions or preset format templates. For parameter integrity check, it can be determined whether the option field selected by the user is empty or whether it meets the specific value range. The verification results can be fed back to the front end, such as prompting the user to enter the missing part through error prompts or form highlights.

[0176] In one specific implementation, the system allows the user to select video generation options through voice input. The voice recognition technology converts the user's voice instructions into text, and the system converts the recognized text into standardized instructions and passes it to the video generation module. For example, the user can say "select a beach background, dance moves, and play cheerful music". The system can combine natural language processing technology to intelligently parse the user's instructions. Even if the user does not input in a predetermined format, the system can infer the user's intention through the context and automatically generate corresponding instructions.

[0177] This embodiment provides users with a variety of video generation options, and the system achieves a high degree of personalization and flexibility in the video generation process. Users can customize video content through simple operations and generate virtual image videos that meet their needs. Parameter verification and integrity checks ensure the accuracy of video generation instructions, avoid errors or omissions in the generation process, and improve system reliability and user experience.

[0178] In one embodiment, after the above S50, the method further includes:

[0179] S801, if the virtual image video generation fails, prompt whether to regenerate the virtual image video;

[0180] S802: If a regeneration instruction is detected, the video generation process is restarted using the virtual image, and the generation progress is polled until the generation result is obtained.

[0181] In this embodiment, when the video generation process fails, the system needs to give an error prompt based on the generation result and provide the user with the option of regenerating the video. The reasons for the generation failure may include video rendering errors, server connection interruption, video parameter mismatch, etc. Prompting the user to regenerate helps solve failures caused by accidental problems and provides a better user experience.

[0182] The system determines whether the generation fails (for example, when the status code is "failure") by capturing the error information or the returned status code during the generation process. If the generation fails, the system pops up a dialog box or a notification to prompt the user, informing the user of the error information and asking the user whether to regenerate the video. The front end can use UI controls (such as pop-up windows and prompt boxes) to guide users to choose to regenerate the video. The specific implementation can be implemented using alerts in JavaScript or native dialog boxes in mobile applications.

[0183] If the user chooses to regenerate the video, the system needs to restart the video generation process. The system can use the previously generated avatar data to avoid the user from re-entering or selecting video generation parameters, simplifying the operation process. This step is intended to reduce the user's repeated operations and improve the user experience.

[0184] The system will retain previously generated avatar data (such as 3D models, animation parameters, background images, etc.) and directly use these cached data when the user chooses to regenerate. When implemented, the system will reload this data into the video generation engine and start a new round of video rendering process. Through the task scheduler or task management system (such as background service), the system can smoothly restart the video generation task.

[0185] During the process of regenerating the video, the system still needs to monitor the progress of the rendering task through a polling mechanism or a callback mechanism. Polling means that the system periodically queries the status of the video generation until the task is completed or fails. The user interface will provide visual feedback based on the progress information, such as a progress bar or status prompt, to ensure that the user can understand the progress of the task in real time.

[0186] The system periodically queries the status of the video generation task through a background process, and the polling frequency can be once per second or every few seconds. By calling the rendering engine's API, the system can obtain the current progress of the task (such as completion percentage, current frame number, etc.). The progress information is displayed to the user through the front-end interface, for example, by displaying the current progress through a progress bar. When the generation task is completed, the system will obtain the final generated video file and notify the user of the completion of the task.

[0187] By prompting the user to regenerate when generation fails, this embodiment enables the system to better handle problems that may occur during the generation process and avoid a decline in user experience due to unexpected errors. Reusing previously generated avatar data reduces the user's repeated operations and improves the convenience and efficiency of generation. By polling the generation progress, the user can understand the progress of the generation in real time, ensuring that the user can continue to receive feedback during the regeneration process.

[0188] In one embodiment, a virtual image video generation device is provided, and the virtual image video generation device corresponds to the virtual image video generation method in the above embodiment. Figure 3 , Figure 3 This is a functional module diagram of a preferred embodiment of the virtual image video generation device of the present invention. Generation instruction receiving module 10, virtual image generation preparation module 20, virtual image generation module 30, video generation module 40, generation result judgment module 50 and video preview module 60. The detailed description of each functional module is as follows:

[0189] A generation instruction receiving module 10 is used to receive a user's video generation instruction and determine whether a virtual image has been generated;

[0190] A virtual image generation preparation module 20 is used to determine the gender of the virtual image based on the video generation instruction if the virtual image is not generated, and receive audio data and a virtual character agent picture;

[0191] A virtual image generation module 30, for generating a virtual image based on the gender of the virtual image, audio data and a virtual character agent picture;

[0192] The video generation module 40 is used to start the video generation process using the virtual image and poll to obtain the generation progress until the generation result is obtained;

[0193] A generation result judgment module 50, used to judge whether the virtual image video is successfully generated based on the generation result;

[0194] The video preview module 60 is used to display a preview of the virtual image video if the virtual image video is successfully generated.

[0195] In one embodiment, the instruction receiving module 10 is generated, specifically for:

[0196] Check whether the generated avatar data exists in the user account;

[0197] If there is no generated virtual image data, it is deemed that no virtual image has been generated;

[0198] If there is generated avatar data, analyzing the validity and integrity of the avatar data;

[0199] Determining whether the virtual image data meets the requirements of current video generation based on the validity and integrity of the virtual image data;

[0200] If the avatar data meets the requirements for video generation, it is deemed that the avatar has been generated;

[0201] If the virtual image data does not meet the requirements for video generation, it is deemed that the virtual image has not been generated.

[0202] In one embodiment, the virtual image generation module 30 is specifically used to:

[0203] Using a convolutional neural network to extract features from the virtual character proxy image to obtain image features;

[0204] Using a recurrent neural network to extract features from the audio data to obtain audio features;

[0205] The image feature and the audio feature are combined and processed by a feature fusion module to generate a fusion feature;

[0206] According to the gender of the avatar, adjusting the fusion feature to match the gender feature;

[0207] A virtual image with natural expressions and synchronized lip sync is generated based on the adjusted fusion features.

[0208] In one embodiment, the video generation module 40 is specifically configured to:

[0209] importing the generated avatar into a real-time rendering engine based on a graphics processing unit;

[0210] Configure the background image, preset action sequence and scene parameters required for video generation, and pass them to the rendering engine through the graphical interface;

[0211] Start the video rendering process and record the unique identifier of the rendering task;

[0212] The current progress information of the rendering task is obtained regularly through the status query interface of the rendering engine until the rendering is completed and the generated result is obtained.

[0213] In one embodiment, the video preview module 60 is specifically used to:

[0214] When an upload instruction is detected, an asynchronous download task is executed to download the virtual image video;

[0215] When the download is complete, the virtual image video is saved in the user's photo album, and an asynchronous upload task is performed to upload the virtual image video to the target platform;

[0216] During the task execution, task progress feedback and completion prompts are provided.

[0217] In one embodiment, the instruction receiving module 10 is generated, specifically for:

[0218] providing a variety of video generation options through a user interface, the video generation options including different background themes, motion styles, and audio types;

[0219] Receiving video generation parameters from the video generation option, and converting the video generation parameters into an instruction format to obtain a video generation instruction;

[0220] Perform format verification and parameter integrity check on the video generation instruction to verify the validity and integrity of the video generation instruction.

[0221] In one embodiment, the generation result judgment module 50 is specifically used to:

[0222] If the virtual image video generation fails, prompt whether to regenerate the virtual image video;

[0223] If a regeneration instruction is detected, the video generation process is restarted using the virtual image, and the generation progress is polled until the generation result is obtained.

[0224] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 4 As shown. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external user terminal through a network connection. When the computer program is executed by the processor, it realizes the functions or steps of a virtual image video generation method server side.

[0225] In one embodiment, a computer device is provided. The computer device may be a user terminal, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it realizes the functions or steps of a user-side of a virtual image video generation method

[0226] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0227] receiving a video generation instruction from a user, and determining whether a virtual image has been generated;

[0228] If the avatar is not generated, determining the gender of the avatar based on the video generation instruction, and receiving audio data and an avatar agent image;

[0229] generating an avatar based on the gender of the avatar, the audio data, and the avatar proxy picture;

[0230] Using the virtual image to start the video generation process, and polling to obtain the generation progress until the generation result is obtained;

[0231] Determining whether the virtual image video is successfully generated based on the generation result;

[0232] If the virtual image video is generated successfully, a preview of the virtual image video is displayed.

[0233] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0234] receiving a video generation instruction from a user, and determining whether a virtual image has been generated;

[0235] If the avatar is not generated, determining the gender of the avatar based on the video generation instruction, and receiving audio data and an avatar proxy image;

[0236] generating an avatar based on the gender of the avatar, the audio data, and the avatar proxy picture;

[0237] Using the virtual image to start the video generation process, and polling to obtain the generation progress until the generation result is obtained;

[0238] Determining whether the virtual image video is successfully generated based on the generation result;

[0239] If the virtual image video is generated successfully, a preview of the virtual image video is displayed.

[0240] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can refer to the relevant descriptions on the server side and the user side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0241] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0242] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0243] It should be noted that if software tools or components other than those of the Company appear in the embodiments of the present application, they are only used for illustration and do not represent actual use. The above-described embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the above-mentioned embodiments, a person of ordinary skill in the art should understand that the technical solutions described in the above-mentioned embodiments can still be modified, or some of the technical features thereof can be replaced by equivalents; and these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.< / video>

Claims

1. A method for generating a virtual image video, characterized in that: The following steps are involved: receiving a video generation instruction from a user, and determining whether a virtual image has been generated; If the avatar is not generated, determining the gender of the avatar based on the video generation instruction, and receiving audio data and an avatar agent image; generating an avatar based on the gender of the avatar, the audio data, and the avatar proxy picture; Using the virtual image to start the video generation process, and polling to obtain the generation progress until the generation result is obtained; Determining whether the virtual image video is successfully generated based on the generation result; If the virtual image video is generated successfully, a preview of the virtual image video is displayed.

2. The method for generating a virtual image video according to claim 1, wherein: Determining whether a virtual image has been generated includes: Check whether the generated avatar data exists in the user account; If there is no generated virtual image data, it is deemed that no virtual image has been generated; If there is generated avatar data, analyzing the validity and integrity of the avatar data; Based on the validity and completeness, determining whether the avatar data meets the requirements of current video generation; If the avatar data meets the requirements for video generation, it is deemed that the avatar has been generated; If the virtual image data does not meet the requirements for video generation, it is deemed that the virtual image has not been generated.

3. The method for generating a virtual image video according to claim 1, wherein: Generating a virtual image based on the gender of the virtual image, the audio data, and the virtual character proxy picture, including: Using a convolutional neural network to extract features from the virtual character proxy image to obtain image features; Using a recurrent neural network to extract features from the audio data to obtain audio features; The image feature and the audio feature are combined and processed by a feature fusion module to generate a fusion feature; According to the gender of the avatar, adjusting the fusion feature to match the gender feature; A virtual image with natural expressions and synchronized lip sync is generated based on the adjusted fusion features.

4. The method for generating a virtual image video according to claim 1, wherein: The video generation process is started by using the virtual image, and the generation progress is obtained by polling until the generation result is obtained, including: importing the generated avatar into a real-time rendering engine based on a graphics processing unit; Configure the background image, preset action sequence and scene parameters required for video generation, and pass them to the rendering engine through the graphical interface; Start the video rendering process and record the unique identifier of the rendering task; Through the status query interface of the rendering engine, the current progress information of the rendering task is obtained regularly until the rendering is completed and the generated result is obtained.

5. The method for generating a virtual image video according to claim 1, wherein: If the virtual image video is successfully generated, after displaying the preview of the virtual image video, the method further includes: When an upload instruction is detected, an asynchronous download task is executed to download the virtual image video; When the download is complete, the virtual image video is saved in the user's photo album, and an asynchronous upload task is performed to upload the virtual image video to the target platform; During the task execution, task progress feedback and completion prompts are provided.

6. The method for generating a virtual image video as claimed in claim 1, characterized in that: Receive the user's video generation instructions, including: providing a variety of video generation options through a user interface, the video generation options including different background themes, motion styles, and audio types; Receiving video generation parameters from the video generation option, and converting the video generation parameters into an instruction format to obtain a video generation instruction; Perform format verification and parameter integrity check on the video generation instruction to verify the validity and integrity of the video generation instruction.

7. The method for generating a virtual image video according to claim 1, wherein: After determining whether the virtual image video is successfully generated based on the generation result, the method further includes: If the virtual image video generation fails, prompt whether to regenerate the virtual image video; If a regeneration instruction is detected, the video generation process is restarted using the virtual image, and the generation progress is polled until the generation result is obtained.

8. A virtual image video generating device, characterized in that: The virtual image video generating device comprises: A generation instruction receiving module is used to receive a user's video generation instruction and determine whether a virtual image has been generated; A virtual image generation preparation module is used to determine the gender of the virtual image based on the video generation instruction if the virtual image has not been generated, and receive audio data and a virtual character agent picture; A virtual image generation module, used to generate a virtual image based on the gender of the virtual image, audio data and a virtual character proxy picture; A video generation module, used to start the video generation process using the virtual image, and poll to obtain the generation progress until the generation result is obtained; A generation result judgment module, used to judge whether the virtual image video is successfully generated based on the generation result; The video preview module is used to display a preview of the virtual image video if the virtual image video is successfully generated.

9. A computer device, characterized in that: The computer device includes a memory, a processor, and a virtual image video generation program stored in the memory and executable on the processor, and the virtual image video generation program, when executed by the processor, implements the steps of the virtual image video generation method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: The storage medium stores a virtual image video generation program, which, when executed by a processor, implements the steps of the virtual image video generation method according to any one of claims 1 to 7.

Images