Video generation method, device and system
By using dynamic models to simulate the physical characteristics and motion trajectory of objects in the video generation method, the problem of the difference between the motion effect of moving objects in the video and the real environment is solved, and the authenticity and rationality of moving objects in the video is improved.
Patent Information
- Application Number
- CN202410878201.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-28
- Publication Date
- 2024-11-12
AI Technical Summary
There is a difference between the movement effect of the moving object in the video generated in the prior art and the movement effect of the object in the real environment, resulting in unnatural and unrealistic movement.
By obtaining material in text, images or videos, using dynamic models to simulate the physical characteristics and motion trajectory of an object, generating videos that match the motion effects of an object in a real environment. The dynamic model includes the dynamic model of the object and the dynamic model of the units in the object, which is used to improve the motion authenticity of moving objects in video.
It enhances the rationality and physical characteristics of the moving objects in the video, improves the authenticity of the moving objects, and makes the generated video effect more in line with the motion performance in the real environment.
Smart Images

Figure CN118945443A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of artificial intelligence, and in particular to a video generation method, device and system. Background Art
[0002] At present, with the development of artificial intelligence, breakthrough progress has been made in the multimedia field in generating videos based on text, images, and videos. However, the motion effect of moving objects in the generated videos is different from the motion effect of objects in the real environment. For example, the motion of moving objects in the generated videos is unnatural and unreal. Therefore, how to improve the motion authenticity of moving objects in the generated videos is an urgent problem to be solved. Summary of the invention
[0003] The present application provides a video generation method, device and system, thereby improving the motion authenticity of moving objects in the generated video.
[0004] In a first aspect, a video generation method is provided, comprising acquiring a material including at least one of text, an image, or a video, and displaying a video in response to a first operation, wherein the video is generated by a dynamic model of an object in the material and a dynamic model of a unit in the object. The dynamic model is used to indicate a motion representation of a simulated physical property of an object and a relative position of the object in space changing over time.
[0005] Since the dynamic model simulates the physical characteristics of the object's movement in the real environment, the movement effect of the object in the video generated based on the dynamic model of the object and the dynamic model of the unit in the object is consistent with the movement effect of the object in the real environment, and the movement effect of the unit in the object is consistent with the movement effect of the unit in the object in the real environment, which enhances the rationality of the movement and the rationality of the physical characteristics of the object in the generated video, thereby effectively improving the movement authenticity of the moving object in the generated video.
[0006] In one possible implementation, the dynamic model matched by the first object and / or the dynamic model matched by the first unit are selected by the user; before displaying the video, the method also includes: displaying the dynamic models matched by multiple objects contained in the material and / or the dynamic models matched by one or more units in the multiple objects, the dynamic models matched by the multiple objects include the dynamic model matched by the first object, and the dynamic models matched by one or more units of the first object include the dynamic model matched by the first unit.
[0007] In another possible implementation, the parameters of the dynamic model matched by the first object and / or the parameters of the dynamic model matched by the first unit are selected by the user; the parameters include properties of the object; before displaying the video, the method also includes: displaying the parameters of the matched dynamic model, the parameters of the matched dynamic model include the parameters of the dynamic model matched by the first object and / or the parameters of the dynamic model matched by the first unit.
[0008] In another possible implementation, the parameter also includes at least one of environmental information or exercise duration.
[0009] By displaying the dynamic model and parameters so that the user can select the dynamic model and parameters, the motion effect of the moving object in the video generated according to the dynamic model and parameters selected by the user meets the user's preferences, thereby improving the user experience.
[0010] In another possible implementation, the first object and / or the first unit are selected by the user; before displaying the video, the method further includes: displaying multiple objects contained in the material and one or more units in the multiple objects, the multiple objects including the first object, and one or more units of the first object including the first unit.
[0011] By displaying objects and the units contained in the objects, users can intuitively see the objects and units provided by the material. When users select objects and / or units, they can realize the movement of any objects and / or units, thereby improving the creativity of the video and enhancing the user experience.
[0012] In another possible implementation manner, the method further includes: generating a video according to the material, the dynamic model matched by the first object, and the dynamic model matched by the first unit included in the first object.
[0013] In another possible implementation, a video is generated based on the material, the dynamic model matching the first object, and the dynamic model matching the first unit contained in the first object, including: generating the motion trajectory of the first object based on the first object, the dynamic model matching the first object, and the corresponding parameters, and generating the motion trajectory of the first unit based on the first unit contained in the first object, the dynamic model matching the first unit contained in the first object, and the corresponding parameters; generating a video based on the material, the motion trajectory of the first object, and the motion trajectory of the first unit.
[0014] By deconstructing the object to obtain the units contained in the object, and generating a video based on the motion trajectory generated by the dynamic model of object matching and the motion trajectory generated by the dynamic model of unit matching, the running effect of the object in the video is consistent with the motion effect of the object in the real environment, and the running effect of the unit in the object is consistent with the motion effect of the unit in the object in the real environment, which enhances the motion rationality and physical property rationality of the object in the generated video. Therefore, the goal of overall and local reasonable and precise motion is achieved, and the motion authenticity of the moving object in the generated video is effectively improved.
[0015] In another possible implementation, the motion trajectory of the first object is generated according to the first object, the dynamic model matched by the first object, and the corresponding parameters, and the motion trajectory of the first unit is generated according to the first unit contained in the first object, the dynamic model matched by the first unit contained in the first object, and the corresponding parameters, including: calling an interface associated with a video generation system and a trajectory generation system, generating the motion trajectory of the first object according to the first object, the dynamic model matched by the first object, and the corresponding parameters, and generating the motion trajectory of the first unit according to the first unit contained in the first object, the dynamic model matched by the first unit contained in the first object, and the corresponding parameters.
[0016] By setting an interface associated with the video generation system and the trajectory generation system, any application with a video generation function can obtain the motion trajectory of the object and the motion trajectory of the unit generated by the trajectory generation system by calling the interface, thereby effectively improving the motion authenticity of the moving object in the generated video and the versatility of the solution provided by the present application.
[0017] In another possible implementation, generating a video according to the material, the motion trajectory of the first object, and the motion trajectory of the first unit includes: generating a video according to the material, the motion trajectory of the first object, the motion trajectory of the first unit, and the depth relationship and / or perspective of at least two objects included in the first object.
[0018] Generate videos based on the depth relationship or perspective of objects so that the movement effects of objects in the video are consistent with the movement effects of objects in the real environment, thereby improving the movement authenticity of moving objects in the generated video.
[0019] In another possible implementation, the method also includes: receiving a second operation of setting video parameters of a video; generating a video based on the material, the dynamic model matched by the first object, and the dynamic model matched by the first unit contained in the first object, including: generating a video based on video parameters, the material, the dynamic model matched by the first object, and the dynamic model matched by the first unit contained in the first object.
[0020] Generate videos that meet user requirements based on the video parameters set by the user, improve the innovation of the video and enhance the user experience.
[0021] In a second aspect, a video generation device is provided, the video generation device comprising modules for executing the method in the first aspect or any possible design of the first aspect. For example, the model training device comprises a communication module, a processing module and a display module.
[0022] The communication module is used to obtain a material including at least one of text, image or video, and the display module is used to respond to the first operation and display a video, wherein the video is generated by a dynamic model of an object in the material and a dynamic model of a unit in the object. The dynamic model is used to indicate the simulated physical characteristics of an object and the motion representation of the relative position of the object in space changing over time.
[0023] In one possible implementation, the dynamic model matched by the first object and / or the dynamic model matched by the first unit are selected by the user; the display module is also used to: display the dynamic models matched by multiple objects contained in the material and / or the dynamic models matched by one or more units in the multiple objects, the dynamic models matched by the multiple objects include the dynamic model matched by the first object, and the dynamic models matched by one or more units of the first object include the dynamic model matched by the first unit.
[0024] In another possible implementation, the parameters of the dynamic model matched by the first object and / or the parameters of the dynamic model matched by the first unit are selected by the user; the parameters include properties of the object; the display module is also used to: display the parameters of the matched dynamic model, and the parameters of the matched dynamic model include the parameters of the dynamic model matched by the first object and / or the parameters of the dynamic model matched by the first unit.
[0025] In another possible implementation, the parameter also includes at least one of environmental information or exercise duration.
[0026] In another possible implementation, the first object and / or the first unit is selected by the user; the display module is further used to: display multiple objects contained in the material and one or more units of the multiple objects, the multiple objects include the first object, and one or more units of the first object include the first unit.
[0027] In another possible implementation, the processing module is used to generate a video according to the material, the dynamic model matched by the first object, and the dynamic model matched by the first unit included in the first object.
[0028] In another possible implementation, the processing module is specifically used to: generate a motion trajectory of the first object according to the first object, a dynamic model matched by the first object, and its corresponding parameters, and generate a motion trajectory of the first unit according to the first unit contained in the first object, a dynamic model matched by the first unit contained in the first object, and its corresponding parameters; generate a video according to the material, the motion trajectory of the first object, and the motion trajectory of the first unit.
[0029] In another possible implementation, the processing module is specifically used to: call the interface associated between the video generation system and the trajectory generation system, generate the motion trajectory of the first object according to the first object, the dynamic model matched by the first object and its corresponding parameters, and generate the motion trajectory of the first unit according to the first unit included in the first object, the dynamic model matched by the first unit included in the first object and its corresponding parameters.
[0030] In another possible implementation manner, the processing module is specifically configured to generate a video according to the material, the motion trajectory of the first object, the motion trajectory of the first unit, and the depth relationship and / or viewing angle of at least two objects included in the first object.
[0031] In another possible implementation, the processing module is further used to: receive a second operation of setting video parameters of a video; and generate a video according to the video parameters, the material, the dynamic model matched by the first object, and the dynamic model matched by the first unit included in the first object.
[0032] In a third aspect, an electronic device is provided, comprising a processor, a memory and a display; the processor is used to execute instructions stored in the memory so that the electronic device performs the operating steps of the method in the first aspect or any possible implementation of the first aspect.
[0033] In a fourth aspect, a data processing system is provided, comprising a client and a server; the client and the server are used to execute the operation steps of the method in the first aspect or any possible implementation of the first aspect, so that the client displays a video generated based on the material.
[0034] In a fifth aspect, a computer-readable storage medium is provided, comprising: computer software instructions; when the computer software instructions are executed in a processor, the processor executes the operating steps of the method described in the first aspect or any possible implementation of the first aspect.
[0035] In a sixth aspect, a computer program product is provided. When the computer program product is run on a computer, the computer is caused to execute the operation steps of the method described in the first aspect or any possible implementation manner of the first aspect.
[0036] The technical effects brought about by any design method in the second to sixth aspects can refer to the technical effects brought about by the first aspect or different design methods in the first aspect, and will not be repeated here.
[0037] Based on the implementations provided in the above aspects, this application can also be further combined to provide more implementations. BRIEF DESCRIPTION OF THE DRAWINGS
[0038] Figure 1 A schematic diagram of the structure of an electronic device provided in this application;
[0039] Figure 2 A schematic diagram of a process of video generation method provided in this application;
[0040] Figure 3 A flowchart of another video generation method provided by this application;
[0041] Figure 4 A flowchart of another video generation method provided by this application;
[0042] Figure 5 A flowchart of another video generation method provided by this application;
[0043] Figure 6 A schematic diagram of a video generation interface provided by this application;
[0044] Figure 7 A schematic diagram of an object and unit selection interface provided by the present application;
[0045] Figure 8 A schematic diagram of a kinetic model selection interface provided in this application;
[0046] Fig. 9 A schematic diagram of a parameter selection interface for a kinetic model provided in this application;
[0047] Fig.10 A schematic diagram of another video generation interface provided for this application;
[0048] Fig.11 A schematic diagram of the structure of a data processing system provided for this application;
[0049] Fig.12 A schematic diagram of the structure of a video generation device provided in this application. DETAILED DESCRIPTION
[0050] To facilitate understanding, the main terms involved in this application are first explained.
[0051] Generative AI (Artificial Intelligence Generated Content, AIGC): refers to the use of artificial intelligence algorithms to generate content with certain creativity and quality. Through training models and learning from large amounts of data, AIGC can generate content related to it based on input conditions or guidance. For example, AIGC refers to the generation of videos based on content described by at least one of text, images, audio, or video. The AIGC model can be embedded in user-generated content (UGC) applications such as short video applications (Application, APP) to allow users to create videos.
[0052] In order to solve the problem that the motion effect of the moving object in the generated video is different from the motion effect of the object in the real environment, the video generation method provided by the present application generates a video based on the dynamic model of the object in the material and the dynamic model of the unit in the object. The dynamic model is used to indicate the simulated physical characteristics of the object and the motion representation of the relative position of the object in space changing over time. Since the dynamic model simulates the physical characteristics of the object's motion in a real environment, the running effect of the object in the video generated based on the dynamic model of the object and the dynamic model of the unit in the object conforms to the motion effect of the object in the real environment, and the running effect of the unit in the object conforms to the motion effect of the unit in the object in the real environment, which enhances the motion rationality of the object in the generated video and the rationality of the physical characteristics, thereby effectively improving the motion authenticity of the moving object in the generated video.
[0053] The video generation method provided in this application can support any set dynamic models and simulation scenes, such as the two-dimensional world, the microscopic world, the macroscopic celestial body movement and the car movement.
[0054] The video generation method provided in the present application can be applied to electronic devices. The electronic device can be a mobile phone, a tablet computer, a notebook computer, a desktop, a laptop, a handheld computer, an ultra-mobile personal computer (UMPC), a netbook, and a personal digital assistant (PDA) and other devices including a display screen. The present application does not impose any special restrictions on the specific form of the electronic device.
[0055] Please refer to Figure 1 , is a schematic diagram of the structure of an electronic device provided by this application. Figure 1As shown, the electronic device includes: a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a power management module 140, an antenna, a wireless communication module 160, an audio module 170, a speaker 170A, a speaker interface 170B, a microphone 170C, a sensor module 180, a button 190, an indicator 191, a display screen 192, and a camera 193, etc. Among them, the above-mentioned sensor module 180 may include sensors such as a distance sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, a touch sensor, and an ambient light sensor.
[0056] The structure illustrated in this embodiment does not constitute a specific limitation on the electronic device. In other embodiments, the electronic device may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.
[0057] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.
[0058] In an embodiment of the present application, the processor 110 is used to generate a video based on the material, the dynamic model matched by the object, and the dynamic model matched by the unit contained in the object.
[0059] In some embodiments, after the processor 110 obtains the material, it identifies the smallest atomic motion unit in the material that cannot be further subdivided. For example, the object contained in the material is identified, and the smallest unit contained in the object that cannot be further subdivided is identified. For example, the object is a car, and the units contained in the object are wheels, body, windows, etc. The object has an independent dynamic model and the unit has an independent dynamic model.
[0060] After the object and the unit are identified, the dynamic model of the object is matched with the dynamic model of the unit, as well as the parameters of the dynamic model. For example, the dynamic model that matches the object identified in the material is determined based on the relationship between the object and the dynamic model set by the system; or the user-defined dynamic model is determined as the dynamic model that matches the object identified in the material.
[0061] The video is generated according to the material, the dynamic model matched by the object, and the dynamic model matched by the unit contained in the object. For example, the motion trajectory of the object and the motion trajectory of the unit can be generated according to the object, the unit, the matched dynamic model and the parameters, and the video is generated according to the material, the motion trajectory of the object and the motion trajectory of the unit.
[0062] The controller can be the nerve center and command center of the electronic device. The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of fetching and executing instructions.
[0063] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.
[0064] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, and / or a USB interface, etc.
[0065] The power management module 140 is used to connect to a power source. The power management module 140 may also be connected to the processor 110, the internal memory 121, the display screen 192, the camera 193, the wireless communication module 160, etc. The power management module 140 receives input of a power source and supplies power to the processor 110, the internal memory 121, the display screen 192, the camera 193, the wireless communication module 160, etc. In some embodiments, the power management module 140 may also be disposed in the processor 110.
[0066] The wireless communication function of the electronic device can be realized through the antenna and the wireless communication module 160. Among them, the wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) network), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication technology (NFC), infrared technology (IR), etc. applied to the electronic device.
[0067] The wireless communication module 160 may be one or more devices that integrate at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via an antenna, modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110. The wireless communication module 160 may also receive a signal to be sent from the processor 110, modulate the frequency of the signal, amplify the signal, and convert it into electromagnetic waves for radiation through the antenna. In some embodiments, the antenna of the electronic device is coupled to the wireless communication module 160, so that the electronic device can communicate with the network and other devices through wireless communication technology.
[0068] In the embodiment of the present application, the electronic device can obtain the material required to generate the video through the wireless communication module 160. In other embodiments, the electronic device can upload the material to the server through the wireless communication module 160, and obtain the video generated according to the material, the dynamic model matched by the object, and the dynamic model matched by the unit contained in the object from the server.
[0069] The electronic device implements the display function through a GPU, a display screen 192, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 192 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.
[0070] The display screen 192 is used to display text, images, and videos. The display screen 192 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light emitting diode (QLED), etc.
[0071] In the embodiment of the present application, the display screen 192 is used to display materials, objects in the materials, units in the objects, dynamic models, parameters, generated videos, etc., so that users can select their favorite content.
[0072] The electronic device can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 192 and application processor, etc. ISP is used to process the data fed back by camera 193. In some embodiments, ISP can be set in camera 193.
[0073] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then transmits the electrical signal to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device may include 1 or N cameras 193, where N is a positive integer greater than 1. For example, this embodiment does not limit the position of the camera 193 on the electronic device.
[0074] Alternatively, the electronic device may not include a camera, that is, the camera 193 is not provided in the electronic device (such as a television). The electronic device may be connected to an external camera 193 via an interface (such as a USB interface 130). The external camera 193 may be fixed to the electronic device via an external fixing member (such as a camera bracket with a clip). For example, the external camera 193 may be fixed to the edge of the display screen 192 of the electronic device, such as the upper edge, via an external fixing member.
[0075] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and continuously self-learn. NPU can realize applications such as intelligent cognition of electronic devices, such as image recognition, facial recognition, voice recognition, text understanding, etc.
[0076] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function. For example, files such as text, images, and videos are stored in the external memory card.
[0077] The internal memory 121 can be used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the electronic device by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data (such as audio data, etc.) created during the use of the electronic device, etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.
[0078] The electronic device can implement audio functions through the audio module 170, the speaker 170A, the microphone 170C, the speaker interface 170B, and the application processor, etc. For example, music playback, recording, etc. In the present application, the microphone 170C can be used to receive the voice instructions issued by the user to the electronic device. The speaker 170A can be used to feedback the decision instructions of the electronic device to the user.
[0079] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be arranged in the processor 110, or some functional modules of the audio module 170 can be arranged in the processor 110. The speaker 170A, also called "speaker", is used to convert audio electrical signals into sound signals. The microphone 170C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals.
[0080] The speaker interface 170B is used to connect a wired speaker. The speaker interface 170B may be a USB interface 130, or a 3.5 mm open mobile terminal platform (OMTP) standard interface or a cellular telecommunications industry association of the USA (CTIA) standard interface.
[0081] The key 190 includes a power key, a volume key, etc. The key 190 can be a mechanical key or a touch key. The electronic device can receive key input and generate key signal input related to user settings and function control of the electronic device.
[0082] The indicator 191 may be an indicator light, which may be used to indicate that the electronic device is in a power-on state, a standby state, or a power-off state, etc. For example, if the indicator light is off, it may indicate that the electronic device is in a power-off state; if the indicator light is green or blue, it may indicate that the electronic device is in a power-on state; if the indicator light is red, it may indicate that the electronic device is in a standby state.
[0083] It is understandable that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device. Figure 1 More or fewer components shown in the figure may be combined with two or more components, or may have different component configurations. For example, the electronic device may also include components such as a speaker. Figure 1 The various components shown in the drawings may be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing or application specific integrated circuits.
[0084] The methods in the following embodiments can all be implemented in an electronic device having the above hardware structure. In the following embodiments, the methods in the embodiments of the present application are described by taking the above electronic device being a mobile phone as an example.
[0085] The video generation method provided by the present application will be described in detail below with reference to the accompanying drawings.
[0086] Figure 2 A schematic diagram of the process flow of the video generation method provided in this application. Figure 1 The electronic device shown in the figure generates a video as an example. Figure 2 As shown, the following steps are included.
[0087] Step 210: Get materials.
[0088] The electronic device can obtain the material from the local storage of the electronic device or obtain the material from the storage outside the electronic device through the network. For example, the electronic device downloads the material from the network or obtains the material from other devices through wireless communication technology (such as Bluetooth, WIFI, etc.). This application does not limit the source of the material and the way in which the electronic device obtains the material. The following embodiments Figure 6 According to the description, a user can operate an electronic device to enable the electronic device to obtain materials.
[0089] The material includes at least one of text, image or video.
[0090] Step 220: Generate a video according to the material, the dynamic model matched by the first object, and the dynamic model matched by the first unit included in the first object.
[0091] Each material can contain one or more objects. For example, any of text, images, or videos can contain one or more objects.
[0092] According to the type classification, objects include but are not limited to animals, plants, and objects. For example, animals include land animals, marine life, birds, insects, etc. Plants include trees, grass, and flowers, etc. Objects include cars, buildings, and items, etc.
[0093] According to the classification of motion state, objects include moving objects and stationary objects. For example, moving objects include cars in motion, running animals, and plants blown by the wind. Stationary objects include buildings and displayed objects.
[0094] The multiple objects in the material described in the embodiment of the present application may include moving objects and / or stationary objects.
[0095] This application does not limit the type and number of objects contained in the material.
[0096] An object can contain one or more units. For example, if the object is a car, the units in the object include the body, wheels, and windows, etc. If the object is a tree, the units in the object include the trunk, branches, and leaves, etc.
[0097] In some embodiments, first, the object contained in the material is identified, and the unit contained in the object is identified, and then the video is generated according to the dynamic model matched by the object and the dynamic model matched by the unit. For example, step 220 includes the following steps 221 to 223.
[0098] Step 221: Identify the objects contained in the material and the units contained in the objects.
[0099] One or more objects contained in the material and one or more units contained in the object are identified according to the model.
[0100] In some embodiments, objects in the material and units in the object are respectively identified based on a plurality of single-task models.
[0101] For example, the material is text or image. The material is input into the first segmentation model, and the object contained in the material is output. The object is input into the second segmentation model, and the unit contained in the object is output.
[0102] Optionally, after identifying the object in the material, the unit of the object is determined by querying the corresponding knowledge graph according to the object. A knowledge graph is a knowledge base of a semantic network, that is, a knowledge base with a directed graph structure. The knowledge graph includes entities, relationships, and attributes. For example, if the object is a car, the knowledge graph of the car includes the various parts of the car, such as the car body, windows, wheels, etc., as well as the attributes of each part and the relationship between each part.
[0103] For another example, the material is a video. The objects contained in the video are identified based on the optical flow method. The optical flow method (optical flow or optical flow) uses the changes in the pixels in the image sequence in the time domain and the correlation between adjacent frames to find the corresponding relationship between the historical frame and the current frame, thereby calculating the motion information of the object between adjacent frames. For example, the basic principle of detecting moving objects: assign a velocity vector to each pixel in the image to form an image motion field. According to the velocity vector characteristics of each pixel, the image can be dynamically analyzed. If there is no moving object in the image, the optical flow vector changes continuously in the entire image area. When there is a moving object in the image, the velocity vector formed by the moving object is different from the velocity vector of the neighborhood background, so the moving object and its position are detected.
[0104] Optionally, environmental information in the material, depth relationships between multiple objects, motion patterns of objects, motion patterns of units, etc. are also identified based on other single-task models. For example, the material is text, image, or video. The material is input into the depth estimation model, and the depth relationship between multiple objects in the material is output. The depth relationship may refer to the positional relationship between multiple objects.
[0105] Movement modes include any movement such as running, jumping, walking or flying.
[0106] The environmental information includes at least one of weather, season, longitude and latitude, road, topography, slope or city.
[0107] In other embodiments, objects in the material, units in the object, depth relationships, motion patterns, environmental information, etc. are identified based on the multi-task model. For example, the material is text, image, or video. The material is input into the multi-task model, and accurate structural information such as objects contained in the material, units contained in the object, motion patterns, depth relationships, environmental information, etc. is output.
[0108] Step 222: Generate the motion trajectory of the object and the motion trajectory of the unit.
[0109] After identifying the objects contained in the material and the units contained in the objects, the dynamic model and parameters of the object matching, as well as the dynamic model and parameters of the unit matching, are determined, and the dynamic model and parameters of the object matching, as well as the dynamic model and parameters of the unit matching are input into the trajectory generation system to obtain the motion trajectory of the object and the motion trajectory of the unit.
[0110] The parameters include the name of the dynamic model and the properties of the moving object. For example, the properties of the moving object include intrinsic properties and motion properties. Intrinsic properties include physical characteristics such as weight, mass, and density. Physical characteristics refer to the properties of a substance without chemical reactions. Motion properties include degree of freedom parameters, etc. The degree of freedom parameters include at least one of the speed of movement, acceleration, rolling resistance coefficient, ground friction coefficient, air resistance, moment of inertia, or heading angle, etc.
[0111] In some embodiments, the dynamics model and parameters recommended by the system are determined as the dynamics model and parameters corresponding to the identified object.
[0112] For example, the system pre-configures a first correspondence relationship of a dynamic model corresponding to an object and a second correspondence relationship of a dynamic model corresponding to a unit. After the electronic device recognizes the object and the unit, it queries the first correspondence relationship according to the recognized object, and determines the dynamic model and parameters corresponding to the recognized object from the first correspondence relationship; it queries the second correspondence relationship according to the recognized unit, and determines the dynamic model and parameters corresponding to the recognized unit from the second correspondence relationship.
[0113] In other embodiments, the system recommends multiple dynamic models and parameters, and determines the dynamic model and parameters selected by the user as the dynamic model and parameters corresponding to the identified object.
[0114] For example, the electronic device can display one or more of the following contents: multiple objects contained in the material, one or more units in the multiple objects, dynamic models matched by the multiple objects, dynamic models matched by one or more units in the multiple objects, and parameters of the matched dynamic models, so that the user can intuitively see the data required to generate the video. The user can also select or set each item according to his or her preferences to generate a video with a customized effect. For example, a static object in a real environment can be made to move to generate an interesting animated video. For example, the display content of the electronic device and the user's interface operation are as follows: Figure 6 's explanation.
[0115] Optionally, the depth relationship of objects in the material, multiple perspectives of the objects, motion patterns and environmental information can also be identified. The identified depth relationship, motion pattern and environmental information can also be used to infer the dynamic model and parameters used to generate the video.
[0116] Since the depth relationship can refer to the positional relationship between multiple objects, and since the size and speed of objects at different positions are different, when generating motion trajectories, the depth relationship of multiple objects is used as a reference factor to make the motion effect of objects in the video more realistic. For example, objects and units at different depths can be identified to achieve object occlusion effects.
[0117] The motion effects of objects are different under different viewing angles. When generating motion trajectories, the different viewing angles of multiple objects are used as reference factors to make the motion effects of objects in the video more realistic.
[0118] The motion effects of objects in different environments are also different. For example, the motion effect of a car driving on a smooth road is different from that of a car driving on a rough road. When generating motion trajectories, the different environments in which multiple objects are located are used as reference factors to make the motion effects of objects in the video more realistic.
[0119] The present application does not limit the motion trajectory generation algorithm. A dynamic model can be selected first according to the method provided in the embodiment of the present application, and a motion trajectory can be generated by referring to a traditional trajectory generation algorithm. The motion trajectory can be uneven and continuous.
[0120] In addition, since the data formats processed by the trajectory generation systems of different objects or units are different, an interface can be established between the video generation system and the trajectory generation system, so that the trajectory generation system can be adapted to the video generation system through the interface, shielding the complexity of multiple trajectory generation systems. The adaptation layer is used to convert the format of the object and the format of the conversion unit, so that the data of the generated video can adapt to the trajectory generation system.
[0121] For example, Figure 3As shown, in the process of video generation, the video generation system can call the interface to run the trajectory generation system corresponding to the object or unit, convert the format of the data such as the motion mode, the parameters of the dynamic model and the environmental information through the adaptation layer, input the format-converted data into the trajectory generation system, and obtain the motion trajectory of the object or the motion trajectory of the unit. Then, the motion trajectory of the object or the motion trajectory of the unit is input into the video generation system so that the video generation system can generate the video.
[0122] The present application does not limit the specific form of the data format. For example, data can be input into the video generation system in the form of a key-value pair or text. Key represents the parameter name, and Value represents the parameter value. The data format includes but is not limited to JSON, YAML, Config, text, and other data formats.
[0123] Step 223: Generate a video according to the motion trajectory of the material, the object and the motion trajectory of the unit.
[0124] The motion trajectory of the material, the object and the unit are input into the video generation system to generate the video. For example, the video generation system includes at least one of the diffusion models, transformer, DiT or SiT.
[0125] For example, the objects and units used to generate the video can be selected by the user. The motion trajectory of the first object is generated according to the first object, the dynamic model matched by the first object and its corresponding parameters, and the motion trajectory of the first unit is generated according to the first unit contained in the first object, the dynamic model matched by the first unit contained in the first object and its corresponding parameters; the video is generated according to the material, the motion trajectory of the first object and the motion trajectory of the first unit. Among them, the first object and / or the first unit are selected by the user. The dynamic model matched by the first object and / or the dynamic model matched by the first unit are selected by the user. The parameters of the dynamic model matched by the first object and / or the parameters of the dynamic model matched by the first unit are selected by the user. For user selection of objects, units, dynamic models and parameters, please refer to the following embodiments Fig. 9 's explanation.
[0126] In some embodiments, video parameters of the video are set. The video parameters include at least one of video size, video duration, or frame rate. The video is generated according to the video parameters, the material, the dynamic model of the object matching, and the dynamic model of the unit matching contained in the object. For example, Figure 3 As shown, after the motion trajectory is generated, the motion trajectory and video parameters are input into the video generation system.
[0127] Optionally, the trajectory generation system can also be embedded in the video generation system, and the material, the dynamic model and parameters of object matching, and the dynamic model and parameters of unit matching are input into the video generation system. The video generation system first generates the object's motion trajectory and the unit's motion trajectory based on the material, the dynamic model and parameters of object matching, and the dynamic model and parameters of unit matching, and then generates a video based on the object's motion trajectory and the unit's motion trajectory.
[0128] Step 230: Display the video.
[0129] After the video is generated, the video is displayed so that the user can intuitively see the video generated based on the material, the dynamic model matched by the first object, and the dynamic model matched by the first unit included in the first object.
[0130] The video generation method provided in the present application provides a simulation enhancement method in the denoising stage of the video generation system, that is, inputting an accurate motion trajectory generated according to a dynamic model. For example, Figure 4 As shown, at each network layer, the precise motion trajectory of the current frame is referenced, including but not limited to the geometric shape of the unit, spatial coordinate geometry, motion direction and other simulated trajectory information. The precise motion trajectory does not change at each step of denoising, maintaining the accuracy of the motion position and the moving object.
[0131] Compared with the traditional video generation model that generates videos freely according to sample data, the generated videos appear unreal. The video generation method provided by the present application uses the motion trajectory of the object and the motion trajectory of the unit generated based on the dynamic model of the object and the dynamic model of the unit in the object as the dynamic model simulates the physical characteristics of the object's motion in a real environment, and uses the motion trajectory as a condition for generating videos to achieve high-precision motion control functions, so that the running effect of the object in the generated video conforms to the motion effect of the object in the real environment, and enhances the motion rationality and physical characteristic rationality of the object in the generated video, thereby effectively improving the motion authenticity of the moving object in the generated video.
[0132] Figure 5 This application provides a flow chart of another video generation method. Here, the object is a video as an example for explanation. Figure 2 The difference between the method embodiments shown is that, in step 510, the object is detected based on the optical flow method to obtain the motion trajectory of the object. For example, the outline, speed, direction and depth of the object are generated frame by frame for the video based on the optical flow method. The motion trajectory of the object can be represented in the form of an array [H, W, 2], i.e., width, height, and direction. The optical flow information is binarized to obtain the motion trajectory of the object. Step 520, the dynamic model of the unit is determined based on the object and its motion trajectory. For example, the unit of the object is parsed, the dynamic model of the matching unit is retrieved, and the motion trajectory of the unit is generated.
[0133] The present application provides a method for supporting enhanced video generation, which inputs the original video, detects the optical flow motion trajectory of the object in the video, matches the dynamic model, and generates the precise motion trajectory of the unit, and generates a video with a reasonable and precise motion trajectory by controlling the reference dynamically reasonable motion trajectory, so that the running effect of the object in the generated video is consistent with the motion effect of the object in the real environment, and enhances the motion rationality and physical property rationality of the object in the generated video, thereby effectively improving the motion authenticity of the moving object in the generated video.
[0134] Next, the display content of the electronic device and the user interface operation during the video generation process provided by this application are explained.
[0135] Figure 6 This is an example of a scene generated by a video provided in this application. Figure 6 As shown in (a) of FIG. 1 , it is a front view of the mobile phone provided in the present application. A video APP icon 601 is displayed on the display screen of the mobile phone. The user can click on the video APP icon 601. Figure 6 As shown in (b) in the figure, the mobile phone displays the user interface of the video APP in response to the click operation, and the user interface may include material icons 602 such as text, image, and video. The user can click the material icon 602 to select the material displayed in the user interface. For example, the user can click the video icon to select a video, and the user interface can also display the video material selected by the user. The user interface may also include a one-click generated control 603. The user clicks the control 603, and the mobile phone generates a video based on identifying the first object in the material selected by the user, the first unit in the first object, and the dynamic model matching the material, the first object, and the dynamic model matching the first unit contained in the first object. As shown Figure 6 As shown in (c) in the figure, the mobile phone displays the generated video in response to the click operation.
[0136] For the specific process of video generation, please refer to the description of the above embodiment and will not be elaborated here.
[0137] The materials include materials stored locally on the mobile phone or materials recommended by the system. The embodiment of the present application does not limit the source of the materials.
[0138] Before the mobile phone displays the generated video, it can also display multiple objects contained in the material, one or more units in the multiple objects, dynamic models matched by the multiple objects, dynamic models matched by one or more units in the multiple objects, and parameters of the matched dynamic models, so that users can select or set various contents according to their preferences.
[0139] In some embodiments, the display material includes multiple objects and one or more units in the multiple objects. The multiple objects include a first object. One or more units of the first object include a first unit. The first object and / or the first unit are selected by a user. The first object includes one or more objects. The first unit includes one or more units. For example, the first object includes a car and a tree. The first unit includes a car body, wheels, windows, leaves, and a tree trunk, etc.
[0140] For example, Figure 7 As shown in (a), the user clicks on the material icon 602, and the mobile phone displays multiple objects contained in multiple materials in response to the click operation. For example, multiple objects contained in multiple texts are displayed. In some embodiments, the user can check the object selection box 604 to select the first object from multiple objects. Optionally, the user can also click on the first object to display one or more units contained in the first object. The user can check the unit selection box 604 to select the first unit from one or more units. The user clicks on the control 603 to generate a video based on the material, the dynamic model matched by the first object, and the dynamic model matched by the first unit contained in the first object, and the generated video is displayed. Among them, the dynamic model matched by the first object and the dynamic model matched by the first unit contained in the first object can be automatically obtained by the system based on the matching of the first object and the first unit.
[0141] For example, Figure 7 As shown in (b) in the figure, multiple materials are displayed. The user can also click on the material, and the mobile phone responds to the click operation. Alternatively, the user can check the material selection box 604, select the first material from the multiple materials, and display the first material. In response to the click operation, the mobile phone displays multiple objects in the first material. For example, multiple objects contained in a text are displayed. The user can check the object selection box 604 to select the first object from the multiple objects.
[0142] In other embodiments, the display material includes a plurality of object matching dynamic models and / or a plurality of object one or more unit matching dynamic models. The plurality of object matching dynamic models include a first object matching dynamic model. The first object one or more unit matching dynamic models include a first unit matching dynamic model. The first object matching dynamic model and / or the first unit matching dynamic model are selected by the user.
[0143] For example, Figure 8As shown, the user clicks on the material icon 602, and the mobile phone displays the dynamic models matching multiple objects in response to the click operation. The user can check the selection box 604 of the dynamic model to select the first object and the dynamic model matching the first object from multiple objects. The user can also click on the first object to display the dynamic model matching one or more units contained in the first object. The user can check the selection box 604 of the dynamic model to select the dynamic model matching the first unit from one or more units. The user clicks on the control 603 to generate a video based on the material, the dynamic model matching the first object, and the dynamic model matching the first unit contained in the first object, and display the generated video.
[0144] In some other embodiments, the parameters of the matched dynamic model are displayed, and the parameters of the matched dynamic model include the parameters of the dynamic model matched by the first object and / or the parameters of the dynamic model matched by the first unit. The parameters of the dynamic model matched by the first object and / or the parameters of the dynamic model matched by the first unit are selected by the user; the parameters include the attributes of the object. The parameters also include at least one of environmental information or motion duration.
[0145] For example, Fig. 9 As shown, the user clicks on the dynamic model 605, and the mobile phone responds to the click operation and displays the parameters of the dynamic model matched by the first object. The user can check the parameter selection box 604 and select the parameters of the dynamic model matched by the first object from multiple parameters. The user clicks on the first object to display the dynamic model matched by the unit contained in the first object. The user can check the dynamic model selection box 604, or click on the dynamic model. The mobile phone responds to the click operation and displays the parameters of the dynamic model matched by the unit contained in the object, and selects the parameters of the dynamic model matched by the first unit from multiple parameters. The user clicks on the control 603 to generate a video based on the material, the dynamic model matched by the first object, and the dynamic model matched by the first unit contained in the first object, and the generated video is displayed.
[0146] Optionally, the selection box may be replaced with a setting box, and the user may set the parameters of the dynamic model by himself, for example, setting the parameters of the dynamic model matched by the first object and the parameters of the dynamic model matched by the first unit according to the range of the parameters of the dynamic model prompted by the system.
[0147] In addition, if there is a large amount of content in each of the objects, units, dynamic models or parameters of the dynamic models to be displayed, the mobile phone can display more content through sliding or turning pages to facilitate the user to select the objects, units, dynamic models and parameters required to generate the video.
[0148] It should be noted that the mobile phone can display at least one of the object, unit, dynamic model or parameter of the dynamic model. Before displaying the generated video, the user can select each of the object, unit, multi-dynamic model and parameter of the dynamic model.
[0149] For example, Fig.10 As shown in (a) of FIG. 5 , the user clicks on the material icon 602, and the mobile phone displays multiple objects contained in multiple materials in response to the click operation. The user can check the object selection box 604 to select the first object from the multiple objects. The user can click on the first object, such as Fig.10 As shown in (b) and (c) of FIG. 1 , the mobile phone displays the dynamic model matched by the first object and the dynamic models of multiple units contained in the first object in response to the click operation. The user can also click on the first object to display the dynamic model matched by one or more units contained in the first object. The user can check the selection box 604 of the dynamic model, or select the dynamic model matched by the first unit from multiple units. Fig.10 As shown in (d) in FIG. 1 , the user clicks on the dynamic model 605, and the mobile phone displays the parameters of the dynamic model matched by the first object in response to the click operation. The user can check the parameter selection box 604 to select the parameters of the dynamic model matched by the first object from multiple parameters. The user clicks on the control 603 to generate a video based on the material, the dynamic model matched by the first object, and the dynamic model matched by the first unit contained in the first object, and the generated video is displayed.
[0150] It should be noted that the present application does not limit the layout of the display interface and the method of user operation. As long as at least one of the parameters of an object, unit, multi-dynamic model or dynamic model can be displayed, and the user can operate at least one of the parameters of an object, unit, multi-dynamic model or dynamic model, so that the mobile phone generates a video based on at least one of the parameters of the object, unit, multi-dynamic model or dynamic model selected by the user, it is within the protection scope of the technical solution provided in the present application.
[0151] In some other embodiments, the video generation method provided by the present application can be applied to Web services. For example, Fig.11 As shown, users can log in to the web page and operate the web page to generate videos. For the operation method of the web page, please refer to the above Figures 6 to 10 The explanation is not limited.
[0152] In addition, the video generation action can also be performed by the cloud service. For example, Fig.11 This is a schematic diagram of the architecture of a data processing system provided in this application. Fig.11 As shown, data processing system 1100 includes a client 1110 , a computing cluster 1120 , and a storage cluster 1130 .
[0153] The storage cluster 1130 includes multiple storage nodes 1131. A storage node 1131 includes one or more controllers, a network card and multiple hard disks. The hard disk is used to store data. The hard disk can be a magnetic disk or other types of storage media, such as a solid-state hard disk or a shingled magnetic recording hard disk. The network card is used to communicate with the computing node 1121 included in the computing cluster 1120. The controller is used to write data to the hard disk or read data from the hard disk according to the read / write data request sent by the computing node 1121. In the process of reading and writing data, the controller needs to convert the address carried in the read / write data request into an address that the hard disk can recognize.
[0154] The computing cluster 1120 includes multiple computing nodes 1121. The computing node 1121 may be a computing device, such as an accelerator card, a server, and the like.
[0155] In some embodiments, the computing cluster 1120 may be a heterogeneous computing architecture to provide high-performance computing. For example, the computing node 1121 may include a computing unit with computing capabilities such as a CPU, a graphics processing unit (GPU), a data processing unit (DPU), a neural processing unit (NPU), and an embedded neural-network processing unit (NPU) to provide high-performance computing.
[0156] For example, computing cluster 1120 includes Figure 2 Multiple accelerator cards are shown, and multiple accelerator cards perform model training.
[0157] In other embodiments, multiple computing nodes 1121 are connected through network devices (such as switches, network cards, etc.) based on high-speed interconnection technology, so that the multiple computing nodes 1121 can communicate with each other.
[0158] The client 1110 communicates with the computing cluster 1120 and the storage cluster 1130 via the network 1140. For example, the client 1110 sends a video generation request to the computing cluster 1120 via the network 1140. For example, the video generation request may include data such as objects, units, dynamic models, or parameters of dynamic models selected by the user. The computing cluster 1120 is requested to generate a video. The network 1140 may refer to an internal network of an enterprise (such as a local area network (LAN)) or the Internet. The client 1110 refers to a computer connected to the network 1140, which may also be referred to as a workstation. Different clients may share resources on the network (such as computing resources and storage resources).
[0159] In some embodiments, the computing cluster 1120 further includes a control node 1122. For example, the control node and the computing node may be independent physical devices. For another example, the control node and the multiple computing nodes may be located on the same physical device. The control node may be a CPU. The multiple computing nodes include computing units such as GPU, NPU, and DPU. The control node 1122 is used to manage and allocate tasks, and multiple computing nodes execute multiple tasks in parallel to increase the data processing rate.
[0160] In the present application, the control node 1122 is used to instruct multiple computing nodes to generate videos in parallel according to a request.
[0161] The computing node 1121 is used to generate a video according to the material, the dynamic model matched by the first object, and the dynamic model matched by the first unit included in the first object.
[0162] The storage cluster 1130 can also be used to store materials, detection objects, models required for units, video generation models, etc. When the computing node 1121 generates a video, it can obtain the model from the storage node 1131, identify data such as objects, units, dynamic models or parameters of dynamic models, and generate a video.
[0163] In other embodiments, the client 1110 is installed with the client program 113, and the client 1110 runs the client program 113 to display a user interface (UI), and the user 1150 operates the user interface to submit a request. For example, the user 1150 operates the user interface to submit a request. After the control node 1122 obtains the request, it loads the model and the material from the storage cluster 1130, identifies the data such as the object, unit, dynamic model or the parameters of the dynamic model, and generates a video.
[0164] Optionally, the system administrator 1160 can call the application platform interface (API) 1112 or the command-line interface (CLI) interface 1113 through the client 1110 to configure system information, such as the correspondence between objects configured for computing nodes and dynamic models and the correspondence between units and dynamic models provided in the present application.
[0165] Fig.11 This is only a schematic diagram. The embodiments of the present application do not limit the device connection mode and the number of devices in the data processing system. For example, the data processing system may include multiple clients. A client may be connected to multiple computing nodes. Different clients establish connections with different computing nodes.
[0166] It is understandable that in order to implement the functions in the above embodiments, the computer device includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the units and method steps of each example described in the embodiments disclosed in this application, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application scenario and design constraints of the technical solution.
[0167] Combined with the above Figures 1 to 11 , describes in detail the video generation method provided by this application, and will be combined with Fig.12 , describing the apparatus provided according to the present application. These apparatuses can be used to implement the functions of the processor in the above method embodiment, and thus can also achieve the beneficial effects possessed by the above method embodiment. In this embodiment, the apparatus can be as follows Figure 1 A processor or a Fig.11 The computing node shown may also be a module (such as a chip) applied to a computer device.
[0168] like Fig.12 As shown, the video generating device 1200 includes a communication module 1201 , a processing module 1202 , a display module 1203 and a storage module 1204 .
[0169] The video generating device 1200 is used to implement the above Figure 2 The functions of the electronic device in the method embodiment shown in FIG.
[0170] The communication module 1201 is used to obtain materials including at least one of text, image or video. For example, the communication module 1201 is used to execute Figure 2 Step 210.
[0171] The processing module 1202 is configured to generate a video according to the material, the dynamic model matched by the first object, and the dynamic model matched by the first unit included in the first object. Figure 2 Step 220.
[0172] Optionally, the processing module 1202 is specifically used to generate a motion trajectory of the first object based on the first object, a dynamic model matching the first object and its corresponding parameters, and to generate a motion trajectory of the first unit based on a first unit contained in the first object, a dynamic model matching the first unit contained in the first object and its corresponding parameters; and to generate a video based on the material, the motion trajectory of the first object and the motion trajectory of the first unit.
[0173] The display module 1203 is used to display the video. For example, the display module 1203 is used to perform Figure 2 Step 230.
[0174] The storage module 1204 is used to store materials, models, etc. to facilitate video generation.
[0175] It should be understood that the video generation device 1200 of the embodiment of the present application can be implemented by an application-specific integrated circuit (ASIC) or a programmable logic device (PLD), and the PLD can be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL) or any combination thereof. It can also be implemented by software. Figure 2 When the method is shown, its various modules may also be software modules, and the video generating device 1200 and its various modules may also be software modules.
[0176] The video generating device 1200 according to the embodiment of the present application may correspond to executing the method described in the embodiment of the present application, and the above and other operations and / or functions of each unit in the video generating device 1200 are respectively to implement Figure 2 For the sake of brevity, the corresponding processes of each method in are not repeated here.
[0177] The method steps in this embodiment can be implemented by hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, mobile hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC. In addition, the ASIC can be located in a computing device. Of course, the processor and the storage medium can also exist in a computing device as discrete components.
[0178] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instruction is loaded and executed on a computer, the process or function described in the embodiment of the present application is executed in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, a network device, a user device or other programmable device. The computer program or instruction may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer program or instruction may be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired or wireless means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium, for example, a floppy disk, a hard disk, a tape; it may also be an optical medium, for example, a digital video disc (DVD); it may also be a semiconductor medium, for example, a solid state drive (SSD). The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.
Claims
1. A video generation method, characterized in that: include: Acquire material, where the material includes at least one of text, image or video; In response to the first operation, a video is displayed, wherein the video is generated based on the material, a dynamic model matching a first object in the material, and a dynamic model matching a first unit contained in the first object, wherein the dynamic model is used to indicate a motion representation of simulated physical properties of an object and a relative position of the object in space that changes over time.
2. The method according to claim 1, characterized in that The dynamic model matched by the first object and / or the dynamic model matched by the first unit are selected by a user; Before displaying the video, the method further includes: The dynamic models matched by multiple objects contained in the material and / or the dynamic models matched by one or more units in the multiple objects are displayed, wherein the dynamic models matched by the multiple objects include the dynamic model matched by the first object, and the dynamic models matched by one or more units in the first object include the dynamic model matched by the first unit.
3. The method according to claim 2, characterized in that The parameters of the dynamic model matched by the first object and / or the parameters of the dynamic model matched by the first unit are selected by the user; the parameters include properties of the object; Before displaying the video, the method further includes: Parameters of the matched kinetic model are displayed, wherein the parameters of the matched kinetic model include parameters of the matched kinetic model of the first object and / or parameters of the matched kinetic model of the first unit.
4. The method according to claim 3, characterized in that The parameters also include at least one of environmental information or exercise duration.
5. The method according to any one of claims 1 to 4, characterized in that The first object and / or the first unit is selected by a user; Before displaying the video, the method further includes: A plurality of objects contained in the material and one or more units in the plurality of objects are displayed, wherein the plurality of objects include the first object, and the one or more units in the first object include the first unit.
6. The method according to any one of claims 1 to 5, characterized in that The method further comprises: The video is generated according to the material, the dynamic model matched by the first object, and the dynamic model matched by the first unit included in the first object.
7. The method according to claim 6, characterized in that Generating the video according to the material, the dynamic model matched by the first object, and the dynamic model matched by the first unit included in the first object includes: Generate a motion trajectory of the first object according to the first object, a dynamic model matched by the first object, and corresponding parameters thereof; and generate a motion trajectory of the first unit according to a first unit included in the first object, a dynamic model matched by the first unit included in the first object, and corresponding parameters thereof; The video is generated according to the material, the motion trajectory of the first object and the motion trajectory of the first unit.
8. The method according to claim 7, characterized in that Generating a motion trajectory of the first object according to the first object, a dynamic model matched by the first object, and corresponding parameters thereof, and generating a motion trajectory of the first unit according to a first unit included in the first object, a dynamic model matched by the first unit included in the first object, and corresponding parameters thereof, including: Call the interface associated between the video generation system and the trajectory generation system to generate the motion trajectory of the first object according to the first object, the dynamic model matched by the first object and its corresponding parameters, and generate the motion trajectory of the first unit according to the first unit included in the first object, the dynamic model matched by the first unit included in the first object and its corresponding parameters.
9. The method according to claim 7 or 8, characterized in that: Generating the video according to the material, the motion trajectory of the first object and the motion trajectory of the first unit includes: The video is generated according to the material, the motion trajectory of the first object, the motion trajectory of the first unit, and the depth relationship and / or viewing angle of at least two objects included in the first object.
10. The method according to any one of claims 6 to 9, characterized in that: The method further comprises: receiving a second operation of setting a video parameter of the video; Generating the video according to the material, the dynamic model matched by the first object, and the dynamic model matched by the first unit included in the first object includes: The video is generated according to the video parameters, the material, the dynamic model matched by the first object, and the dynamic model matched by the first unit included in the first object.
11. An electronic device, characterized in that: The electronic device comprises a processor, a memory and a display; The processor is used to execute instructions stored in the memory so that the electronic device performs the operation steps of the method according to any one of claims 1 to 10, and the display is used to display video.
12. A data processing system, characterized in that: Includes client and server; The client is used to perform the operation steps of the method according to any one of claims 1 to 5, and display the video generated according to the material; The server is used to execute the operation steps of the method according to any one of claims 6 to 10 to generate the video.
13. A computer program product comprising instructions, characterized in that When the instructions are executed by a computing device, the computing device is caused to perform the operation steps of the method according to any one of claims 1 to 10.
14. A computer-readable storage medium, characterized in that: The method comprises computer program instructions. When the computer program instructions are executed by a computing device, the computing device performs the operation steps of the method according to any one of claims 1 to 10.
