Video synthesis method and device, electronic equipment and storage medium

Abstract

The invention discloses a video synthesis method and device, electronic equipment and a storage medium. The method comprises the following steps: acquiring a courseware video to be synthesized, a digital human model and at least one preset scene; analyzing the courseware video to be synthesized, and determining at least one frame of courseware picture and the duration and blank area information of each frame of courseware picture; determining digital human layout data based on the duration time and blank area information of each preset scene and each courseware picture; and performing video synthesis on the courseware video to be synthesized and the digital person model according to the digital person layout data, thereby solving the problem that all pictures in the courseware cannot be adapted due to the fact that a user manually sets a fixed scene in a digital person and video synthesis process. A proper layout mode is selected for each frame of courseware picture of the courseware video to be synthesized through the digital person layout data to carry out digital person layout, so that the shielding condition is effectively avoided, and the teaching quality is improved; a user does not need to manually set and select, user time is saved, and user experience is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a video synthesis method, device, electronic equipment and storage medium. Background Art

[0002] Virtual digital human is a new technology in the AI ​​era. Virtual digital human technology can generate digital human videos based on text, voice and other data combined with trained real people or virtual models. In educational scenarios, the generated digital human videos can be combined with filmed or recorded teaching videos to become teaching videos containing digital humans. In the process of video synthesis, it is often necessary to set the positional relationship between digital humans and teaching courseware videos. The positional relationship between digital humans and teaching courseware videos in different scenes is usually different. Therefore, performing video synthesis in different scenes will result in different video synthesis effects.

[0003] In the current technical solution, the user usually sets the scene manually. For example, the character is in the lower left corner, the character is in the lower right corner, the character portrait is cropped with a rectangular frame, the character portrait is cropped with a circular frame, the courseware is full screen without a character, etc. You can set only one scene, or you can set different scenes according to time periods. However, since the courseware contains different pictures, the size, quantity, position and other courseware contents of the text in different pictures are also different. The fixed scene manually set by the user cannot adapt to all the pictures in the courseware, and may block the text in the teaching courseware, thereby affecting teaching. Summary of the invention

[0004] The present invention provides a video synthesis method, device, electronic device and storage medium to solve the problem that manually set fixed scenes cannot adapt to all pictures in courseware.

[0005] According to one aspect of the present invention, there is provided a video synthesis method, comprising:

[0006] Obtaining a courseware video to be synthesized, a digital human model, and at least one preset scene;

[0007] Analyze the courseware video to be synthesized to determine at least one frame of the courseware picture, as well as the duration and blank area information of each frame of the courseware picture;

[0008] Determine digital human layout data based on the duration and blank area information of each preset scene and each courseware screen;

[0009] The video of the courseware to be synthesized and the digital human model are synthesized according to the digital human layout data.

[0010] According to another aspect of the present invention, there is provided a video synthesis device, comprising:

[0011] A data acquisition module, used to acquire the courseware video to be synthesized, the digital human model and at least one preset scene;

[0012] A courseware analysis module, used for analyzing the courseware video to be synthesized, determining at least one frame of the courseware picture, and the duration and blank area information of each frame of the courseware picture;

[0013] A layout data determination module, used to determine the digital human layout data based on the duration and blank area information of each of the preset scenes and each of the courseware screens;

[0014] The video synthesis module is used to perform video synthesis on the courseware video to be synthesized and the digital human model according to the digital human layout data.

[0015] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0016] at least one processor; and

[0017] a memory communicatively connected to the at least one processor; wherein,

[0018] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the video synthesis method described in any embodiment of the present invention.

[0019] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the video synthesis method described in any embodiment of the present invention when executed.

[0020] The technical solution of the embodiment of the present invention obtains a courseware video to be synthesized, a digital human model and at least one preset scene; analyzes the courseware video to be synthesized to determine at least one frame of the courseware picture, and the duration and blank area information of each frame of the courseware picture; determines the digital human layout data based on the preset scenes and the duration and blank area information of each frame of the courseware picture; performs video synthesis on the courseware video to be synthesized and the digital human model according to the digital human layout data, thereby solving the problem that the digital human and the video synthesis process cannot adapt to all the pictures in the courseware due to the user manually setting a fixed scene, analyzes the courseware video to be synthesized to determine each frame of the courseware picture in the courseware video, analyzes each frame of the courseware picture to determine the duration and blank area information, performs layout based on the preset scene and the duration and blank area information of each frame of the courseware picture, determines the digital human layout data, selects a suitable layout mode for each frame of the courseware picture in the courseware video to be synthesized to perform digital human layout, performs video synthesis on the courseware video to be synthesized and the digital human model based on the digital layout data, effectively avoids the generation of occlusion, and improves the teaching quality; no manual setting and selection by the user is required, thus saving the user time and improving the user experience.

[0021] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0023] Figure 1 is a flowchart of a video synthesis method provided according to Embodiment 1 of the present invention;

[0024] Figure 2 This is an example diagram of a digital human being displayed on a screen according to the prior art;

[0025] Figure 3 is a flowchart of a video synthesis method provided according to Embodiment 2 of the present invention;

[0026] Figure 4 This is a display example diagram of a digital human model and a courseware screen after synthesis according to the second embodiment of the present invention;

[0027] Figure 5This is another example of a display diagram of a digital human model and a courseware screen after synthesis according to the second embodiment of the present invention;

[0028] Figure 6 is a structural schematic diagram of a video synthesis device provided according to Embodiment 3 of the present invention;

[0029] Figure 7 It is a structural schematic diagram of an electronic device for implementing the video synthesis method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0030] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0031] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0032] Embodiment 1

[0033] Figure 1 This is a flowchart of a video synthesis method provided in Example 1 of the present invention. This embodiment is applicable to the situation of synthesizing courseware videos and digital humans. The method can be executed by a video synthesis device. The video synthesis device can be implemented in the form of hardware and / or software. The video synthesis device can be configured in an electronic device.

[0034] It should be noted that in the prior art, in the intelligent teaching scene, the generated digital human video is combined with the filmed or recorded teaching video to form a teaching video containing a digital human. When forming the teaching video, the user manually sets the scene. At this time, the text in the picture is blocked because the scene is not compatible with the picture. For example, Figure 2An example diagram of a digital human being displayed in a screen is provided. It can be seen that the digital human model 11 blocks part of the text in the screen 12 because the digital human model 11 is too large and the setting position is inappropriate.

[0035] The video synthesis method provided in the embodiment of the present application can perform targeted synthesis on each frame of the courseware video and the digital human, avoiding text occlusion, thereby overcoming the problems existing in the existing technical solutions.

[0036] like Figure 1 As shown, the method includes:

[0037] S101, obtaining a courseware video to be synthesized, a digital human model, and at least one preset scene.

[0038] In this embodiment, the courseware video to be synthesized can be specifically understood as a courseware video that has the need for video synthesis with a digital human. The courseware video can be recorded during the courseware playback process; the digital human model can be a static character model or a dynamic character model. The digital human model can be a real person model recorded or photographed by a real person, or it can be an animated character, a virtual character obtained by modeling, etc.; the preset scene can be specifically understood as a pre-set digital human application scene. The setting method of the digital human in different scenes is different. The preset scene can be close-up explanation, full-screen courseware, lecture hall, and character always present priority, etc.

[0039] The courseware video is formed and saved in advance. When performing video synthesis, the courseware video is obtained from the corresponding storage space as the courseware video to be synthesized. When obtaining the courseware video to be synthesized, the storage space can be searched through the name of the courseware video to be synthesized input by the user, and then the courseware video to be synthesized can be read; or different courseware videos are displayed on the interface, and the user selects the courseware video to be synthesized through selection operations, etc. The digital human model can be pre-recorded by real people to generate or pre-build animation or virtual character models. For example, a teacher does not want to be photographed frequently due to personal image considerations, so some videos or images of the teacher in the best state can be taken first; or due to privacy considerations, he is not willing to be photographed. Then, an animated puppet model or other non-real model video or image can be generated, and then the digital human model of the teacher can be generated based on the video or image taken or generated. Subsequently, based on this model, the teacher's teaching video can be combined to synthesize a video containing the teacher's image. Different scenes are set in advance, and the user manually selects one or more scenes as preset scenes; or one or more scenes are automatically selected as preset scenes according to set conditions or rules; or a default scene can be set as a preset scene.

[0040] S102, analyzing the courseware video to be synthesized, and determining at least one frame of the courseware picture, as well as the duration and blank area information of each frame of the courseware picture.

[0041] In this embodiment, the courseware screen can be specifically understood as the screen displayed by the courseware video; the blank area information can be specifically understood as information describing the blank area, for example, the size and position of the blank area.

[0042] The courseware content in the courseware video to be synthesized is generally a presentation, in which a large number of pictures are the same; usually, the courseware video includes several pages of PPT, including title, directory, content, summary page, etc., each page lasts for several minutes. Analyze each frame of the courseware video to be synthesized, filter out repeated pictures, and use the remaining non-repeated pictures as courseware pictures. Count the time when each frame of the courseware picture appears to determine the duration of the courseware picture; determine the blank area by analyzing the place where there are text or pictures in the courseware picture, or analyze the blank area in the courseware picture, and further determine the blank area information.

[0043] S103: Determine digital human layout data based on the duration and blank area information of each preset scene and each courseware screen.

[0044] In this embodiment, the digital human layout information can be understood as information for layout of the digital human model and the courseware screen. For example, the digital human layout information includes the positional relationship between the digital human model and the courseware screen.

[0045] Based on the blank area information of the courseware screen, the position where the digital human model can be set in the courseware screen can be determined. Based on the preset scene, the size, display method, and position information of the digital human model can be determined. Combined with the duration of the courseware screen, the layout information such as the position, size, and display method of the digital human model and the courseware screen at each moment can be determined, and then the digital human layout data can be generated.

[0046] S104, performing video synthesis on the courseware video to be synthesized and the digital human model according to the digital human layout data.

[0047] According to the digital human layout data, the position, size and other information of the digital human model at each moment are determined, the digital human model and the courseware screen are adjusted according to the corresponding sizes, and the adjusted digital human model and the courseware screen are synthesized according to the corresponding positions and durations to form a courseware video containing the digital human model.

[0048] The embodiment of the present invention provides a video synthesis method, which solves the problem that all pictures in the courseware cannot be adapted due to the fixed scene manually set by the user in the process of synthesizing digital humans and videos. The method analyzes the courseware video to be synthesized, determines each frame of the courseware picture in the courseware video, analyzes each frame of the courseware picture to determine the duration and blank area information, performs layout based on the preset scene and the duration and blank area information of each of the courseware pictures, determines the digital human layout data, selects a suitable layout method for each frame of the courseware picture in the courseware video to be synthesized to perform digital human layout, and performs video synthesis on the courseware video to be synthesized and the digital human model based on the digital layout data, effectively avoiding occlusion and improving the teaching quality; there is no need for the user to manually set the selection, saving the user time and improving the user experience.

[0049] Embodiment 2

[0050] Figure 3 This is a flow chart of a video synthesis method provided by Embodiment 2 of the present invention. This embodiment is refined on the basis of the above embodiment. Figure 3 As shown, the method includes:

[0051] S201, obtaining a courseware video to be synthesized, a digital human model, and at least one preset scene.

[0052] As an optional embodiment of the present embodiment, the present optional embodiment further obtains at least one preset scene and is optimized as follows: displaying a scene selection interface; receiving a scene selection operation, where the scene selection operation is a user selecting at least one preset scene from the scene selection interface.

[0053] In this embodiment, the scene selection interface can be specifically understood as a display interface for selecting different scenes; the scene selection operation can be specifically understood as a user operation for selecting a preset scene.

[0054] The scene selection interface can be automatically displayed when performing video synthesis, or can be triggered by the user's operation. For example, a virtual button is displayed on the screen, and the user triggers the virtual button by clicking or sliding. After receiving this operation, the execution device displays the scene selection interface. Different scenes can be displayed in the scene selection interface. Each scene can be arranged in a certain order and displayed in the scene selection interface. The user selects one or more scenes as preset scenes by single-clicking, double-clicking, etc. Different scenes may be exclusive, that is, they cannot exist at the same time. For example, the scene with full-screen priority of the courseware and the scene with priority of the character always existing cannot exist at the same time. After the user selects the scene with the character always existing, the scene with full-screen priority of the courseware that cannot exist at the same time is displayed in gray and locked, and the user cannot select the scene with full-screen priority of the courseware again; in this embodiment, the scene selected by the user can be marked, for example, by color or character, so that the user can clearly see the selected scene; the selected scene, the unselected scene, and the unselectable scene can all be marked in different ways, and the cancel function can also be provided for the selected scene to cancel the selection, etc.

[0055] Optionally, each preset scene is associated with a size range of the digital human model, display information of the digital human model, and position information of the screen.

[0056] In this embodiment, the size range of the digital human model is used to limit the size of the digital human model and can be used to select a blank area; the display information of the digital human model can be the display outline shape of the digital human model, such as square, prototype, ellipse, etc. When displaying the digital human model, the digital human model can be cropped according to the display outline shape. The display information can also be the image type of the digital human model, such as a head portrait or a bust portrait; the position information of the picture includes the size and position of the picture.

[0057] S202, comparing each frame of the courseware video to be synthesized, and extracting the courseware frames that meet the preset difference conditions.

[0058] In this embodiment, the preset difference condition can be specifically understood as a pre-set condition for determining whether the courseware screen is different. For example, when the difference between two frames is represented by pixel values, the number of changed pixel points is greater than a certain value to determine that the preset difference condition is met; or when two frames are compared by a model, the model calculates that the difference value between the two frames is greater than a certain value to determine that the preset difference condition is met, and so on. The courseware video to be synthesized includes multiple frames, and the difference between each frame is compared. For example, comparing the pixel values ​​of the same point in two frames, if the pixel values ​​are different or the difference is greater than a certain difference threshold, it is determined that this point has changed. When the number of changed points in the picture is greater than the preset number threshold, it is determined that the picture meets the preset difference condition, and the two frames are different; otherwise, it is determined that the picture does not meet the preset difference condition, and the two frames are determined to be the same. Compare the differences between the pictures, and extract the pictures with differences as the courseware pictures.

[0059] Since there are repeated pictures in the courseware video to be synthesized, for example, the courseware includes 20 pages of PPT, and the playback time of each page of PPT is about 2 minutes, the pictures in the courseware video to be synthesized will include a large number of repeated pictures. The repeated pictures are filtered out and the pictures with differences are extracted as the courseware pictures, so as to avoid repeated operations when selecting the appropriate position on the picture to place the digital human model later, reduce the data processing amount, and improve the data processing efficiency.

[0060] S203. For each frame of the courseware screen, count the duration of the courseware screen according to the frame number or time corresponding to the courseware screen.

[0061] For each courseware screen, determine the total number of frames that appear in this frame, multiply the total number of frames by the time interval between each frame, and the resulting time is the duration of this frame of courseware screen; or determine the time of the first appearance and the time of the last appearance of this frame of screen when it appears continuously, and the time difference between the two times is the duration. When the courseware screen appears repeatedly at intervals, the duration of each appearance can be counted to obtain the total duration.

[0062] S204, analyzing the blank area of ​​the courseware screen to determine the blank area information.

[0063] The blank area of ​​the courseware screen can be analyzed by pixel value analysis. When the pixel value meets certain conditions, the pixel point is determined to be a background pixel. The background pixel can represent that there is no specific courseware content at this point, that is, this point can be considered a blank point. The positions of all background pixels are statistically analyzed to determine the blank area. For example, the minimum inscribed rectangle is determined according to the coordinates of each background pixel point, and the minimum inscribed rectangle is used as a blank area. The blank area information is further determined, wherein the blank area information includes the size and position of the blank area. For example, when the blank area is a rectangle, the blank area size in the blank area information can be the length and width of the rectangle, and the blank area position can be a vertex or center point of the rectangle, etc., which can uniquely determine the coordinates of a rectangle.

[0064] For each frame of the courseware screen, the duration and blank area information of the courseware screen can be determined in the same manner, that is, the duration of the courseware screen is determined by step S203, and the blank area information of the courseware screen is determined by step S204.

[0065] S205: Determine the target scene corresponding to each courseware screen according to each preset scene and the blank area information of each courseware screen.

[0066] In this embodiment, the target scene can be specifically understood as the scene that the courseware screen matches when it is synthesized with the digital human model. The setting requirements of the digital human model are determined according to the information associated with the preset scene, and whether the digital human model can be set according to the requirements is determined in combination with the blank area information in the courseware screen, and then the target scene corresponding to the courseware screen is determined; or the courseware screen is matched with each preset scene according to the preset scene setting rules, and the matched preset scene is used as the target scene. For example, the preset scene setting rules are that the number of times the same scene appears in a video cannot exceed 10 times, cannot be less than 1 time, etc. On the premise of meeting the scene setting rules, each preset scene and courseware screen is reasonably arranged to obtain the target scene corresponding to each courseware screen. If there are multiple corresponding preset scenes for the courseware screen, the preset scene with the highest matching degree can be selected as the target scene, or a preset scene can be randomly selected as the target scene, or other screening conditions can be set for re-screening.

[0067] As an optional embodiment of this embodiment, this optional embodiment further determines the target scene corresponding to each courseware screen according to each preset scene and the blank area information of each courseware screen, and optimizes it as follows:

[0068] A1. Analyze each courseware screen based on each preset scene and blank area information in the courseware screen to determine at least one scene that matches each courseware screen.

[0069] According to the blank area information, the position and size of the blank area in the courseware screen can be determined, and the size, position and other requirements of the digital human model associated with the preset scene can be analyzed to determine whether the blank area can meet the above requirements. If it does, it means that the digital human model can be set, so it can be determined that this preset scene matches the courseware screen. For example, the requirement of the digital human model associated with the preset scene is that the length and width of the digital human model exceed 100 pixels. According to the blank area information of the courseware screen, it is determined that the width of the largest blank area in the courseware screen is 80 pixels, then it can be determined that this preset scene cannot be matched with the courseware screen; or, the digital human model in the preset scene is placed in the lower right corner of the screen, and the length and width exceed 100 pixels. According to the blank area information of the courseware screen, it is determined that the size and width of the blank area in the lower right corner of the courseware screen exceed 100 pixels, then it can be determined that this preset scene matches the courseware screen. If there is no preset scene that matches the courseware screen, you can select the preset scene with the highest matching degree as the scene that matches the courseware screen, or you can set a default scene as the scene that matches the courseware screen; or analyze the size, quantity, position, etc. of the courseware content such as the text in the courseware screen, and select a suitable scene. For example, when there are many small texts in the courseware screen, and no matching scene is found in the preset scenes, the courseware full-screen priority scene can be used as the scene that matches the courseware screen.

[0070] For each courseware screen, determine in turn whether each preset scene matches this courseware screen, and determine at least one scene that matches each courseware screen.

[0071] A2. Filter the scenes matched by each courseware screen based on preset rules to determine the target scene corresponding to each courseware screen.

[0072] The preset rules can be random selection and rotation of scenes, for example, two different scenes are selected for two adjacent courseware screens; the types of scenes in the video should be as many as possible; the total number of occurrences of each scene is less than a certain number, etc. Preset rules are set in advance according to needs. For example, the preset rule is that the types of scenes in the video should be as many as possible, and the types of scenes matched by all courseware screens and the total number of occurrences of each type are counted. The scene with the least total number of occurrences is selected. This scene may match multiple courseware screens (that is, the scenes matched by multiple courseware screens all include this scene), and the target scene of a courseware screen that matches it is determined as this scene; a new scene is reselected for selection until the corresponding target scene is selected for each courseware screen. In an embodiment of the present application, when the scenes matched by each courseware screen are screened according to the preset rules, it is only necessary to ensure that the target scenes corresponding to each courseware screen finally screened out meet the preset rules. When the preset rules are met, there may be multiple situations, that is, there may be multiple choices for the target scene corresponding to the courseware screen, and one of them can be randomly selected.

[0073] As an optional embodiment of this embodiment, this optional embodiment further screens the scenes matched by each courseware screen based on preset rules, determines the target scene corresponding to each courseware screen, and optimizes it as follows:

[0074] B1. Obtain the first courseware screen, and select a scene from the scenes matched by the first courseware screen as the target scene corresponding to the first courseware screen.

[0075] A courseware video usually includes multiple courseware screens, and each courseware screen can be arranged in the order in which it appears in the video, and the first courseware screen that appears is obtained. All scenes matched by the first courseware screen are determined, and a scene is randomly selected as the target scene corresponding to the first courseware screen.

[0076] B2. The next courseware screen of the first courseware screen is used as the screen to be matched. Based on the target scene corresponding to the previous screen to be matched, a scene is selected from the scenes matched by the screen to be matched as the target scene corresponding to the screen to be matched.

[0077] In this embodiment, the picture to be matched can be specifically understood as a courseware picture with a scene matching requirement; the previous picture to be matched can be specifically understood as a courseware picture previous to the picture to be matched currently being matched.

[0078] The next courseware screen of the first courseware screen, that is, the second courseware screen, is used as the screen to be matched, and the previous screen to be matched of the screen to be matched is the first courseware screen. The target scene corresponding to the previous screen to be matched is determined, and a scene different from the target scene corresponding to the previous screen to be matched is selected from the scenes matched by the screen to be matched as the target scene corresponding to the screen to be matched.

[0079] B3. Take the next courseware screen of the screen to be matched as the new screen to be matched, return to execute the target scene corresponding to the previous screen to be matched based on the screen to be matched, and select a scene from the scenes matched by the screen to be matched as the target scene corresponding to the screen to be matched, until the target scenes corresponding to all courseware screens are determined.

[0080] Determine whether there is a next courseware screen for the screen to be matched. If so, take the next courseware screen of the screen to be matched as the new screen to be matched, return to step B2 and select a scene from the scenes matched by the screen to be matched as the target scene corresponding to the screen to be matched, and repeat the step of determining the target scene corresponding to the new screen to be matched. If there is no target scene, determine that all the target scenes corresponding to the courseware screens have been determined, and end the operation.

[0081] The target scene is one of the preset scenes, which is associated with the size range of the digital human model, the display information of the digital human model and the position information of the screen.

[0082] S206: For each frame of the courseware screen, determine the size and position of the digital human model corresponding to the courseware screen according to the size range of the digital human model associated with the target scene and the blank area information in the screen.

[0083] For each frame of the courseware screen, determine the target scene corresponding to the courseware screen, and determine the size range of the digital human model associated with the target scene. The size range can be used to limit the size of the digital human model that is finally synthesized into the courseware video. The size range of the digital human model associated with the target scene can also be a character with a specific meaning. For example, 0 means no size limit (that is, the size of the digital human model can be set adaptively), and -1 means no digital human model is set, that is, in this target scene, the digital human model is not placed in the courseware screen.

[0084] When the size range of the digital human model associated with the target scene is a numerical range, the blank area in the courseware screen that meets this numerical range is determined according to the blank area information in the courseware screen, the size of the digital human model is determined according to the size of the blank area that meets this numerical range, and a digital human model of appropriate size is generated, and the position of the digital human model is determined according to the position of the blank area; when there are multiple blank areas whose sizes meet this numerical range, a blank area can be randomly selected, or a blank area can be selected according to a certain priority rule, for example, the lower right corner, the lower left corner, the upper left corner, and the upper right corner, with the priority decreasing in sequence. The method of determining the size of the digital human model according to the size of the blank area can be: setting the size of the digital human model according to the size of the blank area, for example, the size range of the digital human model is that the length and width are both greater than 80 pixels, the original ratio of the digital human model is 1:1, if the size of the blank area is 100*150 pixels, the size of the digital human model is finally set to 100*100 pixels, if the size of the blank area is 85*150, the size of the digital human model is finally set to 85*85 pixels, that is, the size of the digital human model can be set according to the size of the blank area.

[0085] When the size range of the digital human model associated with the target scene is an unlimited size represented by characters with specific meanings, a blank area can be selected according to the priority of the size or position of the blank area, and the size and position of the digital human model can be determined according to the size and position of the blank area.

[0086] When the size range of the digital human model associated with the target scene is a character with a specific meaning indicating that no digital human model is set, the size and position of the digital human model can be set to 0 or empty, indicating that there is no digital human model.

[0087] S207: Determine the display information of the digital human model corresponding to the courseware screen according to the display information of the digital human model associated with the target scene.

[0088] Determine the display information of the digital human model associated with the target scene. The display information may be the display shape of the digital human model, the image type of the digital human model, whether it is a bust or a head portrait. The display information of the digital human model associated with the target scene is directly determined as the display information of the digital human model corresponding to the courseware screen.

[0089] S208. Determine the size and position of the courseware screen based on the position information of the screen associated with the target scene.

[0090] Determine the position information of the screen associated with the target scene. The position information of the screen can be the size and position of the screen, for example, the size of the screen is full screen; or the size of the screen is reduced by n times, and the position of the screen is the upper left corner; the position information of the screen is empty, indicating that the size and position of the courseware screen are not adjusted, etc. The position information of the screen associated with the target scene is directly determined as the size and position corresponding to the courseware screen.

[0091] In the embodiment of the present application, a suitable scene can be selected for the courseware screen for video synthesis according to the courseware screen, so that the synthesis effect is better. For example, for some scenes, in order to ensure that the courseware screen can be seen clearly, the screen is adjusted to full screen through the position information of the screen; when the position of the blank area is small and the digital human model cannot be set, the purpose of setting the digital human model is achieved by adjusting the size of the courseware screen, for example, the courseware screen is reduced and placed in the upper left corner, and the digital human model is placed in the lower right corner, or the courseware screen is reduced and placed in the center, and the digital human model is placed in the lower left corner; in specific scenes, the size and position of the courseware screen are adjusted to achieve better visual effects, etc.

[0092] For each frame of the courseware screen, the size and position of the corresponding digital human model, the display information of the digital human model, and the size and position of the courseware screen can be determined through steps S206-S208.

[0093] S209: Generate digital human layout data based on the size and position of the digital human model corresponding to each courseware screen, the display information of the digital human model, the size and position of the courseware screen, and the duration.

[0094] After determining the size and position of the digital human model corresponding to each courseware screen, the display information of the digital human model, and the size and position of the courseware screen, the data corresponding to the courseware screen can be used to adjust the digital human model and the courseware screen to determine the position and size of the final synthesized screen and the digital human model, as well as the display information of the digital human model. Since the courseware screen will last for a period of time, that is, the courseware screen remains unchanged during the duration of this courseware screen, the size, position and display information of the digital human model and the screen during this period remain unchanged, that is, the layout remains unchanged, and the digital human layout data is generated based on the size, position and display information of the digital human model corresponding to each frame of video and the screen corresponding to the digital human model.

[0095] S210, performing video synthesis on the courseware video to be synthesized and the digital human model according to the digital human layout data.

[0096] After the digital human layout data is generated, the size and position of the picture and the digital human model corresponding to each video frame are determined accordingly, and the display information of the digital human model can also be determined. The picture is processed according to the size and position, and the digital human model is processed according to the size, position and display information to form a synthesized picture frame. All the synthesized picture frames are played in sequence to form a synthesized video.

[0097] For example, Figure 4 A display example diagram of the synthesis of a digital human model and a courseware screen is provided. It can be seen that the digital human model 21 is placed in the blank area at the lower left corner of the courseware screen 22, avoiding the situation of blocking text. Figure 5 Another example of a digital human model and a courseware screen after synthesis is provided. During the synthesis, the position and size of the courseware screen 22 are adjusted. This example can be applied in a specific scenario, such as a lecture hall scenario. In this scenario, the digital human model 21 is teaching at the podium, and the courseware screen 22 is displayed on a large screen in the lecture hall.

[0098] The embodiment of the present invention provides a video synthesis method, which solves the problem that all pictures in the courseware cannot be adapted due to the fixed scene manually set by the user in the synthesis process of digital human and video. By analyzing the blank area information and preset scene of each frame of the courseware picture, a suitable target scene is selected for the courseware picture. The selection of the target scene can be dynamically selected according to the user's selection and the courseware picture, and can be dynamically changed according to the preset rules. The digital human layout data is determined based on the data associated with the target scene and the duration of each courseware picture. When setting the digital human model, it can be adaptively set according to the size of the blank area in the courseware picture to better match the courseware picture and avoid occlusion. The video synthesis of the courseware video to be synthesized and the digital human model is performed based on the digital layout data, which effectively avoids occlusion and improves the teaching quality. There is no need for the user to manually set the selection, and the video synthesis scene is automatically and accurately controlled, which saves the user time and improves the user experience.

[0099] Embodiment 3

[0100] Figure 6 FIG. 1 is a schematic diagram of the structure of a video synthesis device provided in Embodiment 3 of the present invention. Figure 6 As shown, the device includes: a data acquisition module 31, a courseware analysis module 32, a layout data determination module 33 and a video synthesis module 34.

[0101] The data acquisition module 31 is used to acquire the courseware video to be synthesized, the digital human model and at least one preset scene;

[0102] The courseware analysis module 32 is used to analyze the courseware video to be synthesized, determine at least one frame of the courseware picture, and the duration and blank area information of each frame of the courseware picture;

[0103] A layout data determination module 33, used to determine the digital human layout data based on the duration and blank area information of each of the preset scenes and each of the courseware screens;

[0104] The video synthesis module 34 is used to perform video synthesis on the courseware video to be synthesized and the digital human model according to the digital human layout data.

[0105] The embodiment of the present invention provides a video synthesis device, which solves the problem that all pictures in the courseware cannot be adapted due to the fixed scene manually set by the user during the synthesis of digital humans and videos. The device analyzes the courseware video to be synthesized, determines each frame of the courseware picture in the courseware video, analyzes each frame of the courseware picture to determine the duration and blank area information, performs layout based on the preset scene and the duration and blank area information of each of the courseware pictures, determines the digital human layout data, selects a suitable layout method for each frame of the courseware picture in the courseware video to be synthesized to perform digital human layout, and performs video synthesis of the courseware video to be synthesized and the digital human model based on the digital layout data, effectively avoiding occlusion and improving the teaching quality; there is no need for the user to manually set and select, saving user time and improving user experience.

[0106] Optionally, the data acquisition module 31 includes:

[0107] An interface display unit, used to display a scene selection interface;

[0108] The operation receiving unit is used to receive a scene selection operation, where the scene selection operation is a user selecting at least one preset scene from the scene selection interface.

[0109] Optionally, the courseware analysis module 32 includes:

[0110] A picture comparison unit, used for comparing each frame of the to-be-synthesized courseware video, and extracting the courseware pictures that meet the preset difference condition;

[0111] A time determination unit, for counting the duration of each courseware picture frame according to the number of frames or time corresponding to the courseware picture frame;

[0112] A blank area determination unit, used to analyze the blank area of ​​the courseware screen and determine blank area information;

[0113] The blank area information includes the size and position of the blank area.

[0114] Optionally, the layout data determination module 33 includes:

[0115] A target scene determination unit, used to determine the target scene corresponding to each of the courseware screens according to each of the preset scenes and the blank area information of each of the courseware screens;

[0116] The model parameter determination unit is used to determine the size and position of the digital human model corresponding to each frame of the courseware screen according to the size range of the digital human model associated with the target scene and the blank area information in the screen; determine the display information of the digital human model corresponding to the courseware screen according to the display information of the digital human model associated with the target scene; determine the size and position of the courseware screen based on the position information of the screen associated with the target scene;

[0117] The layout data generating unit is used to generate the digital human layout data based on the size and position of the digital human model corresponding to each of the courseware screens, the display information of the digital human model, the size and position of the courseware screen, and the duration.

[0118] Optionally, the target scene determination unit includes:

[0119] A scene matching subunit, configured to analyze each of the courseware pictures based on each of the preset scenes and blank area information in the courseware pictures, and determine at least one scene that each of the courseware pictures matches;

[0120] The target scene determination subunit is used to screen the scenes matched by each of the courseware screens based on preset rules, and determine the target scene corresponding to each of the courseware screens.

[0121] Optionally, a scene matching subunit is used to obtain a first courseware screen, select a scene from the scenes matched by the first courseware screen as the target scene corresponding to the first courseware screen; take the next courseware screen of the first courseware screen as the screen to be matched, and based on the target scene corresponding to the previous screen to be matched of the screen to be matched, select a scene from the scenes matched by the screen to be matched as the target scene corresponding to the screen to be matched; take the next courseware screen of the screen to be matched as a new screen to be matched, return to execute the step of selecting a scene from the scenes matched by the screen to be matched as the target scene corresponding to the screen to be matched based on the target scene corresponding to the previous screen to be matched, until the target scenes corresponding to all courseware screens are determined.

[0122] Optionally, each of the preset scenes is associated with a size range of the digital human model, display information of the digital human model, and position information of the screen.

[0123] The video synthesis device provided in the embodiment of the present invention can execute the video synthesis method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0124] Embodiment 4

[0125] Figure 7 A schematic diagram of the structure of an electronic device 40 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0126] like Figure 7 As shown, the electronic device 40 includes at least one processor 41, and a memory connected to the at least one processor 41, such as a read-only memory (ROM) 42, a random access memory (RAM) 43, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 41 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 42 or the computer program loaded from the storage unit 48 to the random access memory (RAM) 43. In the RAM 43, various programs and data required for the operation of the electronic device 40 can also be stored. The processor 41, the ROM 42, and the RAM 43 are connected to each other through a bus 44. An input / output (I / O) interface 45 is also connected to the bus 44.

[0127] A number of components in the electronic device 40 are connected to the I / O interface 45, including: an input unit 46, such as a keyboard, a mouse, etc.; an output unit 47, such as various types of displays, speakers, etc.; a storage unit 48, such as a disk, an optical disk, etc.; and a communication unit 49, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 49 allows the electronic device 40 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0128] The processor 41 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 41 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 41 performs the various methods and processes described above, such as a video synthesis method.

[0129] In some embodiments, the video synthesis method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 48. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 40 via the ROM 42 and / or the communication unit 49. When the computer program is loaded into the RAM 43 and executed by the processor 41, one or more steps of the video synthesis method described above may be performed. Alternatively, in other embodiments, the processor 41 may be configured to perform the video synthesis method in any other suitable manner (e.g., by means of firmware).

[0130] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0131] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0132] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0133] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0134] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0135] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0136] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0137] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A video synthesis method, characterized in that: include: Obtaining a courseware video to be synthesized, a digital human model, and at least one preset scene; Analyze the courseware video to be synthesized to determine at least one frame of the courseware picture, as well as the duration and blank area information of each frame of the courseware picture; Determine digital human layout data based on the duration and blank area information of each preset scene and each courseware screen; The video of the courseware to be synthesized and the digital human model are synthesized according to the digital human layout data.

2. The method according to claim 1, characterized in that Get at least one preset scene, including: Display the scene selection interface; A scene selection operation is received, where the scene selection operation is a user selecting at least one preset scene from the scene selection interface.

3. The method according to claim 1, characterized in that The step of analyzing the courseware video to be synthesized to determine at least one frame of the courseware screen, as well as the duration and blank area information of each frame of the courseware screen, includes: Compare each frame of the to-be-synthesized courseware video to extract the courseware frames that meet the preset difference conditions; For each frame of the courseware screen, the duration of the courseware screen is counted according to the frame number or time corresponding to the courseware screen; Analyze the blank area of ​​the courseware screen to determine the blank area information; The blank area information includes the size and position of the blank area.

4. The method according to claim 1, characterized in that: The determining of the digital human layout data based on the duration and blank area information of each of the preset scenes and each of the courseware screens includes: Determine the target scene corresponding to each of the courseware screens according to each of the preset scenes and the blank area information of each of the courseware screens; For each frame of the courseware screen, determine the size and position of the digital human model corresponding to the courseware screen according to the size range of the digital human model associated with the target scene and the blank area information in the screen; Determine the display information of the digital human model corresponding to the courseware screen according to the display information of the digital human model associated with the target scene; Determine the size and position of the courseware screen based on the position information of the screen associated with the target scene; The digital human layout data is generated based on the size and position of the digital human model corresponding to each of the courseware screens, the display information of the digital human model, the size and position of the courseware screen, and the duration.

5. The method according to claim 4, characterized in that Determining the target scene corresponding to each courseware screen according to each preset scene and the blank area information of each courseware screen includes: Analyze each of the courseware screens based on each of the preset scenes and blank area information in the courseware screen to determine at least one scene that matches each of the courseware screens; The scenes matched by each of the courseware screens are screened based on preset rules to determine the target scene corresponding to each of the courseware screens.

6. The method according to claim 5, characterized in that The step of screening the scenes matched by each of the courseware screens based on a preset rule to determine the target scene corresponding to each of the courseware screens includes: Acquire a first courseware screen, and select a scene from the scenes matched by the first courseware screen as a target scene corresponding to the first courseware screen; The next courseware screen of the first courseware screen is used as the screen to be matched, and based on the target scene corresponding to the previous screen to be matched of the screen to be matched, a scene is selected from the scenes matched by the screen to be matched as the target scene corresponding to the screen to be matched; The next courseware screen of the screen to be matched is taken as the new screen to be matched, and the step of returning to execute the target scene corresponding to the previous screen to be matched based on the screen to be matched, and selecting a scene from the scenes matched by the screen to be matched as the target scene corresponding to the screen to be matched, until the target scenes corresponding to all courseware screens are determined.

7. The method according to any one of claims 1 to 6, characterized in that: Each of the preset scenes is associated with a size range of the digital human model, display information of the digital human model and position information of the screen.

8. A video synthesis device, characterized in that: include: A data acquisition module, used to acquire the courseware video to be synthesized, the digital human model and at least one preset scene; A courseware analysis module, used for analyzing the courseware video to be synthesized, determining at least one frame of the courseware picture, and the duration and blank area information of each frame of the courseware picture; A layout data determination module, used to determine the digital human layout data based on the duration and blank area information of each of the preset scenes and each of the courseware screens; The video synthesis module is used to perform video synthesis on the courseware video to be synthesized and the digital human model according to the digital human layout data.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the video synthesis method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the video synthesis method according to any one of claims 1 to 7 when executed.

Images