A method for a mobile device to obtain a customized background real-time dance video

Abstract

The present invention relates to a method for obtaining a customized background real-time dance video on a mobile device, including: obtaining a user video against a solid background; selecting a background template in response to a user's interaction operation; obtaining a background-free user dance video by using the color median value of the solid background; obtaining a synthesized dance video in real time based on the background template and the background-free user dance video; wherein obtaining the background-free user dance video by using the color median value of the solid background includes: sampling the color values at multiple positions of the solid background to obtain the color median value of the color values at multiple positions; expanding the color value range centered on the color median value; removing the background image within the color value range; displaying the color median value to the user; and in response to the user manually adjusting the color value range, adjusting the removed background color range in real time. This application synthesizes a customized background template in the recorded video in real time when recording a video on a mobile device, reducing the threshold for changing the background and improving the playability of the dance video.

Description

[0001] Relevant information of divisional application

[0002] This case is a divisional application. The parent case of this divisional application is a patent application for invention with the application date of September 10, 2020, application number 202010947691.1, and invention title "A method for obtaining a customized background real-time dance video on a mobile device". Technical field

[0003] The present invention relates to the technical field of video synthesis, and particularly to a method for obtaining a customized background real-time dance video on a mobile device. Background technique

[0004] Currently, various types of short videos are widely liked by users. The video types include music, technology, dance, etc., and the audience is extensive. With the rapid development of short videos, various video editing and processing functions are becoming more and more abundant.

[0005] However, due to the processing power of mobile phones, the processing and synthesis of videos currently rely heavily on computers. Especially for dance short videos, the recording environment directly affects the quality of the recorded video. Therefore, there is an urgent need for a technology to customize the background of dance videos in real time on a mobile device to solve the above problems. Summary of the invention

[0006] In view of the technical problems existing in the prior art, the present invention proposes a method for obtaining a customized background real-time dance video on a mobile device, including: obtaining a user video under a solid color background; selecting a background template in response to a user's interaction operation; obtaining a backgroundless user dance video using the color median value of the solid color background; and obtaining a synthesized dance video in real time based on the background template and the backgroundless user dance video; wherein obtaining a backgroundless user dance video using the color median value of the solid color background includes: sampling the color values at multiple positions of the solid color background to obtain the color median value of the color values at multiple positions; expanding the color value range centered on the color median value; removing the background image within the color value range; displaying the color median value to the user; and in response to the user manually adjusting the color value range, adjusting the removed background color range in real time.

[0007] The method as described above further includes: performing nine-grid convolution on the solid color background image to determine the boundary pixels and adjusting the transparency value of the boundary pixels according to the convolution value.

[0008] The method as described above, wherein before recording the video, the mobile device automatically or the user manually adjusts the focus and / or exposure parameters; during the recording of the video, the focus and / or exposure parameters are locked.

[0009] The method as described above further includes: before recording the video, in response to a user's interaction operation, editing one or more attributes of the synthesized dance video.

[0010] The method as described above further includes: editing the synthesized dance video into a multi-person dance team.

[0011] The method as described above further includes: performing one or more of beauty enhancement, body slimming, and filtering operations on the users in the synthesized dance video.

[0012] The method as described above further includes: adjusting one or more of the size, position, cropping width, and distance of the background-free user dance video.

[0013] The method as described above, wherein the color values are defined using the HSV color model.

[0014] The method as described above, wherein the colors of the solid-color background include green, blue, and red.

[0015] According to another aspect of the present application, a mobile device is provided, including: a camera module configured to acquire a user video under a solid-color background; an interaction module configured to select a background template in response to a user operation; a video processing module configured to obtain a background-free user dance video by using the color median value of the solid-color background; and obtain a synthesized dance video in real time based on the background template and the background-free user dance video; wherein the video processing module includes a video matteing unit configured to obtain a background-free user dance video based on the user video under the solid-color background from the camera module; the video matteing unit is further configured to: sample the color values at multiple positions of the solid-color background to obtain the color median value of the color values at multiple positions; expand the color value range centered on the color median value; and remove the background image within the color value range; display the color median value to the user; and in response to the user manually adjusting the color value range, adjust the removed background color range in real time.

[0016] The mobile device as described above, wherein the video processing module includes: a template providing unit configured to provide multiple background templates; and a video synthesis unit configured to obtain a synthesized dance video by using the background template and the background-free user dance video.

[0017] The mobile device as described above, wherein the video matteing unit is further configured to perform nine-grid convolution on the solid-color background image to determine the boundary pixels and adjust the transparency value of the boundary pixels according to the convolution value.

[0018] The mobile device as described above, wherein the camera module is further configured to automatically or manually adjust the focus and / or exposure parameters by the user before recording the video; and lock the focus and / or exposure parameters during video recording.

[0019] When this application records a video on a mobile device, a customized background template is synthesized in the recorded video in real time, simplifying the operation steps of replacing the background of the video, lowering the threshold for changing the background, being applicable to most mobile devices, and enhancing the playability of dance videos. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Next, the preferred embodiments of the present invention will be further described in detail with reference to the accompanying drawings, where:

[0021] Figure 1 is a schematic structural diagram of a system for generating a dance team combination dance video according to an embodiment of the present invention;

[0022] Figure 2 is a schematic structural diagram of a mobile device according to an embodiment of the present invention;

[0023] Figure 3 is a schematic structural diagram of a server according to an embodiment of the present invention;

[0024] Figure 4 is a flowchart of a method for a mobile device to obtain a customized background real-time dance video according to an embodiment of the present invention;

[0025] Figure 5A -E is a graphical user interface for a mobile device to generate a dance team combination dance video;

[0026] Figure 6 is a flowchart of a method for a mobile device to generate a dance team combination dance video according to an embodiment of the present invention;

[0027] Figure 7 is a schematic diagram of a dance team formation according to an embodiment of the present invention;

[0028] Figure 8 is a schematic diagram of a scene for synthesizing a dance video; and

[0029] Figure 9 is a method for a mobile device to generate a dance video adapted to light and dark environments according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0030] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0031] In the following detailed description, reference is made to the various specification drawings that form a part of the present application and illustrate specific embodiments of the present application. In the drawings, like reference numerals generally describe substantially similar components in different figures. The specific embodiments of the present application are described in sufficient detail below so that those of ordinary skill in the relevant art and technology can implement the technical solutions of the present application. It should be understood that other embodiments may also be utilized or structural, logical, or electrical changes may be made to the embodiments of the present application.

[0032] Figure 1 FIG. 1 is a schematic structural diagram of a system for generating a dance team combination dance video according to an embodiment of the present invention. As shown in the figure, the system includes one or more mobile terminals 10 and a server 20, which are communicatively connected via a wireless network. Among them, the mobile terminal 10 includes a mobile phone, a tablet computer, a personal entertainment device, or a combination thereof, and is used to record a dance video and, based on a specific application, replace the background of the video with a background template in real time. After the recording is completed, the user can store the video in the personal account of the server via the wireless network or share the dance video to show it to more people.

[0033] Figure 2 FIG. 2 is a schematic structural diagram of a mobile terminal according to an embodiment of the present invention. As shown in the figure, the mobile terminal 10 includes a camera module 101, a video processing module 102, a communication module 103, and an interaction module 104. Among them, the video processing module 102 is electrically connected to the camera module 101, the communication module 103, and the interaction module 104 respectively, and the camera module 101 is electrically connected to the interaction module 104.

[0034] The camera module 101 is one or more cameras for acquiring a user video under a solid color background. The camera can be the front camera or the rear camera of the mobile terminal. The camera includes functions of automatic focus and manual focus to meet the needs of different users.

[0035] The video processing module 102 includes a template providing unit 1021, a video matting unit 1022, an attribute editing unit 1023, and a video synthesizing unit 1024. Among them, the template providing unit 1021 is electrically connected to the interaction module 104 and the attribute editing module 1023 respectively; the video matting unit 1022 is electrically connected to the camera module 101 and the attribute editing unit 1023 respectively; the attribute editing unit 1023 is electrically connected to the interaction module 104 and the video synthesizing unit 1024 respectively, and the video synthesizing unit 1024 is electrically connected to the communication module 103.

[0036] The template providing unit 1021 is used to provide a plurality of background templates and / or dance team templates, where the background template can be a video or a picture. Among them, the background template types include but are not limited to beautiful scenery, indoor, stage, festival, atmosphere, and green screen. Among them, the green screen type of background is used as a customized background template for real-time video recording, which can adapt to the entire screen and be synthesized with the background-free user dance video that has been keyed out, making the real recording environment completely invisible. Dynamic elements such as flying birds, waterfalls, rowing boats, etc. can be added to the green screen type of background template to enrich the video content.

[0037] The video keying unit 1022 receives images from the camera module 101 and obtains a background-free user dance video by using the color median value of a solid color background. In some embodiments, the solid color background colors include but are not limited to green, blue, and red. Those skilled in the art should understand that any solid color background that can be used to obtain a background-free user dance video can be applied to the solution of this application.

[0038] The attribute editing unit 1023 receives instructions from the interaction module 104 to adjust video attributes. For example, adjusting parameters such as the hue, contrast, and aspect ratio of the background-free user dance video and / or the background template. In response to instructions generated by the user's interaction on the interaction module 104, beauty effects are applied to the users in the video, or the displayed body shape is adjusted, or the video hue is adjusted, etc.

[0039] In some embodiments, the attribute editing unit 1023 further includes functions such as adjusting the size, position, distance, and combined dance team of the background-free user dance video. When the user records a video, the angle and recording position between the mobile device and the background-free user dance video may not present the best shooting effect. Therefore, the size, position, and distance of the obtained background-free user dance video in the background template can be adjusted to make it blend more naturally with the background template. Among them, size means adjusting the size of the background-free user dance video; distance means simulating the effect of adjusting the camera's far and near distances; position means adjusting the position of the background-free user dance video in the background template. The combined dance team means that when the user shoots a dance video, multiple images of the background-free user dance video are copied and combined into a video of multiple people dancing simultaneously according to the standing positions of the dance team. The movements in the dance are neat and uniform, improving the visibility of the dance video. Among them, the dance team change modes include the 1-person dance team mode, the 2-person dance team mode, the parallel 3-person dance team mode, the triangular 3-person dance team mode, etc.

[0040] The video synthesis unit 1024 is used to synthesize a dance video by obtaining a background template, a dance team template, and a backgroundless user dance video in real time. Among them, the video synthesis unit 1024 is electrically connected to the interaction module 104, and it sends the synthesized video to the interaction module 104, where the user can watch the synthesized video in real time. The video synthesis unit 1024 can also send the synthesized video to the server 20 through the communication module 103 to share the self-made video with more people for viewing.

[0041] The communication module 103 can include one or more wired or wireless communication interfaces. For example, the communication interface can be a network interface card, a wireless modem, or a wired modem. In one application, the communication module 103 can be a WiFi modem. In other applications, the communication module 103 can be a 3G modem, a 4G modem, an LTE modem, a Bluetooth component, a radio frequency receiver, an antenna, or a combination thereof. The communication module 103 can communicate wirelessly with the server 20, and the mobile device 10 can use the communication module 103 to transmit information or receive packets or messages.

[0042] The interaction module 104 is used to display a graphical user interface (GUI) to interact with the user, and can also display functional content to the user, such as function buttons, text boxes, or links for displaying background templates and editing video attributes, etc. The user can select a background template on the interaction module 104 through gestures such as swiping and clicking. The interaction module 104 can be a touch screen display of a liquid crystal display (LCD) or an organic light emitting diode (OLED).

[0043] Figure 3 It is a schematic structural diagram of a server according to an embodiment of the present invention. As shown in the figure, the server includes one or more processing modules 301, a communication module 302, and a storage module 303.

[0044] The processing module 301 can include one or more central processing units (CPUs), graphics processing units (GPUs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or a combination thereof. The processing module 301 can execute software or computer-readable instructions stored in the storage module 303 to perform the methods or operations described herein. The processing module 301 can be implemented in several different ways. For example, the processing module 301 can include one or more embedded processors, processor cores, microprocessors, logic circuits, hardware finite state machines (FSMs), digital signal processors (DSPs), or a combination thereof.

[0045] The communication module 302 can include one or more wired or wireless communication interfaces. For example, the communication interface can be a network interface card, a wireless modem, or a wired modem. In one application, the communication module 302 can be a WiFi modem. In other applications, the communication module 302 can be a 3G modem, a 4G modem, an LTE modem, a Bluetooth component, a radio frequency receiver, an antenna, or a combination thereof.

[0046] The storage module 303 can store software, data, logs, or a combination thereof. The storage module 303 can be an internal memory or an external memory. For example, the memory can be a volatile memory or a non-volatile memory, such as a non-volatile random access memory (NVRAM), a flash memory, a non-volatile memory of a disk memory, or a volatile memory such as a static random access memory (SRAM).

[0047] Figure 4 is a flowchart of a method for a mobile device to obtain a customized background real-time dance video according to an embodiment of the present invention; Figure 5A -E is a graphical user interface for a mobile device to obtain a customized background real-time dance video. In step 410, a user video under a solid color background is acquired. Refer to Figure 5A , in response to the user selecting an accompaniment, the camera of the mobile device acquires a user video under a solid color background. In some embodiments, the user can adjust the position and shooting distance of the mobile device according to the user video displayed on the interaction device 104.

[0048] In step 420, in response to the user's operation on the interaction module, a background template is selected. Refer to Figure 5B , the user selects the "Background" button on the interaction module, and a list of background templates appears. The customizable background templates are in the type of "green screen". Not all background templates can be customized background templates, which need to meet certain requirements, such as not being a transparent background and needing to completely cover the interaction module, etc. In response to the user selecting any background template in the "green screen" background template list, the selected background template is displayed on the interaction module.

[0049] In step 430, a background-free user dance video is obtained by using the color median value of the solid color background. In some embodiments, the specific steps for obtaining a background-free user dance video are as follows:

[0050] 1. Sample the color values at multiple positions of the solid color background to obtain the color median value of the color values at multiple positions;

[0051] 2. Expand the color value range centered on the color median value; and

[0052] 3. Remove the background image within the color value range.

[0053] In some embodiments, the HSV color model is adopted to define color values. Among them, the median color value can be obtained by sorting the color values at multiple positions according to their magnitudes and taking the middle value of the multiple color values. When the user records a video, sometimes due to the uneven illumination of the pure-color background by the light in the recording environment, the color values of the pure-color background in the recorded video are inconsistent, affecting the keying effect. Therefore, multiple color values at different pure-color background positions are obtained, the median value is calculated and extended by a certain range to improve the quality of the background-free user dance video. The extended range is the empirical range of the values of a certain color (such as green / blue / red, etc.) in the HSV color space.

[0054] In some embodiments, a nine-grid convolution is performed on the pure-color background image to determine the boundary pixels and the transparency values of the boundary pixels are adjusted according to the convolution values, thereby effectively reducing the jaggedness of the keying. This application uses a convolutional neural network to judge the boundaries of video pixels, and sets different transparency values for different pixels through the convolution values, making the obtained target user image more complete and more natural.

[0055] In step 440, in response to the user's operation on the interaction module, the video attributes are edited. Refer to Figure 5B , sometimes the quality of the background-free user dance video automatically obtained by the mobile device is not high, and there is still some green background. At this time, the user can click the "green screen" function key on the screen to enter Figure 5C , and the slider of the "chroma key" can be adjusted. As the slider moves, the green part on the screen will increase or decrease. The user stops sliding when the green part completely disappears or is minimized. At this time, the best background-free user dance video is presented. In some embodiments, the mobile device displays the median color value to the user; and in response to the user manually adjusting the color value, the range of the removed background color is adjusted in real time. If the slider on the "chroma key" stays at the median color value, the user only needs to slightly adjust the color value to present the best background-free user dance video, reducing the user's adjustment range. Refer to Figure 5D , by sliding the sliders of different functions, one or more of the size, position, cropping width, and distance of the background-free user dance video can be adjusted to adapt to the background template.

[0056] In some embodiments, editing the video attributes further includes performing one or more of beauty enhancement, slimming, and filtering operations on the user in the synthesized dance video. The user can make personalized adjustments to the user image in the video according to their own preferences, making the user in the video more beautiful.

[0057] In step 450, a synthesized dance video is obtained in real time based on the background template and the background-free user dance video. After the user finishes editing the video attributes, in response to the user clicking the "Shoot" function key, the mobile device starts recording the video, and synthesizes the background template and the background-free user dance video into a video in real time. When the user clicks "End", the synthesized video is saved in the mobile device.

[0058] In some embodiments, before recording the video, the mobile device can automatically or manually adjusted by the user the focus and exposure parameters to make the camera achieve an optimal shooting effect; during the video recording, the mobile device locks the focus and exposure parameters to ensure that a background-free user dance video with better quality is obtained during the recording process. If the auto-focus or auto-exposure function is turned on during the recording process by the camera, the pure color background color value obtained by the mobile device may change, resulting in the background-free user dance video obtained containing a pure color background, causing a problem of degraded matte extraction quality.

[0059] Reference Figure 5E , in response to the user clicking the "Split Screen" function key on the left side of the screen, the synthesized dance video can be edited into a multi-person dance team. In some embodiments, there are multiple dance team templates available for the user to choose from in the split screen function, and their formation positions are fixed. In other embodiments, the user customizes the positions of the background-free user dance videos, and the user can freely select the number and positions of the background-free user dance videos to design the dance team formation. Among them, in the custom dance team, the maximum number of background-free user dance videos is 8.

[0060] For traditional dance videos, to solve the problem of unnatural stacking of background-free user dance videos in the synthesized dance video, different layers of background-free user dance videos are calculated using the pixel blending mode to achieve a realistic overlapping visual effect. However, during the processing, each pixel of adjacent layers needs to be calculated and rendered, resulting in a large amount of calculation. It is difficult to popularize the above method on mobile devices with limited computing power. Therefore, this application designs a method that can perform hybrid optimization on background-free user dance videos of different levels on mobile devices, with advantages such as small amount of calculation and fast response speed.

[0061] Figure 6 is a flowchart of a method for a mobile device to generate a dance team combined dance video according to an embodiment of the present invention. In step 610, a background-free user dance video is obtained under a pure color background. The method for obtaining the background-free user dance video here is the same as the steps for obtaining the background-free user dance in the above Figure 4 method, and will not be elaborated here.

[0062] In step 620, according to the dance team template selected by the user, determine the combination method of the background-free user dance video. Before starting to record the video, the user can operate on the interactive module of the mobile device to select a dance team template. In some embodiments, a list of dance team templates is displayed on the interactive module. After the user selects a dance team template, the dance team formation will be displayed in real time on the interactive module. In other embodiments, the user clicks the function button on the interactive module, and the dance team formation changes in real time on the interactive module. The dance team template includes various dance team formations, and for different dance team formations, the corresponding levels will also be different.

[0063] Among them, the dance team formation is the standing position of the dancers. For different dance team formations, the system will correspondingly display different combination methods of the background-free user dance video. Among them, the dance team formation can three-dimensionalize the space in the video, dividing it into front, back, left, right and other directions. The dance team formation can be combined into a triangle, a pyramid, etc., improving the visual effect of the video.

[0064] Figure 7 It is a schematic diagram of the dance team formation according to an embodiment of the present invention. Generally speaking, the more dancers in the dance team, the more levels. Refer to Figure 7 , taking a three-person dance team with 1 in the front and 2 in the back as an example, the 1 dancer in the front is at level 0, and the 2 parallel dancers in the back are at level 1. Among them, the display ratio of level 0 is greater than that of level 1. The background-free user dance video that protrudes forward enables the user watching the video to clearly appreciate the dance video more, improving the viewing satisfaction. Among them, the dancers in level 0 and level 1 are all obtained by copying multiple background-removed user dance videos and distributing them at different positions. Therefore, the actions of the dancers at different levels are exactly the same. Those skilled in the art should understand that the levels in the dance team template are not limited to 2 levels, and can also be 3 levels or even more levels.

[0065] In step 630, determine one or more overlapping parts after the combination of the background-free user dance videos. In some embodiments, according to the dance team formation in the dance team template, determine the priority of each layer composed of the background-free user dance videos. Among them, the layer of the background-free user dance video located at the back is preferentially drawn. When combining into the dance team formation, it is necessary to layer the background-free user dance videos layer by layer. In some embodiments, the background-free user dance videos can be drawn in sequence from the back to the front. The overlapping parts of the background-free user dance videos are the parts where the background-free user dance videos at different front and back levels overlap, such as Figure 7 the overlapping part 710 and the overlapping part 720 in. Further, the present application can determine the boundaries of one or more overlapping parts, so as to reduce the calculation amount and improve the rendering speed.

[0066] In some embodiments, the boundaries of one or more overlapping portions are determined using the pixel coordinate values of the background-free user dance videos. For example, the coordinate values of the hierarchical pixels of level 0 and level 1 are respectively determined relative to the interaction module. In the same coordinate system, the same coordinates represent the same position on the interaction module. Therefore, by comparing the coordinate values of the user images in multiple background-free user dance videos, the boundaries of the overlapping portions can be determined.

[0067] In other embodiments, the boundaries of one or more overlapping portions are determined using the pixel color values between the background-free user dance videos. Refer to Figure 7 , in the background-free user dance video of level 0, except for the portrait part, the pixels of other parts are transparent. Therefore, by obtaining the video pixels of the level adjacent to level 0, the boundaries of the overlapping portions of the background-free user dance videos can be quickly obtained. Those skilled in the art should understand that any technology that can determine the overlapping portions of the dance videos can be applied to the solution of this application.

[0068] In step 640, a rendering transition process is performed on one or more overlapping portions. Among them, the steps of performing a rendering transition process on one or more overlapping portions include:

[0069] Determine the boundary expansion range based on one or more overlapping portions;

[0070] The boundary expansion range extends 2 - 8 pixels inward from the boundaries of one or more overlapping portions; and

[0071] Use the pixel blending mode to perform a rendering transition process on the pixels within the expansion range.

[0072] Among them, the size of the boundary expansion range is related to the display ratio of the background-free user dance videos. As can be seen from the above, the display ratio of the background-free user dance video arranged in the front layer is greater than that in the back. Therefore, determining the boundary expansion range according to the display ratio of the background-free user dance videos can refine the rendering transition process range and improve the quality of the synthesized dance video.

[0073] In some embodiments, the rendering transition process is a color blending special effect process. For example, OpenGL ES is used to perform a rendering process on the pixels within the expansion range. Further, the function glBlendFunc(GL_ONE, GL_ONE_MINUS_SRC_ALPHA) is used to process the pixels within the expansion range. Among them, the colors in the dance videos of different levels within the expansion range are respectively taken out and multiplied by a coefficient, and then mixed to obtain a new color, making the stacking relationship between the dance videos of different levels more natural and realistic. Among them, the algorithm in the glBlendFunc function can be adjusted according to actual needs and is not limited here.

[0074] In some other embodiments, the rendering transition process is a Gaussian weighted averaging process. Gaussian weighted averaging means using the gray values of multiple pixels in the pixel neighborhood to perform weighted average processing and then replacing the gray value of the original pixel, eliminating the sawtooth and unnatural problems in the overlapping parts of multiple video layers, making the overlapping parts smooth, natural, and realistic. Those skilled in the art should understand that using the above technical means to process the overlapping parts of multiple videos is only an example scheme illustrated in this application. Any technical solution that can solve the above technical problems can be applied in this application, and no limitation is made here.

[0075] Figure 8 It is a schematic diagram of a scene for synthesizing a dance video according to an embodiment of the present invention. As shown in the figure, the dance video includes a background template 810 and one or more backgroundless user dance videos 820, where the background template 810 includes a dancing area 811 of the user. The background template 810 is an outdoor scenery template. Accordingly, the light source is set as the sun 813, which meets the requirements of the outdoor scenery template, and natural scenery such as white clouds 812, rivers 814, and trees 815 are added to make the content of the background template more abundant. Among them, the background template is divided into multiple types. According to the scene, it can be divided into indoor scenery and outdoor scenery; according to the dynamic and static type, it can be divided into dynamic backgrounds and static backgrounds, where the elements in the dynamic background include but are not limited to rivers, flying birds, and waterfalls; according to the environment, it can be divided into mountain and water backgrounds, flower bush backgrounds, and cultural landscape backgrounds, etc. In addition, the user can select a dance team template to determine the standing positions of multiple dancing users in the dancing area. In order to make the synthesis of multiple backgroundless user dance videos and the background template more realistic, this application adds a function of simulating light and shade adaptation to the dance video. By configuring elements such as light sources in the dance video, the three-dimensional sense of the video is increased. The specific method is as follows:

[0076] Figure 9A method for generating a dance video adapted to bright and dark environments on a mobile device according to an embodiment of the present invention. In step 910, information about the light source of the user video is determined based on the user video against a solid-color background. The specific steps are as follows: Collect pixel points of multiple solid-color backgrounds in the user video against a solid-color background; Use a neural network model in deep learning to analyze the pixel points of multiple solid-color backgrounds to determine multiple light paths; Based on the determined multiple light paths, determine the information about the light source of the user video. The template light source information includes one or more of the brightness, position, quantity, color, and dynamic effect of the light source. For example, when the user stands in front of a green screen background to record a dance video and there is a light bulb in the upper left of the user to provide illumination for the video recording of the user; According to the principle that light travels in a straight line, the light hitting the user should be diagonally downward, and when the light hits the user, the color and brightness of the pixels will change. Therefore, use a neural network model in deep learning to analyze the pixel points of multiple solid-color backgrounds to determine the irradiation directions of multiple light paths; The position of the light source of the user video can be determined according to the irradiation directions of the multiple light paths. Those skilled in the art should understand that only approximate information about the light source of the user video needs to be determined in this application, and particularly accurate light source information is not required. As long as the technical solutions that can achieve the above purposes can be applied in this application, no limitations are made here.

[0077] In step 920, the configuration information of the template light source is determined according to the dance team template selected by the user. Different dance team templates have different configured template light sources. As mentioned above, in the outdoor scenery template, the template light source can be the sun; in the indoor background template, the template light source can be a light bulb, etc. And according to the characteristics of different background templates, the information such as the color and lighting of the template light source will also be different.

[0078] In step 930, the information about the light source of the user video or the configuration information of the template light source is modified. In some embodiments, the configuration information of the template light source is kept consistent with the information about the light source of the user video. To ensure that the configuration information of the template light source is kept consistent with the information about the light source of the user video, the information about the light source of the user video can be modified, or the configuration information of the template light source can be modified. Among them, in response to the change in the position of the template light source, the light and dark changes of the user's body in the background-free user dance video are modified. Preferably, the light source configuration information with a small calculation amount is modified. By keeping the position, color, and brightness of the template light source and the light source of the user video consistent in this application, the synthesized dance video is more coordinated and more natural.

[0079] In some embodiments, the configuration information of the template light source is modified based on the mobile system time. Among them, the types of the template light source include natural light sources and artificial light sources. For example, when the user is recording a dance video using the mobile device and the system time of the mobile device is 12 noon, the sun is set in the middle of the sky and the brightness is adjusted to the maximum; when the system time is 8 pm, the sun can be replaced with a street lamp as the light source, and the color tone of the light source is adjusted to a warm color tone. In the solution of the present application, the light emitted by the template light source is all directional light, and the light rays emitted by the directional light create a light and dark line in the synthesized dance video, highlighting the authenticity of the scene.

[0080] In some embodiments, the configuration information of the template light source is modified based on the accompaniment of the dance video. Music and lighting will enhance the dance atmosphere and mobilize the user's emotions. When the user records a dance video, the system will adjust the information such as the brightness, position, color, and dynamic effect of the template light source according to the playing time of the accompaniment music. For example, when the accompaniment music of the user's dance plays to the climax part, the template light source will flash at a certain frequency and change the color of the light source to improve the fun of the video.

[0081] In step 940, the dance team template and the background-free user dance video are combined. In response to adjusting the information of the user video light source or the configuration information of the template light source, the dance team template and the background-free user dance video are combined. Among them, in the combined dance video, there will be light and dark line changes according to the configuration information of the template light source. Among them, in an environment with multiple template light sources and / or multiple background-free user dance videos, when the directional light is projected on the user, the brightness of the illuminated part of the user increases. When there are multiple template light sources in the background template, the light rays will be superimposed on the background-free user dance video. Therefore, the light brightness of the superimposed part is greater than that of the non-superimposed part. In some embodiments, the pixel blending mode is used to blend and render the color of the light emitted by the template light source with the color of the part of the background-free user dance video irradiated by the light.

[0082] In addition, the light source generates a corresponding shape of shadow on the ground according to the contour of the light blockage. For example, when the light source emits directional light rays and irradiates on the user, the audience will see bright parts and shadow parts in the environment, making the whole environment more three-dimensional and real. Among them, when the number of light sources increases by one, the number of generated shadows doubles; multiple background-free user dance videos will generate multiple shadows.

[0083] In some embodiments, the configuration information of the template light source and / or the combination method of the background-free user dance video are used to simulate the user shadows in multiple background-free user dance videos; multiple background-free user dances

[0084] The user shadow in the video generates a shadow layer; and the shadow layer is superimposed and synthesized with multiple backgroundless user dance videos 5 and a background template. Among them, the superimposed mixing process of multiple template light sources and / or the user shadow generated by multiple backgroundless user dance videos generates the color of the superimposed part. This application performs a superimposed mixing process on the superposition of multiple lights and the superposition of multiple shadow parts, making the overall ambient light bright and dark obvious, creating a very realistic dance environment, and improving the user experience.

[0085] 0 The above embodiments are only for illustrating the present invention and are not intended to limit the present invention. Those of ordinary skill in the relevant technical fields can also make various changes and modifications without departing from the scope of the present invention. Therefore, all equivalent technical solutions should also fall within the scope of the disclosure of the present invention.

Claims

1. A method for a mobile device to obtain a customized background real-time dance video, including: Obtaining a user video under a solid-color background; Selecting a background template in response to a user's interaction operation; Obtaining a background-free user dance video by using the color median value of the solid-color background; And Obtaining a synthesized dance video in real time based on the background template and the background-free user dance video; Wherein, obtaining a background-free user dance video by using the color median value of the solid-color background includes: sampling the color values at multiple positions of the solid-color background to obtain the color median value of the color values at multiple positions; expanding the color value range centered on the color median value; removing the background images within the color value range; displaying the color median value to the user; and in response to the user manually adjusting the color value range, adjusting the removed background color range in real time; Wherein, in the process of obtaining a synthesized dance video in real time based on the background template and the background-free user dance video, editing the synthesized dance video into a multi-person dance team and performing light source configuration on the synthesized dance video; Wherein, the editing the synthesized dance video into a multi-person dance team specifically includes: Determining the combination method of the background-free user dance video according to the dance team template selected by the user, and the combination method includes copying multiple images of the background-free user dance video and combining them into a video of multiple people dancing simultaneously according to the positions of the dance team; Determining one or more overlapping parts after combining the background-free user dance videos; Performing rendering transition processing on one or more overlapping parts; Wherein, the performing rendering transition processing on one or more overlapping parts specifically includes: Determining the boundary expansion range based on one or more overlapping parts, and the boundary expansion range is to expand 2-8 pixels inward from the boundaries of one or more overlapping parts; and Performing rendering transition processing on the pixels within the expansion range by using the pixel blending mode; Wherein, the performing light source configuration on the synthesized dance video specifically includes: Determining the information of the light source of the user video according to the user video under the solid-color background; Determining the configuration information of the template light source according to the dance team template selected by the user; Modifying the information of the light source of the user video or the configuration information of the template light source; and Combining the dance team template and the background-free user dance video; Wherein, the determining the information of the light source of the user video specifically includes: Collecting the pixel points of multiple solid-color backgrounds in the user video under the solid-color background; Analyzing the pixel points of multiple solid-color backgrounds by using a neural network model in deep learning to determine multiple light paths; Determining the information of the light source of the user video based on the determined multiple light paths.

2. The method according to claim 1, further comprising: Performing nine-grid convolution on the solid-color background image to determine the boundary pixels and adjusting the transparency values of the boundary pixels according to the convolution values.

3. The method according to claim 1, wherein before recording the video, the mobile device automatically or the user manually adjusts the focus and / or exposure parameters; during recording the video, locking the focus and / or exposure parameters.

4. The method according to claim 1, further comprising: Before recording the video, in response to the user's interaction operation, editing one or more attributes of the synthesized dance video.

5. The method according to claim 4, further comprising: Wherein performing one or more of beauty, slimming, and filtering operations on the users in the synthesized dance video.

6. The method according to claim 4, further comprising: Adjusting one or more of the size, position, cropping width, and distance of the background-free user dance video.

7. The method according to claim 1, wherein the color value is defined using the HSV color model.

8. The method according to claim 1, wherein the colors of the solid-color background include green, blue, and red.

9. A mobile device, comprising: a camera module configured to acquire a user video under a solid-color background; an interaction module configured to select a background template in response to a user operation; a video processing module configured to obtain a background-free user dance video using the color median value of the solid-color background; and, based on the background template and the background-free user dance video, obtain a synthesized dance video in real time; wherein the video processing module includes a video matting unit configured to obtain a background-free user dance video based on the user video under the solid-color background from the camera module; the video matting unit is further configured to: sample the color values at multiple positions of the solid-color background to obtain the color median value of the color values at multiple positions; expand the color value range centered on the color median value; and remove the background image within the color value range; display the color median value to the user; and in response to the user manually adjusting the color value range, adjust the removed background color range in real time; the video processing module is further configured to, during the process of obtaining the synthesized dance video in real time based on the background template and the background-free user dance video, edit the synthesized dance video into a multi-person dance team and perform light source configuration on the synthesized dance video; wherein, the editing the synthesized dance video into a multi-person dance team specifically includes: determining the combination method of the background-free user dance video according to the dance team template selected by the user, and the combination method includes copying multiple images of the background-free user dance video and combining them into a video of multiple people dancing simultaneously according to the standing positions of the dance team; determining one or more overlapping parts after combining the background-free user dance videos; performing rendering transition processing on one or more overlapping parts; wherein, the performing rendering transition processing on one or more overlapping parts specifically includes: determining the boundary expansion range based on one or more overlapping parts, and the boundary expansion range is to expand 2-8 pixels inward from the boundaries of one or more overlapping parts; and, performing rendering transition processing on the pixels within the expansion range using the pixel blending mode; wherein, the performing light source configuration on the synthesized dance video specifically includes: determining the information of the light source of the user video according to the user video under the solid-color background; determining the configuration information of the template light source according to the dance team template selected by the user; modifying the information of the light source of the user video or the configuration information of the template light source; and combining the dance team template and the background-free user dance video; wherein, the determining the information of the light source of the user video specifically includes: collecting the pixel points of multiple solid-color backgrounds in the user video under the solid-color background; analyzing the pixel points of multiple solid-color backgrounds using a neural network model in deep learning to determine multiple light paths; determining the information of the light source of the user video based on the determined multiple light paths.

10. The mobile device according to claim 9, wherein the video processing module includes: a template providing unit configured to provide multiple background templates; and A video synthesis unit configured to obtain a synthesized dance video by using a background template and a user dance video without a background.

11. The mobile device according to claim 9, wherein the video matting unit is further configured to perform a nine-grid convolution on a solid-color background image to determine boundary pixels and adjust the transparency value of the boundary pixels according to the convolution value.

12. The mobile device according to claim 11, wherein the camera module is further configured to automatically or manually adjust the focus and / or exposure parameters by the user before recording a video; and lock the focus and / or exposure parameters during video recording.

Images