Control method and device, electronic equipment and vehicle

Abstract

The invention discloses a control method and device, electronic equipment and a vehicle, and the method comprises the steps: determining a target person and a cover image corresponding to target music; the target person is determined from at least one person related to the target music; the cover image is an image related to the target music; generating a first image containing the target person based on the cover image; and based on the target music, driving the target person in the first image to act, and generating a target video of the target music.

Description

Technical Field

[0001] The present application relates to but is not limited to the field of computer technology, and in particular to a control method, device, electronic equipment and vehicle. Background Art

[0002] The in-vehicle entertainment system is a system used to provide entertainment functions such as audio, video, navigation, and games inside the vehicle. Among them, the audio player is one of the most frequently used applications in the in-vehicle entertainment system. At present, various vehicle manufacturers mostly use the method of directly connecting the car machine version to the audio software to allow users to play on demand.

[0003] Unlike mobile phones, computers and other devices, car displays are always on and have a large screen. Therefore, providing richer, more vivid and interesting interactive displays when playing music has become an important demand of users. Summary of the invention

[0004] In view of this, the present application provides at least one control method, device, equipment and vehicle.

[0005] The technical solution of this application is implemented as follows:

[0006] In one aspect, the present application provides a control method, the method comprising:

[0007] Determine a target person and a cover image corresponding to the target music; the target person is determined from at least one person related to the target music; the cover image is an image related to the target music;

[0008] Based on the cover image, generating a first image including a target person;

[0009] Based on the target music, the target person in the first image is driven to move, and a target video of the target music is generated.

[0010] In some embodiments, generating a first image including a target person based on the cover image includes:

[0011] Determine image style information of the cover image;

[0012] Based on the image style information of the cover image, a first image including the target person is generated.

[0013] In some embodiments, generating a first image including a target person based on the cover image includes:

[0014] Acquire a second image of the target person; the second image represents a front view of the target person;

[0015] Determining first characteristic information of the target person based on the second image;

[0016] Based on the cover image and the first feature information, a first image is generated.

[0017] In some embodiments, the method further comprises:

[0018] Based on the second image, second feature information corresponding to the target person is determined; the second feature information represents more detailed feature information corresponding to the target person relative to the first feature information;

[0019] Based on the target audio, driving the target person's action in the first image to generate a target video of the target music includes:

[0020] Based on the target music, driving the target person in the first image to move, and generating a first video;

[0021] The target person in the first video is redrawn based on the second feature information to obtain a target video.

[0022] In some embodiments, generating a first image including a target person based on the cover image includes:

[0023] Determine third feature information corresponding to the reference image; the third feature information at least includes information about the proportion of the reference person in the reference image;

[0024] Based on the cover image and the third feature information, a first image including the target person is generated.

[0025] In some embodiments, based on the target music, driving the action of the target person in the first image includes at least one of the following:

[0026] Determine the music style of the target music; and determine the dance style of the target person in the first image based on the music style;

[0027] Determine the rhythm characteristics of the target music; determine the movement speed of the target person in the first image based on the rhythm characteristics;

[0028] The lyrics information of the target music is obtained; based on the lyrics information and the playing progress of the target music, the lip movements of the target person in the first image are driven.

[0029] In some embodiments, the method further comprises:

[0030] Based on the target music, an animation effect of the background image in the first image is determined.

[0031] On the other hand, the present application also provides a control device, comprising:

[0032] The first determination module is used to determine a target person and a cover image corresponding to the target music; the target person is determined from at least one person related to the target music; the cover image is an image related to the target music;

[0033] A first generating module, used to generate a first image containing a target person based on the cover image;

[0034] A driving module is used to drive the target person's action in the first image based on the target music to generate a target video of the target music.

[0035] On the other hand, the present application also provides an electronic device, including a processor and a display, wherein:

[0036] The processor is used to determine a target person and a cover image corresponding to the target music; the target person is determined from at least one person related to the target music; the cover image is an image related to the target music; based on the cover image, a first image containing the target person is generated; based on the target music, a target person in the first image is driven to move, and a target video corresponding to the target music is generated;

[0037] The display shown is used to display and output the first video.

[0038] In another aspect, the present application also provides a vehicle, comprising:

[0039] A vehicle body and an in-vehicle entertainment system mounted on the vehicle body; the in-vehicle entertainment system includes a processor and a display; wherein,

[0040] The processor is used to determine a target person and a cover image corresponding to the target music; the target person is determined from at least one person related to the target music; the cover image is an image related to the target music; based on the cover image, a first image containing the target person is generated; based on the target music, a target person in the first image is driven to move, and a target video corresponding to the target music is generated;

[0041] The display shown is used to display and output the first video.

[0042] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and are not intended to limit the technical solutions of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and are used together with the specification to illustrate the technical solution of the present application.

[0044] Figure 1 A schematic diagram of an implementation flow of a control method provided in this application;

[0045] Figure 2 A schematic diagram of the implementation flow of another control method provided in this application;

[0046] Figure 3 A schematic diagram of the implementation flow of another control method provided in this application;

[0047] Figure 4 A schematic diagram of the implementation flow of another control method provided in this application;

[0048] Figure 5 A schematic diagram of an implementation flow of an embodiment of a control method provided by the present application;

[0049] Figure 6 A schematic diagram of the structure of a control device provided in this application;

[0050] Figure 7 A schematic diagram of an embodiment of using the control device provided by the present application to generate a target video corresponding to target music;

[0051] Figure 8 A hardware entity schematic diagram of an electronic device provided in this application;

[0052] Fig. 9 A schematic diagram of a vehicle provided for this application. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solutions and advantages of the present application clearer, the technical solutions of the present application are further elaborated in detail below in conjunction with the drawings and embodiments. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0054] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0055] The terms "first / second / third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It is understandable that "first / second / third" can be interchanged with a specific order or sequence where permitted so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0056] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing this application and are not intended to limit this application.

[0057] In order to enrich the interactive functions of the car display when playing music, most audio software will display the cover of the album to which the music belongs while playing the music. However, since the cover of the music album often has a strong artistic style, the face displayed on the display may be incomplete, too small, missing, or too many characters, resulting in the problem of not meeting the driving image requirements when using music to drive the cover character.

[0058] Based on this, the present application provides a control method, which can be executed by a processor of an electronic device. The electronic device may refer to a server, a laptop, a tablet computer, a desktop computer, a smart TV, a mobile device (such as a mobile phone, a portable video player, a personal digital assistant, a dedicated messaging device, a portable gaming device), a vehicle, or other devices with control capabilities. When the electronic device is a vehicle, the processor of the electronic device may be an electronic control unit (ECU), a vehicle control unit (VCU), a microcontroller unit (MCU), or other units with control functions.

[0059] Figure 1 A schematic diagram of the implementation flow of a control method provided in this application, such as Figure 1 As shown, the method includes the following steps S101 to S103:

[0060] Step S101, determine the target person and cover image corresponding to the target music; the target person is determined from at least one person related to the target music; the cover image is an image related to the target music, and in an optional implementation, the cover image is an image in a music file corresponding to the target music, or an image bound to a music file.

[0061] Here, the target music is the music to be played specified by the user.

[0062] The at least one person related to the target music may be the singer, lyricist, composer, producer or other related person of the target music.

[0063] In some embodiments, the target person is a person determined from at least one person associated with the target music.

[0064] In some embodiments, the target person may be one or more than one person.

[0065] When there are too many target persons, the images of some target persons in the target video generated finally may be too small and difficult to identify. Therefore, in some embodiments, a threshold value of the number of target persons may be preset so that the image of each target person can be clearly displayed in the target video generated finally.

[0066] In some embodiments, the target person may be automatically selected according to a default rule, or may be specified by the user from at least one person.

[0067] In some embodiments, the default rule may be to set a singer of the target music as the target person. For example, if the target music is a work sung by a single singer, the singer is determined as the target person; for another example, if the target music is a work sung by a band, the lead singer of the band can be determined as the target person; for another example, if the target music is a work sung by a group, the leader of the group can be determined as the target person; and so on.

[0068] The cover image is an image related to the target music, for example, the cover image can be a poster of the target music, a poster of the album to which the target music belongs, or a poster of the singer of the target music, etc.

[0069] Step S102: generating a first image including the target person based on the cover image.

[0070] Here, the cover image is processed to obtain a first image including the target person.

[0071] In some embodiments, a target person is determined from the cover image, and characters other than the target person are deleted or all information other than the target person is deleted, thereby generating a first image.

[0072] In some embodiments, the first image is generated using the cover image and at least one image related to the target person.

[0073] In some embodiments, the background image in the cover image is used as the background image of the first image, and feature information of the target person is extracted from at least one image related to the target person, thereby generating the first image by fusing the feature information of the target person and the background image of the first image.

[0074] In some embodiments, since the cover image can convey the theme, style or emotion of the target music or the album to which the target music belongs, a first image including the target person can be generated based on the theme, style or emotion information corresponding to the cover image.

[0075] Step S103, based on the target music, driving the target person's action in the first image to generate a target video of the target music.

[0076] Here, the movement of the limbs, head or facial features of the target person in the first image is driven based on the target music, thereby generating a target video that matches the progress of the playback of the target music.

[0077] In some embodiments, a related model can be used to drive the target person's action in the first image based on the target music. For example, an audio to video model (Audio2Video) can be used to drive the target person's action based on the target music, thereby generating a target video.

[0078] In the control method provided by the present application, after determining the target person and cover image corresponding to the target music, a first image containing the target person is generated based on the cover image, the first image is used as the regenerated cover image, and the target person's action in the first image is driven based on the target music to generate a target video of the target music. In this way, on the one hand, by generating the first image, the cover image can be updated, so that the person image displayed when playing the target music is the target person image, which is more in line with the user's preference or more in line with the theme of the target music; on the other hand, by driving the target person's action in the first image through the target music, the static image can be converted into a dynamic video effect, thereby providing the user with visual interaction while playing the target music, thereby improving the user's experience.

[0079] In some embodiments, Figure 2 As shown, the control method provided by the present application includes the following steps S201 to S204:

[0080] Step S201, determining a target person and a cover image corresponding to a target music; the target person is determined from at least one person related to the target music; and the cover image is an image related to the target music.

[0081] Here, step S201 corresponds to the above step S101, so the specific implementation of step S201 can refer to the specific implementation of step S101 above.

[0082] Step S202: determining image style information of the cover image.

[0083] Here, image style information refers to the unique visual effects and artistic features presented in the image.

[0084] According to different classification methods, image styles can be divided into many categories. For example, according to the art form, image styles can be divided into oil painting style, watercolor style, sketch style, documentary photography, landscape photography, portrait photography, etc.; according to the creative technique, image styles can be divided into figurative style and abstract style; and so on.

[0085] It is understandable that different image styles will make the image appear different, but will not change the content of the image.

[0086] Different image styles are specifically manifested in different image brightness, texture, color temperature, hue, contrast, saturation, lines, composition, and overall expression characteristics.

[0087] In this way, by determining the image style information of the cover image, the image brightness, texture, color temperature, hue, contrast, saturation, lines, composition and overall expression characteristics corresponding to the cover image can be determined.

[0088] Step S203: generating a first image including the target person based on the image style information of the cover image.

[0089] Here, after the image style information is determined, a first image including the target person is generated according to the image style information of the cover image, so that the generated first image has the same or similar image style as the cover image.

[0090] For example, the first image is generated according to at least one of the characteristics of the cover image, such as image brightness, texture, color temperature, hue, contrast, saturation, lines, composition, and overall expression.

[0091] Step S204: based on the target music, driving the target person's movements in the first image to generate a target video of the target music.

[0092] Here, step S204 corresponds to the above step S103, so the specific implementation of step S204 can refer to the specific implementation of step S103 above.

[0093] In the above embodiment, the first image is generated based on the image style information of the cover image, so that the first image has the same or similar image style as the cover image, so that the target video can retain the same or similar visual effects and artistic features as the original cover image.

[0094] In some embodiments, Figure 3 As shown, the control method provided by the present application may include the following steps S301 to S305:

[0095] Step S301, determining a target person and a cover image corresponding to a target music; the target person is determined from at least one person related to the target music; and the cover image is an image related to the target music.

[0096] Here, step S301 corresponds to the above step S101, so the specific implementation of step S301 can refer to the specific implementation of step S101 above.

[0097] Step S302, obtaining a second image of the target person; the second image represents a front view of the target person.

[0098] Here, the second image is an image including a frontal image of the target person.

[0099] In some embodiments, the second image may be a cover image, or may be an image retrieved from a public network or a designated database containing a frontal image of the target person.

[0100] Step S303: determining first characteristic information of the target person based on the second image.

[0101] Here, feature extraction is performed on the second image to determine first feature information related to the target person.

[0102] In some embodiments, the first feature information refers to coarse-grained features of the target person extracted from the second image.

[0103] In some embodiments, the first feature information may include the target person's gender characteristics, racial characteristics, contour characteristics, face shape characteristics, facial features distribution characteristics, age characteristics, etc.

[0104] In some embodiments, the first feature information of the target person can be extracted from the second image using methods such as image segmentation, edge detection, corner detection, feature pyramid, and autoencoder network.

[0105] Step S304: Generate the first image based on the cover image and the first feature information.

[0106] Here, the first image is generated based on the cover image and the first feature information (ie, coarse-grained feature information) of the target person extracted from the second image.

[0107] In some embodiments, the first image is generated based on a background image of the cover image and the first feature information.

[0108] In some embodiments, a first image is generated based on image style information of the cover image (i.e., at least one of image brightness, texture, color temperature, hue, contrast, saturation, lines, composition, and overall expression) and first feature information.

[0109] In some embodiments, the first image is generated based on the cover image and the first feature information using a generative diffusion model. For example, the cover image and the first feature information are input into a graph-generated graph diffusion model, thereby generating a first image having image features of the cover image and the first feature information using the graph-generated graph diffusion model.

[0110] Step S305: Based on the target music, drive the target person in the first image to move, and generate a target video of the target music.

[0111] Here, step S305 corresponds to the above step S103, so the specific implementation of step S305 can refer to the specific implementation of step S103 above.

[0112] In the above embodiment, the first characteristic information of the target person is extracted from the second image containing the front image of the target person. Therefore, the first characteristic information can fully describe the complete characteristics of the front face of the target person, meet the music-driven requirements, and thus enable the target video to display the image of the target person more vividly and completely.

[0113] In some embodiments, the control method provided by the present application further includes the following steps S306:

[0114] Step S306: determining second feature information corresponding to the target person based on the second image; the second feature information represents more detailed feature information corresponding to the target person relative to the first feature information.

[0115] Here, feature extraction is performed on the second image to determine second feature information related to the target person.

[0116] Compared to the first feature information, the second feature information is more detailed feature information of the target person extracted from the second image, that is, fine-grained features of the target person, which can uniquely identify the target person or distinguish the target person from other persons.

[0117] In some embodiments, the first feature information may include texture features of the face and / or limbs of the target person, detailed contour features, etc.

[0118] In this way, based on the target audio, the target person's action in the first image is driven to generate the target video of the target music, that is, the above step S305 can be implemented as the following steps S3051 to S3052:

[0119] Step S3051, based on the target music, driving the target person's movements in the first image to generate a first video.

[0120] Here, based on the target music, a specified audio driving algorithm is used to drive the movement of the target person in the first image to generate a first video.

[0121] For example, a latent space-based image animation method (Latent Image Animate, LIA) is used to drive the movement of a target character in the first image based on target music.

[0122] Here, since the feature information of the target person contained in the first image is the first feature information, that is, the coarse-grained feature information, the character image in the first video generated based on the first image is a character image with the coarse-grained features of the target person.

[0123] Step S3052: redraw the target person in the first video based on the second feature information to obtain the target video.

[0124] Here, based on the second feature information, the target person image in the first video is redrawn using the specified image generation model to obtain the target video. In this way, the target person image in the target video has the first feature information and the second feature information of the target person, that is, the target video can more accurately describe the target person image, thereby obtaining a better driving effect.

[0125] In some embodiments, based on the second feature information, a diffusion model is used to redraw the target person in the first video to obtain a target video.

[0126] In some embodiments, based on the second feature information and the image style information of the cover image, a diffusion model is used to redraw the target person in the first video to obtain a target video.

[0127] When music is used to drive the movements of people in an image, the features of the people in the image will be distorted to a certain extent. Later, the generated video needs to be corrected again using the features of the people, resulting in duplication of work.

[0128] In the above embodiment provided by the present application, the first image is first generated using the coarse-grained features of the target person (i.e., the first feature information), and the target person's movements in the first image are driven based on the target music to obtain the first video, and then the target person in the first video is redrawn and corrected using the fine-grained features of the target person (i.e., the second feature information) to obtain the target video. In this way, when the target music is used to drive the movement of the target person, only the coarse-grained features with a smaller amount of data are used to generate the first image, and the movement of the target person in the first image is driven based on the target music, and then the target person in the first video is redrawn and corrected using the fine-grained features of the target person, thereby avoiding the above-mentioned duplication of work and achieving the purpose of reducing the amount of calculation and improving the efficiency of generating the target video.

[0129] In some embodiments, Figure 4 As shown, the control method provided by the present application can be implemented as the following steps S401 to S404:

[0130] Step S401, determining a target person and a cover image corresponding to a target music; the target person is determined from at least one person related to the target music; and the cover image is an image related to the target music.

[0131] Here, step S401 corresponds to the above step S101, so the specific implementation of step S401 can refer to the specific implementation of step S101 above.

[0132] Step S402, determining third characteristic information corresponding to the reference image; the third characteristic information at least includes information on the proportion of the reference person in the reference image in the reference image.

[0133] Here, the reference image may be a preset image or an image specified by the user.

[0134] The third feature information corresponding to the reference image at least includes information on the proportion of the reference person in the reference image, that is, the ratio of the area occupied by the reference person in the reference image to the total area of ​​the reference image.

[0135] In some embodiments, the third feature information corresponding to the reference image further includes position information of the reference person in the reference image, for example, the reference person is located in the central area, the lower middle area, the area close to the left edge, etc. of the reference image.

[0136] Step S403: Generate a first image including the target person based on the cover image and the third feature information.

[0137] Here, based on the third feature information corresponding to the cover image and the reference image, the first image is generated so that the proportion information of the target person in the first image is the same as or similar to the proportion information of the reference person in the reference image.

[0138] In some embodiments, based on the cover image, image style information of the cover image is determined; based on the image style information of the cover image and the third feature information, a first image including the target person is generated.

[0139] For example, when the image style of the cover image is oil painting style and the third characteristic information represents that the reference character accounts for 1 / 2 of the reference image and is located in the central area of ​​the reference image, then the first image generated based on the cover image and the third characteristic information is an oil painting style image, and the target character accounts for 1 / 2 of the first image and is located in the central area of ​​the first image.

[0140] In some embodiments, based on the cover image, image style information of the cover image is determined; a second image containing a frontal image of a target person is obtained; based on the second image, first feature information of the target person (i.e., coarse-grained features of the target person) is determined; based on the image style information of the cover image, the first feature information and the third feature information of the target person, a first image containing the target person is generated.

[0141] For example, when the image style of the cover image is oil painting style, the first characteristic information of the target person is Han nationality, male, and the third characteristic information represents that the reference person accounts for 1 / 2 of the reference image and is located in the central area of ​​the reference image, then the first image generated based on the cover image, the first characteristic information and the third characteristic information is an oil painting style image, the target person is a character with Han male characteristics, and the target person accounts for 1 / 2 of the first image and is located in the central area of ​​the first image.

[0142] Step S404: based on the target music, driving the target person's movements in the first image to generate a target video of the target music.

[0143] Here, step S404 corresponds to the above step S103, so the specific implementation of step S404 can refer to the specific implementation of step S103 above.

[0144] In some embodiments, the action of the target person in the first image is driven based on the target music, that is, the above step S103 can be implemented as at least one of the following steps S1031 to S1033:

[0145] Step S1031, determining the music style of the target music; and determining the dance style of the target person in the first image based on the music style.

[0146] Here, music style is a way of describing the unique characteristics of music. For example, music styles include classical music, flow music, rock music, country music, jazz, hip-hop music, electronic music, ancient music, folk music, etc.

[0147] In some embodiments, a model with music style recognition capability may be used to determine the music style of the target music.

[0148] In some embodiments, a model based on audio features can be used to identify the music style of the target music. First, key audio features such as rhythm, melody, harmony, timbre, etc. are extracted from the target music; then, machine learning algorithms such as support vector machines, decision trees, and random forests are used to classify and identify the extracted audio features, thereby determining the music style of the target music.

[0149] In some embodiments, a model based on lyrics and audio features can be used to identify the music style of the target music. First, natural language processing is performed on the lyrics to extract lyrics features such as keywords and emotional tendencies, and the audio features of the music are extracted at the same time; then, the audio features and lyrics features are fused, and feature learning and classification are performed using deep learning technology to determine the music style of the target music.

[0150] Dance style refers to the artistic characteristics and creative personality of a dance. For example, dance styles include classical dance, folk dance, modern dance, contemporary dance, and so on.

[0151] In this way, after the music style of the target music is determined, the dance style of the target person in the first image can be determined based on the music style.

[0152] For example, when the music style is classical music, the target person's dance style can be determined as classical dance; for another example, when the music style is flow music, the target person's dance style can be determined as modern dance or contemporary dance; for another example, when the music style is country music, the target person's dance style can be determined as folk dance or ethnic dance; and so on.

[0153] In the above embodiment, by determining the music style of the target music and determining the dance style of the target character based on the music style, the character movements in the target video can be made more suitable for the target music, allowing the user to better experience the artistic characteristics expressed by the target music.

[0154] Step S1032, determining the rhythm characteristics of the target music; and determining the movement speed of the target person in the first image based on the rhythm characteristics.

[0155] Here, rhythmic characteristics refer to the changes and repetitions of the lightness, heaviness, slowness and speed of the sound beats in music, which have time characteristics.

[0156] In some embodiments, first, audio information is extracted from the target music, and the extracted audio information is preprocessed by noise reduction, filtering, etc.; then, the rhythm characteristics of the target music are determined through time domain analysis, frequency domain analysis, energy analysis, etc.

[0157] In some embodiments, the trained machine learning model can be used to process the target music to generate rhythmic features of the target music.

[0158] In some embodiments, deep learning technology can be used to identify rhythm features corresponding to the target music.

[0159] Since the rhythm feature can represent the lightness, heaviness, slowness and speed of the target music's sound beat, the target character's movement speed can be determined based on the rhythm feature. For example, when the target music's rhythm is fast, the target character's movement speed is increased; when the target music's rhythm is slow, the target character's movement speed is decreased.

[0160] In this way, by determining the rhythm characteristics of the target music and determining the movement speed of the target character based on the rhythm characteristics, the movement speed of the target character in the target video can be made to match the rhythm characteristics of the target music, thereby improving the generation effect of the target video.

[0161] Step S1033, obtaining lyrics information of the target music; and driving the lip movements of the target person in the first image based on the lyrics information and the playing progress of the target music.

[0162] Here, the lyrics information of the target music may be acquired from a storage device of the electronic device or a designated database.

[0163] In some embodiments, the emotional characteristics, rhythmic characteristics, energy and corresponding lyrics of the currently playing music can be determined based on the playback progress of the target music, and then the lip reading synthesis model can be used to drive the lip movements of the target character.

[0164] In this way, by combining the lyrics information and playback progress of the target music and driving the lip movements of the target person in the first image, the lip movements of the target person in the target video can be matched with the content of the currently playing target music, reflecting the corresponding emotions, rhythm, energy, lyrics and other characteristics, thereby improving the matching degree between the target video and the target music.

[0165] In some embodiments, the control method provided by the present application further includes the following step S104:

[0166] Step S104: determining an animation effect of a background image in the first image based on the target music.

[0167] Here, the background image in the first image may be a background image of an album image of the target music, or may be an image generated based on the image style of the album image of the target music.

[0168] In some embodiments, the changing speed of the background image in the first image can be adjusted based on the rhythm characteristics of the target music, so as to obtain an animation effect that changes continuously with the lightness, heaviness, speed and slowness of the rhythm.

[0169] In some embodiments, the background image of the first image may be updated based on the lyrics information of the target music. For example, the semantic features of the lyrics of the target music may be obtained and identified; based on the semantic features and the playback progress of the target music, the corresponding background content may be matched. For another example, the lyrics information of the target music may be obtained; based on the playback progress of the target music, the keyword information in the currently playing lyrics may be identified and determined; based on the determined keyword information, the corresponding background image may be matched.

[0170] In the above embodiment, the animation effect of the background image in the first image is determined based on the target music, so that the animation effect of the background image in the target video can match the playback progress of the target music, further improving the matching degree between the target video and the target music.

[0171] Next, combine Figure 5 , an embodiment of the control method provided according to the present application is described.

[0172] like Figure 5 As shown, this embodiment includes the following steps S501 to S506:

[0173] Step S501, in response to the user's operation of playing the target music, obtaining the cover image corresponding to the target music, the singer image corresponding to the target music and the reference image; then, executing step S502;

[0174] Step S502, extracting image style information corresponding to the cover image, coarse-grained features and fine-grained features of the singer in the singer image, and the proportion and area of ​​the reference person in the reference image; then, executing step S503;

[0175] Step S503, regenerating the cover image of the target music based on the extracted image style information, the coarse-grained features of the singer, and the proportion and area of ​​the reference character in the reference image; then, executing step S504;

[0176] Step S504, using the target music to drive the lip movements and body movements of the singer in the regenerated cover image to obtain a first video; then, executing step S505;

[0177] Step S505, based on the singer's fine-grained features and image style information, the singer's image in the first video is corrected to obtain a target video; then, step S506 is executed;

[0178] Step S506: display the output target video.

[0179] It can be seen from the above embodiments that in the control method provided by the present application, the cover image of the target music is redrawn based on the image style information of the target music, the coarse-grained features of the singer in the singer image, and the reference image, and the singer's movements in the redrawn cover image are driven by the target music, and the generated first video is corrected based on the singer's fine-grained features and the image style information of the cover image to obtain the target video. In this way, the redrawn cover image can maintain the same or similar image style as the original cover image, and the position of the singer in the redrawn cover image is more in line with user expectations, and the proportion meets the music driving requirements; in addition, the first video is first generated using the coarse-grained features of the singer, and then the first video is corrected using the fine-grained features of the singer to obtain the target video. In this way, the target video that accurately describes the singer's characteristics can be obtained, and the amount of calculation required for the video generation process can be reduced.

[0180] Based on the foregoing embodiments, the present application provides a control device, which includes the units included and the modules included in the units, and can be implemented by a processor in a computer device; of course, it can also be implemented by a specific logic circuit; in the implementation process, the processor can be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP), a field programmable gate array (FPGA) or a vehicle's MCU, ECU, etc.

[0181] Figure 6 A schematic diagram of the structure of a control device provided in this application is shown in FIG. Figure 6 As shown, the control device 600 includes: a first determination module 610, a first generation module 620 and a driving module 630, wherein:

[0182] The first determination module 610 is used to determine a target person and a cover image corresponding to the target music; the target person is determined from at least one person related to the target music; the cover image is an image related to the target music;

[0183] A first generating module 620, configured to generate a first image including the target person based on the cover image;

[0184] The driving module 630 is used to drive the target person's action in the first image based on the target music to generate a target video of the target music.

[0185] In some embodiments, the first generating module 620 is used to:

[0186] Determining image style information of the cover image;

[0187] Based on the image style information of the cover image, a first image including the target person is generated.

[0188] In some embodiments, the first generating module 620 is used to:

[0189] Acquire a second image of the target person; the second image represents a front view of the target person;

[0190] Determining first characteristic information of the target person based on the second image;

[0191] The first image is generated based on the cover image and the first feature information.

[0192] In some embodiments, the first determining module 610 is further configured to determine second feature information corresponding to the target person based on the second image; the second feature information represents more detailed feature information corresponding to the target person relative to the first feature information;

[0193] The driving module 630 is used to drive the target person in the first image to move based on the target music to generate a first video;

[0194] The device 600 further includes a second generating module 640; the second generating module 640 is used to redraw the target person in the first video based on the second feature information to obtain the target video.

[0195] In some embodiments, the first determining module 610 is used to:

[0196] Determine third feature information corresponding to the reference image; the third feature information at least includes information about the proportion of the reference person in the reference image;

[0197] Based on the cover image and the third feature information, a first image including the target person is generated.

[0198] In some embodiments, the driver module 630 is configured to perform at least one of the following:

[0199] Determining the music style of the target music; and determining the dance style of the target person in the first image based on the music style;

[0200] Determine the rhythm characteristics of the target music; and determine the movement speed of the target person in the first image based on the rhythm characteristics;

[0201] Acquire lyrics information of the target music; and drive lip movements of the target person in the first image based on the lyrics information and the playing progress of the target music.

[0202] In some embodiments, the driving module 630 is further used to determine an animation effect of a background image in the first image based on the target music.

[0203] Next, combine Figure 7 , the process of using the control device provided by the present application to generate a video corresponding to the target music is described. Among them, the first determination module is implemented by using the feature preprocessor 710, the first generation module is implemented by using the first diffusion model 720, the driving module is implemented by using the lip synthesis model 730, and the second generation module is implemented by using the second diffusion model 740.

[0204] like Figure 7 As shown, after detecting the user's operation of playing the target music, the album cover image 701, the reference image 702 and the singer image 703 are input into the feature preprocessor 710.

[0205] The singer image 703 is a frontal image of the singer obtained from a designated database.

[0206] The feature preprocessor 710 extracts the image style information of the cover image 701 , the proportion and area of ​​the reference person in the reference image 702 , and the coarse-grained features and fine-grained features of the singer image 703 .

[0207] The feature preprocessor 710 sends the image style information of the cover image 701, the proportion and area of ​​the reference person in the reference image 702, and the coarse-grained features of the singer image 703 to the first diffusion model 720, and sends the image style information of the cover image 701 and the fine-grained features of the singer image 703 to the second diffusion model 740.

[0208] The first diffusion model 720 redraws the cover image based on the image style information, the proportion of the reference character in the reference image and the area information where it is located, and the coarse-grained features of the singer image 703 to obtain the redrawn cover image 704; the first diffusion module 720 sends the redrawn cover image 704 to the lip reading synthesis model 730.

[0209] The lip-reading synthesis model 730 drives the singer's movements in the redrawn cover image 704 based on the target music, obtains the first video 705, and sends the first video 705 to the second diffusion model 740.

[0210] The second diffusion model 740 uses the image style information of the cover image 701 and the fine-grained features of the singer image 703 to correct the singer image in the first video 705 and obtain the target video 706 of the target music.

[0211] The description of the above device embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiment. For technical details not disclosed in the device embodiment of the present application, please refer to the description of the method embodiment of the present application for understanding.

[0212] Based on the above embodiments, the present application provides an electronic device. Figure 8 As shown, the electronic device 800 includes a processor 810 and a display 820, wherein:

[0213] The processor 810 is used to determine a target person and a cover image corresponding to a target music; the target person is determined from at least one person related to the target music; the cover image is an image displayed and output when the target music is played; based on the cover image, a first image including the target person is generated; based on the target music, a target person in the first image is driven to move, and a target video corresponding to the target music is generated;

[0214] The display 820 shown is used to display and output the first video.

[0215] In some embodiments, the electronic device 800 further includes a communication interface 830 , so that the processor 810 can utilize the communication interface 830 to acquire the second image of the target person from a public network or a designated database.

[0216] In some embodiments, the electronic device 800 further includes a bus 840 , so that the processor 810 , the display 820 , and the communication interface 830 can implement internal communications using the bus 840 .

[0217] It is necessary to explain that the electronic device provided by the present application can be used to implement the above-mentioned method embodiments, and therefore, the electronic device provided by the present application has similar beneficial effects as the method embodiments. In some embodiments, for technical details not disclosed in the electronic device embodiments of the present application, please refer to the description of the method embodiments of the present application for understanding.

[0218] Based on the above embodiments, the present application provides a vehicle. Fig. 9 As shown, the vehicle 900 includes a vehicle body 910 and an in-vehicle entertainment system (not shown in the figure) mounted on the vehicle body 910; the in-vehicle entertainment system includes a processor and a display; wherein,

[0219] The processor is used to determine a target person and a cover image corresponding to a target music; the target person is determined from at least one person related to the target music; the cover image is an image related to the target music; based on the cover image, a first image containing the target person is generated; based on the target music, a target person in the first image is driven to move, and a target video corresponding to the target music is generated;

[0220] The display shown is used to display and output the first video.

[0221] It should be noted that the vehicle provided in this application can be used to implement the above-mentioned method embodiments, and therefore, the vehicle provided in this application has similar beneficial effects as the method embodiments. In some embodiments, for technical details not disclosed in the vehicle embodiments of this application, please refer to the description of the method embodiments of this application for understanding.

[0222] It should be noted that in the embodiment of the present application, if the above-mentioned control method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, which is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a disk or an optical disk. In this way, the embodiment of the present application is not limited to any specific hardware, software or firmware, or any combination of hardware, software, and firmware.

[0223] An embodiment of the present application provides a computer device, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the program, some or all of the steps in the above method are implemented.

[0224] The embodiment of the present application provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, some or all of the steps in the above method are implemented. The computer-readable storage medium can be transient or non-transient.

[0225] An embodiment of the present application provides a computer program, including a computer-readable code. When the computer-readable code is run in a computer device, a processor in the computer device executes some or all of the steps for implementing the above method.

[0226] The embodiment of the present application provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and when the computer program is read and executed by a computer, some or all of the steps in the above method are implemented. The computer program product can be implemented specifically by hardware, software or a combination thereof. In some embodiments, the computer program product is specifically embodied as a computer storage medium, and in other embodiments, the computer program product is specifically embodied as a software product, such as a software development kit (SDK) and the like.

[0227] It should be noted here that the description of the various embodiments above tends to emphasize the differences between the various embodiments, and the same or similar aspects can be referenced to each other. The description of the above device, storage medium, computer program and computer program product embodiments is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the embodiments of the device, storage medium, computer program and computer program product of this application, please refer to the description of the method embodiment of this application for understanding.

[0228] It should be understood that "one embodiment" or "an embodiment" mentioned throughout the specification means that specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present application, the size of the serial number of each step / process mentioned above does not mean the order of execution, and the execution order of each step / process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application. The serial numbers of the embodiments of the present application mentioned above are for description only and do not represent the advantages and disadvantages of the embodiments.

[0229] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0230] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0231] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0232] In addition, all functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may be a separate unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0233] A person skilled in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM), magnetic disks or optical disks, etc., various media that can store program codes.

[0234] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can essentially or in other words, the part that contributes to the relevant technology can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0235] The above is only an implementation method of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application.

Claims

1. A control method, comprising: Determine the target person and cover image corresponding to the target music; The target person is determined from at least one person associated with the target music; The cover image is an image related to the target music; Based on the cover image, generating a first image including the target person; Based on the target music, the target person in the first image is driven to move, and a target video of the target music is generated.

2. The method according to claim 1, wherein: The step of generating a first image including the target person based on the cover image includes: Determining image style information of the cover image; Based on the image style information of the cover image, a first image including the target person is generated.

3. The method according to claim 1, wherein: The step of generating a first image including the target person based on the cover image includes: Acquire a second image of the target person; the second image represents a front view of the target person; Determining first characteristic information of the target person based on the second image; The first image is generated based on the cover image and the first feature information.

4. The method according to claim 3, further comprising: Based on the second image, determining second feature information corresponding to the target person; The second feature information represents more detailed feature information corresponding to the target person relative to the first feature information; The step of driving the target person's action in the first image based on the target audio to generate the target video of the target music includes: Based on the target music, driving the target person in the first image to move, and generating a first video; The target person in the first video is redrawn based on the second feature information to obtain the target video.

5. The method according to any one of claims 1 to 4, wherein: The step of generating a first image including the target person based on the cover image includes: Determine third feature information corresponding to the reference image; the third feature information at least includes information about the proportion of the reference person in the reference image; Based on the cover image and the third feature information, a first image including the target person is generated.

6. The method according to any one of claims 1 to 4, wherein: The step of driving the action of the target person in the first image based on the target music includes at least one of the following: Determining the music style of the target music; and determining the dance style of the target person in the first image based on the music style; Determine the rhythm characteristics of the target music; and determine the movement speed of the target person in the first image based on the rhythm characteristics; Acquire lyrics information of the target music; and drive lip movements of the target person in the first image based on the lyrics information and the playing progress of the target music.

7. The method according to any one of claims 1 to 4, further comprising: Based on the target music, an animation effect of a background image in the first image is determined.

8. A control device comprising: A first determination module is used to determine a target person and a cover image corresponding to a target music; The target person is determined from at least one person related to the target music; the cover image is an image related to the target music; A first generating module, used for generating a first image including the target person based on the cover image; A driving module is used to drive the target character action in the first image based on the target music to generate a target video of the target music.

9. An electronic device comprising a processor and a display, wherein: The processor is used to determine a target person and a cover image corresponding to a target music; the target person is determined from at least one person related to the target music; the cover image is an image related to the target music; Based on the cover image, generating a first image including the target person; Based on the target music, driving the target person in the first image to move, and generating a target video corresponding to the target music; The display shown is used to display and output the first video.

10. A vehicle comprising: A vehicle body and an in-vehicle entertainment system mounted on the vehicle body; The in-vehicle entertainment system includes a processor and a display; wherein, The processor is used to determine a target person and a cover image corresponding to a target music; the target person is determined from at least one person related to the target music; the cover image is an image related to the target music; based on the cover image, a first image containing the target person is generated; based on the target music, a target person in the first image is driven to move, and a target video corresponding to the target music is generated; The display shown is used to display and output the first video.

Images