Cover generation method and device for digital human live broadcast, equipment and medium

Abstract

The invention provides a cover generation method and device for digital human live broadcast, equipment and a medium, and relates to the technical field of computers, in particular to the technical fields of artificial intelligence, digital human, augmented reality and the like. The specific implementation scheme is as follows: acquiring a plurality of image frames from a live video of a digital person; determining a to-be-processed image used for generating a live broadcast cover from the plurality of image frames according to expressions and / or postures of the digital human in the plurality of image frames; for at least one image area in the to-be-processed image, determining an interception box position according to the degree of interest of the at least one image area; and carrying out image interception on the to-be-processed image based on the interception box position to obtain a live broadcast cover image.

Description

Technical Field

[0001] The present disclosure relates to the field of AI (Artificial Intelligence), specifically to technical fields such as artificial intelligence, digital humans, and augmented reality, and can be applied to video and live broadcast scenarios, and in particular to a method, device, equipment, and medium for generating a cover for a digital human live broadcast. Background Art

[0002] With the promotion of digital human live broadcast platform, more and more small and medium-sized businesses choose digital human live broadcast capabilities, and the live broadcast cover plays a vital role as the visual presentation of the live broadcast content. The live broadcast cover is not only the key to attracting the audience's attention, but also can convey the live broadcast subject, enhance the brand image, increase exposure, promote audience retention and influence audience expectations.

[0003] Therefore, how to generate covers for digital human live broadcasts is particularly important. Summary of the invention

[0004] The present disclosure provides a cover generation method, device, equipment and medium for digital human live broadcast.

[0005] According to a first aspect of the present disclosure, a method for generating a cover for a digital human live broadcast is provided, comprising: acquiring multiple image frames from a live broadcast video of a digital human; determining an image to be processed for generating a live broadcast cover from the multiple image frames according to the expression and / or posture of the digital human in the multiple image frames; determining a capture frame position for at least one image area in the image to be processed according to the interest level of the at least one image area; and performing image capture of the image to be processed based on the capture frame position to obtain a live broadcast cover image.

[0006] According to a second aspect of the present disclosure, a cover generation device for digital human live broadcast is provided, comprising: an acquisition module, used to acquire multiple image frames from the live broadcast video of the digital human; a first determination module, used to determine, from the multiple image frames, an image to be processed for generating a live broadcast cover according to the expression and / or posture of the digital human in the multiple image frames; a second determination module, used to determine a capture frame position for at least one image area in the image to be processed according to the interest level of the at least one image area; and a capture module, used to capture the image to be processed based on the capture frame position to obtain a live broadcast cover image.

[0007] According to a third aspect of the present disclosure, there is provided an electronic device, including:

[0008] at least one processor; and

[0009] a memory communicatively connected to the at least one processor; wherein,

[0010] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the cover generation method for digital human live broadcast as described in the first aspect.

[0011] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the cover generation method for digital human live broadcast as described in the first aspect.

[0012] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising computer instructions, which, when executed by a processor, implement the cover generation method for digital human live broadcast as described in the first aspect.

[0013] The present disclosure provides a method, device, equipment and medium for generating a cover for a digital human live broadcast, which has the following beneficial effects:

[0014] Acquire multiple image frames from the live video of the digital human; determine the image to be processed for generating the live cover from the multiple image frames according to the expression and / or posture of the digital human in the multiple image frames; determine the position of the capture frame according to the interest level of at least one image area in the image to be processed; and capture the image to be processed based on the position of the capture frame to obtain the live cover image. The present disclosure can make full use of the dynamic characteristics of the live broadcast of the digital human, capture the moment when the expression and / or posture of the digital human meets the requirements, and determine the position of the capture frame according to the interest level of the image area, so that the generated cover image can focus on the most important content and avoid the interference of irrelevant information. Finally, by capturing the image, the live cover image is automatically generated, which effectively reduces the manual intervention in the live cover generation process, improves the generation efficiency of the live cover, and can realize the personalized customization of the live cover to enhance the user experience.

[0015] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0017] Figure 1 It is a flowchart of a method for generating a cover for a digital human live broadcast provided according to an embodiment of the present disclosure;

[0018] Figure 2is a flow chart of a method for generating a cover for a digital human live broadcast provided according to another embodiment of the present disclosure;

[0019] Figure 3 is a flow chart of a method for generating a cover for a digital human live broadcast provided according to another embodiment of the present disclosure;

[0020] Figure 4 is a flow chart of a method for generating a cover for a digital human live broadcast provided according to another embodiment of the present disclosure;

[0021] Figure 5 is a flow chart of a method for generating a cover for a digital human live broadcast provided according to another embodiment of the present disclosure;

[0022] Figure 6 is a flow chart of a method for generating a cover for a digital human live broadcast provided according to another embodiment of the present disclosure;

[0023] Figure 7 is a flow chart of a method for generating a cover for a digital human live broadcast provided according to another embodiment of the present disclosure;

[0024] Figure 8 A schematic diagram of the support surface of the digital human provided by the present disclosure;

[0025] Fig. 9 is a flow chart of a method for generating a cover for a digital human live broadcast provided according to another embodiment of the present disclosure;

[0026] Fig.10 A schematic structural diagram of a cover generation device for digital human live broadcasting provided according to another embodiment of the present disclosure;

[0027] Fig.11 It is a block diagram of an electronic device used to implement the cover generation method for digital human live broadcast according to the embodiment of the present disclosure. DETAILED DESCRIPTION

[0028] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0029] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information are all carried out with the user's consent, comply with the relevant laws and regulations, and do not violate public order and good morals.

[0030] With the promotion of the digital human live broadcast platform, more and more small and medium-sized businesses are choosing digital human live broadcast capabilities and enjoying the various advantages and conveniences brought by digital human live broadcast. The main reasons include:

[0031] 1. Using digital people for live broadcasting can effectively reduce labor costs, equipment costs, venue costs and time costs;

[0032] 2. The digital human anchor has a more vivid and realistic image and voice, which can improve the viewing experience of the live broadcast. The digital human anchor can simulate a variety of emotions and postures, which enhances the interactivity and fun of the live broadcast;

[0033] Through digital human live broadcast, small and medium-sized businesses can interact with a large number of users in real time, collect user feedback in a timely manner, and further optimize products or services.

[0034] However, while small and medium-sized businesses enjoy the convenience brought by digital live broadcasts, they also face the problems of lack of live broadcast cover materials and insufficient drawing capabilities.

[0035] In response to at least one of the above-mentioned problems, the present disclosure proposes a cover generation method, device, equipment and medium for digital human live broadcast.

[0036] For ease of understanding, firstly, the relevant concepts that may be involved in each embodiment of the present application are briefly described:

[0037] Artificial Intelligence (AI) is an important driving force for a new round of scientific and technological revolution and industrial transformation. It is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence.

[0038] Augmented Reality (AR) technology integrates the virtual environment generated by the computer with the real environment around the user by means of optoelectronic display technology, interactive technology, multiple sensor technologies, computer graphics and multimedia technology, so that the user can be convinced from the sensory effect that the virtual environment is a part of the real environment around him. Augmented reality has the new characteristics of combining virtual and real, real-time interaction, and three-dimensional registration.

[0039] The following describes the cover generation method, device, equipment and medium for digital human live broadcast according to the embodiments of the present disclosure with reference to the accompanying drawings.

[0040] It should be noted that the executor of the cover generation method for digital human live broadcast in this embodiment is a cover generation device for digital human live broadcast. The cover generation device for digital human live broadcast can be implemented by software and / or hardware and can be configured in an electronic device. The electronic device can include but is not limited to a terminal, a server, etc.

[0041] Figure 1 It is a flowchart of a method for generating a cover for a digital human live broadcast provided according to an embodiment of the present disclosure.

[0042] like Figure 1 As shown, the cover generation method for digital human live broadcast includes:

[0043] Step 101: Acquire multiple image frames from the live video of the digital human.

[0044] As a possible implementation method, after obtaining the live video of the digital human, for each image frame in the live video, first, a quality analysis can be performed on the image frame to determine a first index value of the image frame; wherein the first index value is used to indicate the clarity of the corresponding image frame; thereby, the live video of the digital human is screened based on the first index value, for example, if the first index value of the image frame is greater than a first set threshold, the image frame is retained, otherwise, the image frame is discarded.

[0045] Step 102, determining an image to be processed for generating a live broadcast cover from the multiple image frames according to the expressions and / or postures of the digital human in the multiple image frames.

[0046] Among them, expressions can be used to indicate the emotional state of the digital person, such as happiness, sadness, surprise, tension, fear, anger, etc., and the present disclosure does not impose any restrictions on this.

[0047] Among them, the posture can be used to indicate the body movements of the digital human, such as walking, jumping, running, clapping, etc., and the present disclosure does not impose any restrictions on this.

[0048] As a possible implementation method, for any image frame, an expression recognition model can be used to recognize the expression of the digital person in the image frame, for example, a pre-trained deep learning model (such as the VGGFace model (a face recognition model based on a visual geometry group network structure), FaceNet (face recognition network), etc.) can be used.

[0049] As another possible implementation, for any image frame, a posture estimation model may be used to identify the posture of the digital human in the image frame.

[0050] In order to facilitate the recognition of the expression and / or posture of the digital human in the image frame to improve the accuracy and effectiveness of the recognition, in a possible implementation method of the embodiment of the present disclosure, multiple image frames can be pre-processed, wherein the pre-processing may include adjusting the image size, removing noise, enhancing contrast, increasing brightness, soft light processing, etc.

[0051] In the embodiment of the present disclosure, the image to be processed for generating a live broadcast cover can be determined from multiple image frames based on the expressions and / or postures of the digital human in the multiple image frames.

[0052] Optionally, the image frame in which the expression of the digital human in the multiple image frames is the target expression can be determined as the image to be processed for generating the live broadcast cover. The target expression can be a pre-set expression, such as happiness, surprise, etc., which is not limited in the present disclosure.

[0053] Alternatively, the image frame in which the digital human's posture is the target posture among the multiple image frames can be determined as the image to be processed for generating the live broadcast cover. The target posture can be a preset posture, such as clapping, jumping, etc., which is not limited in the present disclosure.

[0054] Optionally, the image frames in which the posture of the digital human is the target posture and the posture of the digital human is the target posture among the multiple image frames can be determined as the images to be processed for generating the live broadcast cover.

[0055] In a possible implementation of the embodiment of the present disclosure, image frames in which the expressions and / or postures of the digital human match the cover theme in multiple image frames can be determined as images to be processed for generating a live broadcast cover.

[0056] As an example, assuming that the cover theme is "Happy Party", the expression of the digital person matching the cover theme is "happy", and the posture is "raising a toast to celebrate, waving to say hello, dancing", and assuming that there is an image frame in which the expression of the digital person is "happy" and the posture is "raising a toast to celebrate", then it can be determined that the image frame is the image to be processed for generating the live broadcast cover.

[0057] Step 103: for at least one image region in the image to be processed, determine a capture frame position according to the interest level of the at least one image region.

[0058] In the embodiment of the present disclosure, the image to be processed may include an image region, and the image region included in the image to be processed may be but is not limited to one, and the present disclosure does not impose any limitation on this.

[0059] In the embodiment of the present disclosure, for any image region, the user's interest level in the image region may be analyzed.

[0060] As an example, for any image area, a first target score of the image area can be determined based on attribute information of the image area; wherein the first target score can be used to indicate the user's interest level in the corresponding image area, and the attribute information of the image area may include, for example, color, texture, shape, etc.

[0061] As another example, for any image region, a saliency evaluation algorithm can be used to evaluate the saliency of the image region, thereby determining a second target score based on the evaluated saliency score of the image region, wherein the second target score can be used to indicate the user's interest level in the corresponding image region.

[0062] As another example, for any image area, a target detection algorithm can be used to identify objects in the image area, and thus, based on the attribute information of the identified objects, a third target score of the image area is determined; wherein the third target score can be used to indicate the user's interest in the corresponding image area, and the attribute information of the object may include, for example, size, position, color, etc., which is not limited in the present disclosure.

[0063] As another example, for any image region, multiple interest level determination methods are used to determine the interest index value of the image region, wherein the interest index value can be used to indicate the user's interest level in the corresponding image region.

[0064] For example, for any image area, each interest level determination method is used to determine the interest score corresponding to the image area, and the interest score can be used to indicate the user's interest level in the corresponding image area under the corresponding method; the interest scores are fused to obtain an interest index value.

[0065] It should be noted that the above-mentioned method for determining the degree of interest in an image region is only exemplary. In practical applications, other methods may be used, and the present disclosure does not impose any limitation on this.

[0066] Thereby, an effective determination of the user's interest level in an image region may be achieved.

[0067] In the embodiment of the present disclosure, for at least one image region in the image to be processed, the position of the capture frame may be determined according to the interest level of the at least one image region.

[0068] Step 104, capturing the image to be processed based on the position of the capture frame to obtain a live broadcast cover image.

[0069] The cover generation method for digital human live broadcast of the embodiment of the present disclosure obtains multiple image frames from the live broadcast video of the digital human; determines the image to be processed for generating the live broadcast cover from the multiple image frames according to the expression and / or posture of the digital human in the multiple image frames; determines the position of the capture frame according to the interest level of at least one image area in the image to be processed; and captures the image to be processed based on the position of the capture frame to obtain the live broadcast cover image, thereby making full use of the dynamic characteristics of the digital human live broadcast, capturing the moment when the expression and / or posture of the digital human meets the requirements, and determining the position of the capture frame according to the interest level of the image area, so that the generated cover image can focus on the most important content and avoid the interference of irrelevant information. Finally, by capturing the image, the live broadcast cover image is automatically generated, which effectively reduces the manual intervention in the live broadcast cover generation process, improves the generation efficiency of the live broadcast cover, and can realize the personalized customization of the live broadcast cover to enhance the user experience.

[0070] It should be noted that in the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information are all carried out with the user's consent, and are in compliance with relevant laws and regulations and do not violate public order and good morals.

[0071] In order to clearly illustrate how, in the above embodiments of the present disclosure, the image to be processed for generating a live broadcast cover is determined from multiple image frames according to the expressions and / or postures of the digital human in the multiple image frames, the present disclosure also proposes a cover generation method for digital human live broadcast.

[0072] Figure 2 It is a flowchart of a method for generating a cover for a digital human live broadcast provided according to another embodiment of the present disclosure.

[0073] like Figure 2 As shown, the cover generation method for digital human live broadcast includes:

[0074] Step 201: Acquire multiple image frames from the live video of the digital human.

[0075] It should be noted that the execution process of step 201 can refer to the execution process of any embodiment of the present disclosure, and will not be repeated here.

[0076] Step 202: for any image frame among the multiple image frames, perform object recognition on the image frame to determine whether the image frame contains a digital human.

[0077] In the embodiments of the present disclosure, for any image frame among the multiple image frames, object recognition can be performed on the image frame to determine whether the image frame contains a digital human. For example, a convolutional neural network (CNN) model can be used to extract image features from the image frame; based on the image features, a trained classifier or object recognition model can be used to determine whether the image frame contains a digital human.

[0078] Step 203: determine the image frame containing the digital human as the first image.

[0079] In the embodiment of the present disclosure, an image frame containing a digital human may be determined as the first image.

[0080] It should be noted that the number of the first images may be but is not limited to one, and the present disclosure does not impose any restriction on this.

[0081] Step 204: Recognize the expression of the digital human in the first image and determine a first score corresponding to the recognized expression.

[0082] As an example, an expression recognition model may be used to recognize the expression of the digital person in the first image; based on the recognized expression, a corresponding score is queried, and the score is determined as the first score corresponding to the recognized expression.

[0083] Step 205: Recognize the posture of the digital human in the first image and determine a second score; wherein the second score is used to indicate the degree of grace of the digital human's posture.

[0084] As an example, a posture estimation model may be used to identify the posture of the digital human in the first image; based on the identified posture, a corresponding score is queried, and the score is used to determine the second score.

[0085] Step 206: Determine an image to be processed from at least one first image based on the first score and / or the second score corresponding to the first image.

[0086] As a possible implementation, the first image with the highest first score among the at least one first image may be determined as the image to be processed, thereby enabling accurate screening of image frames based on the expression of the digital human.

[0087] As another possible implementation, the first image with the highest second score among the at least one first image may be determined as the image to be processed. Thus, accurate screening of image frames may be achieved based on the posture of the digital human.

[0088] As another possible implementation, a first image in which the first score is greater than the second set threshold and the second score is greater than the third set threshold in at least one first image can be determined as an image to be processed. Thus, depending on the expression and posture of the digital human, the images to be processed whose expression and posture meet the requirements can be accurately screened.

[0089] As another possible implementation, for any first image, a quality score of the first image is obtained based on the fusion of the first score and the second score corresponding to the first image; and an image to be processed is determined from at least one first image according to the quality score of each first image. For example, for any first image, a weighted sum of the first score and the second score corresponding to the first image can be performed to obtain the quality score of the first image, thereby determining the first image with the highest and / or lowest quality score among the first images as the image to be processed. Thus, the first score and the second score are effectively fused to accurately determine the quality score of the image, and then based on the quality score of the image, accurate determination of the image to be processed is achieved.

[0090] It should be noted that the image to be processed may be but is not limited to being one, and the present disclosure does not impose any limitation on this.

[0091] Step 207: for at least one image region in the image to be processed, determine a capture frame position according to the interest level of the at least one image region.

[0092] Step 208, capturing the image to be processed based on the position of the capture frame to obtain a live broadcast cover image.

[0093] It should be noted that the execution process of step 207 to step 208 can refer to the execution process of any embodiment of the present disclosure, and will not be repeated here.

[0094] The cover generation method for digital human live broadcast of the disclosed embodiment performs object recognition on any image frame among multiple image frames to determine whether the image frame contains a digital human; determines the image frame containing the digital human as the first image; recognizes the expression of the digital human in the first image to determine the first score corresponding to the recognized expression; recognizes the posture of the digital human in the first image to determine the second score; wherein the second score is used to indicate the degree of beauty of the digital human's posture; and determines the image to be processed from the first image based on the first score and / or the second score corresponding to the first image. Through the object recognition technology, the image frames of the live video are initially screened to obtain the image frames containing the digital human subject, and then the images obtained by the initial screening are automatically screened for the second time based on the expression and / or posture of the digital human, so as to accurately obtain the image to be processed without manual intervention, thereby improving the processing efficiency.

[0095] Figure 3It is a flowchart of a method for generating a cover for a digital human live broadcast provided according to another embodiment of the present disclosure.

[0096] like Figure 3 As shown, the cover generation method for digital human live broadcast includes:

[0097] Step 301: Acquire multiple image frames from the live video of the digital human.

[0098] It should be noted that the execution process of step 201 can refer to the execution process of any embodiment of the present disclosure, and will not be repeated here.

[0099] Step 302: For any image frame among the multiple image frames, perform object recognition on the image frame to determine whether the image frame contains a digital human.

[0100] Step 303: determine the image frame containing the digital human as the first image.

[0101] Step 304: Recognize the expression of the digital human in the first image and determine a first score corresponding to the recognized expression.

[0102] Step 305: Recognize the posture of the digital human in the first image and determine a second score; wherein the second score is used to indicate the degree of grace of the digital human's posture.

[0103] It should be noted that the execution process of step 301 to step 305 can refer to the execution process of any embodiment of the present disclosure, and will not be repeated here.

[0104] Step 306: Determine a first target image from at least one first image according to the first score corresponding to each first image.

[0105] As an example, the first image with the highest first score among the first images may be determined as the first target image.

[0106] As another example, a first image in which a first score is greater than a fourth set threshold value may be determined as a first target image.

[0107] It should be noted that the first target image may be but is not limited to one, and the present disclosure does not impose any limitation on this.

[0108] Step 307: Determine a second target image from at least one first image according to the second score corresponding to each first image.

[0109] As an example, the first image with the highest second score among the first images may be determined as the second target image.

[0110] As another example, a first image whose second score is greater than a fifth set threshold value in the first image may be determined as the second target image.

[0111] It should be noted that the second target image may be but is not limited to one, and the present disclosure does not impose any limitation on this.

[0112] Step 308: both the first target image and the second target image are used as images to be processed.

[0113] In the embodiment of the present disclosure, both the first target image and the second target image may be used as images to be processed.

[0114] Step 309: for at least one image region in the image to be processed, determine a capture frame position according to the interest level of the at least one image region.

[0115] Step 310, capturing the image to be processed based on the position of the capture frame to obtain a live broadcast cover image.

[0116] It should be noted that the execution process of step 309 to step 310 can refer to the execution process of any embodiment of the present disclosure, and will not be repeated here.

[0117] As a possible implementation method, the first target image in the image to be processed can be captured according to the position of the capture box to obtain a first partial image, and the second target image can be captured to obtain a second partial image, and the first partial image and the second partial image can be synthesized to obtain a live cover image.

[0118] That is to say, when the image to be processed includes a first target image and a second target image, the capture frame position includes the capture frame position of the first target image and the capture frame position of the second target image, respectively. Thus, according to the capture frame position of the first target image, the first target image is captured to obtain a first partial image, and according to the capture frame position of the second target image, the second target image is captured to obtain a second partial image; thus, the first partial image and the second partial image are synthesized to obtain a live cover image.

[0119] For example, assuming that the expression of the digital person in the first target image screened out meets the user's expectations, but the posture of the digital person in the first target image does not meet the user's expectations, and the expression of the digital person in the second target image screened out does not meet the user's expectations, but the posture of the digital person in the second target image meets the user's expectations, the position of the capture frame of the first target image is the position corresponding to the human body area of ​​the digital person, and the position of the capture frame of the second target image is the position corresponding to the human body of the digital person. Based on the capture frame position of the first target image, the first target image can be captured to obtain a first partial image, and based on the capture frame position of the second target image, the second target image can be captured to obtain a second partial image; thus, the first partial image and the second partial image are synthesized to obtain a live broadcast cover image. For example, an image segmentation model can be used to extract the head area image of the digital person from the first partial image, and the limb area image of the digital person can be extracted from the second partial image; the head area image and the limb area image can be synthesized using image synthesis technology to obtain a live broadcast cover image. For another example, an image segmentation model may be used to extract a first facial area image of a target size of the digital human from a first partial image, and a second facial area image of a target size of the digital human from a second partial image; and image synthesis technology may be used to synthesize the first facial area image and the area image of the second partial image except the second facial area image to obtain a live cover image.

[0120] Therefore, the required image area can be captured from different images to be processed, and then image synthesis technology can be used to synthesize the captured image area to achieve effective acquisition of the live cover image, thereby improving the flexibility of live cover generation.

[0121] The cover generation method for digital human live broadcast of the embodiment of the present disclosure determines a first target image from at least one first image according to a first score corresponding to each first image; determines a second target image from at least one first image according to a second score corresponding to each first image; and uses both the first target image and the second target image as images to be processed. Thus, accurate determination of the images to be processed can be achieved.

[0122] Based on the above embodiments of the present disclosure, the present disclosure also proposes a cover generation method for digital human live broadcast. Figure 4 It is a flowchart of a method for generating a cover for a digital human live broadcast provided according to another embodiment of the present disclosure.

[0123] like Figure 4 As shown, based on the above embodiments of the present disclosure, the cover generation method for digital human live broadcast may also include:

[0124] Step 401: Recognize the posture of the digital human in the first image to determine the body movements of the digital human.

[0125] Among them, body movements can be, for example, walking, jumping, running, clapping, etc., and the present disclosure does not impose any restrictions on this.

[0126] As an example, a posture estimation model may be used to identify the posture of the digital human in the first image to determine the body movements of the digital human.

[0127] Step 402: Determine a first sub-score corresponding to the action category according to the action category to which the digital human's body action belongs.

[0128] Among them, the action type, as an example, can be divided based on the location where the limb movement occurs. For example, the action categories can include: finger movements, palm movements, arm movements, arm holding movements, upper limb pushing movements, upper limb pulling movements, etc. in upper limb movements; walking movements, running movements, jumping movements, crossing movements, standing movements, squatting movements, stepping movements, kicking movements, lower limb buffering movements, lower limb extension movements, lower limb whipping movements, etc. in lower limb movements; shoulder movements, back movements, chest movements, waist movements, hip movements, turning over movements, sideways movements, etc. in torso movements.

[0129] As another example, the action categories may be divided based on the form of the action, for example, the action categories may include pushing, pulling, whipping, buffering, stretching, swinging, twisting, and the like.

[0130] As another example, the action categories may be divided based on the action functions. For example, the action categories may include supporting actions, moving actions, operating actions, expressing actions, and the like.

[0131] In the disclosed embodiment, the first sub-score corresponding to the action category may be determined according to the action category to which the digital human's body action belongs.

[0132] As an example, assuming that the action type is divided based on the location where the limb action occurs, then based on the action categories of upper limb action, lower limb action and trunk action to which the digital human's limb action belongs, the action scores of the upper limb action, the lower limb action and the trunk action are determined respectively; then, the action scores of the upper limb action, the lower limb action and the trunk action are fused to obtain the first sub-score.

[0133] As another example, assuming that the action types are divided based on the action forms, or the action categories are divided based on the action functions, then based on the action category to which the digital human's limb movements belong, the action score corresponding to the action category can be queried, and the action score can be determined as the first sub-score.

[0134] As another example, assuming that the action type is divided based on the action form and the action category, then based on the action category under the action form to which the digital human's limb action belongs, the first action score corresponding to the action category is queried, and based on the action category under the action function to which the digital human's limb action belongs, the second action score corresponding to the action category is queried; then, the first action score and the second action score queried above are merged to obtain a first sub-score.

[0135] It should be noted that the above examples of the first sub-score are only exemplary. In practical applications, other methods may also be used, which will not be described in detail in this disclosure.

[0136] Therefore, by finely dividing the action categories to which the digital human's body movements belong, the first sub-score can be effectively determined.

[0137] Step 403: Determine a second sub-score according to the movement coordination of the digital human's body movements.

[0138] In the disclosed embodiment, the movement coordination of the digital human's body movements can be analyzed.

[0139] As a possible implementation method, the coordination of the digital human's limb movements can be analyzed based on one or more combinations of the digital human's posture consistency, body symmetry, and body balance in the limb movements to obtain the second sub-score.

[0140] As a possible implementation method, a posture consistency score can be used to indicate the posture consistency of the digital human's limb movements, a symmetry score can be used to indicate the body symmetry of the digital human's limb movements, and a balance score can be used to indicate the body balance of the digital human's limb movements. The second sub-score is determined based on one or more combinations of the posture consistency score, the symmetry score, and the balance score.

[0141] For example, when determining the second sub-score based on the posture consistency score, the posture consistency score can be determined as the second sub-score; when determining the second sub-score based on the symmetry score, the symmetry score can be determined as the second sub-score; when determining the second sub-score based on the balance score, the balance score can be determined as the second sub-score.

[0142] For another example, when the second sub-score is determined based on multiple combinations of the posture consistency score, the symmetry score, and the balance score, for example, a weighted sum of the scores in the combination may be performed to obtain the second sub-score.

[0143] Thus, the second sub-score can be effectively determined based on one or more combinations of posture consistency, body symmetry, and body balance of the digital human in limb movements.

[0144] Step 404: Determine a second score based on the first sub-score and the second sub-score.

[0145] The cover generation method for digital human live broadcast in the disclosed embodiment determines the digital human's body movements by identifying the digital human's posture in the first image; determines the first sub-score corresponding to the action category according to the action category to which the digital human's body movements belong; determines the second sub-score according to the movement coordination of the digital human's body movements; and determines the second score based on the first sub-score and the second sub-score. Thus, the second score can be accurately determined by accurately identifying and comprehensively evaluating the digital human's posture.

[0146] In the case where the second sub-score is determined based on one or more combinations of the posture consistency score, the symmetry score, and the balance score, in order to clearly illustrate how the posture consistency score is determined, the present disclosure also proposes a flow chart of a cover generation method for digital human live broadcast.

[0147] Figure 5 A flowchart of a method for generating a cover for a digital human live broadcast provided in another embodiment of the present disclosure.

[0148] like Figure 5 As shown, based on the above embodiments of the present disclosure, the cover generation method for digital human live broadcast may also include:

[0149] Step 501, identifying the target human body joints of the digital human to determine the position information of the target human body joints; wherein the position information of the target human body joints includes the image coordinates of the corresponding first human body key points.

[0150] The target human joint may be, for example, a knee joint, a hip joint, a shoulder joint, etc., and the present disclosure does not impose any limitation on this.

[0151] In the disclosed embodiments, the target human joints of the digital human can be identified to determine the position information of the target human joints. For example, models such as OpenPose, PoseNet and HRNet (High-Resolution Network) can be used to determine the human key points of the digital human and the image coordinates of the human key points. Based on the human key points, a deep learning model (such as a convolutional neural network model) is used to detect the human joints of the digital human, realize the identification of the target human joints of the digital human, and determine the human key points corresponding to the target human joints. Accordingly, the image coordinates of the human key points corresponding to the target human joints can be determined.

[0152] Step 502, determining a target joint bending angle of the target human joint according to the image coordinates of the human key points corresponding to the target human joint.

[0153] Step 503: Obtain standard joint bending angles of target human joints in standard movements that match the digital human's limb movements.

[0154] The standard movements may be movements performed by professional athletes, professional dancers, etc.

[0155] In the disclosed embodiment, the standard joint bending angle of the target human joint in the standard action matching the digital human's limb action can be obtained. For example, assuming that the digital human's limb action is jumping, image data of a professional athlete's jumping standard action can be collected, and based on the image data, the standard joint bending angle of the human joint can be obtained and saved, so that in subsequent applications, the corresponding standard joint bending angle can be queried based on the target human joint in the standard action matching the limb action.

[0156] Step 504: Determine the posture consistency score of the digital human's limb movements according to the first similarity between the target joint bending angle of the target human joint and the standard joint bending angle.

[0157] As an example, a similarity algorithm may be used to determine a first similarity between a target joint bending angle of a target human joint and a standard joint bending angle, and the first similarity may be determined as a posture consistency score of the digital human's limb motion.

[0158] The cover generation method for digital human live broadcast of the disclosed embodiment identifies the target human joint of the digital human to determine the position information of the target human joint; wherein the position information of the target human joint includes the image coordinates of the corresponding first human key point; according to the image coordinates of the human key point corresponding to the target human joint, the target joint bending angle of the target human joint is determined; the standard joint bending angle of the target human joint under the standard action matching the digital human's limb action is obtained; according to the first similarity between the target joint bending angle of the target human joint and the standard joint bending angle, the posture consistency score of the digital human's limb action is determined. Thus, the posture consistency score of the digital human under the limb action is effectively determined by relying on the joint bending angle of the human joint of the digital human.

[0159] In the case where the second sub-score is determined based on one or more combinations of the posture consistency score, the symmetry score, and the balance score, in order to clearly illustrate how the symmetry score is determined, the present disclosure also proposes a flow chart of a cover generation method for digital human live broadcast.

[0160] Figure 6 A flowchart of a method for generating a cover for a digital human live broadcast provided in another embodiment of the present disclosure.

[0161] like Figure 6 As shown, based on the above embodiments of the present disclosure, the cover generation method for digital human live broadcast may also include:

[0162] Step 601, obtaining the image coordinates of multiple second human body key points of the digital human; wherein the second human body key points include head key points and multiple pairs of limb key points; wherein any pair of limb key points includes a left limb key point and a right limb key point.

[0163] Among them, key points of the head may include eyes, eyebrows, etc.

[0164] Among them, limb key points may include shoulders, elbows, wrists, hips, knees and ankles, etc.

[0165] Step 602, determining the center line of the human body based on the image coordinates of the key points of the head.

[0166] As an example, a geometric algorithm may be used to determine the center line of a human body based on the image coordinates of key points of the head.

[0167] Step 603, for any pair of limb key points, based on the image coordinates of the left limb key point among the limb key points, determine the first distance between the left limb key point among the limb key points and the center line, and based on the image coordinates of the right limb key point among the limb key points, determine the second distance between the right limb key point among the limb key points and the center line.

[0168] Step 604 : determining a second similarity corresponding to the pair of limb key points based on the first distance and the second distance corresponding to the limb key points.

[0169] As an example, a similarity algorithm or a distance algorithm may be used to determine the second similarity corresponding to the pair of limb key points based on the first distance and the second distance corresponding to the pair of limb key points.

[0170] Step 605: Determine the symmetry score of the digital human in the body movements according to the second similarities corresponding to each pair of body key points.

[0171] As an example, the second similarities corresponding to each pair of limb key points may be weighted averaged to obtain the symmetry score of the digital human in limb movements.

[0172] The cover generation method for digital human live broadcast of the disclosed embodiment obtains the image coordinates of multiple second human body key points of the digital human; wherein the second human body key points include head key points and multiple pairs of limb key points; wherein any pair of limb key points include left limb key points and right limb key points; based on the image coordinates of the head key points, the center line of the human body is determined; for any pair of limb key points, based on the image coordinates of the left limb key points in the limb key points, the first distance between the left limb key points and the center line is determined, and based on the image coordinates of the right limb key points in the limb key points, the second distance between the right limb key points and the center line is determined; based on the first distance and the second distance corresponding to the limb key points, the second similarity corresponding to the limb key points is determined; according to the second similarity corresponding to each pair of limb key points, the symmetry score of the digital human under the limb action is determined. Thus, by means of the distance between the limb key points on the left and right sides of the digital human and the center line of the human body, the similarity between the limb key points on the left and right sides is accurately determined, and then, relying on the similarity corresponding to each pair of limb key points, the symmetry score of the digital human under the limb action is accurately determined.

[0173] In the case where the second sub-score is determined based on one or more combinations of the posture consistency score, the symmetry score, and the balance score, in order to clearly illustrate how the balance score is determined, the present disclosure also proposes a flow chart of a cover generation method for digital human live broadcast.

[0174] Figure 7 A flowchart of a method for generating a cover for a digital human live broadcast provided in another embodiment of the present disclosure.

[0175] like Figure 7 As shown, based on the above embodiments of the present disclosure, the cover generation method for digital human live broadcast may also include:

[0176] Step 701, obtaining the image coordinates of a plurality of third-party human key points of the digital human.

[0177] Among them, the third human body key points include heels, ankles, shoulders, elbows, wrists, hips, knees, etc.

[0178] Step 702: Determine the position of the center of mass (also called the center of gravity) of the digital human according to the image coordinates of a plurality of third human key points.

[0179] As an example, assume that the number of third-party human key points is n, where the image coordinates of the i-th third-party human key point are (x i ,y i ), the center of mass position of the digital human can be determined according to the following formula:

[0180]

[0181] Among them, w i represents the weight of the i-th human key point, (x G ,y G ) represents the center of mass position of the digital human.

[0182] Among them, the weight of each third-party human body key point can be pre-set. For example, the weight of the corresponding human body key point can be determined according to the mass proportion of the body part to which the third-party human body key point belongs; for example, when the body part to which the third-party human body key point belongs is the head, the mass proportion of the head is 8%, and the weight of the third-party human body key point is 8%; when the body part to which the third-party human body key point belongs is the torso (including buttocks), the mass proportion of the torso is 50%, and the weight of the third-party human body key point is 50%, and so on.

[0183] Step 703: Determine the line segment corresponding to the supporting surface of the digital human in the first image according to the image coordinates corresponding to the heel of the digital human.

[0184] The support surface is determined based on the projection of the digital human's heel in the horizontal direction of the image.

[0185] As an example, Figure 8 The supporting surface of the digital human is shown. Assume that the image coordinates of the left heel of the digital human are (x 1 ,y 1 ), the image coordinates of the right heel are (x 2 ,y 2 ), then the projection coordinate of the left heel of the digital human in the horizontal direction of the image is (x 1 ,y 1 ), the projection coordinate of the right heel in the horizontal direction of the image is (x 2 ,y 1 ), then the coordinates of the two end points of the line segment corresponding to the supporting surface of the digital human in the first image are (x 1 ,y 1 )、(x 2 ,y 1 ), the line segment is located on the straight line y = y 1 .

[0186] Step 704: determine the center position of the support surface according to the coordinates of the two end points of the line segment corresponding to the support surface.

[0187] Still using the above example, the center position of the support surface can be determined according to the following formula:

[0188]

[0189] Among them, (x 中 ,y中 ) indicates the center position of the support surface.

[0190] Step 705: Determine the balance score of the digital human in the body movements according to the distance between the center position and the center of mass position.

[0191] Among them, the distance between the center position and the center of mass position is negatively correlated with the balance score of the digital human in body movements, that is, the greater the distance between the center position and the center of mass position, the smaller the balance score of the digital human in body movements.

[0192] The cover generation method for digital human live broadcast of the disclosed embodiment obtains the image coordinates of multiple third-party human key points of the digital human; wherein the third-party human key points include the heels; according to the image coordinates of the multiple third-party human key points, the center of mass position of the digital human is determined; according to the image coordinates corresponding to the heels of the digital human, the line segment corresponding to the support surface of the digital human in the first image is determined; according to the coordinates of the two end points of the line segment corresponding to the support surface, the center position of the support surface is determined; wherein the support surface is determined based on the projection of the heels of the digital human in the horizontal direction of the image; according to the distance between the center position and the center of mass position, the balance score of the digital human under body movements is determined. Thus, by analyzing the center of mass of the digital human and the center of the support surface, the balance score of the digital human under body movements can be effectively and accurately determined.

[0193] In order to clearly illustrate how to determine the position of the cropped frame according to the interest level of at least one image area in the image to be processed in the implementation of the present disclosure, the present disclosure also proposes a flow chart of a cover generation method for digital human live broadcast.

[0194] Fig. 9 A flowchart of a method for generating a cover for a digital human live broadcast provided in another embodiment of the present disclosure.

[0195] like Fig. 9 As shown, based on the above embodiments of the present disclosure, the cover generation method for digital human live broadcast may also include:

[0196] Step 901: perform object recognition on an image to be processed to determine at least one target object contained in the image to be processed.

[0197] The target object may be, for example, a digital person, a car, an animal, etc., and the present disclosure does not impose any limitation on this.

[0198] It should be noted that the target object can be but is not limited to one, and the present disclosure does not impose any restrictions on this.

[0199] In the embodiments of the present disclosure, object recognition can be performed on the image to be processed to identify the target object contained in the image to be processed. For example, a deep learning model (such as a YOLO (You Only Look Once) model, a Faster R-CNN (Faster Region-based Convolutional Neural Networks) model, an SSD (Single Shot MultiBox Detector), etc.) can be used to perform object recognition on the image to be processed to identify the target object contained in the image to be processed.

[0200] Step 902 : Based on at least one target object contained in the image to be processed, the image to be processed is segmented to obtain image regions corresponding to each target object.

[0201] As an example, a detection box of each target object in the image to be processed can be generated, and then an image segmentation model (such as Mask R-CNN (Mask Region-based Convolutional Neural Network, region-based mask convolutional neural network or mask region convolutional neural network) model, FCN (Fully Convolutional Networks, full convolutional neural network) model, U-Net (Convolutional Networks for Biomedical Image Segmentation, U-type convolutional neural network) model, etc.) can be used to perform image segmentation on the image to be processed based on the detection box of each target object to obtain the image area corresponding to each target object.

[0202] Step 903 : determining a target image region from among the image regions based on the interest level of each image region in the image to be processed.

[0203] As an example, a heat map of the image to be processed may be generated based on the interest level of each image region in the image to be processed; and a target image region may be determined from each image region based on the interest level indicated by the heat map. Specifically, the index value of the interest level corresponding to each image region in the image to be processed may be mapped to the brightness value of the pixel to generate a heat map of the image to be processed, and the heat map of the image to be processed may be displayed to the user; and the user may determine the target image region from each image region based on the displayed heat map.

[0204] As another example, an image region in the to-be-processed image whose index value of the interest level is greater than a fifth set threshold value may be determined as the target image region.

[0205] Step 904: Determine the position of the cropping frame of the image to be processed based on the target image region.

[0206] As an example, assume that the target image region is one. The position of the cropping frame of the image to be processed can be determined based on the position information of the detection frame of the target image region in the image to be processed. For example, the position indicated by the position information of the detection frame of the target image region in the image to be processed is determined as the position of the cropping frame of the image to be processed.

[0207] As another example, assume that there are multiple target image regions. The position of the cropping frame of the image to be processed can be determined based on the positions of the detection frames of the multiple target image regions in the image to be processed. For example, in the plane coordinate system of the image to be processed, which is the xoy coordinate system, where the x-axis represents the horizontal direction, the y-axis represents the vertical direction, and the coordinate origin (0, 0) is located at the upper left corner of the image to be processed, and the position of the detection frame includes the upper left corner coordinates and the lower right corner coordinates. In this case, the upper left corner coordinates of the position of the cropping frame of the image to be processed can be determined based on the minimum value of the abscissa in the position information of the detection frames corresponding to each target image region, and the minimum value of the ordinate in the position information of the detection frames corresponding to each target image region. The lower right corner coordinates of the position of the cropping frame of the image to be processed can be determined based on the maximum value of the abscissa in the position information of the detection frames corresponding to each target image region, and the maximum value of the ordinate in the position information of the detection frames corresponding to each target image region. For example, assume that the position of the detection frame of target image region 1 in the image to be processed is {(x1, y1), (x2, y2)}, and the position of the detection frame of target image region 2 in the image to be processed is {(x3, y3), (x4, y4)}, where (x1, y1) is the upper left corner coordinates of target image region 1, (x2, y2) is the lower right corner coordinates of target image region 1, (x3, y3) is the upper left corner coordinates of target image region 2, (x4, y4) is the lower right corner coordinates of target image region 2, x1 < x2 < x3 < x4, and y3 < y1 < y4 < y2. Then the position of the cropping frame of the image to be processed is {(x1, y3), (x4, y2)}.

[0208] The method for generating a cover for digital human live broadcast according to the embodiments of the present disclosure identifies at least one target object included in the image to be processed by performing object recognition on the image to be processed; performs image segmentation on the image to be processed based on at least one target object included in the image to be processed to obtain the image regions corresponding to the respective target objects; determines the target image region from the respective image regions based on the degree of interest of the respective image regions in the image to be processed; and determines the position of the cropping frame of the image to be processed based on the target image region. Thus, the image region of interest to the user in the image to be processed can be automatically located, facilitating the personalized customization of the cover image for the user.

[0209] With the above Figures 1 to 9Corresponding to the cover generation method for digital human live broadcast provided in the embodiment, the present disclosure also provides a cover generation device for digital human live broadcast. Figures 1 to 9 The cover generation method for digital human live broadcast provided in the embodiment corresponds to the embodiment, so the implementation method of the cover generation method for digital human live broadcast is also applicable to the cover generation device for digital human live broadcast provided in the embodiment of the present disclosure, and will not be described in detail in the embodiment of the present disclosure.

[0210] Fig.10 It is a structural schematic diagram of a cover generation device for digital human live broadcast provided according to another embodiment of the present disclosure.

[0211] like Fig.10 As shown, the cover generation device 1000 for digital human live broadcast may include: an acquisition module 1010, a first determination module 1020, a second determination module 1030 and an interception module 1040;

[0212] The acquisition module 1010 is used to acquire multiple image frames from the live video of the digital human.

[0213] The first determination module 1020 is used to determine the image to be processed for generating the live broadcast cover from multiple image frames according to the expression and / or posture of the digital human in the multiple image frames.

[0214] A second determination module 1030 is used to determine the position of the capture frame according to the interest level of at least one image area in the image to be processed;

[0215] The capture module 1040 is used to capture the image to be processed based on the capture frame position to obtain a live broadcast cover image.

[0216] In a possible implementation of the embodiment of the present disclosure, the first determination module 1020 is used to: for any image frame among multiple image frames, perform object recognition on the image frame to determine whether the image frame contains a digital human; determine the image frame containing the digital human as the first image; recognize the expression of the digital human in the first image, and determine a first score corresponding to the recognized expression; recognize the posture of the digital human in the first image, and determine a second score; wherein the second score is used to indicate the degree of beauty of the digital human's posture; and determine an image to be processed from at least one first image based on the first score and / or the second score corresponding to the first image.

[0217] In a possible implementation of the embodiments of the present disclosure, the first determination module 1020 is used to: for any first image, obtain a quality score of the first image based on a fusion of a first score and a second score corresponding to the first image; and determine an image to be processed from at least one first image according to the quality scores of each first image.

[0218] In a possible implementation of the embodiments of the present disclosure, the first determination module 1020 is used to: determine a first target image from at least one first image based on a first score corresponding to each first image; determine a second target image from at least one first image based on a second score corresponding to each first image; and use both the first target image and the second target image as images to be processed.

[0219] In a possible implementation of the embodiment of the present disclosure, the capture module 1040 is used to: capture the first target image in the image to be processed according to the capture frame position to obtain a first partial image, and capture the second target image to obtain a second partial image; and synthesize the first partial image and the second partial image to obtain a live cover image.

[0220] In a possible implementation of the embodiment of the present disclosure, the first determination module 1020 is used to: recognize the posture of the digital human in the first image to determine the digital human's body movements; determine a first sub-score corresponding to the action category according to the action category to which the digital human's body movements belong; determine a second sub-score according to the movement coordination of the digital human's body movements; and determine a second score based on the first sub-score and the second sub-score.

[0221] In a possible implementation of the embodiment of the present disclosure, the first determination module 1020 is used to analyze the coordination of the digital human's limb movements based on one or more combinations of posture consistency, body symmetry, and body balance in the digital human's limb movements to obtain a second sub-score.

[0222] In a possible implementation of the embodiment of the present disclosure, the first determination module 1020 is used to: identify the target human joint of the digital human to determine the position information of the target human joint; wherein the position information of the target human joint includes the image coordinates of the corresponding first human key point; determine the target joint bending angle of the target human joint according to the image coordinates of the human key point corresponding to the target human joint; obtain the standard joint bending angle of the target human joint under the standard action matching the limb action of the digital human; determine the posture consistency score of the limb action of the digital human according to the first similarity between the target joint bending angle of the target human joint and the standard joint bending angle.

[0223] In a possible implementation of the embodiment of the present disclosure, body symmetry is evaluated using a symmetry score of the digital human under body movements. The first determination module 1020 is used to: obtain image coordinates of multiple second human body key points of the digital human; wherein the second human body key points include head key points and multiple pairs of limb key points; wherein any pair of limb key points include left limb key points and right limb key points; determine the center line of the human body based on the image coordinates of the head key points; for any pair of limb key points, determine a first distance between the left limb key point in the limb key points and the center line based on the image coordinates of the left limb key point in the limb key points, and determine a second distance between the right limb key point in the limb key points and the center line based on the image coordinates of the right limb key point in the limb key points; determine a second similarity corresponding to the limb key points based on the first distance and the second distance corresponding to the limb key points; and determine the symmetry score of the digital human under body movements according to the second similarity corresponding to each pair of limb key points.

[0224] In a possible implementation of the embodiment of the present disclosure, body balance is evaluated using a balance score of the digital human under body movements. The first determination module 1020 is used to: obtain image coordinates of multiple third-body key points of the digital human; wherein the third-body key points include heels; determine the center of mass position of the digital human according to the image coordinates of the multiple third-body key points; determine a line segment corresponding to a support surface of the digital human in the first image according to the image coordinates corresponding to the heels of the digital human; determine the center position of the support surface according to the coordinates of two end points of the line segment corresponding to the support surface; wherein the support surface is determined based on the projection of the heels of the digital human in the horizontal direction of the image; and determine the balance score of the digital human under body movements according to the distance between the center position and the center of mass position.

[0225] In a possible implementation of the embodiments of the present disclosure, the second determination module 1030 is used to: perform object recognition on the image to be processed to determine at least one target object contained in the image to be processed; perform image segmentation on the image to be processed based on the at least one target object contained in the image to be processed to obtain image areas corresponding to each target object; determine a target image area from each image area based on the degree of interest of each image area in the image to be processed; and determine a capture frame position of the image to be processed based on the target image area.

[0226] The cover generation device for digital human live broadcast of the embodiment of the present disclosure obtains multiple image frames from the live broadcast video of the digital human; determines the image to be processed for generating the live broadcast cover from the multiple image frames according to the expression and / or posture of the digital human in the multiple image frames; determines the position of the interception frame according to the interest level of at least one image area in the image to be processed; and intercepts the image to be processed based on the position of the interception frame to obtain the live broadcast cover image. Thus, the large model can be automatically trained based on the feedback information, so that the trained large model can learn and generalize the cover generation strategy for digital human live broadcast that adapts to various complex user emotional states, and improve the user satisfaction of the determined cover generation strategy for digital human live broadcast. Thus, the dynamic characteristics of the digital human live broadcast are fully utilized, the moment when the expression and / or posture of the digital human meets the requirements is captured, and the position of the interception frame is determined according to the interest level of the image area, so that the generated cover image can focus on the most important content, avoid the interference of irrelevant information, and finally automatically generate the live broadcast cover image by intercepting the image, effectively reducing the manual intervention in the live broadcast cover generation process, improving the generation efficiency of the live broadcast cover, and realizing the personalized customization of the live broadcast cover, enhancing the user experience.

[0227] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0228] Fig.11 A schematic block diagram of an example electronic device 1100 that can be used to implement an embodiment of the present disclosure is shown. The electronic device 1100 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0229] like Fig.11As shown, the electronic device 1100 includes a computing unit 1101, which can perform various appropriate actions and processes according to a computer program stored in a ROM (Read-Only Memory) 1102 or a computer program loaded from a storage unit 1108 to a RAM (Random Access Memory) 1103. In the RAM 1103, various programs and data required for the operation of the electronic device 1100 can also be stored. The computing unit 1101, the ROM 1102, and the RAM 1103 are connected to each other via a bus 1104. An I / O (Input / Output) interface 1105 is also connected to the bus 1104.

[0230] Multiple components in the electronic device 1100 are connected to the I / O interface 1105, including: an input unit 1106, such as a keyboard, a mouse, etc.; an output unit 1107, such as various types of displays, speakers, etc.; a storage unit 1108, such as a disk, an optical disk, etc.; and a communication unit 1109, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1109 allows the electronic device 1100 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0231] The computing unit 1101 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1101 include, but are not limited to, a CPU (Central Processing Unit), a GPU (Graphic Processing Units), various dedicated AI (Artificial Intelligence) computing chips, various computing units running machine learning model algorithms, a DSP (Digital Signal Processor), and any appropriate processor, controller, microcontroller, etc. The computing unit 1101 performs the various methods and processes described above, such as a cover generation method for a digital human live broadcast. For example, in some embodiments, the cover generation method for a digital human live broadcast may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 1108. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 1100 via the ROM 1102 and / or the communication unit 1109. When the computer program is loaded into RAM 1103 and executed by the computing unit 1101, one or more steps of the cover generation method for digital human live broadcast described above may be performed. Alternatively, in other embodiments, the computing unit 1101 may be configured to execute the cover generation method for digital human live broadcast in any other appropriate manner (e.g., by means of firmware).

[0232] Various embodiments of the systems and techniques described above herein may be implemented in digital electronic circuit systems, integrated circuit systems, FPGAs (Field Programmable Gate Arrays), ASICs (Application-Specific Integrated Circuits), ASSPs (Application Specific Standard Products), SOCs (System On Chips), CPLDs (Complex Programmable Logic Devices), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include: being implemented in one or more computer programs that may be executed and / or interpreted on a programmable system including at least one programmable processor that may be a dedicated or general-purpose programmable processor that may receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0233] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0234] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. More specific examples of machine-readable storage media may include electrical connections based on one or more lines, portable computer disks, hard disks, RAM, ROM, EPROM (Electrically Programmable Read-Only-Memory) or flash memory, optical fiber, CD-ROM (Compact Disc Read-Only Memory), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0235] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (Cathode-Ray Tube) or LCD (Liquid Crystal Display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0236] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: LAN (Local Area Network), WAN (Wide Area Network), the Internet, and blockchain networks.

[0237] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server may also be a server of a distributed system, or a server combined with a blockchain.

[0238] It should be noted that artificial intelligence is a discipline that studies how computers can simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, planning, etc.), and includes both hardware-level and software-level technologies. Artificial intelligence hardware technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, and big data processing; artificial intelligence software technologies mainly include computer vision technology, speech recognition technology, natural language processing technology, as well as machine learning / deep learning, big data processing technology, knowledge graph technology, and other major directions.

[0239] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0240] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for generating a cover for a digital human live broadcast, the method comprising: Acquire multiple image frames from a live video of a digital human; Determining, from the plurality of image frames, an image to be processed for generating a live broadcast cover according to the expression and / or posture of the digital human in the plurality of image frames; For at least one image area in the image to be processed, determining a capture frame position according to the interest level of the at least one image area; The image to be processed is captured based on the capture frame position to obtain a live cover image.

2. The method according to claim 1, wherein: The step of determining, from the plurality of image frames, an image to be processed for generating a live broadcast cover according to the expression and / or posture of the digital human in the plurality of image frames comprises: For any of the plurality of image frames, perform object recognition on the image frame to determine whether the image frame contains a digital human; Determine the image frame containing the digital human as the first image; Recognizing the expression of the digital person in the first image, and determining a first score corresponding to the recognized expression; Recognize the posture of the digital human in the first image and determine a second score; wherein the second score is used to indicate the degree of beauty of the posture of the digital human; The image to be processed is determined from at least one of the first images based on the first score and / or the second score corresponding to the first images.

3. The method according to claim 2, wherein: The determining the image to be processed from at least one of the first images based on the first score and / or the second score corresponding to the first image comprises: For any of the first images, obtaining a quality score of the first image based on a fusion of a first score and a second score corresponding to the first image; An image to be processed is determined from the at least one first image according to the quality score of each of the first images.

4. The method according to claim 2, wherein: The determining the image to be processed from at least one of the first images based on the first score and / or the second score corresponding to the first image comprises: determining a first target image from the at least one first image according to a first score corresponding to each of the first images; determining a second target image from the at least one first image according to a second score corresponding to each of the first images; The first target image and the second target image are both used as the images to be processed.

5. The method according to claim 4, wherein: The image to be processed is captured based on the capture frame position to obtain a live broadcast cover image, including: According to the position of the capture frame, the first target image in the image to be processed is captured to obtain a first partial image, and the second target image is captured to obtain a second partial image; The first partial image and the second partial image are synthesized to obtain the live broadcast cover image.

6. The method according to claim 2, wherein: The step of recognizing the posture of the digital human in the first image and determining a second score includes: Recognizing the posture of the digital human in the first image to determine the body movements of the digital human; According to the action category to which the limb action of the digital human belongs, determining a first sub-score corresponding to the action category; determining the second sub-score according to the movement coordination of the digital human's limb movements; Based on the first sub-score and the second sub-score, the second score is determined.

7. The method according to claim 6, wherein: Determining the second sub-score according to the movement coordination of the digital human's limb movements includes: Based on one or more combinations of posture consistency, body symmetry, and body balance of the digital human in the limb movements, the coordination of the limb movements of the digital human is analyzed to obtain the second sub-score.

8. The method according to claim 7, wherein: The posture consistency is evaluated by using the posture consistency score of the digital human in the limb motion, wherein the process of determining the posture consistency score of the digital human in the limb motion includes: Identify the target human joint of the digital human to determine the position information of the target human joint; wherein the position information of the target human joint includes the image coordinates of the corresponding first human key point; Determining a target joint bending angle of the target human joint according to the image coordinates of the human body key point corresponding to the target human joint; Acquire the standard joint bending angle of the target human joint under the standard action matching the limb action of the digital human; According to a first similarity between the target joint bending angle of the target human joint and the standard joint bending angle, a posture consistency score of the limb movement of the digital human is determined.

9. The method according to claim 7, wherein: The body symmetry is evaluated by using the symmetry score of the digital human in the limb movements, wherein the process of determining the symmetry score of the digital human in the limb movements includes: Acquire the image coordinates of a plurality of second human body key points of the digital human; wherein the second human body key points include a head key point and a plurality of pairs of limb key points; wherein any pair of limb key points includes a left limb key point and a right limb key point; Determine the center line of the human body based on the image coordinates of the key points of the head; For any pair of the limb key points, based on the image coordinates of the left limb key point among the limb key points, determine a first distance between the left limb key point among the limb key points and the center line, and based on the image coordinates of the right limb key point among the limb key points, determine a second distance between the right limb key point among the limb key points and the center line; Determine a second similarity corresponding to the limb key point based on the first distance and the second distance corresponding to the limb key point; The symmetry score of the digital human in the limb movement is determined according to the second similarities corresponding to each pair of limb key points.

10. The method according to claim 7, wherein: The body balance is evaluated by using the balance score of the digital human in the limb movements, wherein the process of determining the balance score of the digital human in the limb movements includes: Acquire the image coordinates of a plurality of third body key points of the digital human; wherein the third body key points include the heel; Determining the center of mass position of the digital human according to the image coordinates of the plurality of third human key points; Determine, according to the image coordinates corresponding to the heel of the digital human, a line segment corresponding to the supporting surface of the digital human in the first image; Determine the center position of the support surface according to the coordinates of the two end points of the line segment corresponding to the support surface; wherein the support surface is determined based on the projection of the heel of the digital human in the horizontal direction of the image; The balance score of the digital human in the limb movement is determined according to the distance between the center position and the center of mass position.

11. The method according to any one of claims 1 to 10, wherein: The step of determining the position of the capture frame according to the interest level of at least one image area in the image to be processed comprises: Performing object recognition on the image to be processed to determine at least one target object contained in the image to be processed; Based on at least one target object contained in the image to be processed, performing image segmentation on the image to be processed to obtain image regions corresponding to each target object; Determining a target image region from among the image regions based on the interest level of each of the image regions in the image to be processed; Based on the target image area, a capture frame position of the image to be processed is determined.

12. A cover generation device for digital human live broadcast, wherein: The device comprises: An acquisition module, used for acquiring multiple image frames from the live video of the digital human; A first determination module, configured to determine, from the plurality of image frames, an image to be processed for generating a live broadcast cover according to the expression and / or posture of the digital human in the plurality of image frames; A second determination module is used to determine the position of the capture frame according to the interest level of at least one image area in the image to be processed; The capture module is used to capture the image to be processed based on the capture frame position to obtain a live broadcast cover image.

13. The device according to claim 12, wherein: The first determining module is used to: For any of the plurality of image frames, perform object recognition on the image frame to determine whether the image frame contains a digital human; Determine the image frame containing the digital human as the first image; Recognizing the expression of the digital person in the first image, and determining a first score corresponding to the recognized expression; Recognize the posture of the digital human in the first image and determine a second score; wherein the second score is used to indicate the degree of beauty of the posture of the digital human; The image to be processed is determined from at least one of the first images based on the first score and / or the second score corresponding to the first images.

14. The device according to claim 13, wherein: The first determining module is used to: For any of the first images, obtaining a quality score of the first image based on a fusion of a first score and a second score corresponding to the first image; An image to be processed is determined from the at least one first image according to the quality score of each of the first images.

15. The device according to claim 13, wherein: The first determining module is used to: determining a first target image from the at least one first image according to a first score corresponding to each of the first images; determining a second target image from the at least one first image according to a second score corresponding to each of the first images; The first target image and the second target image are both used as the images to be processed.

16. The device according to claim 15, wherein: The interception module is used for: According to the position of the capture frame, the first target image in the image to be processed is captured to obtain a first partial image, and the second target image is captured to obtain a second partial image; The first partial image and the second partial image are synthesized to obtain the live broadcast cover image.

17. The device according to claim 13, wherein: The first determining module is used to: Recognizing the posture of the digital human in the first image to determine the body movements of the digital human; According to the action category to which the limb action of the digital human belongs, determining a first sub-score corresponding to the action category; determining the second sub-score according to the movement coordination of the digital human's limb movements; Based on the first sub-score and the second sub-score, the second score is determined.

18. The device according to claim 17, wherein: The first determining module is used to: Based on one or more combinations of posture consistency, body symmetry, and body balance of the digital human in the limb movements, the coordination of the limb movements of the digital human is analyzed to obtain the second sub-score.

19. The device according to claim 18, wherein: The posture consistency is evaluated by using the posture consistency score of the digital human in the limb movement. The first determination module is used to: Identify the target human joint of the digital human to determine the position information of the target human joint; wherein the position information of the target human joint includes the image coordinates of the corresponding first human key point; Determining a target joint bending angle of the target human joint according to the image coordinates of the human body key point corresponding to the target human joint; Acquire the standard joint bending angle of the target human joint under the standard action matching the limb action of the digital human; According to a first similarity between the target joint bending angle of the target human joint and the standard joint bending angle, a posture consistency score of the limb movement of the digital human is determined.

20. The device according to claim 18, wherein The body symmetry is evaluated by using the symmetry score of the digital human in the limb movements. The first determination module is used to: Acquire the image coordinates of a plurality of second human body key points of the digital human; wherein the second human body key points include a head key point and a plurality of pairs of limb key points; wherein any pair of limb key points includes a left limb key point and a right limb key point; Determine the center line of the human body based on the image coordinates of the key points of the head; For any pair of the limb key points, based on the image coordinates of the left limb key point among the limb key points, determine a first distance between the left limb key point among the limb key points and the center line, and based on the image coordinates of the right limb key point among the limb key points, determine a second distance between the right limb key point among the limb key points and the center line; Determine a second similarity corresponding to the limb key point based on the first distance and the second distance corresponding to the limb key point; The symmetry score of the digital human in the limb movement is determined according to the second similarities corresponding to each pair of limb key points.

21. The device according to claim 18, wherein The body balance is evaluated by using the balance score of the digital human under the limb movements. The first determination module is used to: Acquire the image coordinates of a plurality of third human body key points of the digital human; wherein the third human body key points include the heel; Determining the center of mass position of the digital human according to the image coordinates of the plurality of third human key points; Determine, according to the image coordinates corresponding to the heel of the digital human, a line segment corresponding to the supporting surface of the digital human in the first image; Determine the center position of the support surface according to the coordinates of the two end points of the line segment corresponding to the support surface; wherein the support surface is determined based on the projection of the heel of the digital human in the horizontal direction of the image; The balance score of the digital human in the limb movement is determined according to the distance between the center position and the center of mass position.

22. The device according to any one of claims 12 to 21, wherein: The second determining module is used to: Performing object recognition on the image to be processed to determine at least one target object contained in the image to be processed; Based on at least one target object contained in the image to be processed, performing image segmentation on the image to be processed to obtain image regions corresponding to each target object; Determining a target image region from among the image regions based on the interest level of each of the image regions in the image to be processed; Based on the target image area, a capture frame position of the image to be processed is determined.

23. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 11.

24. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 11.

25. A computer program product, comprising a computer program, which, when executed by a processor, implements the method of any one of claims 1 to 11.

Images