Virtual image follow-up control method, device, electronic device and readable storage medium

Abstract

The present application provides a method, device, electronic device and readable storage medium for virtual image tracking control, which obtains the host's facial image and obtains multiple facial features contained in the facial image, wherein for each facial feature, a plurality of preset expression base templates corresponding thereto can be obtained, and then the template coefficient of the facial feature relative to each expression base template is determined based on the facial feature and the multiple expression base templates. Based on each expression base template and its corresponding template coefficient, the state of the virtual features corresponding to the facial features in the constructed virtual image is controlled for tracking. This solution, by embodying the host's facial features with an expression base template carrying a template coefficient, and then controlling the virtual image, can achieve tracking control of the virtual image without the need for other auxiliary equipment, and this solution is controlled based on the information of the expression base template, so that the control information is more standardized and the control is more precise.

Description

Technical Field

[0001] The present application relates to the field of live broadcast technology, and in particular to a method, device, electronic device and readable storage medium for controlling a virtual image. Background Art

[0002] With the popularity of the live streaming industry, more and more people want to enter the live streaming industry, but some anchors are not confident about their appearance. If virtual images can be used to assist these anchors and let the virtual images follow the anchors to perform vividly, then the threshold for live streaming will be greatly lowered.

[0003] In the existing solutions, some anchors directly use avatar stickers or wear headgear for live broadcasts, but these static stickers or headgear cannot make vivid expressions with the anchors, which greatly reduces the interaction effect with the audience. Another solution is to let the anchor wear professional motion capture equipment, but this method is expensive, and the anchor has to wear a helmet at all times, which causes great discomfort to the anchor. Summary of the invention

[0004] The purpose of the present application includes, for example, providing a method, device, electronic device and readable storage medium for virtual image follow-up control, which can accurately, standardizedly and without interference realize the follow-up control of the virtual image.

[0005] The embodiments of the present application can be implemented as follows:

[0006] In a first aspect, the present application provides a method for controlling a virtual character following motion, the method comprising:

[0007] Obtaining a facial image of the anchor, and obtaining multiple facial features contained in the facial image;

[0008] For each of the facial features, obtaining a plurality of preset corresponding expression base templates;

[0009] Determining, according to the facial feature and the plurality of expression base templates, a template coefficient of the facial feature relative to each of the expression base templates;

[0010] Based on the expression base templates and their corresponding template coefficients, the virtual features corresponding to the facial features in the constructed virtual image are controlled in a state-following manner.

[0011] In an optional implementation, the step of determining the template coefficient of the facial feature relative to each of the expression base templates based on the facial feature and the multiple expression base templates includes:

[0012] According to the pre-established objective function, the facial features and the multiple expression base templates, the objective function is optimized until the multiple template coefficients contained in the objective function are obtained when the preset requirements are met, wherein the template coefficients correspond one-to-one to the expression base templates.

[0013] In an optional implementation, the step of optimizing the objective function according to the pre-established objective function, the facial features and the plurality of expression base templates comprises:

[0014] Each of the expression base templates is projected and transformed, and the objective function is optimized according to a pre-established objective function, the facial features and the multiple expression base templates after the projection transformation.

[0015] In an optional embodiment, the step of projecting each of the expression base templates and optimizing the objective function according to a pre-established objective function, the facial features, and the plurality of expression base templates after the projection transformation comprises:

[0016] Performing a primary projection transformation on each of the expression base templates using a preset initial transformation coefficient, and obtaining a plurality of template coefficients included in the objective function according to a pre-established objective function, the facial features, and the plurality of expression base templates after the primary projection transformation;

[0017] The obtained template coefficients are used as the updated transformation coefficients of each expression base template, and the updated transformation coefficients are used to re-project the expression base templates. The objective function is optimized according to the objective function, the facial features and the multiple expression base templates after the re-projection transformation until the preset requirements are met.

[0018] In an optional implementation, the facial features include a plurality of facial feature points, and each of the expression base templates includes a plurality of expression feature points;

[0019] The step of projecting each of the expression base templates comprises:

[0020] For each of the facial feature points in the facial features, determining an expression feature point in each of the expression base templates corresponding to the facial feature point;

[0021] Projection transformation is performed on each of the expression base templates according to the coordinate values ​​of the expression feature points in each of the expression base templates and the coordinate values ​​of the corresponding facial feature points.

[0022] In an optional implementation, the facial features include a plurality of facial feature points, and each of the expression base templates includes a plurality of expression feature points;

[0023] The step of optimizing the objective function according to the pre-established objective function, the facial features and the multiple expression base templates after projection transformation comprises:

[0024] Obtaining a first array constructed by the plurality of facial feature points, and obtaining a second array constructed by the plurality of expression feature points included in each of the expression base templates;

[0025] Based on a pre-established objective function, a plurality of second arrays are weightedly accumulated according to corresponding template coefficients and then the first array is subtracted, and when the objective function meets preset requirements, each template coefficient is determined.

[0026] In an optional implementation, the step of performing state tracking control on virtual features corresponding to the facial features in the constructed virtual image based on the expression base templates and the template coefficients corresponding to each of the expression base templates comprises:

[0027] Performing weighted accumulation of the multiple expression base templates according to corresponding template coefficients to obtain control information;

[0028] The state of the virtual features corresponding to the facial features in the constructed virtual image is controlled according to the control information.

[0029] In an optional implementation, the control information includes a plurality of coordinate information, each of which is a weighted result of coordinates of corresponding expression feature points included in a plurality of expression base templates;

[0030] The step of performing state tracking control on the virtual features corresponding to the facial features in the constructed virtual image according to the control information comprises:

[0031] Obtaining a virtual feature corresponding to the facial feature in the constructed virtual image, and obtaining a plurality of virtual feature points included in the virtual feature;

[0032] Each virtual feature point in the virtual feature is controlled according to the corresponding coordinate information in the control information to drive the virtual image to follow the state.

[0033] In a second aspect, the present application provides a virtual image follow-up control device, the device comprising:

[0034] The first acquisition module is used to obtain the host's facial image and obtain multiple facial features contained in the facial image;

[0035] A second acquisition module is used to obtain a plurality of corresponding expression base templates for each of the facial features;

[0036] A determination module, configured to determine a template coefficient of the facial feature relative to each of the expression base templates based on the facial feature and the multiple expression base templates;

[0037] The control module is used to perform state tracking control on the virtual features corresponding to the facial features in the constructed virtual image based on the expression base templates and the corresponding template coefficients.

[0038] In a third aspect, the present application provides an electronic device comprising one or more storage media and one or more processors communicating with the storage media, wherein the one or more storage media store machine-executable instructions executable by the processor, and when the electronic device is running, the processor executes the machine-executable instructions to perform the method steps described in any one of the aforementioned embodiments.

[0039] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores machine-executable instructions, and when the machine-executable instructions are executed, the method steps described in any one of the aforementioned embodiments are implemented.

[0040] The beneficial effects of the embodiments of the present application include, for example:

[0041] The present application provides a method, device, electronic device and readable storage medium for virtual image tracking control, which obtains the host's facial image and obtains multiple facial features contained in the facial image, wherein for each facial feature, a plurality of preset corresponding expression base templates can be obtained, and then the template coefficient of the facial feature relative to each expression base template is determined based on the facial feature and the multiple expression base templates. Based on each expression base template and its corresponding template coefficient, the state of the virtual features corresponding to the facial features in the constructed virtual image is controlled by tracking. This solution, by embodying the host's facial features with an expression base template carrying a template coefficient, and then controlling the virtual image, can achieve tracking control of the virtual image without the need for other auxiliary equipment, and this solution is controlled based on the information of the expression base template, so that the control information is more standardized and the control is more precise. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0043] Figure 1 A schematic diagram of an application scenario of the virtual character follow-up control method provided in an embodiment of the present application;

[0044] Figure 2 A flowchart of a virtual character follow-up control method provided in an embodiment of the present application;

[0045] Figure 3 A schematic diagram of a face template provided in an embodiment of the present application;

[0046] Figure 4 A schematic diagram of facial feature points provided in an embodiment of the present application;

[0047] Figure 5 A schematic diagram of associating feature points between an expression base template and a facial feature in an embodiment of the present application;

[0048] Figure 6 for Figure 2 Flow chart of sub-steps included in step S130;

[0049] Figure 7 for Figure 2 Flow chart of sub-steps included in step S140;

[0050] Figure 8 A schematic diagram of a virtual image provided in an embodiment of the present application;

[0051] Fig. 9 A structural block diagram of an electronic device provided in an embodiment of the present application;

[0052] Fig.10 This is a functional module block diagram of the virtual image follow-up control device provided in an embodiment of the present application.

[0053] Icons: 100 - live broadcast provider; 200 - live broadcast server; 300 - live broadcast receiver; 410 - memory; 420 - processor; 430 - virtual image follow-up control device; 431 - first acquisition module; 432 - second acquisition module; 433 - determination module; 434 - control module. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.

[0055] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for which protection is sought, but merely represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0056] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0057] In the description of the present application, it should be noted that the terms "first", "second", etc., if used, are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0058] It should be noted that, in the absence of conflict, the features in the embodiments of the present application may be combined with each other.

[0059] like Figure 1 As shown, it is a schematic diagram of an application scenario of the virtual image follow-up control method provided in an embodiment of the present application. The application scenario may include a live broadcast providing end 100, a live broadcast receiving end 300 and a live broadcast server 200 which is respectively communicatively connected to the live broadcast providing end 100 and the live broadcast receiving end 300.

[0060] Among them, the live broadcast provider 100 can be a terminal device (such as a mobile phone, tablet computer, computer, etc.) used by the anchor during the live broadcast, and the live broadcast receiver 300 can be a terminal device (such as a mobile phone, tablet computer, computer, etc.) used by the audience when watching the live broadcast.

[0061] In this embodiment, the scene may also include a video acquisition device for acquiring video frames of the host. The video acquisition device may be, but is not limited to, a camera, a lens of a digital camera, a surveillance camera, or a webcam.

[0062] The video acquisition device may be directly installed or integrated in the live broadcast provider 100. For example, the video acquisition device may be a camera configured on the live broadcast provider 100, and other modules or components in the live broadcast provider 100 may receive videos, images, etc. sent from the video acquisition device via an internal bus. Alternatively, the video acquisition device may also be independent of the live broadcast provider 100, and the two communicate via wired or wireless means.

[0063] The live broadcast provider 100 can send the live broadcast video stream to the live broadcast server 200, and the audience can access the live broadcast server 200 through the live broadcast receiver 300 to watch the live broadcast video.

[0064] Combination Figure 2The embodiment of the present application also provides a virtual image follow-up control method applicable to an electronic device, which is used to follow-up control of a virtual image in a live video. The electronic device may be the live broadcast provider 100 or the live broadcast server 200. The method steps defined in the process related to the virtual image follow-up control method may be implemented by the electronic device. Figure 2 The specific process shown is described in detail.

[0065] Step S110, obtaining a facial image of the anchor, and obtaining a plurality of facial features contained in the facial image.

[0066] Step S120, for each of the facial features, obtaining a plurality of preset corresponding expression base templates.

[0067] Step S130: determining a template coefficient of the facial feature relative to each of the expression base templates according to the facial feature and the multiple expression base templates.

[0068] Step S140, based on each of the expression base templates and the corresponding template coefficients, the virtual features corresponding to the facial features in the constructed virtual image are controlled in state tracking.

[0069] When the anchor uses the live broadcast provider 100 to broadcast live, the video capture device can capture the anchor's live video frames. The live broadcast provider 100 can send the captured live video frames to the live broadcast server 200, and push the live broadcast server 200 to send it to the live broadcast receiving terminal 300 held by each audience watching the anchor.

[0070] During the live broadcast of the anchor, the anchor's facial information is the information that mainly reflects the anchor's status. In actual applications, the anchor may need to use a virtual image for various reasons, such as a cartoon character, and control the virtual image to synchronously display or replace the display of his or her own facial status to broadcast live with the help of the virtual image.

[0071] In this process, it is particularly important to make the virtual image accurately synchronize with the real-time status of the anchor.

[0072] In this embodiment, the live broadcast provider 100 can collect real-time video frames of the anchor, and the anchor provider can perform subsequent processing based on the collected video frames, or send the collected video frames to the live broadcast server 200, and the live broadcast server 200 will perform subsequent processing based on the received video frames. This embodiment does not make any specific restrictions on this.

[0073] By performing face recognition processing on the acquired video frame, a face image contained in the video frame is obtained, wherein the process may include identifying a face region in the video frame, extracting the face region to form a face image, and the like.

[0074] The face image contains multiple face features, such as left eye, right eye, nose, mouth, eyebrows, etc. When controlling the virtual image to follow the host's state, the following calculation can be performed for each face feature.

[0075] In this embodiment, the electronic device also pre-stores a plurality of preset face templates. Optionally, the face templates may adopt general open source models. Each face template includes a plurality of expression base templates, for example, Figure 3 As shown in , there are ten different face templates, among which, the expression base template of the first face template includes left eye template, right eye template, mouth template, nose template, etc.

[0076] It should be noted that each face template contains the same type of expression base template, but the corresponding expression base templates in different face templates may be in different states. For example, the left eye templates in some face templates are in an open state, and the left eye templates in some face templates are in a closed state.

[0077] For each facial feature of the host obtained, multiple preset corresponding expression base templates can be obtained. For example, for the left eye feature of the host, the left eye template in each facial template can be obtained. The obtained multiple left eye templates can be in different states, such as open eyes, closed eyes, half-open eyes, etc.

[0078] By analyzing the information of the acquired facial features and the corresponding multiple expression base templates, the template coefficients of the facial features relative to each expression base template can be determined. The value of each template coefficient is between the range (0,1), that is, each template coefficient can represent the probability with which the corresponding expression base template reflects the facial features.

[0079] Through the above calculation, the host's facial features can be transformed into multiple expression base templates, specifically, multiple expression base templates carrying respective template coefficients.

[0080] On this basis, the electronic device also includes pre-built virtual images, such as cartoon images, loli images, hero images, etc. Specifically, corresponding settings can be made according to needs.

[0081] Similarly, the virtual image includes multiple virtual features, and the virtual features may include a left eye, a right eye, a nose, a mouth, etc. The facial features of the host correspond one-to-one to the virtual features of the virtual image. For example, the left eye of the virtual image may be controlled based on the feature information of the host's left eye. Of course, in the actual process, it is generally adopted to synchronously and parallelly analyze and process the various facial features of the host, and finally synchronously control the various virtual features of the virtual image, that is, the full facial features of the virtual image are controlled based on the full facial features of the host. It is just that when the scheme is explained in this embodiment, the processing of each facial feature is explained.

[0082] After obtaining multiple expression base templates corresponding to a certain facial feature of the host and their corresponding template coefficients through the above process, when controlling the virtual image, the state of the corresponding virtual features in the virtual image can be controlled based on the obtained expression base templates and their corresponding template coefficients.

[0083] Since the host's facial features are converted into the corresponding expression base template carrying the template coefficient, the virtual image can be controlled based on the expression base template carrying the template coefficient to achieve the effect of the virtual image following the host's facial state. In addition, since the expression base template is a template with preset information specifications, the control in actual application based on the preset template information can make the control information more standardized and the control more accurate.

[0084] In this embodiment, the anchor's facial features are transformed into information of an expression base template carrying a template coefficient, wherein the template coefficient represents the probability with which the expression base template reflects the facial features, and the accurate determination of the template coefficient is particularly important for whether the expression base template can accurately reflect the facial features. Because, in this embodiment, in order to accurately determine the template coefficient, the following method is used to determine the template coefficient:

[0085] According to the pre-established objective function, facial features and multiple expression base templates, the objective function is optimized until multiple template coefficients included in the objective function are obtained when the preset requirements are met.

[0086] Among them, the template coefficient corresponds one-to-one to the expression base template.

[0087] In this embodiment, an objective function may be pre-constructed, and the objective function includes two components, the first component is the expression base template information and the template coefficient, and the second component is the facial feature information. When the expression base template and the facial features are known, the expression base template and the facial feature information are introduced into the objective function, and the optimization of the objective function is essentially the optimization of the template coefficients contained therein.

[0088] In one implementation, during the optimization process of the objective function, when the preset requirements are met, it can be determined that the template coefficients at this time meet the preset requirements, and the template coefficients can be fixed. The method for determining that the preset requirements are met can be that the number of iterations reaches a preset number, or the function value of the objective function is lower than a preset threshold, etc., which can be specifically set according to needs.

[0089] In this embodiment, the face template pre-constructed in the electronic device is a 3D model. Accordingly, the information of the expression base template in the face template is 3D information, and the face image obtained based on the video frame is a two-dimensional image. Therefore, in order to accurately optimize the objective function based on the expression base template and the face features, it is first necessary to keep the 3D model information consistent with the two-dimensional information of the face features. Therefore, in this embodiment, the process of optimizing the objective function can be implemented in the following manner:

[0090] Each expression base template is projected and transformed, and the objective function is optimized according to the pre-established objective function, the facial features and the multiple expression base templates after the projection transformation.

[0091] In this embodiment, it is considered that in actual application, the host's face may not be facing the video acquisition device in many cases during live broadcast, that is, the face area in the obtained host's face image may not be a frontal face image. The various face templates pre-established in the electronic device are generally established based on the frontal face image, that is, the various expression base templates are also features under the perspective of facing the face.

[0092] Therefore, when the expression base template is projected and transformed, in addition to projecting the expression base template from the 3D model to the two-dimensional coordinate system to maintain the consistency of the coordinate system information, in order to align the expression base template with the facial features of the host as much as possible, in this embodiment, after the expression base template is projected, the projected expression base template can be offset, scaled, etc., so as to minimize the gap between the projected expression base template and the facial features.

[0093] In this embodiment, the host's facial image contains multiple facial features, and each facial feature contains multiple facial feature points. For example, for the host's left eye, the left eye can be composed of multiple eye feature points, such as multiple feature points that constitute the eye socket contour, the center point that locates the eyeball position, etc.

[0094] Similarly, each face template contains multiple expression base templates, and each expression base template contains multiple expression feature points. Similarly, for the left eye template in the expression base template, each left eye template is composed of multiple feature points. In addition, in order to facilitate the management of expression base templates of the same type (such as left eye) but in different states (such as open eyes, closed eyes, half-open eyes, etc.), the feature points of the same type of expression base templates are encoded in the same encoding method. For example, the corresponding expression feature points in each expression base template are marked with the same serial number. That is, the points, edges, and point connection methods that represent the feature points in each expression base template are consistent, and points with a certain serial number represent the same semantics. For example, the Nth point in each expression base template is fixedly represented as the eye corner point.

[0095] Therefore, in the process of projecting each expression base template onto the facial features, first, for each facial feature of the anchor, multiple facial feature points constituting the facial features can be extracted. The multiple facial feature points are 2D feature points, for example Figure 4 It should be noted that the identification method of the facial feature points can be consistent with the identification method of the template feature points in the expression base template.

[0096] On this basis, for each facial feature point in the facial features, the expression base feature point corresponding to the facial feature point in each expression base template can be determined, that is, the facial feature point is bound to the corresponding expression base feature point one by one. Figure 5 As shown in , the left side shows the facial feature points in the facial features, and the right side shows the 3D template feature points in the face template.

[0097] Then, projection transformation is performed on each expression base template according to the coordinate values ​​of the expression feature points in each expression base template and the coordinate values ​​of the corresponding facial feature points.

[0098] That is, first extract each facial feature point in the facial features, then bind the facial feature points with the corresponding template feature points, and implement the projection transformation of the expression base template based on the binding relationship and the coordinate values ​​of the template feature points and the coordinate values ​​of the facial feature points. In this way, the projection transformation is implemented based on the coordinate values ​​of the corresponding feature points, which can minimize the gap between the expression base template and the facial features after the projection transformation.

[0099] In this embodiment, in the process of projecting the expression base template, corresponding transformation coefficients need to be adopted for each expression base template for transformation. The transformation results are also different under different transformation coefficients, and the transformation results will further affect the optimization process of the objective function, that is, the projection transformation of the expression base template and the optimization of the objective function are mutually influential. Based on this consideration, in this embodiment, the respective optimization results of the projection transformation and the objective function are used to further optimize each other, so that the preset requirements are finally met.

[0100] In detail, in this embodiment, when performing the optimization processing of the objective function, first, for the projection transformation of the expression base template, the preset initial transformation coefficients can be used to perform an initial projection transformation on each expression base template, and based on the pre-established objective function, facial features and multiple expression base templates after the initial projection transformation, multiple template coefficients included in the objective function are obtained.

[0101] In this embodiment, when performing the projection transformation of the expression base template for the first time, for example, for a plurality of expression base templates in the standard state, the closed eye state, the blinking state, etc., the initial transformation coefficients may be a coefficient group such as (1, 0, 0, ...). The initial transformation coefficients are used to perform the first projection transformation.

[0102] After the projection transformation, the projection transformed expression base template can be brought into the objective function to calculate the template coefficient of this round.

[0103] On the basis of calculating the template coefficients, the obtained template coefficients are used as the updated transformation coefficients of each expression base template, and the updated transformation coefficients are used to perform re-projection transformation on each expression base template, and the polarity objective functions of multiple expression base templates after re-projection transformation are optimized according to the objective function, facial features and the optimization process until the preset requirements are met.

[0104] That is, the expression base template is projected and transformed using the template coefficients calculated in the previous round, and then the expression base template after projection transformation is brought into the target function to calculate the template coefficients of the new round. In this way, after multiple rounds of iterative calculations, when the number of iterations reaches a preset number, or the function value of the target function is lower than a preset threshold, it can be determined that the preset requirements are met, and the template coefficients in the target function at this time can be obtained as the final template coefficients.

[0105] In this embodiment, the obtained template coefficients can be further optimized by optimizing the above-mentioned projection transformation and the objective function under the influence of each other.

[0106] In this embodiment, it can be seen from the above that the objective function is composed of template coefficients, expression base templates, and face features, and the face features include multiple face feature points, and each expression base template includes multiple template feature points. Figure 6 , the objective function is optimized to determine the template coefficients as follows:

[0107] Step S131, obtaining a first array constructed by the plurality of facial feature points, and obtaining a second array constructed by the plurality of expression feature points included in each of the expression base templates.

[0108] Step S132, based on a pre-established objective function, weightedly add the plurality of second arrays according to the corresponding template coefficients and then subtract the first array, and determine each template coefficient when the objective function meets the preset requirements.

[0109] In this embodiment, a plurality of facial feature points may constitute a first array. For example, if the number of facial feature points of the extracted facial features is 100, then a two-dimensional 100*2 first array may be obtained by combining the coordinate values ​​of each facial feature point. Accordingly, since the definition of expression feature points is similar to that of facial feature points, each expression base template may correspond to a two-dimensional 100*2 second array.

[0110] As an implementation, the pre-built objective function can be as follows:

[0111] f=||P(V)-L|| 2

[0112] Where V = template coefficient 1*expression base template 1+template coefficient 2*expression base template 2+...

[0113] In the above formula, P(V) represents the result of weighted accumulation of each expression base template (each second array) after projection transformation according to the template coefficient, and L represents the first array composed of facial feature points. Each template coefficient is determined by fitting the objective function.

[0114] After the facial features are reflected in the expression base template in the above manner, the constructed virtual image can be further controlled based on the determined expression base template and template coefficient. Figure 7 , this step can be achieved by:

[0115] Step S141, performing weighted accumulation on the multiple expression base templates according to corresponding template coefficients to obtain control information.

[0116] Step S142, performing state tracking control on the virtual features corresponding to the facial features in the constructed virtual image according to the control information.

[0117] In this embodiment, the multiple expression base templates obtained above can be weighted and accumulated according to the corresponding template coefficients as control information. For example, the control information can be the accumulation result calculated according to the following formula:

[0118] Template coefficient 1*expression base template 1+template coefficient 2*expression base template 2+...

[0119] It can be seen from the above that each expression base template includes multiple template feature points, and the virtual features in the virtual image also include multiple virtual feature points. The control of the virtual image is actually the control of each virtual feature point.

[0120] That is, the obtained control information includes multiple coordinate information, each of which is a weighted result of the coordinates of the corresponding expression feature points included in multiple expression base templates. For example, if the left eye in the facial feature is followed, the obtained coordinate information is the coordinate information of the corresponding feature points in multiple left eye templates.

[0121] When the virtual image is controlled, the virtual features corresponding to the facial features in the constructed virtual image can be obtained, and multiple virtual feature points contained in the virtual features can be obtained, for example, the left eye feature of the virtual image, and multiple feature points contained in the left eye of the virtual image.

[0122] Then, each virtual feature point in the virtual feature is controlled according to the corresponding coordinate information in the control information to drive the virtual image to follow the state. That is, the coordinate information of the corresponding template feature point in the expression base template weighted by the template coefficient is used to control the coordinates of the corresponding virtual feature point in the virtual feature.

[0123] For example, Figure 8 As shown in , when the host's mouth is in a certain state, the state may be jointly reflected by multiple expression base templates (mouth templates) according to different template coefficients, for example, by the template of the open mouth state and the template of the pouting state. The state can be represented by 0.2*open mouth expression base template+0.5*pouting expression base template. Among them, 0.2 is the template coefficient of the expression base template of the open mouth state, and 0.5 is the template coefficient of the expression base template of the pouting state.

[0124] Based on the above results, the virtual image can be controlled to follow the host's mouth state accordingly to achieve mouth state tracking.

[0125] Through the above method, the host's facial features can be converted into an expression base template carrying a template coefficient, and then the virtual image can be controlled based on the expression base template and weighted by the template coefficient, so as to achieve the purpose of state tracking control of the virtual image following the host's facial features.

[0126] In one implementation, if the electronic device is the above-mentioned live broadcast provider 100, the live broadcast provider 100 can generate a corresponding live video stream and send it to the live broadcast server 200 after collecting the video frame of the anchor and controlling the virtual image accordingly, and the live broadcast server 200 pushes it to each live broadcast receiving terminal 300 for the audience to watch.

[0127] In another implementation, if the electronic device is the above-mentioned live broadcast server 200, the live broadcast server 200 can generate a push stream containing the video frames of the anchor and the virtual image after realizing the follow-up control of the virtual image, and push it to each live broadcast receiving terminal 300 for the audience to watch.

[0128] The virtual image follow-up control method provided in the embodiment of the present application can convert the host's facial features into a preset standard expression base template, and then perform follow-up control on the virtual image based on the corresponding expression base template. Without the need for auxiliary equipment, the purpose of following the host's state for the virtual image to perform follow-up control can be successfully achieved, and the conversion to the preset standard expression base template for control can make the control information more standardized and the control more accurate.

[0129] See also Fig. 9 The embodiment of the present application also provides an electronic device. The electronic device can be used as a live broadcast device, for example, it can be a terminal device used by the anchor during live broadcast (such as the above-mentioned live broadcast provider 100), or it can be a server (such as the above-mentioned live broadcast server 200) that is connected to the terminal device used by the anchor during live broadcast.

[0130] In detail, the electronic device may include a memory 410, a processor 420 and a virtual image following control device 430. The memory 410 and the processor 420 are electrically connected directly or indirectly to realize data transmission or interaction. For example, they may be electrically connected to each other through one or more communication buses or signal lines. The virtual image following control device 430 includes at least one software function module that can be stored in the memory 410 in the form of software or firmware. The processor 420 is used to execute an executable computer program stored in the memory 410, for example, the software function module and computer program included in the virtual image following control device 430, so as to realize the virtual image following control method, thereby ensuring that the virtual image is controlled to follow the host state for live broadcast display, and ensuring that the following effect of the virtual image is more standardized and accurate.

[0131] The memory 410 may be, but is not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable read-only memory (EEPROM), etc. The memory 410 is used to store a program, and the processor 420 executes the program after receiving an execution instruction.

[0132] The processor 420 may be an integrated circuit chip with signal processing capability, for example, a central processing unit (CPU), a network processor (NP), a system on chip (SoC), a digital signal processor (DSP), etc., to implement or execute the methods and steps disclosed in the embodiments of the present application.

[0133] Understandably, Fig. 9 The structure shown is for illustration only, and the electronic device may also include Fig. 9 More or fewer components as shown, or with Fig. 9 The different configurations shown, for example, may also include a communication unit for exchanging information with other live broadcast devices. Fig. 9 Each component shown in the figure can be implemented by hardware, software or a combination thereof.

[0134] See also Fig.10 , is a functional module block diagram of a virtual image following control device 430 provided in an embodiment of the present application. The virtual image following control device 430 includes a first obtaining module 431, a second obtaining module 432, a determining module 433 and a controlling module 434.

[0135] The first acquisition module 431 is used to obtain the host's facial image and obtain multiple facial features contained in the facial image.

[0136] In this embodiment, the first obtaining module 431 may be used to execute Figure 2 As shown in step S110, the relevant contents of the first obtaining module 431 can refer to the above description of step S110.

[0137] The second obtaining module 432 is used to obtain a corresponding plurality of expression base templates for each of the facial features.

[0138] In this embodiment, the second obtaining module 432 may be used to execute Figure 2 As shown in step S120, the relevant contents of the second obtaining module 432 can refer to the above description of step S120.

[0139] The determination module 433 is used to determine the template coefficient of the facial feature relative to each of the expression base templates according to the facial feature and the multiple expression base templates.

[0140] In this embodiment, the determination module 433 may be used to execute Figure 2 As shown in step S130, the relevant contents of the determination module 433 can refer to the above description of step S130.

[0141] The control module 434 is used to perform state tracking control on the virtual features corresponding to the facial features in the constructed virtual image based on the expression base templates and their corresponding template coefficients.

[0142] In this embodiment, the control module 434 can be used to execute Figure 2 As shown in step S140, the relevant contents of the control module 434 can refer to the above description of step S140.

[0143] In a possible implementation, the determination module 433 may be specifically used to:

[0144] According to the pre-established objective function, the facial features and the multiple expression base templates, the objective function is optimized until the multiple template coefficients contained in the objective function are obtained when the preset requirements are met, wherein the template coefficients correspond one-to-one to the expression base templates.

[0145] In a possible implementation, the determination module 433 may be used to perform optimization processing in the following manner:

[0146] Each of the expression base templates is projected and transformed, and the objective function is optimized according to a pre-established objective function, the facial features and the multiple expression base templates after the projection transformation.

[0147] In a possible implementation, the determination module 433 may be specifically used to:

[0148] Performing a primary projection transformation on each of the expression base templates using a preset initial transformation coefficient, and obtaining a plurality of template coefficients included in the objective function according to a pre-established objective function, the facial features, and the plurality of expression base templates after the primary projection transformation;

[0149] The obtained template coefficients are used as the updated transformation coefficients of each expression base template, and the updated transformation coefficients are used to re-project the expression base templates. The objective function is optimized according to the objective function, the facial features and the multiple expression base templates after the re-projection transformation until the preset requirements are met.

[0150] In a possible implementation, the facial features include multiple facial feature points, each of the expression base templates includes multiple expression feature points, and the determination module 433 can be used to implement the projection transformation in the following manner:

[0151] For each of the facial feature points in the facial features, determining an expression feature point in each of the expression base templates corresponding to the facial feature point;

[0152] Projection transformation is performed on each of the expression base templates according to the coordinate values ​​of the expression feature points in each of the expression base templates and the coordinate values ​​of the corresponding facial feature points.

[0153] In a possible implementation, the determination module 433 may be specifically used to:

[0154] Obtaining a first array constructed by the plurality of facial feature points, and obtaining a second array constructed by the plurality of expression feature points included in each of the expression base templates;

[0155] Based on a pre-established objective function, a plurality of second arrays are weightedly accumulated according to corresponding template coefficients and then the first array is subtracted, and when the objective function meets preset requirements, each template coefficient is determined.

[0156] In a possible implementation, the control module 434 may be specifically used to:

[0157] Performing weighted accumulation of the multiple expression base templates according to corresponding template coefficients to obtain control information;

[0158] The state of the virtual features corresponding to the facial features in the constructed virtual image is controlled according to the control information.

[0159] In a possible implementation, the control information includes a plurality of coordinate information, each of which is a weighted result of coordinates of corresponding expression feature points included in a plurality of expression base templates. The control module 434 may be specifically used for:

[0160] Obtaining a virtual feature corresponding to the facial feature in the constructed virtual image, and obtaining a plurality of virtual feature points included in the virtual feature;

[0161] Each virtual feature point in the virtual feature is controlled according to the corresponding coordinate information in the control information to drive the virtual image to follow the state.

[0162] For descriptions of the processing flow of each module in the device and the interaction flow between each module, reference may be made to the relevant descriptions in the above method embodiment, which will not be described in detail here.

[0163] In an embodiment of the present application, corresponding to the above-mentioned virtual image following control method, a computer-readable storage medium is also provided, and a computer program is stored in the computer-readable storage medium. When the computer program is run, each step of the above-mentioned virtual image following control method is executed.

[0164] Among them, the steps executed when the aforementioned computer program is running will not be repeated here one by one, and reference may be made to the explanation of the virtual image follow-up control method in the previous text.

[0165] In summary, the virtual image tracking control method, device, electronic device and readable storage medium provided in the embodiments of the present application obtain the host's facial image and obtain multiple facial features contained in the facial image, wherein for each facial feature, a preset multiple expression base templates corresponding thereto can be obtained, and then the template coefficient of the facial feature relative to each expression base template is determined based on the facial feature and the multiple expression base templates. Based on each expression base template and its corresponding template coefficient, the state of the virtual features corresponding to the facial features in the constructed virtual image is controlled by tracking. This scheme, by embodying the host's facial features with an expression base template carrying a template coefficient, and then controlling the virtual image, can achieve tracking control of the virtual image without the need for other auxiliary equipment, and this scheme is controlled based on the information of the expression base template, so that the control information is more standardized and the control is more precise.

[0166] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A method for controlling a virtual image, characterized in that: The method comprises: Obtaining a facial image of the anchor, and obtaining multiple facial features contained in the facial image; For each of the facial features, obtaining a plurality of preset corresponding expression base templates; Determining, according to the facial feature and the plurality of expression base templates, a template coefficient of the facial feature relative to each of the expression base templates; Based on the expression base templates and the corresponding template coefficients, the virtual features corresponding to the facial features in the constructed virtual image are controlled in a state-following manner; Wherein, the step of determining the template coefficient of the facial feature relative to each of the expression base templates comprises: Using preset initial transformation coefficients, each of the expression base templates is subjected to an initial projection transformation, and based on a pre-established objective function, the facial features, and the multiple expression base templates after the initial projection transformation, a plurality of template coefficients included in the objective function are obtained; using the obtained template coefficients as the updated transformation coefficients of each of the expression base templates, using the updated transformation coefficients to perform a re-projection transformation on each of the expression base templates, and optimizing the objective function based on the objective function, the facial features, and the multiple expression base templates after the re-projection transformation until the preset requirements are met.

2. The method for controlling the virtual image according to claim 1, characterized in that: The facial features include a plurality of facial feature points, and each of the expression base templates includes a plurality of expression feature points; The step of projecting each of the expression base templates comprises: For each of the facial feature points in the facial features, determining an expression feature point in each of the expression base templates corresponding to the facial feature point; Projection transformation is performed on each of the expression base templates according to the coordinate values ​​of the expression feature points in each of the expression base templates and the coordinate values ​​of the corresponding facial feature points.

3. The method for controlling the virtual image according to claim 1, characterized in that: The facial features include a plurality of facial feature points, and each of the expression base templates includes a plurality of expression feature points; The step of optimizing the objective function according to the pre-established objective function, the facial features and the multiple expression base templates after projection transformation comprises: Obtaining a first array constructed by the plurality of facial feature points, and obtaining a second array constructed by the plurality of expression feature points included in each of the expression base templates; Based on a pre-established objective function, a plurality of second arrays are weightedly accumulated according to corresponding template coefficients and then the first array is subtracted, and when the objective function meets preset requirements, each template coefficient is determined.

4. The method for controlling the virtual image according to claim 1, characterized in that: The step of performing state following control on the virtual features corresponding to the facial features in the constructed virtual image based on the expression base templates and the template coefficients corresponding to each of them comprises: Performing weighted accumulation of the multiple expression base templates according to corresponding template coefficients to obtain control information; The state of the virtual features corresponding to the facial features in the constructed virtual image is controlled according to the control information.

5. The method for controlling the virtual image according to claim 4, characterized in that: The control information includes a plurality of coordinate information, each of which is a weighted result of the coordinates of a corresponding expression feature point included in a plurality of expression base templates; The step of performing state tracking control on the virtual features corresponding to the facial features in the constructed virtual image according to the control information comprises: Obtaining a virtual feature corresponding to the facial feature in the constructed virtual image, and obtaining a plurality of virtual feature points included in the virtual feature; Each virtual feature point in the virtual feature is controlled according to the corresponding coordinate information in the control information to drive the virtual image to follow the state.

6. A virtual image follow-up control device, characterized in that: The device is used to implement the virtual image follow-up control method according to any one of claims 1 to 5, comprising: The first acquisition module is used to obtain the host's face image and obtain multiple face features contained in the face image; A second acquisition module is used to obtain a plurality of corresponding expression base templates for each of the facial features; A determination module, configured to determine a template coefficient of the facial feature relative to each of the expression base templates based on the facial feature and the multiple expression base templates; The control module is used to perform state tracking control on the virtual features corresponding to the facial features in the constructed virtual image based on the expression base templates and the corresponding template coefficients.

7. An electronic device, characterized in that: It includes one or more storage media and one or more processors communicating with the storage media, the one or more storage media store machine executable instructions executable by the processor, and when the electronic device is running, the processor executes the machine executable instructions to execute the virtual image follow-up control method described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores machine-executable instructions, and when the machine-executable instructions are executed, the virtual image follow-up control method described in any one of claims 1-5 is implemented.

Images