Eye drive control method and device, storage medium and electronic device

Abstract

The present invention discloses an eye drive control method and device, a storage medium and an electronic device. The method comprises: displaying a target picture pre-configured for an anchor object in the current live broadcast screen, wherein the target picture displays the face of a virtual animal object; based on the collected face image of the anchor object, identifying the monocular closure feature of the anchor object, wherein the monocular closure feature is calculated based on the position offset of multiple closed point groups collected on a single eye of the anchor object, and a single closed point group includes an upper eyelid point and a lower eyelid point intersecting with the same straight line; according to the degree of eye closure indicated by the monocular closure feature, driving and controlling the eyes of the virtual animal object in the target picture to perform synchronous closure adjustment. The present invention solves the technical problem that the existing technology cannot guarantee the accuracy of eye drive control.

Description

Technical Field

[0001] The present invention relates to the field of computers, and in particular to an eye drive control method and device, a storage medium and an electronic device. Background Art

[0002] In order to enrich the live broadcast screen, developers of live broadcast applications often set up various live broadcast special effects. For example, based on a three-dimensional (3D) face model, a virtual portrait that matches the host's face is created, and the virtual portrait replaces the host's real head portrait. Or, a virtual animal image displayed in a picture is used to replace the host's real head portrait. The above methods are used to hide the host's real portrait information and protect the host's privacy.

[0003] During the live broadcast process, in order to ensure that the facial expressions of the virtual human or animal image are synchronized with those of the host, the facial expressions of the virtual human or animal image are usually driven by the host's actual facial expressions.

[0004] Among them, the eyes are among the multiple facial key points used to determine facial expressions. However, since the eyes are very sensitive and the eye muscles often shake in a small range, it is easy to lose eye detection temporarily due to frequent changes, resulting in the problem of being unable to guarantee the accuracy of eye drive control.

[0005] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0006] The embodiments of the present invention provide an eye drive control method and device, a storage medium and an electronic device, so as to at least solve the technical problem that the existing technology cannot guarantee the accuracy of eye drive control.

[0007] According to one aspect of an embodiment of the present invention, an eye drive control method is provided, comprising: displaying a target picture pre-configured for an anchor object in a current live broadcast screen, wherein the target picture displays the face of a virtual animal object; based on a collected face image of the anchor object, identifying a monocular closure feature of the anchor object, wherein the monocular closure feature is calculated based on a position offset of a plurality of closure point groups collected on a single eye of the anchor object, wherein the single closure point group includes an upper eyelid point and a lower eyelid point intersecting with the same straight line; and according to the degree of eye closure indicated by the monocular closure feature, driving and controlling the eyes of the virtual animal object in the target picture to perform synchronous closure adjustment.

[0008] According to another aspect of an embodiment of the present invention, an eye drive control device is also provided, including: a display unit, used to display a target picture pre-configured for an anchor object in a current live broadcast screen, wherein the target picture displays the face of a virtual animal object; an identification unit, used to identify a monocular closure feature of the anchor object based on a collected facial image of the anchor object, wherein the monocular closure feature is calculated based on the position offset of multiple closure point groups collected on a single eye of the anchor object, and the single closure point group includes an upper eyelid point and a lower eyelid point intersecting with the same straight line; a drive control unit, used to drive and control the eyes of the virtual animal object in the target picture to perform synchronous closure adjustment according to the degree of eye closure indicated by the monocular closure feature.

[0009] According to another aspect of an embodiment of the present invention, there is further provided a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the above-mentioned eye drive control method when running.

[0010] According to another aspect of an embodiment of the present invention, there is provided an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the eye drive control method through the computer program.

[0011] In an embodiment of the present invention, a target image pre-configured for an anchor object is displayed in the current live screen, and based on the collected facial image of the anchor object, a monocular closure feature of the anchor object is identified, wherein the monocular closure feature is calculated based on the position offset of multiple closed point groups collected on a single eye of the anchor object, and the single closed point group includes an upper eyelid point and a lower eyelid point intersecting with the same straight line, and then according to the degree of eye closure indicated by the monocular closure feature, the eyes of the virtual animal object in the target image are driven and controlled to perform synchronous closure adjustment, thereby avoiding the problem of eye detection loss caused by a small range of eye muscle shaking, thereby improving the accuracy of eye drive control of the animal image displayed in the live screen, and further solving the technical problem that the existing technology cannot guarantee the accuracy of eye drive control. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0013] Figure 1 is a schematic diagram of an application environment of an optional eye drive control method according to an embodiment of the present invention;

[0014] Figure 2is a flow chart of an optional eye drive control method according to an embodiment of the present invention;

[0015] Figure 3 is a schematic diagram of an optional eye drive control method according to an embodiment of the present invention;

[0016] Figure 4 is a schematic diagram of another optional eye drive control method according to an embodiment of the present invention;

[0017] Figure 5 is a schematic diagram of another optional eye drive control method according to an embodiment of the present invention;

[0018] Figure 6 is a schematic diagram of another optional eye drive control method according to an embodiment of the present invention;

[0019] Figure 7 is a flow chart of an optional eye drive control method according to an embodiment of the present invention;

[0020] Figure 8 is a schematic structural diagram of an optional eye drive control device according to an embodiment of the present invention;

[0021] Fig. 9 It is a schematic diagram of the structure of an optional electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0022] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0023] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0024] According to one aspect of an embodiment of the present invention, an eye drive control method is provided. Optionally, as an optional implementation, the eye drive control method can be but is not limited to being applied to: Figure 1 The eye drive control system in the hardware environment shown in FIG. The eye drive control system may include but is not limited to a terminal device 102, a network 104, a server 106, a database 108, and a terminal device 109. The terminal device 102 runs a target client (such as Figure 1 The target client may be a live broadcast interface of a live broadcast platform). The terminal device 102 includes a human-computer interaction screen, a processor and a memory. The human-computer interaction screen is used to display the live broadcast interface of the anchor client (such as Figure 1 The terminal device 109 also includes a human-computer interaction screen, a processor and a memory, wherein the human-computer interaction screen is used to display the live broadcast interface of the audience client.

[0025] The specific process is as follows: as in step S102, a target picture pre-configured for the anchor object is displayed in the current live screen in the client running in the terminal device 102, wherein the face of the virtual animal object is displayed in the target picture; S104, the face image of the anchor object is collected; S106, the terminal device 102 sends the face image of the anchor object to the server 106 through the network 104. The server 106 will execute steps S108-S112, based on the collected face image of the anchor object, identify the monocular closure feature of the anchor object, and according to the degree of eye closure indicated by the monocular closure feature, drive and control the eyes of the virtual animal object in the target picture to synchronize the closure adjustment, and encapsulate the adjusted target picture as the live streaming data to be pushed. Finally, as in step S114, the server 106 pushes the live streaming data to the terminal device 102 and the terminal device 109 through the network 104.

[0026] As another optional implementation, when the terminal device 102 has a relatively strong computing and processing capability, the above steps S108-S112 may also be completed by the terminal device 102. This is an example, and this embodiment does not impose any limitation on this.

[0027] Optionally, in this embodiment, the terminal device may be a terminal device configured with a target client, which may include but is not limited to at least one of the following: a mobile phone (such as an Android phone, an iOS phone, etc.), a laptop, a tablet computer, a PDA, a MID (Mobile Internet Devices), a PAD, a desktop computer, a smart TV, etc. The target client may be a video client, an instant messaging client, a browser client, an education client, etc. The network may include but is not limited to: a wired network, a wireless network, wherein the wired network includes: a local area network, a metropolitan area network and a wide area network, and the wireless network includes: Bluetooth, WIFI and other networks that realize wireless communication. The server may be a single server, or a server cluster consisting of multiple servers, or a cloud server. The above is only an example, and this embodiment does not impose any limitation on this.

[0028] As an optional implementation, Figure 2 As shown, the above-mentioned eye drive control method includes:

[0029] S202, displaying a target picture pre-configured for the anchor object in the current live broadcast screen, wherein the target picture displays the face of a virtual animal object;

[0030] S204, based on the collected face image of the anchor object, identifying a monocular closure feature of the anchor object, wherein the monocular closure feature is calculated based on the position offset of multiple closure point groups collected on a single eye of the anchor object, and the single closure point group includes an upper eyelid point and a lower eyelid point intersecting with the same straight line;

[0031] S206: According to the eye closure degree indicated by the single eye closure feature, the eyes of the virtual animal object in the target image are driven and controlled to be synchronously closed and adjusted.

[0032] As Figure 3 Taking the interface shown as an example, it can be understood that the above live broadcast screen is the interface displayed when the anchor client applied to the terminal performs live broadcast. The interface can be divided into four display areas, including live broadcast area 301: used to display the screen broadcast by the anchor user to the audience user during the live broadcast; interactive area 302: used to display the interactive information of the audience user; setting area 303: used to display the controls for setting the live broadcast parameters; search area 304: used to display the controls for obtaining live broadcast pictures. Among them, as shown in the live broadcast area 301, the target picture pre-configured for the anchor object is displayed in this area, and the face of the virtual animal object is displayed in the target picture.

[0033] Optionally, the above-mentioned method of collecting the face image of the anchor object can be as follows: Figure 1The terminal device 102 shown in the figure obtains the facial image of the host object through the device's own shooting device, or the host object uses a third-party device to obtain the facial image of the host object during the live broadcast through the terminal device 102, and then imports the above-obtained facial image into the terminal device 102 for live broadcast, which is not limited here.

[0034] The above-mentioned single eye closure feature is further explained. It can be understood that the single eye closure feature indicates the closure degree of a single eye in the face image, and the eye of the corresponding virtual animal object is driven and controlled to be closed and adjusted according to the eye closure degree indicated by the single eye closure feature.

[0035] In the above method, after obtaining the face image of the anchor object, the eye image in the face image of the anchor object is first identified. As an optional method, the eye image can be cut out separately for more accurate processing. In another optional method, it can also be directly processed in the face image together with other facial feature extraction operations.

[0036] It is understandable that, since human eye muscles are relatively sensitive, small-scale shaking is prone to occur. In order to more accurately detect such eye changes, the single eye closure features are extracted for each eye to quantify the eye change features. Figure 4 As shown, in the individually captured human eye image, the right eye is taken as an example as the object of monocular closure feature extraction. As shown in the three groups of closed point groups, a single closed point group includes upper eyelid points and lower eyelid points intersecting the same straight line. That is, upper eyelid point 401 and lower eyelid point 402 are the first closed point group, upper eyelid point 403 and lower eyelid point 404 are the second closed point group, and upper eyelid point 405 and lower eyelid point 406 are the third closed point group. The three closed point groups shown in the figure are only examples. In actual applications, more closed point groups can be used according to the accuracy requirements to obtain more accurate monocular closure features.

[0037] In an embodiment of the present invention, a target image pre-configured for an anchor object is displayed in the current live screen, and based on the collected facial image of the anchor object, a monocular closure feature of the anchor object is identified, wherein the monocular closure feature is calculated based on the position offset of multiple closed point groups collected on a single eye of the anchor object, and the single closed point group includes an upper eyelid point and a lower eyelid point intersecting with the same straight line, and then according to the degree of eye closure indicated by the monocular closure feature, the eyes of the virtual animal object in the target image are driven and controlled to perform synchronous closure adjustment. That is to say, in this embodiment, the degree of eye closure is indirectly indicated according to the monocular closure feature, and then the eyes of the virtual animal are synchronously closed and adjusted, thereby avoiding direct feedback adjustment of the eyes of the virtual animal object with the actually obtained eye image information, thereby solving the technical problem that the abnormal tremor of the eyes of the virtual animal object caused by the tremor of the eye muscles of the real user is solved, and achieving the technical effect of improving the accuracy of the eye drive control of the animal image displayed in the live screen.

[0038] As an optional implementation manner, based on the collected facial image of the anchor object, identifying the monocular closure feature of the anchor object includes:

[0039] S1, identifying an eye key point sequence of the anchor object from the face image, wherein the eye key point sequence includes position information of each eyelid point in the plurality of closed point groups;

[0040] S2, calculating the monocular closure coefficient of the anchor object according to the eye key point sequence;

[0041] S3, obtaining the pupil distance of the anchor object;

[0042] S4, determining the monocular closure feature according to the ratio between the monocular closure degree coefficient and the pupil distance.

[0043] by Figure 5 The above embodiment is described by taking the eye image of the anchor shown in FIG. Figure 5 Taking the right eye shown in FIG. 1 as an example, the pupil distance is the distance L between the left pupil 501 and the right pupil 502 in the eye image. In another optional embodiment, the pupil distance may also be based on the distance between the two inner eye corners in the eye image.

[0044] Continue with Figure 5 Take this as an example to illustrate: Figure 53 sets of key point sequences on the right eye image are shown, namely, upper eyelid point 401 and lower eyelid point 402, upper eyelid point 403 and lower eyelid point 404, upper eyelid point 405 and lower eyelid point 406. Any set of key point sequences are upper eyelid points and lower eyelid points intersecting with the same straight line. Optionally, the straight line where any set of key point sequences is located is perpendicular to the straight line where the left eye pupil 501 and the right eye pupil 502 are located ( Figure 5 The straight line where the key point sequence is located is not shown in the figure). It can be understood that the number of monocular key point sequences can be determined according to actual needs. Figure 5 The three groups of key point sequences shown in FIG. 1 are only exemplary and do not limit the present embodiment.

[0045] Optionally, after the eye key point sequence is identified, the position information of the key point sequence can be indicated by the pixel position of the key point sequence in the captured eye image. Figure 5 When the eye image size is shown to be 300px (pixel) * 60px, the position of the lower eyelid point 402 in the above eye image can be determined to be (200px, 20px). Similarly, the position of the upper eyelid point 401 can be determined to be (200px, 40px), and further the position information of all the above key points can be determined.

[0046] After calculating the monocular closure coefficient m of the anchor object according to the identified eye key point sequence, combined with the obtained pupil distance L of the anchor object, the monocular closure feature s can be determined according to the ratio of the above two.

[0047] In this embodiment, the eye key point sequence of the above-mentioned host object is identified from the above-mentioned face image, and the monocular closure degree coefficient of the above-mentioned host object is calculated according to the above-mentioned eye key point sequence; then the pupil distance of the above-mentioned host object is obtained; finally, the monocular closure feature is determined according to the ratio between the above-mentioned monocular closure degree coefficient and the above-mentioned pupil distance, thereby determining a specific method for calculating the monocular closure feature, that is, determining the monocular closure feature according to the pupil distance and the monocular closure coefficient, and then using it to control the synchronous closure adjustment of the eyes of the virtual animal object, thereby achieving the technical effect of improving the accuracy of eye drive control.

[0048] As an optional implementation manner, calculating the monocular closure coefficient of the above-mentioned anchor object according to the above-mentioned eye key point sequence includes:

[0049] S1, determining the eye-opening distance and eye-closing offset distance corresponding to each of the above-mentioned closed point groups, wherein the eye-opening distance is the spatial distance between the above-mentioned upper eyelid point and the above-mentioned lower eyelid point in the above-mentioned closed point group when one eye is open, and the eye-closing offset distance is the offset distance between the above-mentioned upper eyelid point and the above-mentioned lower eyelid point in the above-mentioned closed point group when one eye is closed;

[0050] S2, calculating the distance difference corresponding to each of the above-mentioned closed point groups, and obtaining a plurality of distance difference values, wherein the above-mentioned distance difference value is the difference between the above-mentioned open eye distance and the closed eye offset distance of the same closed point group;

[0051] S3, performing weighted sum calculation on the multiple distance differences to obtain the monocular closure coefficient.

[0052] The following combination Figure 6 The above implementation is described in detail. Figure 6 4 shows the open state and closed state of a single eye, and a set of key point sequences of upper eyelid point 401 and lower eyelid point 402 in the single eye schematic diagrams of the open state and the closed state. It can be understood that in the open state of a single eye, as Figure 6 As shown in the left figure, the eye opening distance between the upper eyelid point 401 and the lower eyelid point 402 is a 1 For example, the position of the upper eyelid point 401 is (200px, 40px), and the position of the lower eyelid point 402 is (200px, 20px). 1 That is 20px; Figure 6 The right figure in the figure shows a set of key point sequences in the state of one eye closed. It can be seen from the figure that the upper eyelid point 401 and the lower eyelid point 402 in the state of one eye open fall on the upper eyelid point 601 and the lower eyelid point 602 respectively after the single eye is closed. It can be seen from the figure that in the closed state, there may be a horizontal offset distance Δx i Assuming that the position of the upper eyelid point 601 in the closed state is (197px, 50px), and the position of the lower eyelid point 602 is (202px, 50px), then the above closed eye offset distance Δx i It is understandable that the above-mentioned method for calculating the spatial distance and the offset distance is only an example. In the actual application scenario of the above-mentioned method, the horizontal coordinates of the upper and lower eyelid points may be inconsistent when the single eye is open, and the vertical coordinates of the upper and lower eyelid points may be inconsistent when the single eye is closed (i.e., the single eye is not completely closed). In the above-mentioned cases, the above-mentioned eye-opening distance a is determined by calculating the arithmetic distance according to the position information. 1 And the closed eye offset distance Δx i value.

[0053] According to the above method, the eye opening distance a of the i-th group of closed points can be determined: i And the closed eye offset distance Δx i , and the distance difference between the two a i -Δx i . After calculating the distance difference a corresponding to each closed point group i -Δx i Then, the distance difference of each closed point group is weighted and calculated to obtain the monocular closure coefficient m.

[0054] It is understandable that if Figure 5 Due to the differences in the positions of the three closed point groups shown, it can be considered that the changes in the positions at different positions have different degrees of response to the monocular closure feature. Therefore, the distance difference a of the closed point groups at different positions can be calculated. i -Δx i The weights are assigned so that the distance difference reflected by the distance difference of the closed point group can more accurately characterize the eye closure degree of the anchor object.

[0055] In an optional embodiment, when there are n groups of monocular closure point groups, the distance difference of each group of monocular closure groups is weighted It can be determined that the above monocular closure coefficient m can be obtained by the following method:

[0056] Through this embodiment, by determining the eye-opening distance and eye-closing offset distance corresponding to each of the above-mentioned closing point groups, then calculating the distance difference corresponding to each of the above-mentioned closing point groups, and finally performing weighted summation calculation on the above-mentioned multiple distance differences, the above-mentioned monocular closure coefficient is obtained. In other words, this embodiment provides a method for accurately determining the monocular closure coefficient, that is, simultaneously examining the position change of the closing point group in the open state and the closed state of the monocular eye, and assigning weights to the closing point groups in different positions, so as to more accurately reflect the monocular closure coefficient through the distance difference parameter of the closing point group, thereby achieving the technical effect of improving the accuracy of eye detection.

[0057] As an optional implementation manner, determining the monocular closure feature according to the ratio between the monocular closure degree coefficient and the pupil distance includes:

[0058] S1, determining a compensation coefficient matching the above single eye;

[0059] S2, taking the product of the compensation coefficient and the ratio as the monocular closure feature.

[0060] It can be understood that in this embodiment, a method for accurately determining the monocular closure feature is provided, that is, after determining the ratio between the monocular closure degree coefficient and the pupil distance, a compensation coefficient λ matching the monocular closure degree coefficient is further determined, and then the product of the compensation coefficient and the ratio between the monocular closure degree coefficient and the pupil distance is used as the monocular closure feature. It can be understood that the compensation coefficient λ here can be set according to actual needs, and this application does not impose any limitation on the value of λ.

[0061] Next, combine Figure 5 , Figure 6 The above method for determining the monocular closure feature s is described in detail. Figure 5 In the eye image of the anchor object shown, three groups of closed points are determined, the first group is upper eyelid point 401 and lower eyelid point 402, the second group is upper eyelid point 403 and lower eyelid point 404, and the third group is upper eyelid point 405 and lower eyelid point 406;

[0062] First, according to the determined n groups of monocular closure point groups (3 groups are shown in the figure), the position information of each point is obtained, and according to the position information of each point, the position information of each monocular closure point group is obtained in each group of closure point groups. Figure 6 The eye opening distance a in the open state shown in i And the closed eye offset distance Δx in the closed state i , and the distance difference a of each closed point group is obtained based on the difference between the two i -Δx i ;

[0063] After determining the weight of each closed point group In the case of, the above monocular closure coefficient m is determined as

[0064] Next, according to Figure 5 The pupil distance L shown and the compensation coefficient λ obtained determine the above-mentioned monocular closure characteristics

[0065] Through the above-mentioned embodiments of the present application, by determining multiple groups of closed point groups in a monocular image and simultaneously examining the positional relationship of the multiple groups of closed point groups in the open eye state and the closed eye state, it is possible to more accurately quantify the degree of eye closure of the host object, thereby achieving more precise control and driving of the eyes of the virtual animal object, thereby achieving the technical effect of improving the accuracy of eye driving of the virtual animal object.

[0066] As an optional implementation, according to the eye closure degree indicated by the single eye closure feature, driving and controlling the eyes of the virtual animal object in the target image to perform synchronous closure adjustment includes:

[0067] S1, determining the eye shaking state of the anchor object according to the change of the single eye closure feature of at least three adjacent frames;

[0068] S2, adjusting the eyes of the virtual animal object to close synchronously according to a driving control method that matches the eye shaking state.

[0069] It is understandable that during the live broadcast of the host object, the video data collected usually includes the face of the host object. The face image of the host object corresponding to each frame can be obtained by extracting the video data by frame, and then the monocular closure feature can be determined according to the above method through the face image of each frame.

[0070] According to the above method, when a segment of video data including the face of the host object is obtained, the eye shaking state of the host object is determined based on at least three adjacent frames of facial images. The so-called eye shaking state includes at least two states: the normal eye opening or closing state of the host object, and the eye muscle shaking state with a weak amplitude. In the first state, the eyes of the animal object can be synchronously closed according to the eye closure indicated by the single eye closure feature of the host object; in the second case, since the host object only has a weak muscle shaking state in the eye, if the eye closure adjustment of the virtual animal object is driven directly by the single eye closure feature, the adjustment abnormality and unstable eye closure of the virtual animal object will occur.

[0071] Furthermore, according to the above method of this embodiment, the eye shaking state of the host object is determined according to the changes in the single eye closure characteristics of at least three adjacent frames, and the matching drive control method is adjusted according to the eye shaking state, thereby adjusting the synchronous closure of the eyes of the virtual animal object.

[0072] Through the above-mentioned embodiments of the present application, when the eye shaking state of the host object is determined, the eyes of the virtual animal object are adjusted to be closed synchronously according to the driving control method matching the eye shaking state, thereby achieving the situation that when the eyes of the host object are only in a state of slight muscle shaking, the eyes of the virtual animal object can still be accurately controlled to be closed synchronously, thereby solving the technical problem of unstable eye driving of the virtual animal object.

[0073] As an optional implementation, adjusting the eyes of the virtual animal object to close synchronously according to a driving control method matching the eye shaking state includes:

[0074] S1, when the eye jitter state indication is in a weak jitter, obtaining the eye jitter amplitude value between the current face image frame and the previous two face images before the current face image frame;

[0075] S2, calculating an eye closure adjustment value according to the jitter coefficient matched with the current frame face image and the eye jitter amplitude value;

[0076] S3, adjusting the monocular closure feature according to the eye closure adjustment value, and performing mean filtering on the monocular closure feature to obtain the processed monocular closure feature;

[0077] S4, according to the processed single eye closure feature, synchronously closing the eyes of the virtual animal object in the target image corresponding to the next frame of human face image after the current frame of human face image.

[0078] In an optional embodiment, the eye jitter amplitude value may be represented by the monocular closure coefficient m. In another optional embodiment, the eye jitter amplitude value may also be represented by the monocular closure feature s.

[0079] The eye jitter state of the current frame is determined by comparing the eye jitter amplitude value of the current frame with a predetermined threshold. In an optional embodiment, when the eye jitter amplitude value is within a first predetermined threshold range, the eye jitter of the current frame is determined to be in a weak jitter state; when the eye jitter amplitude value is within a second predetermined threshold range, the eye jitter of the current frame is determined to be in an obvious jitter state.

[0080] The above method is described in detail below. In this method, the above eye jitter amplitude value is characterized by the above monocular closure feature s. By comparing the monocular closure feature s of the current frame determined by the above method with the first threshold range of 0.3 to 0.85, when the monocular closure feature s is within the interval of 0.3 to 0.85, it is determined that the eye jitter of the current frame is in a weak eye muscle jitter state; comparing the monocular closure feature s of the current frame with the second threshold range of 0 to 0.3 and above 0.85, when the monocular closure feature s is within the second threshold range of 0 to 0.3 and above 0.85, it is determined that the eye jitter of the current frame is in an obvious jitter state;

[0081] Furthermore, when it is determined by the above method that the eyes of the current frame are in a weak shaking state, the single eye closure characteristic value indicating the eye shaking amplitude of the current frame is adjusted by the following formula:

[0082]

[0083] It needs further explanation that the above a is the eye opening distance of the i-th frame: a i+1 =(s i+1 -s i ) / A, the numerator is the jitter of the current frame relative to the previous frame, and A is the jitter interval. It can be understood that when 0.3 to 0.85 is the weak jitter state determination interval, A = 0.35. The s obtained by the above adjustment is the eye closure adjustment value;

[0084] Next, after obtaining the eye closure adjustment value, for the consistency of eye driving, the monocular closure feature value s is further subjected to mean filtering to obtain the processed monocular closure feature;

[0085] Finally, the eyes of the virtual animal are synchronously closed according to the adjusted monocular closure characteristic value.

[0086] Optionally, in this embodiment, the obtained anchor object and virtual animal face key points can be used to construct a grid, and the key point data can be used as the texture coordinates for graphics rendering. Then, the offset is calculated using the positions of the key points of the anchor face in the previous and next frames. Then, the key point positions of the pet face are updated, and the updated key point data are used as the vertex coordinates for graphics rendering. Finally, the DirectX or OpenGL API is called to generate the rendered image.

[0087] Through the above-mentioned embodiments of the present application, when it is determined that the eye closure state of the current frame is in a weak jitter, the eye jitter value of the current frame is adjusted according to the eye jitter amplitude values ​​of the previous two frames to obtain the processed monocular closure feature, and at the same time, for the consistency of driving, the eye closure feature of the current frame is processed by mean filtering, thereby achieving adaptive adjustment of the monocular closure feature of the host object when there is a weak jitter in the eyes, thereby avoiding the technical problem of inaccurate eye closure driving of the virtual animal object, and achieving the technical effect of improving the accuracy of eye drive control.

[0088] As an optional implementation manner, adjusting the eyes of the virtual animal object to synchronously close according to the driving control mode matching the eye shaking state includes:

[0089] S1, when the eye shaking state indication is in a non-shaking state, performing mean filtering on the monocular closure feature corresponding to the current frame face image to obtain the processed monocular closure feature

[0090] S2, according to the processed single eye closure feature, synchronously closing the eyes of the virtual animal object in the target image corresponding to the next frame of face image after the current frame of face image

[0091] The above method is described in detail below. The eye jitter amplitude value of the current frame is represented by the single eye closure feature value s. By comparing the single eye closure feature s of the current frame determined by the above method with the first threshold range of 0.3 to 0.85, when the single eye closure feature s is not in the range of 0.3 to 0.85, but in the range of 0 to 0.3 and above 0.85, it is determined that the current eye is in a non-jittering state;

[0092] When it is determined that the current eye is in a non-jittering state, directly continue to perform mean filtering on the above-mentioned monocular closure feature value s, thereby obtaining a processed monocular closure feature;

[0093] Finally, the eyes of the virtual animal are synchronously closed according to the adjusted monocular closure characteristic value.

[0094] Through the above-mentioned embodiments of the present application, when it is determined that the eye closure state of the current frame is non-jittering, the eye closure features of the current frame are directly processed by mean filtering, thereby achieving the technical effect of maintaining the consistency of the eye drive of the virtual animal image when the eyes of the host object are non-jittering.

[0095] The following combination Figure 7 The specific implementation methods of the present application are described.

[0096] like Figure 7 As shown, executing step S702, displaying the target image;

[0097] As Figure 3 Taking the interface shown as an example, it can be understood that the above live broadcast screen is the interface displayed when the anchor client applied to the terminal is broadcasting live. The interface can be divided into four display areas, including live broadcast area 301: used to display the screen broadcast by the anchor user to the audience user during the live broadcast; interactive area 302: used to display the interactive information of the audience user; setting area 303: used to display the controls for setting the live broadcast parameters; search area 304: used to display the controls for obtaining the target picture. Among them, as shown in the live broadcast area 301, the target picture pre-configured for the anchor object is displayed in this area, and the face of the virtual animal object is displayed in the target picture. Method two: obtaining the above target animal picture from the locally stored image set;

[0098] Then, step S704 is executed to collect a face image;

[0099] It is understandable that during the live broadcast of the host object, the video data collected usually includes the face of the host object. The face image of the host object corresponding to each frame can be obtained by extracting the video data by frame.

[0100] In step S706, determining a monocular closure coefficient;

[0101] Then, step S708 is executed to determine the monocular closure feature;

[0102] Combination Figure 5 , Figure 6 The above method for determining the monocular closure feature s is described in detail. Figure 5 In the eye image of the anchor object shown, three groups of closed points are determined, the first group is upper eyelid point 401 and lower eyelid point 402, the second group is upper eyelid point 403 and lower eyelid point 404, and the third group is upper eyelid point 405 and lower eyelid point 406;

[0103] First, according to the determined n groups of monocular closure point groups (3 groups are shown in the figure), the position information of each point is obtained, and according to the position information of each point, the position information of each monocular closure point group is obtained in each group of closure point groups. Figure 6 The eye opening distance a in the open state shown in i And the closed eye offset distance Δx in the closed state i , and the distance difference a of each group of closed points is obtained based on the difference between the two i -Δx i ;

[0104] After determining the weight of each closed point group In the case of, the above monocular closure coefficient m is determined as

[0105] Next, according to Figure 5 The pupil distance L shown and the compensation coefficient λ obtained determine the above-mentioned monocular closure characteristics

[0106] Then, step S710 is executed to detect whether there is shaking. If eye shaking is detected, step S712 is executed. If no eye shaking is detected, step S714 is executed.

[0107] As shown in step S712, when eye shaking is detected, eye closure is adjusted.

[0108] By comparing the monocular closure feature s of the current frame determined by the above method with the first threshold range of 0.3 to 0.85, when the monocular closure feature s is within the interval of 0.3 to 0.85, it is determined that the eye jitter of the current frame is in a weak eye muscle jitter state; comparing the monocular closure feature s of the current frame with the second threshold range of 0 to 0.3 and above 0.85, when the monocular closure feature s is within the second threshold range of 0 to 0.3 and above 0.85, it is determined that the eye jitter of the current frame is in an obvious jitter state;

[0109] Furthermore, when it is determined by the above method that the eyes of the current frame are in a weak shaking state, the monocular closure feature representing the eye shaking amplitude value of the current frame is adjusted by the following formula:

[0110]

[0111] It needs further explanation that the above a is the eye opening distance of the i-th frame: a i+1 =(s i+1 -s i ) / A, the numerator is the jitter of the current frame relative to the previous frame, and A is the jitter interval. It can be understood that when 0.3 to 0.85 is the weak jitter state determination interval, A = 0.35. The s obtained by the above adjustment is the eye closure adjustment value;

[0112] Then execute step S714 to perform mean processing;

[0113] It can be understood that when it is detected that the eyes are in a shaking state, the eyes are first adjusted to be closed and then the average processing is performed. When it is detected that the eyes are not shaking, the eye closure features of the current frame are directly processed by the average.

[0114] Finally, step S716 is executed to render the virtual animal object.

[0115] The above steps are to adjust the virtual animal's eyes synchronously according to the adjusted single eye closure feature value. The obtained anchor object and virtual animal face key points can be used to construct a mesh, and the key point data can be used as the texture coordinates for graphics rendering. Then the offset is calculated using the position of the anchor's face key points in the previous and next frames. Then the key point position of the pet's face is updated, and the updated key point data is used as the vertex coordinates for graphics rendering. Finally, the DirectX or OpenGL API is called to generate the rendered image.

[0116] In an embodiment of the present invention, a target image pre-configured for an anchor object is displayed in the current live screen, and based on the collected facial image of the anchor object, a monocular closure feature of the anchor object is identified, wherein the monocular closure feature is calculated based on the position offset of multiple closed point groups collected on a single eye of the anchor object, and a single closed eyelid includes an upper eyelid point and a lower eyelid point where a straight line of the eyelids intersects, and then according to the degree of eye closure indicated by the monocular closure feature, the eyes of the virtual animal object in the target image are driven and controlled to perform synchronous closure adjustment, thereby avoiding the problem of eye detection loss caused by a small range of eye muscle shaking, thereby improving the accuracy of eye drive control of the animal image displayed in the live screen, and further solving the technical problem that the existing technology cannot guarantee the accuracy of eye drive control.

[0117] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should know that the present invention is not limited by the described action sequence, because according to the present invention, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.

[0118] According to another aspect of the embodiment of the present invention, an eye drive control device for implementing the above eye drive control method is also provided. As shown in the figure, the device includes:

[0119] The display unit 804 is used to display a target picture pre-configured for the anchor object in the current live broadcast screen, wherein the target picture displays the face of the virtual animal object;

[0120] The recognition unit 804 is used to recognize the closed features of a single eye of the anchor object based on the collected face image of the anchor object, wherein the closed features of a single eye are calculated based on the position offset of a plurality of closed point groups collected on a single eye of the anchor object, and a single closed eyelid includes an upper eyelid point and a lower eyelid point where a straight line of eyelids intersects;

[0121] The driving control unit 806 is used to drive and control the eyes of the virtual animal object in the target image to perform synchronous closing adjustment according to the eye closing degree indicated by the single eye closing feature.

[0122] Optionally, in this embodiment, the embodiments to be implemented by the above-mentioned various unit modules can refer to the above-mentioned various method embodiments, which will not be repeated here.

[0123] According to another aspect of the embodiment of the present invention, an electronic device for implementing the above-mentioned eye drive control method is also provided. The electronic device may be Figure 1 The terminal device or server shown in the figure. This embodiment is described by taking the electronic device as a terminal device as an example. Fig. 9 As shown, the electronic device includes a display 901, a transmission device 902, a processor 903 and a memory 904, wherein the memory 904 stores a computer program, and the processor 903 is configured to execute the steps in any of the above method embodiments through the computer program.

[0124] Optionally, in this embodiment, the electronic device may be located in at least one network device among a plurality of network devices of a computer network.

[0125] Optionally, in this embodiment, the processor may be configured to perform the following steps through a computer program:

[0126] S1, displaying a target picture pre-configured for the anchor object in the current live broadcast screen, wherein the target picture displays the face of a virtual animal object;

[0127] S2, based on the collected face image of the anchor object, identifying the monocular closure feature of the anchor object, wherein the monocular closure feature is calculated based on the position offset of multiple closure point groups collected on a single eye of the anchor object, and the single closure point group includes an upper eyelid point and a lower eyelid point intersecting with the same straight line;

[0128] S3, according to the eye closure degree indicated by the single eye closure feature, driving and controlling the eyes of the virtual animal object in the target image to perform synchronous closure adjustment.

[0129] Alternatively, a person skilled in the art may understand that: Fig. 9 The structure shown is for illustration only, and the electronic device may also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, a mobile Internet device (MID), a PAD, and other terminal devices. Fig. 9 The electronic device and the electronic equipment described above are not limited in structure. Fig. 9 More or fewer components (such as network interfaces, etc.) as shown in, or with Fig. 9 Different configurations are shown.

[0130] Among them, the memory 904 can be used to store software programs and modules, such as the program instructions / modules corresponding to the eye drive control method and device in the embodiments of the present invention. The processor 903 executes various functional applications and data processing by running the software programs and modules stored in the memory 904, that is, realizing the above-mentioned eye drive control method. The memory 904 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 904 may further include a memory remotely located relative to the processor 903, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof. As an example, if Fig. 9As shown, the memory 904 may include, but is not limited to, the display unit 802, the identification unit 804, and the drive control unit 806 in the eye drive control device. In addition, it may also include, but is not limited to, other module units in the eye drive control device, which will not be repeated in this example.

[0131] Optionally, the transmission device 902 is used to receive or send data via a network. Specific examples of the network may include wired networks and wireless networks. In one example, the transmission device 902 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices and routers via a network cable so as to communicate with the Internet or a local area network. In one example, the transmission device 902 is a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0132] In addition, the electronic device also includes: a display 901, which is used to display a target picture pre-configured for the anchor object in the current live broadcast screen, wherein the target picture displays the face of a virtual animal object; and a connection bus 910, which is used to connect the various module components in the electronic device.

[0133] In other embodiments, the terminal device or server may be a node in a distributed system, wherein the distributed system may be a blockchain system, and the blockchain system may be a distributed system formed by connecting the multiple nodes through network communication. Among them, the nodes may form a peer-to-peer (P2P, Peer To Peer) network, and any form of computing device, such as a server, terminal and other electronic devices, may become a node in the blockchain system by joining the peer-to-peer network.

[0134] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0135] According to one aspect of the present application, a computer-readable storage medium is provided, and a processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the above-mentioned eye drive control method.

[0136] Optionally, in this embodiment, the computer-readable storage medium may be configured to store a computer program for performing the following steps:

[0137] S1, displaying a target picture pre-configured for the anchor object in the current live broadcast screen, wherein the target picture displays the face of a virtual animal object;

[0138] S2, based on the collected face image of the anchor object, identifying the monocular closure feature of the anchor object, wherein the monocular closure feature is calculated based on the position offset of multiple closure point groups collected on a single eye of the anchor object, and the single closure point group includes an upper eyelid point and a lower eyelid point intersecting with the same straight line;

[0139] S3, according to the eye closure degree indicated by the single eye closure feature, driving and controlling the eyes of the virtual animal object in the target image to perform synchronous closure adjustment.

[0140] Optionally, in this embodiment, a person of ordinary skill in the art may understand that all or part of the steps in the various methods of the above embodiments may be completed by instructing hardware related to the terminal device through a program, and the program may be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc.

[0141] If the integrated units in the above embodiments are implemented in the form of software functional units and sold or used as independent products, they can be stored in the above computer-readable storage medium. Based on such understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling one or more computer devices (which can be personal computers, servers or network devices, etc.) to perform all or part of the steps of the above methods of various embodiments of the present invention.

[0142] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0143] In the several embodiments provided in the present application, it should be understood that the disclosed client can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0144] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0145] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0146] The above are only preferred embodiments of the present invention. It should be pointed out that, for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. An eye drive control method, characterized in that: include: Displaying a target picture pre-configured for the anchor object in the current live broadcast screen, wherein the target picture displays the face of the virtual animal object; Identify an eye key point sequence of the anchor object from the face image of the anchor object, wherein the eye key point sequence includes position information of each eyelid point in a plurality of closed point groups, wherein a single closed point group includes an upper eyelid point and a lower eyelid point intersecting with the same straight line; Determine the eye-opening distance and eye-closing offset distance corresponding to each of the closed point groups, wherein the eye-opening distance is the spatial distance between the upper eyelid point and the lower eyelid point in the closed point group when one eye is open, and the eye-closing offset distance is the offset distance between the upper eyelid point and the lower eyelid point in the closed point group when one eye is closed; Calculating the distance difference corresponding to each of the closed point groups to obtain a plurality of distance difference values, wherein the distance difference value is the difference between the eye-opening distance and the eye-closing offset distance of the same closed point group; Performing weighted sum calculation on the multiple distance differences to obtain a monocular closure degree coefficient; Obtaining the pupil distance of the anchor object; Determining the monocular closure feature according to the ratio between the monocular closure degree coefficient and the pupil distance; According to the eye closure degree indicated by the single eye closure feature, the eyes of the virtual animal object in the target image are driven and controlled to be synchronously closed and adjusted.

2. The method according to claim 1, characterized in that Determining the monocular closure feature according to the ratio between the monocular closure degree coefficient and the pupil distance includes: determining compensation factors matched to individual eyes; The product of the compensation coefficient and the ratio is used as the monocular closure feature.

3. The method according to claim 1, characterized in that According to the eye closure degree indicated by the single eye closure feature, driving and controlling the eyes of the virtual animal object in the target image to perform synchronous closure adjustment includes: Determining the eye shaking state of the anchor object according to the change of the single eye closure feature of at least three adjacent frames; The eyes of the virtual animal object are adjusted to be closed synchronously according to a driving control mode that matches the eye shaking state.

4. The method according to claim 3, characterized in that Adjusting the eyes of the virtual animal object to perform synchronous closing adjustment according to a driving control method matching the eye shaking state includes: When the eye jitter state indicates that the eye jitter is in a weak jitter state, obtaining an eye jitter amplitude value between a current face image frame and two previous face images before the current face image frame; Calculating an eye closure adjustment value according to a jitter coefficient matched with the current frame face image and the eye jitter amplitude value; adjusting the monocular closure feature according to the eye closure adjustment value, and performing mean filtering on the monocular closure feature to obtain the processed monocular closure feature; According to the processed single eye closure feature, the eyes of the virtual animal object in the target image corresponding to the next frame of face image after the current frame of face image are synchronously closed and adjusted.

5. The method according to claim 3, characterized in that: Adjusting the eyes of the virtual animal object to perform synchronous closing adjustment according to a driving control method matching the eye shaking state includes: When the eye shaking state indication is in a non-shaking state, performing mean filtering processing on the monocular closure feature corresponding to the current frame face image to obtain the processed monocular closure feature; According to the processed single eye closure feature, the eyes of the virtual animal object in the target image corresponding to the next frame of face image after the current frame of face image are synchronously closed and adjusted.

6. An eye drive control device, characterized in that: include: A display unit, used to display a target picture pre-configured for the anchor object in the current live broadcast screen, wherein the target picture displays the face of the virtual animal object; The recognition unit is used to recognize the eye key point sequence of the anchor object from the face image of the anchor object, wherein the eye key point sequence includes the position information of each eyelid point in a plurality of closed point groups, wherein a single closed point group includes an upper eyelid point and a lower eyelid point intersecting with the same straight line; determine the eye-opening distance and the eye-closing offset distance corresponding to each of the closed point groups, wherein the eye-opening distance is the spatial distance between the upper eyelid point and the lower eyelid point in the closed point group when one eye is open, and the eye-closing offset distance is the offset distance between the upper eyelid point and the lower eyelid point in the closed point group when one eye is closed; calculate the distance difference corresponding to each of the closed point groups to obtain a plurality of distance difference values, wherein the distance difference is the difference between the eye-opening distance and the eye-closing offset distance of the same closed point group; perform weighted sum calculation on the plurality of distance differences to obtain a single eye closure degree coefficient; obtain the pupil distance of the anchor object; determine the single eye closure feature according to the ratio between the single eye closure degree coefficient and the pupil distance; A driving control unit is used to drive and control the eyes of the virtual animal object in the target image to perform synchronous closing adjustment according to the eye closing degree indicated by the single eye closing feature.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein the program executes the method described in any one of claims 1 to 5 when executed.

8. An electronic device, comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to execute the method according to any one of claims 1 to 5 through the computer program.

Images