Virtual object processing method, electronic device and storage medium

Abstract

The present application discloses a virtual object processing method, an electronic device, and a storage medium. The virtual object processing method includes: obtaining first feature data obtained by encoding a driving object; obtaining change data of the first feature data relative to a reference feature data of the driving object; fusing the change data and second feature data obtained by encoding the virtual object to obtain update data; and obtaining an updated virtual object based on the update data. In the above manner, the present application can improve the efficiency of virtual object processing.

Description

Technical Field

[0001] The present application relates to the technical field of video image processing, and in particular to a virtual object processing method, an electronic device and a storage medium. Background Art

[0002] With the popularization of smart terminals and the development of communication technology, more and more people are enjoying the fun of surfing high-speed networks on smart terminals. The continuous iteration of smart terminal hardware and software and the continuous upgrading of network bandwidth enable people to watch high-definition images and videos on smart terminals.

[0003] In order to diversify the content presented, videos or images are often processed accordingly. Virtual objects are presented in videos or images. Common virtual objects are generally created through design and 3D modeling, and then the movement changes of virtual objects are realized through real-life motion capture, or virtual objects in corresponding states are designed, and objects in videos or images are replaced through post-manual image processing, or AI technology is used to extract action sequences from words and expressions to realize the movement changes of virtual objects. Summary of the invention

[0004] The main technical problem solved by the present application is to provide a virtual object processing method, an electronic device and a storage medium, which can adjust and improve the processing efficiency.

[0005] In order to solve the above technical problems, a technical solution adopted in the present application is: to provide a virtual object processing method, the method comprising: obtaining first feature data obtained by encoding a driving object; obtaining change data of the first feature data relative to a reference feature data of the driving object; fusing the change data and the second feature data obtained by encoding the virtual object to obtain update data; and obtaining the updated virtual object based on the update data.

[0006] In order to solve the above technical problems, another technical solution adopted in the present application is: to provide an electronic device, which includes a processor and a memory; the memory is coupled to the processor, and a computing program is stored in the memory, and the processor can execute the computer program to implement the above virtual object processing method.

[0007] In order to solve the above technical problem, another technical solution adopted by the present application is: providing a computer-readable storage medium storing a computer program, which can be executed by a processor to implement the above virtual object processing method.

[0008] The beneficial effects of the present application are: different from the prior art, by encoding the driving object to obtain the first feature data, obtaining the change data of the first feature data relative to the reference data, so that different first feature data of the driving object can obtain change data based on a relatively unified standard, and can quickly obtain effective change data to more effectively characterize the feature changes of the driving object. This type of processing has low calculation pressure and a low error rate. Moreover, since the change data is simple to obtain and can effectively display the feature changes, the accuracy of the updated data obtained by fusing the change data with the second feature data of the virtual object is also high, and thus the updated virtual object can be obtained more quickly and effectively. The process is simple and the error rate is low. There is no need to use complex motion capture methods to obtain motion changes, and the virtual object obtained by extracting feature changes from text or expressions can more accurately display feature changes and have a higher accuracy rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 This is a schematic diagram of the system composition of an exemplary implementation scenario of the first embodiment of the present application;

[0010] Figure 2 This is a schematic diagram of a first process of an embodiment of a virtual object processing method of the present application;

[0011] Figure 3 It is a second flow chart of an embodiment of the virtual object processing method of the present application;

[0012] Figure 4 It is a third flow chart of an embodiment of the virtual object processing method of the present application;

[0013] Figure 5 is a schematic diagram of an exemplary circuit structure of an electronic device embodiment of the present application;

[0014] Figure 6 It is a schematic diagram of an exemplary circuit structure of a computer-readable storage medium embodiment of the present application. DETAILED DESCRIPTION

[0015] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0016] After long-term research, the inventor of the present application found that the current production of virtual images generally requires design and 3D modeling, and then the prepared virtual image is mapped to the virtual image using motion capture, so that the virtual image can produce the desired corresponding expressions, movements and other changes, or input words or voices corresponding to certain expressions, movements, etc. through AI, so that the model can make movements or expressions corresponding to the words or voices. It is necessary to build a virtual image of a 3D model, and the use of motion capture requires a lot of manpower, which is difficult and the production process is complicated. Although the motion mapping is more accurate, the cost is extremely high and the efficiency is low. The actions derived from words or voices have a high error rate, which indirectly leads to low efficiency and makes it difficult to present good movements and expression changes on the virtual image. To this end, the present application proposes the following embodiments.

[0017] The following embodiments of the present application can be applied to scenarios such as live broadcast and video processing. Taking the live broadcast scenario as an example, Figure 1 As shown, the live broadcast system 1 may include a server 10, an audience terminal 20 and an anchor terminal 30. The server 10, the audience terminal 20 and the anchor terminal 30 may be electronic devices, and specifically the audience terminal 20 and the anchor terminal 30 may be mobile terminals, computers, servers or other terminals, etc. The mobile terminal may be a mobile phone, a laptop computer, a tablet computer, a smart wearable device, etc., and the computer may be a desktop computer, etc. The server 10 may pull the live broadcast data stream from the anchor terminal 30, and push the acquired live broadcast data stream to the audience terminal 20. After the audience terminal 20 acquires the live broadcast data stream, it can watch the live broadcast process of the anchor or the guest. The mixed stream of the live broadcast data stream may occur in at least one of the server 10, the anchor terminal 30 and the audience terminal 20. Video or voice connection can be performed between the anchor terminal 30 and the anchor terminal 30, and between the anchor terminal 30 and the audience terminal 20. In the video connection, the connection party may push the live broadcast data stream including the video stream to the server 10. Taking video processing as an example, the screen or object of an ordinary video file may be processed by an electronic device.

[0018] The following embodiments may perform corresponding processing on the video data including the video files or live video streams exemplified above. Of course, the following embodiments may also perform corresponding processing on images.

[0019] like Figure 2 As shown, the virtual object processing method embodiment of the present application may include the following steps:

[0020] S100: Acquire first feature data obtained by encoding a driving object.

[0021] The driving object may refer to an object used to generate corresponding changes. The virtual object may refer to an object on which the corresponding changes are applied so that it can also have corresponding changes. In other words, the corresponding changes generated by the driving object are used to drive the virtual object to also generate corresponding changes. The above changes may include changes in expressions and postures. For example, the driving object can be displayed in the screen of the video data, and the corresponding changes generated by the driving object can be used to drive the virtual object outside the video data to generate corresponding changes, so that the virtual object can replace the driving object and be presented in the screen corresponding to the video data.

[0022] The virtual object may include at least facial features, such as a human face or other animal face. Similarly, the driven object may include facial features, such as a human face or other animal face. Optionally, the face may be a cartoon face or an ordinary face.

[0023] The first feature data obtained by encoding the driving object may refer to data used to characterize the features of the driving object. Taking a human face as an example, the first feature data may include data characterizing features such as face shape, skin color, hairstyle, hair color, eyes and their opening and closing states, mouth and its opening and closing states, and head posture. Of course, other objects besides human faces can also obtain first feature data characterizing the features of the object through encoding.

[0024] like Figure 3 As shown, taking video data as an example, the first feature data can be obtained specifically by the following method included in step S100:

[0025] S110: Acquire video data.

[0026] For live broadcast, the server 10 or the anchor terminal 30 can obtain the video data generated during the live broadcast, that is, receive the video frames of the video data in sequence during the live broadcast. During the live broadcast, for example, the anchor's face can be used as the driving object.

[0027] S120: Encoding the video frame having the driving object in the video data to obtain first feature data of the driving object of the corresponding video frame.

[0028] For video frames in which a driving object appears during the live broadcast, the video frames can be encoded to obtain the overall feature data of the video frames, including the first feature data of the driving object. During the whole process, the live video frames can be received in sequence to detect whether the currently received video frame has a driving object. If so, the currently received video frame is encoded. For ordinary video files, the video frames with driving objects are encoded in the same way to obtain the first feature data.

[0029] For specific detection of the driving object and the encoding method, reference may be made to the following steps included in step S120:

[0030] S121: sequentially identifying a driving object in each video frame of the video data.

[0031] If the driving object is a face, the driving object can be identified in each video frame in turn using face recognition technology. The face recognition technology can be an existing face recognition technology. Of course, if the driving object is another object, a recognition model can be trained by deep learning or other methods, and the driving object can be identified using the recognition model. Of course, the face recognition technology can be implemented in a generative adversarial network, and then each video frame can be input into the generative adversarial network model frame by frame to identify the driving object.

[0032] S122: Encode the video frame in which the driving object is identified by using a generative adversarial network to obtain corresponding first feature data expressed by a multi-dimensional feature vector.

[0033] Generative adversarial networks are a type of deep learning model and a type of artificial intelligence algorithm used for unsupervised machine learning. Generative adversarial networks generally consist of two parts: a generator and a discriminator. The generator and the discriminator implement corresponding functions by fitting corresponding generation and discriminant functions. Specifically, the generator can be a neural network model, and the discriminator can also be a neural network model.

[0034] Optionally, the generator may have a coding grid and a decoding grid, etc., and the coding grid of the generator may be used to encode the video frame that identifies the driving object to obtain a multidimensional feature vector, that is, the first feature data. The first feature data may include, for example, data of dimensions such as face shape, skin color, hair color, eyes and their opening and closing states, mouth and their opening and closing states, and head posture. Specifically, after the training is completed, the video frame with the driving object is encoded using a generative adversarial network, and feature data of the video frame can be obtained, and the feature data includes the first feature data. For example, the feature data of the video frame includes background feature data corresponding to the background of the picture and first feature data corresponding to the driving object. The video frame with the driving object is encoded using a generative adversarial network, and the first feature data obtained has good decoupling characteristics between the feature dimensions, which can reduce the mutual influence and interference between the data of each feature dimension, and is convenient for reducing the influence of other data on the change data obtained by subsequent calculation, which is more conducive to the subsequent fusion. Optionally, the generative adversarial network is a StyleGAN2 model, and can also be a StyleGAN model or other generative adversarial network models.

[0035] Optionally, each video frame is input into the generative adversarial network frame by frame for processing, and if the current frame does not identify the driving object, it is directly output. If the current frame identifies the driving object, the current frame is encoded to obtain the first feature data corresponding to the frame.

[0036] Of course, in addition to generating adversarial networks, other existing deep learning algorithms, neural network algorithms, etc. can also be used to implement tools or models that encode images into vectors.

[0037] S200: Acquire change data of the first characteristic data relative to the reference characteristic data of the driving object.

[0038] The reference feature data of the driving object can be used as a benchmark, which is used to characterize a benchmark state of the driving object. The reference feature data can be consistent or substantially consistent with the first feature data in feature dimension or structure. The change of the first feature data relative to the reference data can be considered as the change between the state corresponding to the first feature data and the state corresponding to the reference data. Taking the human face as an example, generally speaking, expressions, postures, etc. are easy to change, which belongs to the state of change. By setting the reference data of the driving object, and then calculating the change data of the first feature data relative to the driving object and the reference data characteristics, the change data can be obtained more accurately in a more unified manner, with a low error rate, and can more effectively characterize the changes in the state, action, etc. of the driving object.

[0039] Before step S200, it is necessary to first obtain reference feature data of the driving object. For details, please refer to the following steps:

[0040] S210: Acquire a reference image having a driving object.

[0041] The reference image must contain the driving object, in other words, the driving object is displayed in the reference image. The driving object in the reference image is used as a benchmark. The reference image can be obtained from the video data, that is, one frame of the image is captured from the video data as the reference image. The reference image can also be obtained outside the video data, or can be shot in advance. Taking live broadcast as an example, the reference image can be obtained before the live broadcast starts, and then it can be compared with the video frame of the video stream during the live broadcast. Optionally, the reference image and the live broadcast data correspond to the same scene, that is, they are both shot in the same scene.

[0042] Optionally, in order to enable the changed data to be better reflected in the updated virtual object, the reference image can be selected or determined by means of the virtual object. Specifically, step S210 includes the following steps:

[0043] S211: Acquire an image to be processed having a driving object.

[0044] The image to be processed can be selected by the user from a photo album or the Internet, or can be taken by the user using a camera and input into the electronic device. Of course, the image to be processed can also be automatically acquired by the electronic device, or taken by the user. The number of images to be processed can be multiple, and a reference image can be determined from the multiple images to be processed. The image to be processed can be acquired outside the video data, or can be one of the frames captured from the video data. Specifically, it can be implemented by the following steps included in step S211.

[0045] S2111: Acquire an image to be processed outside the video data that is in the same scene as the video data; or, acquire the image to be processed in the video data.

[0046] Acquired outside the video data, for example, taken by the user himself, or taken by the electronic device using a camera outside the video data, or acquired locally or on the Internet. Taking live broadcast as an example, before the live broadcast, you can automatically or manually take an image of the anchor in the live broadcast scene as the image to be processed. Acquiring the image to be processed in the same scene as the video data makes it easier to calculate the change between the first feature data and the reference feature data later, which makes the calculation simpler and the processing pressure less. Of course, the image to be processed may not be in the same scene as the video data.

[0047] like Figure 4 As shown, as to how to determine the reference image from the image to be processed, a virtual object may be used. Specifically, this embodiment may include the following steps:

[0048] S010: Acquire a preset static image with a virtual object.

[0049] The preset static image is an ordinary two-dimensional image with a virtual object displayed on it. Taking a human face as an example, the virtual object can be a cartoon face or an ordinary human face. During the live broadcast, the user can select and determine the preset static image by himself, or it can be automatically generated by an electronic device, automatically obtained in an album or on the Internet. The purpose of obtaining the preset static image is to obtain the relevant data of the virtual object therein, so as to facilitate the subsequent fusion processing with the relevant data of the driving object, and then the virtual object can be further presented in the video data.

[0050] After obtaining the preset static image, the reference image can be determined using the virtual object in the preset static image. For details, refer to the following steps:

[0051] S212: Calculating the similarity between the driving object of the image to be processed and the virtual object of the preset static image.

[0052] That is, the reference image is determined based on whether the driving object of the image to be processed is similar to the virtual object of the preset static image, so that the motion changes of the driving object of the video frame relative to the driving object of the image to be processed can be approximated or equal to the motion changes of the driving object of the video frame relative to the virtual object, thereby enabling these motion changes to be better integrated into the virtual object later.

[0053] Taking a human face as an example, if the similarity is greater than or equal to a preset threshold, it can be considered that the driving object in the image to be processed and the virtual object in the preset static image are similar or roughly consistent in expression and posture, etc. For example, if the virtual object in the preset static image is a frontal neutral expression, then the posture and expression in the reference image should also be a frontal neutral expression.

[0054] There are many ways to calculate the similarity between images. The following is an exemplary introduction of one of them. Specifically, step S212 includes the following steps:

[0055] S2121: Acquire a plurality of first key feature points of the driving object in the image to be processed, and generate a first key feature vector representing the plurality of first key feature points.

[0056] A plurality of first key feature points are acquired in the image to be processed by a key point detection and positioning method, and the plurality of first key feature points can characterize the driving object in the image to be processed.

[0057] Taking the face as an example, the key points are located using the face key point detection algorithm to obtain the first key feature points. For example, the first key feature points include eye feature points, nose feature points, face contour feature points, mouth feature points, and forehead feature points. The first key feature vector is used to characterize multiple first key feature points. Multiple key feature points are digitized, and then the first key feature vector is obtained in the form of a vector. For example, the coordinates and other data of each of the multiple first key feature points are obtained, and then the coordinates and other data of each first key feature point are normalized, and the normalized data is used to form the first key feature vector.

[0058] S2122: Acquire multiple second key feature points of the virtual object in a preset static image, and generate a second key feature vector representing the multiple second key feature points.

[0059] Similarly, a plurality of second key feature points are acquired in the preset static image through the key point detection and positioning method, and the plurality of second key feature points can characterize the driving object in the image to be processed.

[0060] Taking the face as an example, the key points are located using the face key point detection algorithm to obtain the second key feature points. For example, the second key feature points include eye feature points, nose feature points, face contour feature points, mouth feature points, and forehead feature points. The second key feature vector is used to characterize multiple second key feature points. Multiple key feature points are digitized, and then the second key feature vector is obtained in the form of a vector. For example, the coordinates and other data of each of the multiple second key feature points are obtained, and then the coordinates and other data of each second key feature point are normalized, and the normalized data is used to form the second key feature vector.

[0061] S2123: Calculate the distance between the first key feature vector and the second key feature vector as the similarity.

[0062] Optionally, the Euclidean distance between the first key feature vector and the second key feature vector is calculated, and the Euclidean distance is used as the similarity. It is determined whether the similarity is greater than or equal to a preset threshold, and then it is determined whether the driving object of the image to be processed is similar to the virtual object in the preset static image. If the similarity is greater than or equal to the preset threshold, the following step S213 is executed. If the similarity is less than the preset threshold, the following step S214 is executed.

[0063] S213: If the similarity is greater than or equal to a preset threshold, the image to be processed is determined as a reference image.

[0064] S214: If the similarity is less than the preset threshold, re-execute step S211.

[0065] By selecting a reference image whose driving object is similar to the virtual object of the preset static image, on the one hand, the reference image can unify the standard for calculating the change data, thereby facilitating the subsequent use of the change data for corresponding calculations. On the other hand, selecting a reference image in this way can make the changes in the driving object between the video frames of the video data relative to the reference image better "transplanted" to the virtual object, with better compatibility, thereby making the updated virtual object more realistic and the processing speed faster.

[0066] After determining the reference image with the driving object, the reference image needs to be encoded. For details, refer to the following steps:

[0067] S220: Encode the reference image to obtain reference feature data of the driving object.

[0068] The reference image is encoded into data to obtain reference feature data of the driving object. The reference image is encoded using the same encoding method as encoding the video frame of the video data having the driving object, and the reference feature data is consistent or substantially consistent with the first feature data in feature dimension or structure.

[0069] Specifically, the reference image can also be encoded using a generative adversarial network, as shown in the following steps:

[0070] S221: Encode the reference image using a generative adversarial network to obtain reference feature data expressed as a multi-dimensional feature vector.

[0071] In this embodiment, the same generative adversarial network can be used for encoding and decoding. That is, the reference image and the video frame of the video data are respectively encoded using the coding grid of the same generative adversarial network, and then the first feature data and the reference feature data with the same or substantially the same feature dimensions can be formed. In this way, encoding the reference image can obtain a multi-dimensional feature vector, that is, the reference feature data expressed in a multi-dimensional feature vector. The reference feature data may also include data of dimensions such as face shape, skin color, hair color, eyes and their opening and closing states, mouth and its opening and closing states, and head posture.

[0072] Based on the above content, the first feature data and the reference feature data can be compared to obtain the change data. That is, step S200 may include the following steps:

[0073] S230: Compare the first feature data with the reference feature data to obtain change data.

[0074] Specifically, the first feature data and the reference feature data are compared on the same feature dimension to obtain the change data corresponding to the changed feature dimension. The first feature data and the reference feature data having the same or substantially the same data structure are compared. If the data of some feature dimensions are changed, the change data corresponding to the changed feature dimension can be obtained.

[0075] Taking the human face as an example, when the driven object is talking, turning the head, lowering the head and raising the head, the facial shape, skin color, hairstyle and hair color are generally unchanged, and what changes are the posture and expression (mouth and / or eyes) of the head. In this way, the data difference between the first feature data and the reference data in these feature dimensions can be used as change data to facilitate the subsequent driving of the virtual face in these dimensions.

[0076] The following is a mathematical description of the above process:

[0077] Assume that the first feature data corresponding to the video frame at time t is The reference feature data corresponding to the reference image is Then the expression of the change data can be referred to as follows:

[0078]

[0079] Where F(,) is the motion change function, For changing data.

[0080] The first feature data and reference feature data expressed as a multi-dimensional feature vector are obtained by encoding through a generative adversarial network. As mentioned above, the feature dimensions have good decoupling characteristics, and the interference and mutual influence between the feature dimensions are small. For some feature dimensions, they generally remain unchanged, while some feature dimensions will change. The change data corresponding to the changed feature dimensions can be obtained by comparison. The specific motion change function can be designed as follows:

[0081]

[0082] This design is just as mentioned above. For the same driving object, some feature dimensions in the video frames at different times will remain unchanged, such as face shape, skin color, hairstyle, hair color, etc., while some feature dimensions may change, such as the degree of eye closure and mouth and head posture. After the first feature data and the reference feature data are subtracted, the unchanged attribute difference is 0, and the changed attribute difference is not 0, so it can better reflect the motion changes between the reference image and the video frame. In this way, the calculation process is simpler and more efficient, effectively reducing processing pressure and increasing processing speed. For live broadcasts, live broadcast delays can be reduced.

[0083] After the preset static image is acquired in step S010, before executing the following step S300, the second feature data needs to be calculated, and the details can refer to the following steps included in this embodiment:

[0084] S020: Encoding the preset static image to obtain second feature data of the virtual object.

[0085] After acquiring the preset static image, the preset static image is encoded to encode the image into data to obtain second feature data of the virtual object. The second feature data characterizes each feature of the virtual object in the preset static image. The second feature data and the first feature data may be consistent or substantially consistent in data structure or feature dimension, so that the change data can be effectively merged with the second feature data later.

[0086] For encoding the preset static image, the following steps may be specifically referred to in step S020:

[0087] S021: Encode a preset static image using a generative adversarial network to obtain second feature data expressed as a multi-dimensional feature vector.

[0088] The preset static image is input into the generative adversarial network, and the preset static image is encoded by the generative adversarial network. Specifically, the preset static image is encoded by the encoder of the generative adversarial network to obtain a multi-dimensional feature vector, that is, the second feature data. Taking the face as an example, the second feature data may also include data of dimensions such as face shape, skin color, hair color, eyes and their opening and closing states, mouth and its opening and closing states, and head posture.

[0089] The second feature data, the reference feature data and the first feature data can all be obtained through the same generative adversarial network encoding, that is, the preset static image, the reference image and the video frame of the video data are input into the same generative adversarial network for encoding acquisition.

[0090] For the processing of video data, the second feature data can be generated by calculation in advance and stored in advance so as to be reused in the fusion processing of step S300. In this way, the second feature data can be repeatedly fused with the change data corresponding to each video frame with the driving object in the video data to obtain the update data corresponding to each video frame, thereby effectively improving the processing speed.

[0091] After the change data and the second characteristic data are calculated, the change data and the second characteristic data may be processed as follows:

[0092] S300: Fusing the change data and the second feature data obtained by encoding the virtual object to obtain update data.

[0093] The fusion processing of the change data and the second feature data may refer to updating, replacing or superimposing the corresponding data in the second feature data using the change data, so that the updated data can simultaneously have at least some of the features corresponding to the change data and the second feature data, and the feature changes corresponding to the change data can be added to the virtual object, so as to facilitate the subsequent updated virtual object. The fusion of the change data and the second feature data can be performed in a neural network model, such as a generative adversarial network.

[0094] There are many ways to perform fusion processing. The following is an exemplary description of one of them. For details, please refer to the following steps included in step S300:

[0095] S310: Using the changed data, update the data of the feature dimension corresponding to the changed data in the second feature data to obtain updated data.

[0096] Taking the human face as an example, as mentioned above, the characteristic dimensions such as the degree of eye opening and closing, the degree of mouth opening and closing, and the head posture are prone to change. Assume that the change data corresponds to the above characteristic dimensions. Use the change data to update the data of the corresponding characteristic dimensions in the second characteristic data, that is, update the data of the characteristic dimensions such as eyes, mouth, and head posture in the second characteristic data, so that the data of the characteristic dimensions such as eyes, mouth, and head posture in the second characteristic data become change data, and then obtain the updated data to achieve the fusion of the second characteristic data and the change data. Then the updated data can make the other characteristic dimensions of the virtual object remain unchanged, and what changes are the degree of eye opening and closing and the degree of mouth opening and closing, so that the virtual object has the characteristic changes of the driving object.

[0097] Specifically, a generative adversarial network is used to update the data of the feature dimension corresponding to the changed data in the second feature data, thereby obtaining updated data.

[0098] Based on the above, the following will explain the above process in data form, assuming that the mathematical form of the updated data is:

[0099]

[0100] in is the second feature data of the virtual object, is the changing data, and H(,) is the fusion processing function.

[0101] The second feature data expressed as a multi-dimensional feature vector is obtained by encoding through a generative adversarial network. The feature dimensions have good decoupling characteristics, and the interference and mutual influence between the feature dimensions are small. Some feature dimensions are generally maintained unchanged, while some feature dimensions change, and the corresponding feature dimensions in the second feature data can be updated with the changed data. The specific fusion processing function can be designed as follows:

[0102]

[0103] Taking the human face as an example, the rationality of this design lies in the fact that the change data has nothing to do with the identity characteristics of the face, but only with the movement changes of the face. Therefore, the addition can maintain the identity characteristics of the virtual object while changing the facial expression and movement.

[0104] The update data is obtained by fusing the change data and the second characteristic data. The purpose of the update data is to reflect the action change of the driving object on the virtual object. For details, please refer to the following steps:

[0105] S400: Obtaining an updated virtual object based on the update data.

[0106] The updated virtual object is obtained based on the updated data, so that the updated virtual object has a change corresponding to the change action data. In this way, the updated virtual object can retain the original virtual object's own characteristics and have the change characteristics of the driving object.

[0107] For video data, the updated virtual object needs to be reflected in each video frame, and it can be considered that the updated virtual object is used to "replace" the driving object in the original video frame. Specifically, the following steps can be referred to in step S400:

[0108] S410: Using the update data, update the first feature data in the feature data of the corresponding video frame.

[0109] Each video frame having a driving object has its corresponding update data. The update data is used to update the first feature data in the feature data of the corresponding video frame. Optionally, the first feature data in the feature data of the corresponding video frame is replaced by the update data, so that the updated feature data of the video frame includes the update data.

[0110] For example, the Nth video frame is encoded to obtain feature data including the first feature data of the driving object. The change data corresponding to the Nth video frame is obtained by using the first feature data and the reference feature data. The change data corresponding to the Nth video frame is fused with the second feature data to obtain the update data corresponding to the Nth video frame. The first feature data in the feature data of the Nth video frame is updated by using the update data corresponding to the Nth video frame, thereby obtaining the updated feature data of the Nth video frame.

[0111] S420: Decoding the updated feature data of the video frame using a generative adversarial network to obtain a video frame with an updated virtual object.

[0112] The decoder of the generative adversarial network is used to decode the updated feature data, and then the corresponding image is obtained, that is, the video frame with the updated virtual object. In other words, the update data updates the first feature data corresponding to the driving object, and the update data carries the identity characteristics of the virtual object and the change characteristics of the driving object, so that the virtual object is displayed in the final video frame. The virtual object is the updated virtual object, that is, it has the identity characteristics of the virtual object and the change characteristics of the driving object. Finally, it is achieved that the action of the driving object in the video data is retained and the driving object is replaced by a virtual object.

[0113] By using a generative adversarial network to decode the updated feature data of the video frame, the decoded image has a better effect in terms of fidelity and generates a high-definition and realistic image.

[0114] Optionally, the generative adversarial network may include multiple upsampling operators. For example, the generative adversarial network may adopt a generative adversarial network such as StyleGAN2. The decoding structure of a generative adversarial network such as StyleGAN2 is composed of a cascade of multiple upsampling operators and is trained in a progressive manner, which can further improve the fidelity of the generated image and the resolution of the image, thereby making the image presentation high-definition and realistic.

[0115] In the processing process, after the video data is acquired, it is input frame by frame for virtual object processing. The video frames without the driving object are not processed but directly output, while the video frames with the driving object are processed and then output as updated video frames. Finally, the updated video frames and the directly output video frames are output frame by frame, that is, the updated video frames and the directly output video frames are output according to the previous frame sequence to form updated video data.

[0116] Optionally, this embodiment can be specifically implemented in a generative adversarial network. Specifically, a preset static image is encoded in the generative adversarial network to obtain second feature data, and the data is pre-stored; a reference image is obtained and input into the generative adversarial network to obtain reference feature data, and the data is pre-stored. Then, the video data is input into the generative adversarial network frame by frame to calculate the change data and fusion processing, and then the updated video frames are output frame by frame.

[0117] To summarize, the driving object is encoded to obtain the first feature data, and the change data of the first feature data relative to the reference data is obtained, so that different first feature data of the driving object can obtain change data based on a relatively unified standard, and effective change data can be quickly obtained, and the feature changes of the driving object can be more effectively characterized. The calculation and processing pressure is small, and the error rate is low. Moreover, since the change data is simple to obtain and can effectively display the feature changes, the accuracy of the updated data obtained by fusing the change data with the second feature data of the virtual object is also high, and thus the updated virtual object can be obtained more quickly and effectively. The process is simple and the error rate is low. There is no need to use complex motion capture methods to obtain motion changes, and the virtual object obtained by extracting feature changes from text or expressions can more accurately display feature changes and have a higher accuracy rate.

[0118] This embodiment can be applied in the live broadcast system 1. In some embodiments, the execution process of the above embodiment can be completed at the host end 30, and the host end 30 pushes the video stream after the virtual object processing to the server 10, and the server 10 pushes the video stream to the viewer end 20. In other embodiments, the execution process of this embodiment can also be completed at the server 10, and the host end 30 pushes the original video stream to the server 10, and the server 10 performs virtual object processing, generates an updated video stream, and pushes the updated video stream to the host end 30 and the viewer end 20. In some other embodiments, the execution process of this embodiment can also be completed in the server 10 and the host end 30, and the host end 30 performs some steps, and after performing some steps, pushes the results to the server 10, and the server 10 performs another part of the steps, thereby obtaining an updated video stream and pushing it to the host end 30 and the viewer end 20.

[0119] Both the host terminal 30 and the server 10 can be electronic devices.

[0120] like Figure 5 As shown, the electronic device 100 described in the electronic device embodiment of the present application includes a processor 110 and a memory 120. The memory 120 is coupled to the processor 110.

[0121] The memory 120 is used to store computer programs, and may be a RAM, a ROM, or other types of storage devices. Specifically, the memory may include one or more computer-readable storage media, which may be non-transitory. The memory may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory is used to store at least one program code.

[0122] The processor 110 is used to control the operation of the electronic device 100. The processor 110 may also be referred to as a CPU (Central Processing Unit). The processor 110 may be an integrated circuit chip having signal processing capabilities. The processor 110 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. A general-purpose processor may be a microprocessor or the processor 110 may also be any conventional processor, etc.

[0123] The processor 110 is used to execute the computer program stored in the memory 120 to implement the virtual object processing method described in the embodiment of the virtual object processing method of the present application.

[0124] In some embodiments, the electronic device 100 may further include: a peripheral device interface 130 and at least one peripheral device. The processor 110, the memory 120 and the peripheral device interface 130 may be connected via a bus or a signal line. Each peripheral device may be connected to the peripheral device interface 130 via a bus, a signal line or a circuit board. Specifically, the peripheral device includes: at least one of a radio frequency circuit 140, a display screen 150, an audio circuit 160 and a power supply 170.

[0125] The peripheral device interface 130 may be used to connect at least one peripheral device related to I / O (Input / output) to the processor 110 and the memory 120. In some embodiments, the processor 110, the memory 120, and the peripheral device interface 130 are integrated on the same chip or circuit board; in some other implementations, any one or two of the processor 110, the memory 120, and the peripheral device interface 130 may be implemented on a separate chip or circuit board, which is not limited in this embodiment.

[0126] The radio frequency circuit 140 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 140 communicates with communication networks and other communication devices through electromagnetic signals. The radio frequency circuit 140 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals into electrical signals. Optionally, the radio frequency circuit 140 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, and the like. The radio frequency circuit 140 can communicate with other terminals through at least one wireless communication protocol. The wireless communication protocol includes but is not limited to: the World Wide Web, a metropolitan area network, an intranet, various generations of mobile communication networks (2G, 3G, 4G and 5G), a wireless local area network and / or a WiFi (Wireless Fidelity) network. In some embodiments, the radio frequency circuit 140 may also include circuits related to NFC (Near Field Communication), which is not limited in this application.

[0127] The display screen 150 is used to display the UI (User Interface). The UI may include graphics, text, icons, videos, and any combination thereof. When the display screen 150 is a touch display screen, the display screen 150 also has the ability to collect touch signals on the surface or above the surface of the display screen 150. The touch signal can be input to the processor 110 as a control signal for processing. At this time, the display screen 150 can also be used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards. In some embodiments, the display screen 150 can be one, arranged on the front panel of the electronic device 100; in other embodiments, the display screen 150 can be at least two, respectively arranged on different surfaces of the electronic device 100 or in a folding design; in other embodiments, the display screen 150 can be a flexible display screen, arranged on a curved surface or a folding surface of the electronic device 100. Even, the display screen 150 can also be set to a non-rectangular irregular shape, that is, a special-shaped screen. The display screen 150 can be made of materials such as LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode, organic light-emitting diode).

[0128] The audio circuit 160 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals and input them into the processor 110 for processing, or input them into the radio frequency circuit 140 to achieve voice communication. For the purpose of stereo acquisition or noise reduction, there can be multiple microphones, which are respectively arranged at different parts of the electronic device 100. The microphone can also be an array microphone or an omnidirectional acquisition microphone. The speaker is used to convert the electrical signal from the processor 110 or the radio frequency circuit 140 into sound waves. The speaker can be a traditional film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert the electrical signal into sound waves audible to humans, but also convert the electrical signal into sound waves inaudible to humans for purposes such as ranging. In some embodiments, the audio circuit 160 may also include a headphone jack.

[0129] The power supply 170 is used to power various components in the electronic device 100. The power supply 170 can be an alternating current, a direct current, a disposable battery, or a rechargeable battery. When the power supply 170 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery that is charged through a wired line, and a wireless rechargeable battery is a battery that is charged through a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0130] For a detailed description of the functions and execution processes of each functional module or component in the electronic device embodiment of the present application, reference can be made to the description in the above-mentioned virtual object processing method embodiment of the present application, which will not be repeated here.

[0131] In the several embodiments provided in the present application, it should be understood that the disclosed electronic device and virtual object processing method can be implemented in other ways. For example, the various embodiments of the electronic device described above are only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0132] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0133] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0134] See also Figure 6 , if the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium 200. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions / computer programs to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of each embodiment of the present invention. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks or optical disks, and electronic devices such as computers, mobile phones, laptops, tablet computers, cameras, etc. having the above-mentioned storage media.

[0135] The description of the execution process of the program data in the computer-readable storage medium can refer to the description in the above-mentioned embodiment of the virtual object processing method of the present application, which will not be repeated here.

[0136] The above descriptions are merely embodiments of the present application and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A virtual object processing method, characterized in that: include: Acquire first characteristic data obtained by encoding the driving object; Acquire change data of the first feature data relative to reference feature data of the driving object, where the reference feature data of the driving object is obtained by encoding a reference image having the driving object; The change data and second feature data obtained by encoding the virtual object are merged to obtain update data; the second feature data is obtained by encoding a preset static image with the virtual object, and the similarity between the driving object of the reference image and the virtual object of the preset static image is greater than or equal to a preset threshold; The virtual object is updated based on the update data.

2. The virtual object processing method according to claim 1, characterized in that: Before acquiring the change data of the first characteristic data relative to the reference characteristic data of the driven object, the method includes: Acquire a reference image having the driven object; Wherein, acquiring a reference image having the driving object includes: Acquire an image to be processed having the driving object; Calculating the similarity between the driving object of the image to be processed and the virtual object of the preset static image; If the similarity is greater than or equal to a preset threshold, the image to be processed is determined as the reference image.

3. The virtual object processing method according to claim 2, characterized in that: The calculating the similarity between the driving object of the image to be processed and the virtual object of the static image includes: Acquire a plurality of first key feature points of the driving object in the image to be processed, and generate a first key feature vector representing the plurality of first key feature points; Acquire a plurality of second key feature points of the virtual object in the preset static image, and generate a second key feature vector representing the plurality of second key feature points; The distance between the first key feature vector and the second key feature vector is calculated as the similarity.

4. The virtual object processing method according to claim 1, characterized in that: The step of obtaining first characteristic data obtained by encoding the driving object includes: Get video data; The video frame having the driving object in the video data is encoded to obtain first feature data of the driving object corresponding to the video frame.

5. The virtual object processing method according to claim 4, characterized in that: The encoding process of the video frame having the driving object in the video data to obtain the first feature data of the driving object in the video frame includes: sequentially identifying the driving object in each video frame of the video data; Encoding the video frame in which the driving object is identified using a generative adversarial network to obtain the first feature data expressed as a corresponding multi-dimensional feature vector; The obtaining of change data of the first characteristic data relative to the reference characteristic data of the driven object comprises: The reference image is encoded using the generative adversarial network to obtain the reference feature data expressed as a multi-dimensional feature vector.

6. The virtual object processing method according to claim 5, characterized in that: The acquiring change data of the first characteristic data relative to the reference characteristic data of the driven object includes: The first feature data and the reference feature data are compared at the same feature dimension to obtain the change data corresponding to the changed feature dimension.

7. The virtual object processing method according to claim 6, characterized in that: The step of fusing the change data and the second feature data obtained by encoding the virtual object to obtain the update data comprises: The preset static image is encoded using the generative adversarial network to obtain the second feature data expressed as a multi-dimensional feature vector.

8. The virtual object processing method according to claim 7, characterized in that: The step of fusing the change data and the second feature data obtained by encoding the virtual object to obtain the update data includes: The data of the characteristic dimension corresponding to the changed data in the second characteristic data is updated by using the changed data to obtain the updated data.

9. The virtual object processing method according to any one of claims 4 to 8, characterized in that: The virtual object updated based on the update data includes: Using the update data to update the first feature data in the feature data of the corresponding video frame; The updated feature data of the video frame is decoded using a generative adversarial network to obtain a video frame with the updated virtual object.

10. The virtual object processing method according to claim 9, characterized in that: The generative adversarial network is a StyleGAN model or a StyleGAN2 model.

11. The virtual object processing method according to claim 4, characterized in that: Obtaining an image to be processed with a driver object includes: The image to be processed that is in the same scene as the video data is obtained outside the video data; or, the image to be processed is obtained in the video data.

12. The virtual object processing method according to claim 4, characterized in that: The virtual object and / or the driven object at least include facial features; and / or the video data is a video stream collected during a live broadcast.

13. An electronic device, characterized in that: include: Processor and memory; The memories are respectively coupled to the processors, and a computer program is stored in the memories. The processors can execute the computer program to implement the virtual object processing method according to any one of claims 1 to 12.

14. A computer-readable storage medium, characterized in that: A computer program is stored, and the computer program can be executed by a processor to implement the virtual object processing method according to any one of claims 1 to 12.

Images