Virtual object driving method and device

Abstract

The invention provides a virtual object driving method and device, electronic equipment and a storage medium. The method comprises the following steps: acquiring a first video; the first video comprises dynamic information of a target object; based on a received first action instruction, acquiring reference information corresponding to the first action instruction according to the first video; the first action instruction is used for driving a virtual object to execute a target action corresponding to the first action instruction; and obtaining an image corresponding to the first action instruction based on the first action instruction, the reference information and the virtual object.

Description

Technical Field

[0001] The present disclosure relates to the field of virtual reality technology, and in particular to a virtual object driving method and device. Background Art

[0002] With the rapid development of digital technology, the modeling and driving of digital humans have been widely used in many fields, such as virtual character creation, film and television special effects production, game development, and online interactive services. However, the current mainstream methods of digital human modeling have significant limitations. In the existing digital human modeling process, a digital human model is usually constructed by relying on an image of the target object as the original character information. This method can capture the basic appearance features of the target object to a certain extent, but because the information conveyed by the image itself is very limited, it is difficult to fully and accurately present the rich characteristics of the target object. Summary of the invention

[0003] The present disclosure provides a virtual object driving method and device to at least solve the above technical problems existing in the prior art.

[0004] According to a first aspect of the present disclosure, a virtual object driving method is provided, comprising:

[0005] Acquire a first video; the first video includes dynamic information of a target object;

[0006] Based on the received first action instruction, obtaining reference information corresponding to the first action instruction according to the first video; the first action instruction is used to drive the virtual object to perform a target action corresponding to the first action instruction;

[0007] Based on the first action instruction, the reference information and the virtual object, an image corresponding to the first action instruction is obtained.

[0008] In the above solution, the obtaining of the first video includes one of the following:

[0009] Acquire the first uploaded video through the interface;

[0010] Acquire the first video through an image acquisition device and guidance information;

[0011] The guidance information includes text guidance information, animation guidance information and audio guidance information; the guidance information is used to instruct the target object to perform a corresponding action.

[0012] In the above solution, the step of obtaining reference information corresponding to the first action instruction based on the received first action instruction according to the first video includes:

[0013] Based on the first action instruction, matching and obtaining a target image from the first video;

[0014] The reference information is obtained based on the target image.

[0015] In the above solution, the target image is an image frame in the first video; and obtaining the reference information based on the target image includes:

[0016] Determine part information corresponding to the first action instruction, where the part information corresponds to a part of the virtual object;

[0017] A target sub-image corresponding to the part information is intercepted from the target image as the reference information.

[0018] In the above scheme, the method further includes:

[0019] Determine action parameter information according to the first action instruction;

[0020] According to the part information and the action parameter information, an image frame is matched from the first video as the target image.

[0021] In the above solution, obtaining the reference information based on the target image includes:

[0022] Obtaining a plurality of sub-image sets obtained according to the first video, each of the sub-image sets corresponding to a part of the target object;

[0023] Determine part information and action parameter information according to the first action instruction, wherein the part information corresponds to a part of the virtual object;

[0024] Determine a target sub-image set from the plurality of sub-image sets according to the part information;

[0025] A target sub-image is determined from the target sub-image set as the reference information according to the action parameter information.

[0026] In the above solution, the step of obtaining reference information corresponding to the first action instruction from the first video based on the received first action instruction includes:

[0027] Modeling the dynamic information included in the first video to obtain a target model;

[0028] An output image corresponding to the first action instruction is outputted by the target model as the reference information.

[0029] In the above solution, the target model includes a plurality of sub-models; and outputting an output image corresponding to the first action instruction through the target model as the reference information includes:

[0030] Determine the part information and the action parameter information according to the first action instruction;

[0031] Determining a target sub-model from the multiple sub-models according to the part information;

[0032] The output image is obtained according to the action parameter information and the target sub-model.

[0033] In the above solution, obtaining the image corresponding to the first action instruction based on the first action instruction, the reference information and the virtual object includes:

[0034] Acquire a sub-image corresponding to the reference information;

[0035] Acquire a main image corresponding to the first virtual image executing the first action instruction;

[0036] The sub-image is superimposed on the main image according to the position to obtain an image corresponding to the first action instruction.

[0037] According to a second aspect of the present disclosure, there is provided a virtual object driving device, comprising:

[0038] An acquisition unit, configured to acquire a first video; the first video includes dynamic information of a target object;

[0039] a matching unit, configured to obtain reference information corresponding to the first action instruction according to the first video based on the received first action instruction; the first action instruction is used to drive the virtual object to perform a target action corresponding to the first action instruction;

[0040] A driving unit is configured to obtain an image corresponding to the first action instruction based on the first action instruction, the reference information and the virtual object.

[0041] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The above and other objects, features and advantages of the exemplary embodiments of the present disclosure will become readily understood by reading the detailed description below with reference to the accompanying drawings. In the accompanying drawings, several embodiments of the present disclosure are shown in an exemplary and non-limiting manner, in which:

[0043] In the drawings, the same or corresponding reference numerals represent the same or corresponding parts.

[0044] Figure 1 A first optional flow chart of a virtual object driving method provided by an embodiment of the present disclosure is shown;

[0045] Figure 2 A second optional flow chart of the virtual object driving method provided by the embodiment of the present disclosure is shown;

[0046] Figure 3 A third optional flow chart of the virtual object driving method provided by the embodiment of the present disclosure is shown;

[0047] Figure 4 A fourth optional flow chart of the virtual object driving method provided by the embodiment of the present disclosure is shown;

[0048] Figure 5 A fifth optional flow chart of the virtual object driving method provided in the embodiment of the present disclosure is shown;

[0049] Figure 6 A sixth optional flow chart of the virtual object driving method provided by the embodiment of the present disclosure is shown;

[0050] Figure 7 An optional structural schematic diagram of a virtual object driving device provided in an embodiment of the present disclosure is shown;

[0051] Figure 8 A schematic diagram of the structure of an electronic device according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0052] In order to make the purpose, features, and advantages of the present disclosure more obvious and easy to understand, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present disclosure.

[0053] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it can be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0054] In the following description, the terms "first\second" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present disclosure described herein can be implemented in an order other than that illustrated or described herein.

[0055] Unless otherwise defined, all technical and scientific terms used in this disclosure have the same meaning as those commonly understood by those skilled in the art to which this disclosure belongs. The terms used in this disclosure are only for the purpose of describing the embodiments of this disclosure and are not intended to limit this disclosure.

[0056] It should be understood that in the various embodiments of the present disclosure, the size of the serial number of each implementation process does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present disclosure.

[0057] Digital humans can also be called virtual objects. In the existing digital human modeling process, digital human models are usually built by relying only on an image of the target object as the original character information. Although this method can capture the basic appearance characteristics of the target object to a certain extent, it is difficult to fully and accurately present the rich characteristics of the target object because the information conveyed by the image itself is very limited.

[0058] On the one hand, modeling based on a single image alone cannot fully demonstrate the unique personality characteristics of the target object in different postures, expressions, and movements. For example, key information such as the target object's facial muscle movement pattern, subtle changes in body posture, and characteristics of emotional expression are difficult to fully present in a static image, resulting in the constructed digital human appearing stiff and unnatural in the process of driving movements and expressions, lacking realism and vividness.

[0059] On the other hand, due to the limitations of images, existing modeling methods are powerless to capture the potential features of the target object that are not shown in the image, such as specific behavioral habits, personality style, etc. This makes it impossible for the generated digital human model to fully restore the overall appearance of the target object, greatly limiting the performance and scope of digital humans in practical applications.

[0060] In summary, the existing digital human modeling methods based on a single image have obvious deficiencies in presenting the characteristics of the target object and mining potential features, and cannot meet the growing demand for highly realistic and personalized digital humans. Therefore, developing a technical method that can more comprehensively and accurately obtain the characteristic information of the target object and effectively apply it to digital human modeling has important practical significance and broad application prospects.

[0061] Figure 1 A first optional flow chart of the virtual object driving method provided in the embodiment of the present disclosure is shown, and will be explained according to each step.

[0062] Step S101, obtaining a first video.

[0063] In some embodiments, the carrier implementing the virtual object driving method (hereinafter referred to as the carrier) obtains a first video; the first video includes dynamic information of the target object; the target object is the prototype of the virtual object to be driven, which can be a person, an animal or an object that can move. That is, the virtual object is generated and driven based on the target object; the dynamic information includes information such as the action, posture, expression and mouth shape output when the target object performs different actions. The carrier can be a computer program, an electronic circuit, a database, a mobile application, an electronic device, a cloud computing platform, a distributed system, an artificial intelligence framework, a mathematical model, an automation tool and a microcontroller, etc., which can implement software or hardware of the algorithm and method flow.

[0064] Step S102: based on the received first action instruction, obtaining reference information corresponding to the first action instruction according to the first video.

[0065] In some embodiments, the carrier receives a first action instruction, and the first action instruction is used to drive the virtual object to perform a target action corresponding to the first action instruction; the target action includes at least one of a body movement and an expression movement.

[0066] In some embodiments, the carrier determines the target action corresponding to the first action instruction, and obtains reference information corresponding to the target action from the first video; the reference information may include an image or action parameters. The action parameters may include a part and an action amplitude, such as the size of a mouth opening, the angle of an arm raising, etc.

[0067] Step S103: obtaining an image corresponding to the first action instruction based on the first action instruction, the reference information and the virtual object.

[0068] In some embodiments, the carrier controls the virtual object to implement a target action corresponding to the first action instruction based on the first action instruction and the reference information, and obtains an image corresponding to the first action instruction based on the target action. The image includes an image obtained by allowing the virtual object to complete the first action instruction according to the reference information.

[0069] In other embodiments, the carrier may also regenerate a virtual object based on the first action instruction and the reference information to replace the original virtual object.

[0070] In some optional embodiments, the method further includes displaying an image corresponding to the first action instruction on a carrier that displays the virtual object.

[0071] In some embodiments, there is a scene instructing a virtual object to perform a series of actions, such as instructing a virtual object to laugh from not smiling. During this process, the process from not smiling to laughing can be divided into multiple first action instructions, and the mouth opening amplitude corresponding to each action instruction is different. Optionally, the multiple first action instructions can be divided according to the mouth opening amplitude from small to large, and multiple corresponding target actions are obtained from the first video based on the multiple first action instructions. Then, based on the multiple first action instructions, the corresponding multiple target actions and the virtual object, images corresponding to the multiple first action instructions are obtained, and the images are displayed in the order of the first action instructions, so as to obtain the dynamic action of the virtual object from not smiling to laughing.

[0072] In this way, through the virtual object driving method provided by the embodiment of the present disclosure, the first video includes the target object's actions, expressions and posture changes over a period of time, which can provide rich dynamic information or continuity information, and provide sufficient data basis for fully understanding and presenting the characteristics of the target object; in addition, the subtle differences in actions, subtle changes in expressions and overall behavior patterns of the target object in different situations can be captured, and reference information corresponding to the first action instruction can be obtained from the first video. The virtual object is driven based on the reference information, so that the virtual object is no longer limited to a single static appearance presentation, but can vividly and realistically simulate various characteristics of the target object; finally, the virtual object is driven based on the reference information of the video, which greatly improves the realism and credibility of the digital human. Whether in terms of the smoothness of the action or the naturalness of the expression, it can be closer to the real target object, bringing a more immersive and satisfying experience to the user.

[0073] Figure 2 A second optional flow chart of the virtual object driving method provided in the embodiment of the present disclosure is shown, and will be explained according to each step.

[0074] Step S201, obtaining a first video.

[0075] In some embodiments, the carrier may obtain the first video uploaded through an interface, wherein the first video is pre-recorded and then uploaded through the interface. The first video includes dynamic information of the target object.

[0076] In other embodiments, the carrier may also obtain the first video through an image acquisition device and guidance information. The guidance information includes text guidance information, animation guidance information, and audio guidance information; the guidance information is used to instruct the target object to perform a corresponding action, and the corresponding action includes a target action corresponding to the first action instruction. That is, the carrier directly records the process of the target object performing different actions based on the guidance information through the image acquisition device. The guidance information is information that guides the target object to perform corresponding actions when recording the first video, such as blinking, opening the mouth, turning the head left, raising the left arm, and raising the right arm.

[0077] Step S202: Based on the first action instruction, obtain a target image by matching from the first video.

[0078] In some embodiments, the carrier may match the first video frame by frame based on the target action corresponding to the first action instruction to obtain a target image. For example, if the target action corresponding to the first action instruction is the left arm stretching upwards close to the ear, then based on the target action, frame by frame matching is performed to obtain an image including the target action, which is the target image.

[0079] In other embodiments, in response to the first video including guidance information, the time interval including the first action instruction in the first video is confirmed based on the guidance information, and the video frames in the time interval are matched frame by frame to obtain the target image. For example, if the target action corresponding to the first action instruction is to stretch the left arm upward to the ear, then based on the guidance information, it is determined that the video frames from the mth millisecond to the m+ath millisecond in the video frame are video frames that may include the target action, and the carrier matches the first action instruction and the video frames from the mth millisecond to the m+ath millisecond to obtain the target image. Wherein, m and a are positive integers.

[0080] In some further embodiments, in response to the first video including guidance information, multiple video frames in the first video that may include the first action instruction are confirmed based on the guidance information, and the first action instruction and the multiple video frames are matched frame by frame to obtain a target image. For example, if the target action corresponding to the first action instruction is to stretch the left arm upward to the ear, then based on the guidance information, it is determined that the video frames from the nth frame to the n+bth frame are video frames that may include the target action, and then the carrier matches the first action instruction and the video frames from the nth frame to the n+bth frame to obtain a target image. Wherein, n and b are positive integers.

[0081] It should be noted that, before executing step S202, it is not necessary to perform any preprocessing on the first video; the preprocessing may include classifying the video frames in the first video based on the guidance information.

[0082] Step S203: obtaining the reference information based on the target image.

[0083] In some embodiments, the carrier determines the part information corresponding to the first action instruction, and captures a target sub-image corresponding to the part information from the target image as the reference information.

[0084] In specific implementation, the reference information may include the target sub-image, and may also include the features of the corresponding parts in the target sub-image; for example, if the part information is mouth information, the mouth information (i.e., reference information) may include lip feature information (lip shape, lip size, lip color, lip thickness, lip texture and lip fullness, etc.), tooth feature information (delayed color, number of teeth exposed and tooth defects, etc.); or, if the part information is; if the part information is facial information, the features of the face may include orientation features (front, side, looking down or looking up) and exposed facial features information.

[0085] The part information corresponds to information of a part of the virtual object. For example, if the first action instruction is to stretch the left arm upward close to the ear, the part corresponding to the first action instruction is the left arm. The part information includes anatomical part information of a human or animal body. Alternatively, if the first action instruction is to laugh, the part information corresponding to the first action instruction is the mouth (lips) and teeth.

[0086] Step S204: obtaining an image corresponding to the first action instruction based on the first action instruction, the reference information and the virtual object.

[0087] In some embodiments, the carrier can obtain a sub-image of the corresponding part of the virtual object based on the reference information and the first action instruction, and superimpose the sub-image on the main image of the virtual object to obtain an image corresponding to the first action instruction. The superposition can include covering, that is, the sub-image covers the corresponding part of the main image; it can also be after deleting the image of the corresponding part of the main image, superimposing the sub-image on the main image.

[0088] In other embodiments, the carrier can also drive the virtual object based on the first action instruction and the reference information, so that the virtual object implements the target action corresponding to the first action instruction based on the reference information, and captures an image of the virtual object to obtain an image corresponding to the first action instruction.

[0089] In some further embodiments, the carrier may also regenerate a virtual object based on the first action instruction and the reference information to replace the original virtual object; or, the carrier may generate a corresponding part of the virtual object based on the first action instruction and the reference information to replace the corresponding part of the original virtual object to obtain a new virtual object.

[0090] In some embodiments, after obtaining the image corresponding to the first action instruction, the carrier displays the image corresponding to the first action instruction on the carrier displaying the virtual object. Alternatively, the carrier reconstructs and drives the virtual object based on the image corresponding to the first action instruction.

[0091] In some embodiments, there is a scene instructing a virtual object to perform a series of actions, such as instructing a virtual object to laugh from not smiling. During this process, the process from not smiling to laughing can be divided into multiple first action instructions, and the mouth opening amplitude corresponding to each action instruction is different. Optionally, the multiple first action instructions can be divided according to the mouth opening amplitude from small to large, and multiple corresponding target actions are obtained from the first video based on the multiple first action instructions. Then, based on the multiple first action instructions, the corresponding multiple target actions and the virtual object, images corresponding to the multiple first action instructions are obtained, and the images are displayed in the order of the first action instructions, so as to obtain the dynamic action of the virtual object from not smiling to laughing.

[0092] In this way, through the virtual object driving method provided by the embodiment of the present disclosure, the first video includes the target object's actions, expressions and posture changes over a period of time, which can provide rich dynamic information or continuity information, and provide sufficient data basis for fully understanding and presenting the characteristics of the target object; in addition, the subtle differences in actions, subtle changes in expressions and overall behavior patterns of the target object in different situations can be captured, and reference information corresponding to the first action instruction can be obtained from the first video. The virtual object is driven based on the reference information, so that the virtual object is no longer limited to a single static appearance presentation, but can vividly and realistically simulate various characteristics of the target object; finally, the virtual object is driven based on the reference information of the video, which greatly improves the realism and credibility of the digital human. Whether in terms of the smoothness of the action or the naturalness of the expression, it can be closer to the real target object, bringing a more immersive and satisfying experience to the user.

[0093] Figure 3 A third optional flow chart of the virtual object driving method provided in the embodiment of the present disclosure is shown, and will be explained according to each step.

[0094] Step S301, obtaining a first video.

[0095] The specific step flow of step S301 is the same as that of step S201 and will not be repeated here.

[0096] Step S302: classify video frames in the first video based on the parts to obtain a plurality of sub-image sets.

[0097] In some embodiments, the carrier can classify the video frames in the first video according to the parts based on the guidance information in the first video to obtain multiple sub-image sets, each of which includes the same target part. The target part includes the part that changes. For example, if the other parts of the body do not move and only the left arm moves (such as swaying, stretching forward, stretching upward), the left arm is the target part.

[0098] Furthermore, the carrier can also classify the video frames in the first video based on the parts and actions, so that each sub-image set includes the same target part and the actions of the target part are the same or similar. For example, one sub-image set contains video frames of the left arm stretching upward, and another sub-image set contains video frames of laughing.

[0099] Step S303: determining a target sub-image set from the plurality of sub-image sets based on the part information corresponding to the first action instruction.

[0100] In some embodiments, the carrier determines a target sub-image set from a plurality of sub-image sets according to the part information. The target sub-image set includes all video frames corresponding to the part information. The part information may be information of the target part.

[0101] Step S304: determining a target sub-image from the target sub-image set as reference information based on the action parameter information corresponding to the first action instruction.

[0102] In some embodiments, the action parameter information may include at least one of an action direction, an action speed, and an action identifier, wherein the action identifier is the name of the action, such as stretching, jumping, laughing, etc. According to the action parameter information, a target sub-image with the same action parameter information is matched from multiple video frames included in the target sub-image set as reference information.

[0103] Step S305: obtaining an image corresponding to the first action instruction based on the first action instruction, the reference information and the virtual object.

[0104] The specific steps of step S305 are the same as those of step S204 and will not be repeated here.

[0105] In this way, through the virtual object driving method provided by the embodiment of the present disclosure, the first video is pre-processed and divided into multiple sub-image sets. The target sub-image set is determined based on the part information corresponding to the first action instruction, and then the target sub-image is determined based on the action parameter information corresponding to the first action instruction. This can improve the speed of finding reference information and thereby improve the driving speed of the virtual object.

[0106] Figure 4A fourth optional flow chart of the virtual object driving method provided in the embodiment of the present disclosure is shown, and will be explained according to various parts.

[0107] Step S401, obtaining a first video.

[0108] The specific step flow of step S401 is the same as that of step S201 and will not be repeated here.

[0109] Step S402: Based on the action parameter information corresponding to the first action instruction, a target image is matched from the first video.

[0110] In some embodiments, the carrier determines the action parameter information corresponding to the first action instruction; based on the action parameter information, the target image is matched from the first video. The action parameter information includes at least one of the action part, action direction, action speed, action angle, action amplitude and action identification, and the action identification is the name of the action, such as stretching legs, jumping, laughing, raising left arm, and stretching left arm upward.

[0111] In a specific implementation, the carrier may match, from multiple video frames included in the first video, an image frame of an action performed by a target object that is identical to the action parameter information, as the target image.

[0112] For example, if the first action instruction includes raising the left arm 30° to the side, then the image frame in which the left arm is raised 30° to the side in the first video is matched as the target image.

[0113] In some other embodiments, the carrier may further classify the video frames in the first video based on the parts to obtain a plurality of sub-image sets (refer to step S302). Based on the action parts included in the action parameter information, a target sub-image set is determined, and matching is performed in the target sub-image set to obtain a target image.

[0114] Step S403: obtaining the reference information based on the target image.

[0115] The specific step flow of step S403 is the same as that of step S203 and will not be repeated here.

[0116] Step S404: obtaining an image corresponding to the first action instruction based on the first action instruction, the reference information and the virtual object.

[0117] The specific step flow of step S404 is the same as that of step S204 and will not be repeated here.

[0118] In this way, through the virtual object driving method provided by the embodiment of the present disclosure, the action parameter information corresponding to the first action instruction is matched with the video frame in the first video to obtain reference information, and then the virtual object is driven based on the reference information, so that the virtual object is no longer limited to a single static appearance presentation, but can vividly and realistically simulate various features of the target object; finally, the virtual object is driven based on the reference information of the video, which greatly improves the realism and credibility of the digital human.

[0119] Figure 5 A fifth optional flow chart of the virtual object driving method provided in the embodiment of the present disclosure is shown, and will be explained according to each step.

[0120] Step S501, obtaining a first video.

[0121] The specific step flow of step S501 is the same as that of step S201 and will not be repeated here.

[0122] Step S502: Modeling is performed based on the first video to obtain a target model.

[0123] In some embodiments, the first video includes different action information corresponding to multiple parts. The carrier can obtain a target model based on the first video, or can obtain multiple sub-models based on different parts and different actions in the first video. Each sub-model corresponds to an action and a part, and at least one of the actions and parts corresponding to different sub-models is different.

[0124] Step S503: outputting an output image corresponding to the first action instruction through the target model as the reference information.

[0125] In some embodiments, the carrier determines the part information and the action parameter information according to the first action instruction; determines the sub-model corresponding to the part information and the action parameter information of the first action instruction from the multiple sub-models as the target sub-model according to the part information; determines that when the parameter information of the action output by the target sub-model is consistent with the action parameter information, the corresponding image is the output image; and determines that the output image is the reference information.

[0126] Step S504: obtaining an image corresponding to the first action instruction based on the first action instruction, the reference information and the virtual object.

[0127] The specific step flow of step S504 is the same as that of step S204 and will not be repeated here.

[0128] In this way, through the virtual object driving method provided by the embodiment of the present disclosure, a target model is obtained based on the first video modeling, and the target model includes the rich appearance features and action features displayed by the target object in the first video; when driving the virtual object, the target model is first driven to obtain reference information, which must include the appearance features and action features of the target object for the first action instruction, and the image is obtained based on the reference information to perfect the virtual object, so that the virtual object can comprehensively and accurately program the rich characteristics of the target object.

[0129] Figure 6 A sixth optional flow chart of the virtual object driving method provided in the embodiment of the present disclosure is shown, and will be explained according to each step.

[0130] Step S601, obtaining a first video.

[0131] The specific step flow of step S601 is the same as that of step S201 and will not be repeated here.

[0132] Step S602: acquiring feature information from the first video based on the first action instruction.

[0133] In some embodiments, the carrier determines the part information and action parameter information corresponding to the first action instruction; based on the part information and the action parameter information, determines a video frame consistent with the part information and the action parameter information from the first video.

[0134] Acquire feature information from the video frame. The feature information includes facial feature information, tooth feature information and lip feature information. The tooth feature information includes at least one of tooth color, tooth shape, tooth arrangement information, tooth defect information and tooth exposure information; the lip feature information includes at least one of lip shape, lip size, lip color, lip thickness, lip texture and lip fullness; the facial feature information includes side facial features and front facial features.

[0135] Step S603: driving the virtual object based on the feature information.

[0136] In some embodiments, the carrier can create a blank virtual object (i.e., only limbs without details such as expressions and muscles), add the feature information to the blank virtual object to obtain the virtual object, and drive the virtual object based on the first action instruction.

[0137] In other embodiments, after creating the virtual object, the carrier may also cover the corresponding part of the virtual object based on the feature information to obtain the virtual object, and drive the virtual object based on the first action instruction.

[0138] For example, the feature information includes tooth feature information, and the virtual object is driven based on at least one of tooth color, tooth shape, tooth arrangement information, tooth defect information, and tooth exposure information obtained in the first video.

[0139] Alternatively, if the feature information includes lip feature information, the virtual object is driven based on at least one of the lip shape, lip size, lip color, lip thickness, lip texture and lip fullness obtained in the first video.

[0140] In this way, through the virtual object driving method provided by the embodiment of the present disclosure, more personalized feature information of the target object can be obtained from the first video, such as tooth feature information, side face feature information, etc., so that the character can be better modeled and driven.

[0141] Figure 7 An optional structural schematic diagram of a virtual object driving device provided in an embodiment of the present disclosure is shown, and will be described according to each part.

[0142] In some embodiments, the virtual object driving device 700 includes an acquisition unit 701 , a matching unit 702 , and a driving unit 703 .

[0143] The acquisition unit 701 is used to acquire a first video; the first video includes dynamic information of a target object;

[0144] The matching unit 702 is configured to obtain reference information corresponding to the first action instruction according to the first video based on the received first action instruction; the first action instruction is used to drive the virtual object to perform a target action corresponding to the first action instruction;

[0145] The driving unit 703 is configured to obtain an image corresponding to the first action instruction based on the first action instruction, the reference information and the virtual object.

[0146] The acquisition unit 701 is specifically used for one of the following:

[0147] Obtain the first uploaded video through the interface;

[0148] Acquire the first video through an image acquisition device and guidance information;

[0149] The guidance information includes text guidance information, animation guidance information and audio guidance information; the guidance information is used to instruct the target object to perform a corresponding action.

[0150] The matching unit 702 is specifically configured to obtain a target image by matching from the first video based on the first action instruction;

[0151] The reference information is obtained based on the target image.

[0152] The matching unit 702 is specifically configured to determine part information corresponding to the first action instruction, where the part information corresponds to a part of the virtual object;

[0153] A target sub-image corresponding to the part information is intercepted from the target image as the reference information.

[0154] The matching unit 702 is further configured to determine action parameter information according to the first action instruction;

[0155] According to the part information and the action parameter information, an image frame is matched from the first video as the target image.

[0156] The matching unit 702 is specifically configured to obtain a plurality of sub-image sets obtained according to the first video, each of the sub-image sets corresponding to a part of the target object;

[0157] Determine part information and action parameter information according to the first action instruction, wherein the part information corresponds to a part of the virtual object;

[0158] Determine a target sub-image set from the plurality of sub-image sets according to the part information;

[0159] A target sub-image is determined from the target sub-image set as the reference information according to the action parameter information.

[0160] The matching unit 702 is specifically configured to model the dynamic information included in the first video to obtain a target model;

[0161] An output image corresponding to the first action instruction is outputted by the target model as the reference information.

[0162] The matching unit 702 is specifically configured to determine the part information and the action parameter information according to the first action instruction;

[0163] Determining a target sub-model from the multiple sub-models according to the part information;

[0164] The output image is obtained according to the action parameter information and the target sub-model.

[0165] The driving unit 703 is specifically configured to obtain a sub-image corresponding to the reference information;

[0166] Acquire a main image corresponding to the first virtual image executing the first action instruction;

[0167] The sub-image is superimposed on the main image according to the position to obtain an image corresponding to the first action instruction.

[0168] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device and a readable storage medium.

[0169] Figure 8 A schematic block diagram of an example electronic device 800 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0170] like Figure 8 As shown, the electronic device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the electronic device 800 can also be stored. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0171] Multiple components in the electronic device 800 are connected to the I / O interface 805, including: an input unit 806, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a disk, an optical disk, etc.; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the electronic device 800 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0172] The computing unit 801 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 801 performs the various methods and processes described above, such as a virtual object driving method. For example, in some embodiments, the virtual object driving method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of the virtual object driving method described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to perform the virtual object driving method in any other appropriate manner (e.g., by means of firmware).

[0173] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0174] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0175] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0176] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0177] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0178] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0179] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0180] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of the present disclosure, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0181] The above is only a specific embodiment of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any person skilled in the art who is familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present disclosure, which should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. A virtual object driving method, the method comprising: Get the first video; The first video includes dynamic information of the target object; Based on the received first action instruction, obtaining reference information corresponding to the first action instruction according to the first video; The first action instruction is used to drive the virtual object to perform a target action corresponding to the first action instruction; Based on the first action instruction, the reference information and the virtual object, an image corresponding to the first action instruction is obtained.

2. The method according to claim 1, wherein obtaining the first video comprises one of the following: Acquire the first uploaded video through the interface; Acquire the first video through an image acquisition device and guidance information; The guidance information includes text guidance information, animation guidance information and audio guidance information; the guidance information is used to instruct the target object to perform a corresponding action.

3. The method according to claim 1, wherein based on the received first action instruction, obtaining reference information corresponding to the first action instruction according to the first video comprises: Based on the first action instruction, matching and obtaining a target image from the first video; The reference information is obtained based on the target image.

4. The method according to claim 3, wherein the target image is an image frame in the first video; The obtaining the reference information based on the target image comprises: Determine part information corresponding to the first action instruction, where the part information corresponds to a part of the virtual object; A target sub-image corresponding to the part information is intercepted from the target image as the reference information.

5. The method according to claim 4, further comprising: Determine action parameter information according to the first action instruction; According to the part information and the action parameter information, an image frame is matched from the first video as the target image.

6. The method according to claim 3, wherein obtaining the reference information based on the target image comprises: Obtaining a plurality of sub-image sets obtained according to the first video, each of the sub-image sets corresponding to a part of the target object; Determine part information and action parameter information according to the first action instruction, wherein the part information corresponds to a part of the virtual object; Determine a target sub-image set from the plurality of sub-image sets according to the part information; A target sub-image is determined from the target sub-image set as the reference information according to the action parameter information.

7. The method according to claim 1, wherein the step of obtaining reference information corresponding to the first action instruction from the first video based on the received first action instruction comprises: Modeling the dynamic information included in the first video to obtain a target model; An output image corresponding to the first action instruction is outputted through the target model as the reference information.

8. The method according to claim 7, wherein the target model includes a plurality of sub-models; and outputting an output image corresponding to the first action instruction through the target model as the reference information comprises: Determine the part information and the action parameter information according to the first action instruction; Determining a target sub-model from the multiple sub-models according to the part information; The output image is obtained according to the action parameter information and the target sub-model.

9. The method according to claim 1, wherein obtaining the image corresponding to the first action instruction based on the first action instruction, the reference information and the virtual object comprises: Acquire a sub-image corresponding to the reference information; Acquire a main image corresponding to the first virtual image executing the first action instruction; The sub-image is superimposed on the main image according to the position to obtain an image corresponding to the first action instruction.

10. A virtual object driving device, the device comprising: An acquisition unit, used for acquiring a first video; The first video includes dynamic information of the target object; a matching unit, configured to obtain reference information corresponding to the first action instruction according to the first video based on the received first action instruction; The first action instruction is used to drive the virtual object to perform a target action corresponding to the first action instruction; A driving unit is configured to obtain an image corresponding to the first action instruction based on the first action instruction, the reference information and the virtual object.

Images