Action control method and device of virtual object, equipment and storage medium

Abstract

The invention provides an action control method and device for a virtual object, equipment and a storage medium. The method comprises the steps that a second action executed next time is determined based on a first action currently executed by the virtual object; obtaining a target video slice identifier corresponding to the second action, and obtaining a target video slice of the virtual object executing the second action based on the target video slice identifier; determining at least one second video frame based on all the first video frames included in the target video slice; and sending the at least one second video frame to the front-end equipment for playing. According to the invention, the action switching delay of the virtual object can be reduced, and the action fluency of the virtual object is improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method, device, equipment and storage medium for controlling the motion of a virtual object. Background Art

[0002] With the rapid development of artificial intelligence technology, digital virtual humans have become an important branch of the field of artificial intelligence, and are widely used in customer service, education and training, entertainment interaction, etc. In these applications, the motion control of digital virtual humans is one of the key technologies to achieve a smooth interactive experience.

[0003] In the prior art, for each action, a dynamic image is generated based on the image sequence corresponding to the action, such as a GIF format, and then the dynamic images corresponding to each action are integrated based on a pre-defined action sequence to obtain a dynamic image resource. During the digital virtual human interaction process, the front-end device needs to download all these dynamic image resources to the local memory according to the preset action flow in order to perform streaming action output.

[0004] However, when the movements of the virtual digital human are updated or new movements are added, the front-end device needs to re-download the corresponding animated image resources, causing a delay in the movement switching of the virtual digital human, resulting in unsmooth movements of the virtual digital human. Summary of the invention

[0005] The present invention provides a method, device, equipment and storage medium for controlling the motion of a virtual object, so as to solve the defect in the prior art that when the motion of a virtual digital human is updated or a new motion is added, the motion switching of the virtual digital human is delayed, resulting in unsmooth motion of the virtual digital human, thereby reducing the motion switching delay of the virtual digital human and improving the motion smoothness of the virtual digital human.

[0006] The present invention provides a method for controlling a motion of a virtual object, comprising: Determining a second action to be performed next based on a first action currently performed by the virtual object; Obtaining a target video slice identifier corresponding to the second action, and obtaining a target video slice for the virtual object to perform the second action based on the target video slice identifier; Determine at least one second video frame based on all first video frames included in the target video slice; The at least one second video frame is sent to the front-end device for playback.

[0007] According to a method for controlling an action of a virtual object provided by the present invention, the method of determining a second action to be performed next based on a first action currently performed by the virtual object comprises: searching for a node corresponding to the first action in a preset action tree, and determining an action corresponding to a next node connected to the node as the second action; or,

[0008] When an action instruction input by a user and sent by a front-end device is received, the second action is determined based on the action instruction.

[0009] According to a method for controlling an action of a virtual object provided by the present invention, obtaining a target video slice for the virtual object to perform the second action based on the target video slice identifier includes: Obtaining the offset of each target key frame corresponding to the target video slice identifier from the index file; the index file includes the offset of each key frame corresponding to each video slice identifier; The starting position of the target video slice in the memory is obtained based on the offset of each target key frame, and the target video slice is obtained based on the starting position.

[0010] According to a method for controlling a motion of a virtual object provided by the present invention, the method further comprises: Get multiple video frames; Based on the actions of the virtual objects in the video frames, classifying all the video frames to obtain video frames corresponding to the actions; For each of the actions, a video frame corresponding to the action is determined as a video slice corresponding to the action.

[0011] According to a method for controlling a motion of a virtual object provided by the present invention, the step of acquiring a plurality of video frames comprises: Acquire at least one frame of initial image including the virtual object; For each frame of the initial image, obtaining color value information and transparency information of each pixel in the initial image; Encoding the color value information of each pixel to obtain first encoding information; Processing the transparency information of each pixel point, and encoding the processed transparency information of each pixel point to obtain second encoding information; Based on the first encoding information and the second encoding information, a video frame corresponding to the initial image is determined.

[0012] According to a method for controlling a motion of a virtual object provided by the present invention, the determining of at least one second video frame based on all first video frames included in the target video slice comprises: Acquire first position information of the virtual object included in the last video frame in the video slice corresponding to the first action; Determine a distance between second position information of a virtual object included in each key frame in all the first video frames and the first position information; A target key frame corresponding to the minimum distance and all video frames after the target key frame in the at least one first video frame are determined as the second video frames.

[0013] According to a method for controlling a motion of a virtual object provided by the present invention, sending the at least one second video frame to the front-end device for playing includes: When receiving the audio data sent by the front-end device, synchronizing the audio data and each of the second video frames based on the timestamp of the audio data and the timestamp of each of the second video frames to obtain synchronized audio and video; The synchronized audio and video are sent to the front-end device for playback.

[0014] The present invention also provides a motion control device for a virtual object, comprising: A determination module, configured to determine a second action to be performed next based on a first action currently performed by the virtual object; an acquisition module, configured to acquire a target video slice identifier corresponding to the second action, and acquire a target video slice for the virtual object to perform the second action based on the target video slice identifier; The determination module is further configured to determine at least one second video frame based on all first video frames included in the target video slice; A sending module is used to send the at least one second video frame to the front-end device for playback.

[0015] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method for controlling the motion of a virtual object as described above is implemented.

[0016] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method for controlling the motion of a virtual object as described in any one of the above is implemented.

[0017] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method for controlling the motion of a virtual object as described above is implemented.

[0018] The method, device, equipment and storage medium for controlling the action of a virtual object provided by the present invention determine the second action to be performed next based on the first action currently performed by the virtual object, obtain the target video slice identifier corresponding to the second action, obtain the target video slice for the virtual object to perform the second action based on the target video slice identifier, and determine at least one second video frame based on all first video frames included in the target video slice, and send at least one second video frame to the front-end device for playback. Since it is only necessary to determine the second action to be performed next by the virtual object and obtain at least one second video frame corresponding to the second action, only these second video frames need to be sent to the front-end device. Even if the action of the virtual object has changed or a new action has been added, it is only necessary to obtain the video frame corresponding to the changed or newly added second action to be performed, thereby avoiding the phenomenon of downloading all animated image resources, reducing the delay when the action of the virtual object is switched, and improving the action fluency of the virtual object. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0020] Figure 1 A schematic flow chart of a method for controlling a motion of a virtual object provided by an embodiment of the present invention.

[0021] Figure 2 A schematic diagram of the structure of an action tree provided by an embodiment of the present invention.

[0022] Figure 3 A schematic diagram of video slicing provided by an embodiment of the present invention.

[0023] Figure 4 A schematic diagram of an index file provided by an embodiment of the present invention.

[0024] Figure 5 A schematic diagram of image encoding provided by an embodiment of the present invention.

[0025] Figure 6 A schematic diagram of processing transparency information provided by an embodiment of the present invention.

[0026] Figure 7 A schematic diagram of the position of a virtual object provided by an embodiment of the present invention.

[0027] Figure 8 A schematic diagram of the structure of a video frame provided by an embodiment of the present invention.

[0028] Fig. 9 A schematic diagram of audio and video synchronization provided by an embodiment of the present invention.

[0029] Fig.10 A schematic diagram of newly added video slices provided in an embodiment of the present invention.

[0030] Fig.11 An architectural diagram of a method for controlling a virtual object's motion according to an embodiment of the present invention.

[0031] Fig.12 A schematic diagram of the structure of a virtual object motion control device provided by an embodiment of the present invention.

[0032] Fig.13 A schematic diagram of the physical structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0033] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0034] At present, for each action, a corresponding animated image will be produced in advance based on the image sequence corresponding to the action. For the digital virtual human interaction mode without lip drive, the produced animated images will be integrated according to the preset action order or action sequence to obtain the animated image resources corresponding to the action sequence. During the interaction process, the front-end device will download the animated image resources corresponding to the action sequence from the back-end server and add them to the local memory, and respond to the user's interactive instructions, and manage the loading and unloading of animated image resources in the content in real time to simulate streaming action output.

[0035] During the interaction process, if the action of the virtual digital human needs to be changed or a new action needs to be added to the virtual digital human, the previously downloaded dynamic image resources will no longer be applicable, and all dynamic image resources corresponding to the action sequence after the action is changed or added need to be downloaded again from the backend server. In this process, since all dynamic image resources need to be downloaded again, the action switching of the virtual digital human will be delayed, resulting in unsmooth movements of the virtual digital human.

[0036] In view of the above problems, an embodiment of the present invention provides a method for controlling the action of a virtual object, in which a second action to be performed next time can be determined based on the first action currently performed by the virtual object, and a target video slice corresponding to the second action can be obtained. Based on all the first video frames included in the target video slice, at least one second video frame can be determined, and at least one second video frame can be sent to a front-end device for playback to control the action of the virtual object. Since it is only necessary to determine the second action to be performed next time by the virtual object and obtain at least one second video frame corresponding to the second action, and only these second video frames need to be sent to the front-end device, even if the action of the virtual object has changed or a new action has been added, it is only necessary to obtain the video frame corresponding to the changed or newly added second action to be executed, thereby avoiding the phenomenon of downloading all animated image resources, reducing the delay when the action of the virtual object is switched, and improving the action smoothness of the virtual object.

[0037] Combine the following Figures 1 to 11 The method for controlling the motion of a virtual object provided by an embodiment of the present invention is described. The embodiment of the present invention can be applied to scenarios where digital virtual humans need to interact, and can be particularly applied to digital virtual humans interacting without lip drive. The execution subject of the method can be an electronic device such as a server, a server cluster, or a specially designed virtual object motion control device, or a virtual object motion control device set in the electronic device, and the virtual object motion control device can be implemented by software, hardware, or a combination of the two.

[0038] Figure 1 A flowchart of a method for controlling a virtual object's motion provided by an embodiment of the present invention is shown in FIG. Figure 1 As shown, the method includes: Step 101: Determine a second action to be performed next based on a first action currently performed by a virtual object.

[0039] In this step, the virtual object includes a digital virtual person, or other objects that need to be controlled by action. The first action currently performed by the virtual object can be understood as the action of the virtual object being played or displayed by the front-end device, and the server will record it when sending the video frame corresponding to the first action at the previous moment. It should be understood that for the first action of the virtual object, it can be a preset action or after the front-end device sends a request message to the server, the server sends the video frame corresponding to the first action to the front-end device, so that the front-end device controls the virtual object to perform the action by playing the video frame corresponding to the first action.

[0040] After obtaining the first action currently being performed by the virtual object, the server can determine the second action to be performed next time based on the preset interaction logic or rules. The interaction logic or rules may include, for example, the natural transition of actions, instructions input by the user, or the need to adapt to changes in the environment. For example, if the action currently being performed by the virtual object is "standing up", and the user asks a question through voice or text input, the second action to be performed next time is determined to be "nodding".

[0041] Step 102: Obtain a target video slice identifier corresponding to the second action, and obtain a target video slice for the virtual object to perform the second action based on the target video slice identifier.

[0042] In this step, the video slice corresponding to each action is pre-set in the material library, each video slice includes multiple video frames, and each video slice represents an independent and complete action of the virtual object, that is, the multiple video frames contained in the video slice can be combined to obtain a complete action of the virtual object. In addition, each video slice will be set with its corresponding video slice identifier to represent the action contained in the video slice, and the video slice corresponding to a specific action can be quickly located and retrieved in the material library through the identifier.

[0043] After determining the second action that the virtual object is about to perform, the material library is queried to obtain the target video slice identifier corresponding to the second action from the material library, and the target video slice corresponding to the target video slice identifier is obtained. The target video slice represents the second action performed by the virtual object.

[0044] Step 103: Determine at least one second video frame based on all first video frames included in the target video slice.

[0045] In this step, after acquiring all the first video frames included in the target video slice, the second video frame can be determined from the first video frame. In one possible implementation, all the first video frames can be directly determined as the second video frames, so that the integrity of the virtual object action can be guaranteed. In another possible implementation, part of the second video frames can be selected from the first video frames according to the position of the virtual object in each first video frame, and the position of the virtual object in the selected part of the second video frame is close to the position of the virtual object in the video frame corresponding to the first action, so that when the front-end device plays the video frame, the virtual object switches from the first action to the second action, and the position teleportation phenomenon occurs. Of course, at least one second video frame can also be selected from all the first video frames according to the real-time feedback of the user or the changes in the environment.

[0046] Step 104: Send at least one second video frame to a front-end device for playback.

[0047] In this step, after determining at least one second video frame from all the first video frames, the server can send the at least one second video frame to the front-end device, wherein the front-end device includes a smart phone, a tablet computer or a terminal, etc. The player or rendering engine on the front-end device processes the received second video frame and displays it, so that the virtual object will perform the second action.

[0048] It should be noted that at the next moment, the second action will become the new first action currently being performed by the virtual object. The server will re-determine the new second action that the virtual object is about to perform based on the new first action, and thereby obtain the target video slice corresponding to the new second action in the manner described in the aforementioned embodiment, thereby achieving continuous action control of the virtual object.

[0049] The action control method of a virtual object provided by an embodiment of the present invention determines the second action to be performed next based on the first action currently performed by the virtual object, obtains the target video slice identifier corresponding to the second action, obtains the target video slice for the virtual object to perform the second action based on the target video slice identifier, and determines at least one second video frame based on all first video frames included in the target video slice, and sends at least one second video frame to the front-end device for playback. Since it is only necessary to determine the second action to be performed next by the virtual object and obtain at least one second video frame corresponding to the second action, only these second video frames need to be sent to the front-end device. Even if the action of the virtual object has changed or a new action has been added, it is only necessary to obtain the video frame corresponding to the changed or newly added second action to be performed, thereby avoiding the phenomenon of downloading all animated image resources, reducing the delay when the action of the virtual object is switched, and improving the action fluency of the virtual object.

[0050] In addition, in this embodiment, since it is not necessary to download all the animated image resources, the server only needs to send the video frame corresponding to the second action to be executed next time to the front-end device. Therefore, it can not only reduce the memory usage of the front-end device and the system overhead of the front-end device, but also support high concurrency, that is, it can support the scene where multiple front-end devices control the virtual object at the same time. In addition, when the action of the virtual object changes or a new action is added, only the video slice corresponding to the changed action or the new action needs to be added, and the video slices of other actions can still be used. Therefore, the reusability of resources is improved, and the flexibility of scene or virtual object action expansion is higher.

[0051] Exemplarily, based on the above embodiments, when determining the second action to be performed next based on the first action currently performed by the virtual object, it is possible to search for the node corresponding to the first action in the preset action tree, and determine the action corresponding to the next node connected to the node as the second action; or, when receiving an action instruction input by the user sent by the front-end device, determine the second action based on the action instruction.

[0052] Specifically, Figure 2 A schematic diagram of the structure of an action tree provided by an embodiment of the present invention, such as Figure 2 As shown, the switching order between actions can be set in advance according to the interaction logic of the virtual object, a preset action tree can be generated, and the generated preset action tree can be pre-loaded into the memory of the server. Among them, the interaction logic can include, for example, the natural transition of the virtual object action or the user's historical interaction mode. The generated preset action tree is a data structure used to define all actions that the virtual object may perform and the conversion relationship between them. Figure 2 A, B and C in the figure represent different nodes respectively. Each node represents an action, and the action has its corresponding video slice.

[0053] After obtaining the first action currently executed by the virtual object, the server searches for the node corresponding to the first action based on the pre-loaded preset action tree. If the node corresponding to the first action is found to be A, the action corresponding to the next node connected to the node corresponding to the first action in the preset action tree is determined as the second action to be executed by the virtual object. If the next node connected to node A is C, the action corresponding to node C is determined as the second action.

[0054] Among them, the second action can be quickly determined through the preset action tree, and the determined second action will make the overall behavior of the virtual object smoother when switching from the first action to the second action, which can improve the user's visual experience.

[0055] In addition, the user may also input an action instruction to the front-end device through voice or text, and the front-end device will send the received action instruction to the server, thereby determining the second action based on the action instruction. For example, during the interaction between the user and the virtual object, the user may input "goodbye" through voice, and the front-end device will send an action instruction to the server, and the action instruction includes "goodbye", so the server will use "goodbye" as the second action.

[0056] It should be understood that the action instruction can also be an instruction generated by the front-end device or server based on the voice or text input by the user. For example, if the user inputs the voice "Do you think so?", the front-end device or server can generate an action instruction representing "nodding" based on this information, and use "nodding" as the second action.

[0057] Among them, when an action instruction sent by the front-end device is received, a second action can be determined based on the action instruction, so as to flexibly and naturally respond to user needs and provide a richer and more dynamic interactive experience.

[0058] Exemplarily, based on the above embodiments, when obtaining the target video slice for the virtual object to perform the second action based on the target video slice identifier, the offset of each target key frame corresponding to the target video slice identifier can be obtained from the index file, and the index file includes the offset of each key frame corresponding to each video slice identifier, and the starting position of the target video slice in the memory is obtained based on the offset of each target key frame, and the target video slice is obtained based on the starting position.

[0059] Specifically, after determining the video slice corresponding to each action, it is also necessary to record the offset of at least one key frame included in each video slice. Figure 3 A schematic diagram of a video slice provided by an embodiment of the present invention is shown in FIG. Figure 3 As shown, for video slice i, the offset of the first key frame included in the recorded video slice i is 0, and the offset of the second key frame is 28103. The offset can be understood as the offset of the video frame relative to the start position of the first video slice, that is, the offset relative to the first video frame in the first video slice. In addition, a delimiter can be set between two adjacent video frames in the video slice, and the end position of each video frame is determined by the delimiter.

[0060] Figure 4 A schematic diagram of an index file provided by an embodiment of the present invention, such as Figure 4 As shown, the offsets of the key frames of all video slices are stored in the form of a structure to obtain an index file. The index file includes multiple video slice identifiers, such as video slice 1, video slice 2... video slice n. In addition, the index file also includes the offsets of each key frame contained in the video slice corresponding to each video slice identifier, wherein the offset of the first key frame of the first video slice is 0, and the offset of each key frame of the subsequent video slices needs to be superimposed on the size of all the previous video slices. In addition, a delimiter can be set between each video frame in each video slice included in the index file to identify the end position of each video frame.

[0061] In order to reduce system overhead, the determined index file can also be loaded into the memory of the server in advance. After the target video slice identifier is determined, the offset of each target key frame corresponding to the target video slice identifier can be queried from the index file pre-loaded in the memory, so as to determine the starting position of the target video slice in the memory based on the offset of each target key frame, and obtain the target video slice based on the starting position.

[0062] In this embodiment, by obtaining the offsets of each target key frame corresponding to the target video slice identifier from the index file, the target video slice corresponding to the target video slice identifier can be accurately and quickly obtained based on these offsets, thereby improving the efficiency and accuracy of obtaining the target video slice.

[0063] Exemplarily, based on the above embodiments, when determining the video slice corresponding to each action, it can be done in the following way: obtain multiple video frames, classify all video frames based on the action of the virtual object in each video frame, obtain the video frame corresponding to each action, and for each action, determine the video frame corresponding to the action as the video slice corresponding to the action.

[0064] Specifically, multiple video frames can be obtained by real-time capture or pre-recording, each of which contains the action of the virtual object. The action of the virtual object in each video frame can be identified by key point detection, posture estimation or deep learning model.

[0065] After identifying the action of the virtual object in each video frame, multiple video frames that complete the same independent and complete action will be classified according to the boundary range of each action. For example, all video frames representing "waving" actions are classified into one category, and all video frames representing "dancing" actions are classified into one category. In this way, the video frames corresponding to each action can be obtained.

[0066] For each action, all video frames corresponding to the action can be combined into a complete video segment as the video slice corresponding to the action. Therefore, a video slice can represent the complete process of a virtual object completing an independent action, for example, the entire sequence from the start of waving to the end of waving.

[0067] In this embodiment, after all video frames are classified based on the actions of virtual objects in each video frame, the classified video frames corresponding to each action can be determined as video slices of the corresponding action. This method can not only improve the management efficiency of virtual object action data, but also enhance the flexibility of virtual objects in the interaction process, providing a richer and more realistic interactive experience.

[0068] Exemplarily, based on the above-mentioned embodiment, when acquiring multiple video frames, at least one initial image frame including a virtual object can be acquired first, and for each initial image frame, the color value information and transparency information of each pixel in the initial image can be acquired, and the color value information of each pixel can be encoded to obtain first encoding information, and then the transparency information of each pixel can be processed, and the processed transparency information of each pixel can be encoded to obtain the second encoding information, and then the video frame corresponding to the initial image can be determined based on the first encoding information and the second encoding information.

[0069] Specifically, at least one frame of initial image including the virtual object can be extracted from the virtual object data source, wherein the virtual object data source is a video or image set obtained after pre-shooting and editing. For each frame of the initial image obtained, the initial image is converted into an RGBA data format, and the color value information and transparency information of each pixel in the initial image are extracted based on the converted format, wherein the color value information is usually represented by an RGB model, which includes the intensity values ​​of red, green and blue, and the transparency information is usually represented by an Alpha channel, which characterizes the opacity of each pixel.

[0070] Figure 5 A schematic diagram of image encoding provided by an embodiment of the present invention, such as Figure 5 As shown, the color value information and transparency information of each pixel are divided into two channels, namely, the image channel and the Alpha channel, and the image is converted into an I420 structure based on the two channels and encoded into a WebM Project Video Codec (VPX). The color value information is encoded to obtain the first encoding information.

[0071] Currently, for scenes that require the background of virtual objects to be transparent, the front-end device usually needs to perform pixel-level transparency processing on each image, resulting in large front-end software and hardware resource overhead and high risk of CPU and UI blocking. Figure 6 A schematic diagram of processing transparency information provided by an embodiment of the present invention, such as Figure 6 As shown, for the transparency information of each pixel, the server can perform secondary calculations or customized preprocessing separately, such as matching cropping, contour cutting, anti-aliasing and other graphics optimization algorithms for actual scenes, so as to support complex 2D real-person interactive actions, such as "dancing", "number 1", "bowing", etc. After processing the transparency information of each pixel, the processed transparency information of each pixel is encoded to obtain the second encoding information, which can greatly reduce the resource overhead of the front-end device and reduce the risk of CPU and UI blocking.

[0072] Furthermore, the obtained first encoding information and the second encoding information may be combined to obtain a video frame corresponding to the initial image.

[0073] In this embodiment, for each initial image obtained, the color value information and transparency information of each pixel in the initial image can be encoded respectively. In this way, in the scene where background transparent rendering is required, the transparency information is processed and calculated by the server, and the front-end device can directly process based on the video frame, which reduces the calculation amount of the front-end device and reduces the system resource overhead of the front-end device. In addition, since the transparency information of the initial image can be processed separately, the flexibility of the virtual object movement is improved.

[0074] Exemplarily, based on the above embodiments, when determining at least one second video frame based on all first video frames included in the target video slice, it can be performed in the following manner: Obtain the first position information of the virtual object included in the last video frame in the video slice corresponding to the first action, and determine the distance between the second position information of the virtual object included in each key frame in all the first video frames and the first position information, and determine the target key frame corresponding to the minimum distance and all video frames after the target key frame in at least one first video frame as the second video frame.

[0075] Specifically, Figure 7 A schematic diagram of the position of a virtual object provided by an embodiment of the present invention, such as Figure 7 As shown, while analyzing the initial image and obtaining the color value information and transparency information of each pixel, it is also necessary to calculate and record the position information (X, Y) of the virtual object in the initial image.

[0076] Figure 8 A schematic diagram of the structure of a video frame provided by an embodiment of the present invention, such as Figure 8 As shown, an additional field of the frame header can be added to the first preset number of video frames and the last preset number of video frames of each video slice, and the additional field includes whether there are coordinates of a virtual object, that is, position information, whether it is a key frame, the timestamp corresponding to the video frame, and the coordinate header, wherein the timestamp can be determined according to the actual frame rate and clock scale. Since the key frames in the first preset number of video frames and the last preset number of video frames of each video slice are relatively dense, adding the position information of the virtual object in the additional field of the header of these video frames can ensure a smooth frame skipping effect of the virtual object displacement.

[0077] In addition, in order to further avoid the situation where the virtual object moves instantly, the position of the virtual object in each key frame in each video slice may also be determined and added to the additional field of the header.

[0078] In the prior art, since it is impossible to constrain the position of the virtual object, it is easy to cause teleportation due to excessive displacement deviation of the virtual object before and after the action switch. In order to solve this problem, in this embodiment, the first position information of the virtual object included in the last video frame in the video slice corresponding to the first action currently being performed by the virtual object can be obtained, that is, the position of the virtual object after performing the first action is determined. After determining the second action to be performed, the second position of the virtual object in the additional field of the header of each key frame in each first video frame can be obtained, and the distance between the first position and each second position can be calculated, and the minimum distance can be determined from multiple distances, and the target key frame corresponding to the minimum distance and all video frames after the target key frame are determined as the second video frame that needs to be sent to the front-end device. In this way, the front-end device can start playing from the target key frame, thereby reducing the displacement difference of the virtual object.

[0079] In this embodiment, by obtaining the first position information of the virtual object contained in the last video frame in the video slice corresponding to the first action, and searching for the target video frame from the video slice corresponding to the second action, the distance between the second position information of the virtual object in the target video frame and the first position information is the smallest. Therefore, after the front-end device plays the last video frame, it will immediately play the target video frame, thereby greatly reducing the displacement change of the virtual object and preventing the virtual object from teleporting.

[0080] Exemplarily, based on the above-mentioned embodiments, when at least one second video frame is sent to the front-end device for playback, upon receiving the audio data sent by the front-end device, the audio data and each second video frame may be synchronized based on the timestamp of the audio data and the timestamp of each second video frame to obtain the synchronized audio and video, and the synchronized audio and video may be sent to the front-end device for playback.

[0081] Specifically, Figure 5 As shown, when encoding the original image, a timestamp may be set for each video frame according to the actual frame rate and the clock scale. Fig. 9 A schematic diagram of audio and video synchronization provided by an embodiment of the present invention, such as Fig. 9As shown in the figure, during the transmission of the real-time video slice stream, if the server receives the audio data sent by the front-end device, it will process the audio data into the Opus format and store it in each buffer of the circular queue. The circular queue includes N buffers, such as T1, T2...Tn. After that, the audio data will be extracted from each buffer of the circular queue according to the timestamp of each video frame, so as to synchronize the audio and video, and encapsulate the synchronized audio and video to obtain the webW video slice, which will be sent to the front-end device in real time through the same channel. The front-end device will play the webW video slice to achieve the consistency of voice broadcast and the performance action of the virtual object.

[0082] In this embodiment, when the audio data sent by the front-end device is received, the audio data and each second video frame will be synchronized based on the timestamp of the audio data and the timestamp of each second video frame. In this way, when the front-end device performs voice broadcast and video playback, it can ensure that the voice and the movement of the virtual object are consistent, thereby improving the visual effect.

[0083] It needs to be explained that Fig.10 A schematic diagram of a newly added video slice provided in an embodiment of the present invention, such as Fig.10 As shown, when the scene of the virtual object changes, such as adding a video slice corresponding to a new action or changing the video slice, so that the switching logic of the action changes, it is only necessary to load the new video slice at the end of the original video slice file group memory on the server, such as Fig.10 The video slice 16 in the video slice 16 and the action tree and index file can be refreshed. In this way, the front-end device does not need to download or perceive the video slice resources, and the video slice resources can be reused by multiple users concurrently. It is suitable for high-concurrency scenarios processed by pure central processing unit (CPU).

[0084] Fig.11 The architecture diagram of the method for controlling the motion of a virtual object provided by an embodiment of the present invention is as follows: Fig.11As shown, each user's session request will open a session for tracking and processing the user's interaction process. The session records the video slice ID corresponding to the first action currently executed by the virtual object and its two subscript indexes La and Lb in the action tree and index file, where La represents the video slice ID to be switched, that is, the ID of the next video slice, and Lb represents the memory pointer position to be moved, that is, the memory pointer of the next video slice. Front-end device 1 sends a request message to the server, which includes the first action currently executed by the virtual object in front-end device 1, and session 1 is maintained between front-end device 1 and the server. Front-end device 2 sends a request message to the server, which includes the first action currently executed by the virtual object in front-end device 2, and session 2 is maintained between front-end device 2 and the server.

[0085] Taking the example of the front-end device 1 sending a request message to the server, the server determines the ID 1 of the video slice corresponding to the first action currently performed by the virtual object from session 1, and queries the ID of the next video slice according to the action tree. After the ID of the next video slice is queried, the offset of each key frame corresponding to the ID of the next video slice will be queried from the file index, where the position of the virtual object in the returned key frame is the closest to the position of the virtual object in the last video frame in the current video slice. After finding the key frame with the closest position, all video frames in the next video slice starting from this key frame are regarded as video frames that need to be returned to the front-end device, and response information will be returned to the front-end device, and the response information includes the above-mentioned video frames found.

[0086] The following is a description of a motion control device for a virtual object provided by the present invention. The motion control device for a virtual object described below and the motion control method for a virtual object described above can be referred to each other.

[0087] Fig.12 A schematic diagram of a virtual object motion control device according to an embodiment of the present invention, referring to Fig.12 As shown, the motion control device 1200 of the virtual object includes: A determination module 11, configured to determine a second action to be performed next based on a first action currently performed by the virtual object; An acquisition module 12 is used to acquire a target video slice identifier corresponding to the second action, and acquire a target video slice for the virtual object to perform the second action based on the target video slice identifier; The determination module 11 is further configured to determine at least one second video frame based on all first video frames included in the target video slice; The sending module 13 is used to send the at least one second video frame to the front-end device for playback.

[0088] In an exemplary embodiment, the determination module 11 is specifically configured to: searching for a node corresponding to the first action in a preset action tree, and determining an action corresponding to a next node connected to the node as the second action; or,

[0089] When an action instruction input by a user and sent by a front-end device is received, the second action is determined based on the action instruction.

[0090] In an exemplary embodiment, the acquisition module 12 is specifically configured to: Obtaining the offset of each target key frame corresponding to the target video slice identifier from the index file; the index file includes the offset of each key frame corresponding to each video slice identifier; The starting position of the target video slice in the memory is obtained based on the offset of each target key frame, and the target video slice is obtained based on the starting position.

[0091] In an example embodiment, the apparatus further comprises a classification module, wherein: The acquisition module 12 is also used to acquire multiple video frames; A classification module, used for classifying all the video frames based on the actions of the virtual objects in the video frames to obtain video frames corresponding to the actions; The determination module 11 is further configured to determine, for each of the actions, a video frame corresponding to the action as a video slice corresponding to the action.

[0092] In an exemplary embodiment, the acquisition module 12 is specifically configured to: Acquire at least one frame of initial image including the virtual object; For each frame of the initial image, obtaining color value information and transparency information of each pixel in the initial image; Encoding the color value information of each pixel to obtain first encoding information; Processing the transparency information of each pixel point, and encoding the processed transparency information of each pixel point to obtain second encoding information; Based on the first encoding information and the second encoding information, a video frame corresponding to the initial image is determined.

[0093] In an exemplary embodiment, the determination module 11 is specifically configured to: Acquire first position information of the virtual object included in the last video frame in the video slice corresponding to the first action; Determine a distance between second position information of a virtual object included in each key frame in all the first video frames and the first position information; A target key frame corresponding to the minimum distance and all video frames after the target key frame in the at least one first video frame are determined as the second video frames.

[0094] In an exemplary embodiment, the sending module 13 is specifically configured to: When receiving the audio data sent by the front-end device, synchronizing the audio data and each of the second video frames based on the timestamp of the audio data and the timestamp of each of the second video frames to obtain synchronized audio and video; The synchronized audio and video are sent to the front-end device for playback.

[0095] The device of this embodiment can be used to execute the method of any embodiment in the embodiment of the method for controlling the action of a virtual object. Its specific implementation process and technical effects are similar to those in the embodiment of the method for controlling the action of a virtual object. For details, please refer to the detailed introduction in the embodiment of the method for controlling the action of a virtual object, which will not be repeated here.

[0096] Fig.13 A schematic diagram of the physical structure of an electronic device provided by an embodiment of the present invention, such as Fig.13 As shown, the electronic device may include: a processor 1310, a communication interface 1320, a memory 1330 and a communication bus 1340, wherein the processor 1310, the communication interface 1320 and the memory 1330 communicate with each other through the communication bus 1340. The processor 1310 may call the logic instructions in the memory 1330 to execute the action control method of the virtual object, the method comprising: determining the second action to be executed next based on the first action currently executed by the virtual object; obtaining the target video slice identifier corresponding to the second action, and obtaining the target video slice for the virtual object to execute the second action based on the target video slice identifier; determining at least one second video frame based on all the first video frames included in the target video slice; and sending the at least one second video frame to the front-end device for playback.

[0097] In addition, the logic instructions in the above-mentioned memory 1330 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0098] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the virtual object action control method provided by the above methods, and the method includes: determining a second action to be performed next based on a first action currently performed by the virtual object; obtaining a target video slice identifier corresponding to the second action, and obtaining a target video slice for the virtual object to perform the second action based on the target video slice identifier; determining at least one second video frame based on all first video frames included in the target video slice; and sending the at least one second video frame to the front-end device for playback.

[0099] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, it is implemented to execute the action control method of the virtual object provided by the above-mentioned methods, the method comprising: determining a second action to be performed next based on a first action currently performed by the virtual object; obtaining a target video slice identifier corresponding to the second action, and obtaining a target video slice for the virtual object to perform the second action based on the target video slice identifier; determining at least one second video frame based on all first video frames included in the target video slice; and sending the at least one second video frame to the front-end device for playback.

[0100] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0101] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0102] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for controlling a motion of a virtual object, characterized in that: include: Determining a second action to be performed next based on a first action currently performed by the virtual object; Obtaining a target video slice identifier corresponding to the second action, and obtaining a target video slice for the virtual object to perform the second action based on the target video slice identifier; Determine at least one second video frame based on all first video frames included in the target video slice; The at least one second video frame is sent to a front-end device for playback.

2. The method for controlling the motion of a virtual object according to claim 1, characterized in that: The determining of a second action to be performed next based on a first action currently performed by the virtual object comprises: searching for a node corresponding to the first action in a preset action tree, and determining an action corresponding to a next node connected to the node as the second action; or, When an action instruction input by a user and sent by a front-end device is received, the second action is determined based on the action instruction.

3. The method for controlling the motion of a virtual object according to claim 1, characterized in that: The acquiring, based on the target video slice identifier, a target video slice for the virtual object to perform the second action comprises: Obtaining the offset of each target key frame corresponding to the target video slice identifier from the index file; the index file includes the offset of each key frame corresponding to each video slice identifier; The starting position of the target video slice in the memory is obtained based on the offset of each target key frame, and the target video slice is obtained based on the starting position.

4. The method for controlling the motion of a virtual object according to claim 1, characterized in that: The method further comprises: Get multiple video frames; Based on the actions of the virtual objects in the video frames, classifying all the video frames to obtain video frames corresponding to the actions; For each of the actions, a video frame corresponding to the action is determined as a video slice corresponding to the action.

5. The method for controlling the motion of a virtual object according to claim 4, characterized in that: The acquiring of multiple video frames comprises: Acquire at least one frame of initial image including the virtual object; For each frame of the initial image, obtaining color value information and transparency information of each pixel in the initial image; Encoding the color value information of each pixel to obtain first encoding information; Processing the transparency information of each pixel point, and encoding the processed transparency information of each pixel point to obtain second encoding information; Based on the first encoding information and the second encoding information, a video frame corresponding to the initial image is determined.

6. The method for controlling the motion of a virtual object according to claim 1, characterized in that: The determining at least one second video frame based on all first video frames included in the target video slice comprises: Acquire first position information of the virtual object included in the last video frame in the video slice corresponding to the first action; Determine a distance between second position information of a virtual object included in each key frame in all the first video frames and the first position information; A target key frame corresponding to the minimum distance and all video frames after the target key frame in the at least one first video frame are determined as the second video frames.

7. The method for controlling the motion of a virtual object according to any one of claims 1 to 6, characterized in that: The sending the at least one second video frame to the front-end device for playing includes: When receiving the audio data sent by the front-end device, synchronizing the audio data and each of the second video frames based on the timestamp of the audio data and the timestamp of each of the second video frames to obtain synchronized audio and video; The synchronized audio and video are sent to the front-end device for playback.

8. A motion control device for a virtual object, characterized in that: include: A determination module, configured to determine a second action to be performed next based on a first action currently performed by the virtual object; an acquisition module, configured to acquire a target video slice identifier corresponding to the second action, and acquire a target video slice for the virtual object to perform the second action based on the target video slice identifier; The determination module is further configured to determine at least one second video frame based on all first video frames included in the target video slice; The sending module is used to send the at least one second video frame to the front-end device for playback.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method for controlling the motion of a virtual object according to any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for controlling the motion of a virtual object according to any one of claims 1 to 7 is implemented.

Images