A video processing method, apparatus, device, medium and product

Abstract

The present disclosure provides a video processing method, device, equipment, medium and product, relating to the technical field of video processing, which comprises: acquiring a first image frame of a video to be processed; identifying the first image frame according to a first pose recognition algorithm to obtain an overall pose of a user; when the confidence of the overall pose is lower than a preset threshold, identifying the first image frame according to a second pose recognition algorithm to obtain a local pose of the user; and then updating the first image frame by using a three-dimensional local model of a virtual object according to the local pose, wherein the updated first image frame comprises a local part of the virtual object. In this way, the method can meet the shooting needs of the user and improve the shooting experience of the user.

Description

Technical Field

[0001] This disclosure relates to the field of video processing technology, and in particular to a video processing method, apparatus, device, computer-readable storage medium, and computer program product. Background Technology

[0002] With the continuous development of video processing technology, short video applications (APPs) have emerged. As a type of entertainment application, short video applications are becoming increasingly popular among users. These short video applications can be installed on mobile phones and other terminals, which can then publish or play short videos.

[0003] To enhance the quality and entertainment value of short videos captured by short video apps, virtual object effects are often added. For example, by recognizing the user's body movements in the video and using a 3D model of a virtual object, the virtual object can follow the user's movements, effectively making the user "transform" into the virtual object. These virtual objects can be virtual cartoon characters, such as cartoon figures or animals.

[0004] In real-world scenarios, most short videos shot using short video apps are close-up selfies. However, close-up selfies often miss some of the user's body movements; for example, they may not include the movements of the user's legs or feet. During the recognition process for close-up selfies, the user's body movements may not be recognized, causing virtual object effects to disappear from the video.

[0005] Therefore, the above-mentioned methods of adding virtual object effects to short videos are relatively limited, making it difficult to meet users' shooting needs and reducing their shooting experience. Summary of the Invention

[0006] The purpose of this disclosure is to provide a video processing method, apparatus, device, computer-readable storage medium, and computer program product that can improve the user's shooting experience.

[0007] In a first aspect, this disclosure provides a video processing method, the method comprising: acquiring a first image frame of a video to be processed; identifying the first image frame according to a first pose recognition algorithm to obtain the overall pose of a user; when the confidence level of the overall pose is lower than a preset threshold, identifying the first image frame according to a second pose recognition algorithm to obtain the local pose of the user; and updating the first image frame according to the local pose using a three-dimensional local model of a virtual object, wherein the updated first image frame includes a local part of the virtual object.

[0008] Secondly, this disclosure provides a video processing apparatus, the apparatus comprising:

[0009] The acquisition module is used to acquire the first image frame of the video to be processed;

[0010] The recognition module is used to recognize the first image frame according to the first pose recognition algorithm to obtain the user's overall pose; when the confidence level of the overall pose is lower than a preset threshold, the first image frame is recognized according to the second pose recognition algorithm to obtain the user's local pose.

[0011] An update module is used to update the first image frame based on the local pose using a three-dimensional local model of the virtual object, wherein the updated first image frame includes a local part of the virtual object.

[0012] Thirdly, this disclosure provides a computer-readable medium having a computer program stored thereon, which, when executed by a processing device, implements the steps of the method described in any of the first aspects of this disclosure.

[0013] Fourthly, this disclosure provides an electronic device, comprising:

[0014] A storage device on which computer programs are stored;

[0015] A processing apparatus for executing the computer program in the storage device to implement the steps of the method according to any one of the first aspects of this disclosure.

[0016] Fifthly, this disclosure provides a computer program product containing instructions that, when run on a device, causes the device to perform the method described in any of the implementations of the first aspect above.

[0017] As can be seen from the above technical solutions, this disclosure has the following advantages:

[0018] This method provides multiple pose recognition algorithms. After acquiring image frames from the video to be processed, a first pose recognition algorithm is used to identify the image frame and obtain the user's overall pose. When the confidence level of this overall pose is lower than a preset threshold, it indicates that the user's overall limb movements cannot be recognized (e.g., only the user's head movements are recognized). Then, a second pose recognition algorithm is used to identify the image frame and obtain the user's local pose. Next, based on this local pose, the image frame is updated using a 3D local model of a virtual object, so that the updated image frame includes the local part of the virtual object. In this way, this method can render virtual objects based on the 3D model of the virtual object, even when the user's overall limb movements cannot be recognized, by utilizing the user's local (e.g., head) pose, thus meeting the user's shooting needs and improving the user's shooting experience.

[0019] Other features and advantages of this disclosure will be described in detail in the following detailed description section. Attached Figure Description

[0020] To more clearly illustrate the technical methods of the embodiments of this disclosure, the accompanying drawings used in the embodiments will be briefly described below.

[0021] Figure 1 A flowchart of a video processing method provided in an embodiment of this disclosure;

[0022] Figure 2A A schematic diagram of a client-side shooting interface provided in an embodiment of this disclosure;

[0023] Figure 2B A schematic diagram illustrating a plurality of candidate props provided in an embodiment of this disclosure;

[0024] Figure 3 A schematic diagram of key points of the human body provided in an embodiment of this disclosure;

[0025] Figure 4 A schematic diagram of key points in the head provided in an embodiment of this disclosure;

[0026] Figure 5A A schematic diagram of an updated first image frame provided in an embodiment of this disclosure;

[0027] Figure 5B A schematic diagram of another updated first image frame provided in an embodiment of this disclosure;

[0028] Figure 5C A schematic diagram of yet another updated first image frame provided in an embodiment of this disclosure;

[0029] Figure 6A A schematic diagram of an image frame associated with a virtual object provided in an embodiment of this disclosure;

[0030] Figure 6B A schematic diagram of another image frame associated with a virtual object provided in an embodiment of this disclosure;

[0031] Figure 7A A schematic diagram illustrating a preset action provided in an embodiment of this disclosure;

[0032] Figure 7B A schematic diagram illustrating a target special effect provided in an embodiment of this disclosure;

[0033] Figure 8 A schematic diagram of a video recording interface provided in an embodiment of this disclosure;

[0034] Figure 9 A schematic diagram of a video processing apparatus provided in an embodiment of this disclosure;

[0035] Figure 10 This is a schematic diagram of the structure of a device provided in an embodiment of the present disclosure. Detailed Implementation

[0036] The terms "first" and "second" used in the embodiments of this disclosure are for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature.

[0037] First, some technical terms involved in the embodiments of this disclosure will be introduced.

[0038] Virtual objects refer to virtual cartoon characters, such as cartoon figures and animals. Adding effects to these virtual objects in short videos allows them to move in sync with the user, effectively making the user "transform" into the virtual object. This enhances the quality and entertainment value of short videos created with short video apps.

[0039] Specifically, the system first identifies the user's body movements in the image frame, then renders the virtual object using a 3D model, and adds the virtual object to the image frame using a texture. However, in some cases, not all of the user's body movements may be recognized, such as in close-up selfies or half-body shots.

[0040] Taking close-up selfies as an example, short videos of close-up selfies generally do not include the user's leg and foot movements. After recognizing these short videos, there were instances where the user's body movements could not be recognized, causing the virtual object effects to disappear from the short video. It is evident that the methods described above for adding virtual object effects to short videos are rather limited, failing to meet users' shooting needs and reducing their shooting experience.

[0041] In view of the above, this disclosure provides a video processing method, which includes: acquiring a first image frame of a video to be processed; then recognizing the first image frame according to a first pose recognition algorithm to obtain the overall pose of a user; when the confidence level of the overall pose is lower than a preset threshold, recognizing the first image frame using a second pose recognition algorithm to obtain the local pose of the user; then updating the first image frame according to the local pose using a three-dimensional local model of a virtual object, so that the updated first image frame includes the local part of the virtual object; wherein the three-dimensional local model is a three-dimensional model corresponding to the local part of the user.

[0042] As can be seen, this method can render virtual objects based on the three-dimensional model of the virtual object by utilizing the user's local pose (such as the head) even when the user's overall body movements cannot be recognized, thus meeting the user's shooting needs and improving the user's shooting experience.

[0043] This method can be applied to the client side of short video applications, such as those installed on mobile phones and other terminals. When applied to the client side of a short video application, the method is specifically implemented as a computer program. In some embodiments, the computer program can be independent, for example, it can be a standalone application with corresponding functions. In other embodiments, the computer program can be a functional module or plug-in, attached to an existing short video application or other applications and running there.

[0044] The video processing method provided in this disclosure can be executed independently by a client (e.g., a client of a short video application) or collaboratively by a client and a server (e.g., a server of a short video application). When the video processing method is executed independently by the client, it indicates that the short video application can run offline. The client can be deployed on a terminal, including but not limited to smartphones, tablets, laptops, personal digital assistants (PDAs), or smart wearable devices. The server can be a cloud server, such as a central server in a central cloud computing cluster or an edge server in an edge cloud computing cluster. Alternatively, the server can be a server in a local data center. A local data center refers to a data center directly controlled by the user.

[0045] To facilitate understanding, the following example illustrates how video processing methods are executed by the client.

[0046] like Figure 1 As shown, this figure is a flowchart of a video processing method provided in an embodiment of this disclosure, the method including:

[0047] S101: The client obtains the first image frame of the video to be processed.

[0048] The video to be processed can be a video captured by a client (such as a client for shooting short video applications). Figure 2A As shown in the figure, this figure is a schematic diagram of a client's shooting interface provided in an embodiment of this disclosure.

[0049] The shooting interface includes a shooting control 211, a prop control 220, and a photo album control 230. The shooting control 211 is used to start recording video. Users can trigger the shooting operation through this control. After the client receives the shooting operation, it begins recording video, for example, by encoding the video stream captured by the terminal to obtain a video file.

[0050] It should be noted that the video to be processed can be a video file corresponding to the video file captured in the above manner, or a video previewed after the client enters the shooting interface (e.g., a video stream captured by the camera), or a pre-captured video file. When the video to be processed is a pre-captured video file, the user can select the pre-captured video file through the album control 230. Based on this, the client can obtain the first image frame of the video to be processed.

[0051] The prop control 220 is used to add prop effects to the video. Users can trigger a prop selection operation through this control 220. After receiving the prop selection operation, the client displays multiple candidate props. For example... Figure 2B As shown in the figure, this is a schematic diagram of multiple candidate props provided in an embodiment of this disclosure. A user can trigger an "add prop" operation by clicking on any one of the candidate props. The client can then add the prop's effects to the video based on this "add prop" operation. For example, if the user selects candidate prop 221, the client can add the effects of candidate prop 221 to the video.

[0052] Among them, the candidate props can be virtual objects, such as virtual cartoon characters. After the user selects the candidate prop, the client can add the special effects of the virtual object to the video, so that the virtual object follows the user's movements and achieves the effect of the user "transforming" into the virtual object.

[0053] S102: The client identifies the first image frame based on the first pose recognition algorithm to obtain the user's overall pose.

[0054] The first pose recognition algorithm can be an Avatar 3D algorithm. By using this algorithm to identify the first image frame, the user's overall pose can be obtained. The overall pose can be represented by quaternions of the user's key points. Quaternions are hypercomplex numbers used to characterize the rotation of objects (such as the user's key points mentioned above) in space.

[0055] like Figure 3 As shown in the figure, this is a schematic diagram of key points of the human body provided in an embodiment of this disclosure. The key points of the human body include: pelvis 0, left hip 1, right hip 2, lower spine 3, left knee 4, right knee 5, middle spine 6, left ankle 7, right ankle 8, upper spine 9, left foot 10, right foot 11, neck 12, left collar 13, right collar 14, head 15, left shoulder 16, right shoulder 17, left elbow 18, right elbow 19, left wrist 20, right wrist 21, left hand 22, and right hand 23.

[0056] It should be noted that, Figure 3This is merely one method of dividing human body key points according to the embodiments of this disclosure. In other examples, other methods of division may be used, such as dividing into 17 key points more coarsely or 28 key points more finely.

[0057] S103: When the confidence level of the overall pose is lower than the preset threshold, the client identifies the first image frame according to the second pose recognition algorithm to obtain the user's local pose.

[0058] The confidence level of the overall pose refers to the completeness of the quaternions used to represent the overall pose. In some examples, the overall pose can be represented by the quaternions of the aforementioned 24 key points. In real-world scenarios, the overall pose obtained by the first pose recognition algorithm is represented by only the quaternions of 14 key points, meaning that there are cases where the quaternions of some key points are not recognized. Therefore, the confidence level of the overall pose is 58.3%.

[0059] The embodiments disclosed herein do not specifically limit the preset threshold. In some examples, the preset threshold may be 50%, 55%, 60%, etc. Those skilled in the art can select the size of the preset threshold according to actual needs.

[0060] In some possible implementations, the preset threshold can be 60%. After the client identifies the first image frame using the first pose recognition algorithm, the confidence level of the user's overall pose is 58.3%. That is, the confidence level of the overall pose obtained by the first pose recognition algorithm is lower than the preset threshold. This indicates that the first image frame only includes a part of the user, such as the user's head as a key point. Then, the client switches to using the second pose recognition algorithm to identify the first image frame and obtain the user's local pose. For example, the local pose can be one or more combinations of head pose, hand pose, and leg pose.

[0061] To facilitate understanding, the following explanation will use the head pose as an example of a local pose.

[0062] Among these, head pose can be represented by quaternions of head key points; based on this, the second pose recognition algorithm can be the Face106 algorithm. For example... Figure 4 As shown, this figure is a schematic diagram of a head key point provided in an embodiment of this disclosure.

[0063] It should be noted that, Figure 4 The head key points shown are merely an example; in other embodiments, other segmentation methods may be used to determine the head key points.

[0064] After the client uses the second pose recognition algorithm described above to identify the first image frame, it can obtain the user's head pose, such as the size, orientation, and position of the head in the first image frame.

[0065] S104: The client updates the first image frame based on the local pose using the 3D local model of the virtual object. The updated first image frame includes the local part of the virtual object.

[0066] Continuing the previous example, the local pose is the head pose, and the 3D local model of the virtual object can be the 3D head model of the virtual object. After obtaining the head pose, the client updates the first image frame by combining it with the 3D head model of the virtual object, so that the updated first image frame includes the head of the virtual object.

[0067] This disclosure does not specifically limit the method of adding the virtual object's head to the first image frame. In some examples, the client can add the virtual object's head to the first image frame using a texture, allowing the virtual object's head to cover the user's head in the first image frame, thereby achieving the effect of making the user "transform" into a virtual object. In some possible implementations, the size and position of the virtual object in the first image frame can be adjusted according to the size and position of the user's head in the first image frame to achieve a better coverage effect. Furthermore, the client can also erase the user portion not covered by the virtual object, thereby improving the effect of the virtual object's special effects in the first image frame.

[0068] S105: When the confidence level of the overall pose is higher than or equal to the preset threshold, the client updates the first image frame based on the overall pose using the three-dimensional overall model of the virtual object. The updated first image frame includes the entire virtual object.

[0069] In some examples, the overall pose obtained by the first pose recognition algorithm is represented by quaternions of 20 keypoints, resulting in a confidence level of 83%. This means the confidence level of the overall pose obtained by the first pose recognition algorithm is higher than a preset threshold, indicating that the first image frame includes the user as a whole (e.g., by default, including all or most of the user's keypoints). After obtaining the user's overall pose, the client updates the first image frame by combining it with the 3D overall model of the virtual object, ensuring that the updated first image frame includes the entire virtual object.

[0070] This disclosure does not specifically limit the method of adding the entire virtual object to the first image frame. In some examples, the client can add the entire virtual object to the first image frame using a texture, so that the entire virtual object covers the entire user (e.g., the user's whole body) in the first image frame, thereby achieving the effect of making the user "transform" into a virtual object. In some possible implementations, the size and position of the virtual object in the first image frame can be adjusted according to the size and position of the user's entire body in the first image frame to achieve a better coverage effect. Furthermore, the client can also erase the user parts not covered by the virtual object, thereby improving the effect of the virtual object's special effects in the first image frame.

[0071] It should be noted that the embodiments disclosed herein do not specifically limit the execution order of S103-S104 and S105. For example, the client may execute S105 first, and then execute S103-S104. In other examples, the client may execute only S105, or only S103-S104, etc.

[0072] S106: The client presents the updated first image frame.

[0073] In some embodiments, the client can display an updated first image frame on the shooting interface after the user selects candidate prop 221. In other embodiments, the client can display an updated first image frame on the shooting interface after the user selects candidate prop 221 and the user clicks the shooting control 211 on the shooting interface. This disclosure does not specifically limit the scope of the embodiments.

[0074] like Figure 5A As shown, this figure is a schematic diagram of an updated first image frame provided in an embodiment of this disclosure. From Figure 5A As can be seen, the virtual object corresponding to candidate prop 221 is a cartoon image of an "alien." In the updated first image frame, the "alien's" head covers the user's head. Furthermore, the "alien's" head effect supports rotating with the user's head, thus achieving the effect of the user "transforming" into an "alien." In some possible implementations, the first image frame includes not only the user's head but also the user's upper body (such as...). Figure 5B (As shown in the dashed box). Furthermore, the client can also use the upper body of the "alien" to cover the upper body of the user, such as... Figure 5B As shown, this further enhances the special effects of the "alien" and improves the user's shooting experience.

[0075] like Figure 5C As shown, this figure is a schematic diagram of another updated first image frame provided in an embodiment of this disclosure. Figure 5C and Figures 5A-5B The difference is that, Figure 5C The updated first image frame presented includes the entirety (e.g., the whole body) of the "alien".

[0076] In some embodiments, after the user selects candidate prop 221, the client can also obtain a second image frame with a timestamp earlier than the first image frame. Based on the user's preset body parts in the second image frame, such as the user's left eye, right eye, and pelvic area, multiple consecutive image frames associated with the virtual object (such as "alien" in the example above) are generated. Then, the multiple consecutive image frames associated with the virtual object are presented sequentially. When the presentation is complete, the client presents the image as described above. Figure 5A , Figure 5B or Figure 5C The shooting interface shown.

[0077] In some examples, considering the positional relationship between the left eye, right eye, and pelvic area and other parts of the body, or the positional relationship between the left eye, right eye, and pelvic area, applying special effects to these three parts is more likely to attract user attention, improve user perception, and thus enhance user experience. Furthermore, calculations based on the left eye, right eye, and pelvic area are beneficial for the virtual object's partial or overall coverage of the user. Moreover, compared to other parts of the body, the left eye, right eye, and pelvic area are easier to identify, improving recognition accuracy and enhancing user experience.

[0078] like Figure 6A and Figure 6B As shown, Figure 6A and Figure 6B This is a schematic diagram of an image frame associated with a virtual object, provided as an embodiment of this disclosure. Figures 6A to 6B As can be seen, this image frame adds a highlight effect to the user's preset part in the second image frame, darkens other parts except the preset part, and gradually brightens the virtual object (while other parts remain dark), enhancing the user's perception of "transforming" into a virtual object and improving interactivity and fun.

[0079] It should be noted that, Figure 6A and Figure 6B This is merely one transformation effect provided by this disclosure. In other embodiments, other transformation effects may be used, and this disclosure does not specifically limit them.

[0080] In some embodiments, the client may also generate multiple consecutive image frames associated with the virtual object based on the user's preset part in the second image frame when the preset conditions are met, and present the multiple consecutive image frames in sequence.

[0081] The preset conditions can be at least one of the following: the user action detected by the client is a preset action, the user voice detected by the client is a preset voice, and the user operation detected by the client is a preset operation.

[0082] As an example, the preset voice can be the voice corresponding to the text "transformed" or other voices. Those skilled in the art can set the preset voice according to actual needs.

[0083] As an example, the preset action of sliding both hands from the sides of the body to above the head can be any other action. For example... Figure 7A As shown, this figure is a schematic diagram of a preset action provided by an embodiment of this disclosure. Those skilled in the art can set preset actions according to actual needs.

[0084] As an example, a preset action can be a user-triggered click action, such as a user clicking on the shooting interface. A preset action can also be an action triggered by a user through a hardware button, such as a user pressing and holding (for more than 1 second) the volume down button.

[0085] In some embodiments, the client detects the aforementioned preset voice and / or Figure 7A After the preset action shown, the following will be displayed. Figure 6A and Figure 6B The shooting interface shown.

[0086] Similarly, after the user "transforms" into a virtual object, the client can also display target effects on the shooting interface after detecting special effects voice (such as the voice corresponding to the text "ultimate move") and / or special effects actions (such as the action of raising one hand upwards), such as the virtual object emitting light waves. Figure 7B As shown, this figure is a schematic diagram of a target effect provided in an embodiment of this disclosure.

[0087] S107: The client encodes the updated first image frame to obtain a video file.

[0088] After receiving a shooting operation triggered by the user through the shooting control 211, the client obtains the updated first image frame. For example... Figure 8 As shown, after the user clicks the shooting control 211, the shooting control is switched to the end control 212. The user can trigger the end shooting operation through the end control 212. The client encodes the updated first image frame to obtain a video file.

[0089] In some embodiments, a user can first select a candidate prop 221 via prop control 220, and then trigger a shooting operation via shooting control 211. The client then acquires the video stream captured by the camera and the effects added by the client to the image frames of the video stream based on the shooting operation. After the user clicks the shooting control 211, the user can trigger a preset action and / or a preset voice. When the client recognizes that the user's action is a preset action and / or the voice is a preset voice, it adds effects such as... to the image frames of the video stream. Figure 6A and Figure 6B The transformation effect shown is followed by the effect of adding virtual objects in subsequent image frames. After the user clicks the end control 212, the client encodes the video stream captured by the camera and the effects added by the client in the image frames of the video stream, thereby obtaining a video file including the virtual object effect.

[0090] It should be noted that the above embodiments are only described with the example of the first image frame including one user. Of course, in other embodiments, the first image frame may include multiple users. The specific processing method is similar to that described in the above embodiments, and will not be repeated here.

[0091] Based on the above description, the video processing method provided in this disclosure effectively solves the problem of the disappearance of special effects of virtual objects in some shooting scenarios (such as close-up selfies) by alternately using multiple algorithms. Specifically, after acquiring the image frame of the video to be processed, the first pose recognition algorithm is used to identify the image frame to obtain the user's overall pose. When the confidence of the overall pose is lower than a preset threshold, it indicates that the user's overall limb movements cannot be recognized (e.g., only the user's head movements are recognized), and then the algorithm switches to the second pose recognition algorithm to identify the image frame to obtain the user's local pose. Then, based on the local pose, the image frame is updated using the three-dimensional local model of the virtual object, so that the updated image frame includes the local part of the virtual object.

[0092] Thus, this method can render virtual objects based on the 3D model of the virtual object by utilizing the user's local pose (such as head pose) even when the user's overall body movements cannot be recognized, thereby meeting the user's shooting needs and improving the user's shooting experience.

[0093] Figure 9 This is a schematic diagram of a video processing apparatus according to an exemplary disclosed embodiment, such as... Figure 9 As shown, the video processing device 900 includes:

[0094] Acquisition module 901 is used to acquire the first image frame of the video to be processed;

[0095] The recognition module 902 is used to recognize the first image frame according to the first pose recognition algorithm to obtain the user's overall pose; when the confidence level of the overall pose is lower than a preset threshold, the first image frame is recognized according to the second pose recognition algorithm to obtain the user's local pose.

[0096] The update module 903 is used to update the first image frame based on the local pose using a three-dimensional local model of the virtual object, wherein the updated first image frame includes a local part of the virtual object.

[0097] Optionally, the update module 903 is further configured to update the first image frame according to the overall pose using the three-dimensional overall model of the virtual object when the confidence level of the overall pose is lower than a preset threshold, wherein the updated first image frame includes the entirety of the virtual object.

[0098] Optionally, the device may also include a display module;

[0099] The display module is used to present the updated first image frame.

[0100] Optionally, the acquisition module 901 is further configured to acquire a second image frame with a timestamp earlier than the first image frame; and generate multiple consecutive image frames associated with the virtual object based on the user's preset part in the second image frame;

[0101] The display module is specifically used to present the updated first image frame when the presentation of multiple consecutive image frames associated with the virtual object is completed.

[0102] Optionally, the acquisition module 901 is specifically used to generate multiple consecutive image frames associated with the virtual object based on the user's preset part in the second image frame when preset conditions are met.

[0103] Optionally, the preset conditions include at least one of the following:

[0104] The detected user action is a preset action; and,

[0105] The detected user voice is a preset voice; and

[0106] The user actions detected were preset actions.

[0107] Optionally, the device further includes an encoding module;

[0108] The encoding module is used to encode the updated first image frame to obtain a video file.

[0109] Optionally, the user's preset location includes:

[0110] Left eye area, right eye area, and pelvic area.

[0111] The functions of each module have been described in detail in the method steps of the previous embodiment, and will not be repeated here.

[0112] The following is for reference. Figure 10 It shows a schematic diagram of the structure of an electronic device 1000 suitable for implementing embodiments of the present disclosure. The electronic device may be a terminal, which is used to implement, for example... Figure 9 The functions corresponding to the video processing device 900 shown. Figure 10 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0113] like Figure 10 As shown, the electronic device 1000 may include a processing unit (e.g., a central processing unit, a graphics processor, etc.) 1001, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1008 into a random access memory (RAM) 1003. The RAM 1003 also stores various programs and data required for the operation of the electronic device 1000. The processing unit 1001, ROM 1002, and RAM 1003 are interconnected via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0114] Typically, the following devices can be connected to the I / O interface 1005: input devices 1006 including, for example, a touchscreen, touchpad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; output devices 1007 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1008 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 allows electronic device 1000 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 10 An electronic device 1000 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0115] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 1009, or installed from storage device 1008, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of embodiments of this disclosure.

[0116] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0117] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0118] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0119] The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to: acquire a first image frame of a video to be processed; identify the first image frame according to a first pose recognition algorithm to obtain the user's overall pose; when the confidence level of the overall pose is lower than a preset threshold, identify the first image frame according to a second pose recognition algorithm to obtain the user's local pose; and update the first image frame according to the local pose using a three-dimensional local model of a virtual object, wherein the updated first image frame includes a part of the virtual object.

[0120] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0121] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0122] The modules described in the embodiments of this disclosure can be implemented in software or in hardware. The name of a module does not necessarily limit the module itself; for example, the first acquisition module can also be described as "a module for acquiring at least two Internet Protocol addresses".

[0123] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0124] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0125] According to one or more embodiments of this disclosure, Example 1 provides a video processing method, including:

[0126] Obtain the first image frame of the video to be processed;

[0127] The first image frame is identified using the first pose recognition algorithm to obtain the user's overall pose.

[0128] When the confidence level of the overall pose is lower than a preset threshold, the first image frame is identified according to the second pose recognition algorithm to obtain the user's local pose.

[0129] Based on the local pose, the first image frame is updated using the three-dimensional local model of the virtual object, and the updated first image frame includes a local part of the virtual object.

[0130] According to one or more embodiments of this disclosure, Example 2 provides the method of Example 1, the method further comprising:

[0131] When the confidence level of the overall pose is lower than a preset threshold, the first image frame is updated using the three-dimensional overall model of the virtual object based on the overall pose, and the updated first image frame includes the entirety of the virtual object.

[0132] According to one or more embodiments of this disclosure, Example 3 provides the method of Example 1 or 2, the method further comprising:

[0133] The updated first image frame is then displayed.

[0134] According to one or more embodiments of this disclosure, Example 4 provides the method of Example 3, the method further comprising:

[0135] Obtain the second image frame whose timestamp is earlier than the first image frame;

[0136] Based on the user's preset part in the second image frame, generate multiple consecutive image frames associated with the virtual object;

[0137] Multiple consecutive image frames associated with the virtual object are presented sequentially;

[0138] Presenting the updated first image frame includes:

[0139] When the presentation of the plurality of consecutive image frames associated with the virtual object is completed, the updated first image frame is presented.

[0140] According to one or more embodiments of this disclosure, Example 5 provides the method of Example 4, wherein generating a plurality of consecutive image frames associated with the virtual object based on a preset part of the user in the second image frame includes:

[0141] When the preset conditions are met, multiple consecutive image frames associated with the virtual object are generated based on the preset part of the user in the second image frame.

[0142] According to one or more embodiments of this disclosure, Example 6 provides the method of Example 5, wherein the preset conditions include at least one of the following:

[0143] The detected user action is a preset action; and,

[0144] The detected user voice is a preset voice; and

[0145] The user actions detected were preset actions.

[0146] According to one or more embodiments of this disclosure, Example 7 provides the methods of Examples 1-6, the methods further comprising:

[0147] The updated first image frame is encoded to obtain a video file.

[0148] According to one or more embodiments of this disclosure, Example 8 provides the methods of Examples 4-7, wherein the user's preset location includes:

[0149] Left eye area, right eye area, and pelvic area.

[0150] According to one or more embodiments of this disclosure, Example 9 provides a video processing apparatus, including:

[0151] The acquisition module is used to acquire the first image frame of the video to be processed;

[0152] The recognition module is used to recognize the first image frame according to the first pose recognition algorithm to obtain the user's overall pose; when the confidence level of the overall pose is lower than a preset threshold, the first image frame is recognized according to the second pose recognition algorithm to obtain the user's local pose.

[0153] An update module is used to update the first image frame based on the local pose using a three-dimensional local model of the virtual object, wherein the updated first image frame includes a local part of the virtual object.

[0154] According to one or more embodiments of this disclosure, Example 10 provides the apparatus of Example 9, wherein the update module is further configured to update the first image frame based on the overall pose using the three-dimensional overall model of the virtual object when the confidence level of the overall pose is lower than a preset threshold, wherein the updated first image frame includes the entirety of the virtual object.

[0155] According to one or more embodiments of this disclosure, Example 11 provides an apparatus of Example 9 or 10, the apparatus further comprising a display module;

[0156] The display module is used to present the updated first image frame.

[0157] According to one or more embodiments of this disclosure, Example 12 provides an apparatus of Example 11, wherein the acquisition module is further configured to acquire a second image frame with a timestamp earlier than the first image frame; and generate a plurality of consecutive image frames associated with the virtual object based on a preset part of the user in the second image frame;

[0158] The display module is specifically used to present the updated first image frame when the presentation of multiple consecutive image frames associated with the virtual object is completed.

[0159] According to one or more embodiments of this disclosure, Example 13 provides the apparatus of Example 12, wherein, optionally, the acquisition module is specifically configured to generate a plurality of consecutive image frames associated with the virtual object based on a preset part of the user in the second image frame when preset conditions are met.

[0160] According to one or more embodiments of this disclosure, Example 14 provides an apparatus of Example 13, wherein the preset conditions include at least one of the following:

[0161] The detected user action is a preset action; and,

[0162] The detected user voice is a preset voice; and

[0163] The user actions detected were preset actions.

[0164] According to one or more embodiments of this disclosure, Example 15 provides an apparatus of Examples 9 to 14, the apparatus further comprising an encoding module;

[0165] The encoding module is used to encode the updated first image frame to obtain a video file.

[0166] According to one or more embodiments of this disclosure, Example 16 provides the apparatus of Examples 12-15, wherein the user's preset location includes:

[0167] Left eye area, right eye area, and pelvic area.

[0168] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0169] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0170] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative forms of implementing the claims. Regarding the apparatus in the above embodiments, the specific manner in which the various modules perform their operations has been described in detail in the embodiments relating to the method, and will not be elaborated upon here.

Claims

1. A video processing method, characterized in that, include: Obtain the first image frame of the video to be processed; The first image frame is identified using the first pose recognition algorithm to obtain the user's overall pose. When the confidence level of the overall pose is lower than a preset threshold, the first image frame is identified according to the second pose recognition algorithm to obtain the user's local pose. Based on the local pose, the first image frame is updated using the three-dimensional local model of the virtual object, and the updated first image frame includes the local part of the virtual object; Obtain the second image frame whose timestamp is earlier than the first image frame; Based on the user's preset location in the second image frame, multiple consecutive image frames associated with the virtual object are generated.

2. The method according to claim 1, characterized in that, The method further includes: When the confidence level of the overall pose is higher than or equal to a preset threshold, the first image frame is updated using the three-dimensional overall model of the virtual object based on the overall pose, and the updated first image frame includes the entirety of the virtual object.

3. The method according to claim 1, characterized in that, The method further includes: The updated first image frame is then displayed.

4. The method according to claim 3, characterized in that, The method further includes: sequentially presenting multiple consecutive image frames associated with the virtual object; Presenting the updated first image frame includes: When the presentation of the plurality of consecutive image frames associated with the virtual object is completed, the updated first image frame is presented.

5. The method according to claim 1, characterized in that, The step of generating multiple consecutive image frames associated with the virtual object based on the user's preset location in the second image frame includes: When the preset conditions are met, multiple consecutive image frames associated with the virtual object are generated based on the preset part of the user in the second image frame.

6. The method according to claim 5, characterized in that, The preset conditions include at least one of the following: The detected user action is a preset action; and, The detected user voice is a preset voice; and, The user actions detected were preset actions.

7. The method according to any one of claims 1 to 6, characterized in that, The method further includes: The updated first image frame is encoded to obtain a video file.

8. The method according to any one of claims 7, characterized in that, The user's preset body parts include: Left eye area, right eye area, and pelvic area.

9. A video processing apparatus, characterized in that, include: The acquisition module is used to acquire the first image frame of the video to be processed; The recognition module is used to recognize the first image frame according to the first pose recognition algorithm to obtain the user's overall pose; When the confidence level of the overall pose is lower than a preset threshold, the first image frame is identified according to the second pose recognition algorithm to obtain the user's local pose. An update module is used to update the first image frame based on the local pose using a three-dimensional local model of the virtual object, wherein the updated first image frame includes a local part of the virtual object.

10. An electronic device, characterized in that, include: A storage device on which computer programs are stored; A processing device for executing the computer program in the storage device to implement the steps of the method according to any one of claims 1 to 8.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by the processing device, the program implements the steps of the method according to any one of claims 1 to 8.

12. A computer program product, characterized in that, When the computer program product is run on a computer, it causes the computer to perform the steps of the method as described in any one of claims 1 to 8.

Images