Image processing method and device, storage medium and computer device

Abstract

This invention discloses an image processing method, apparatus, storage medium, and computer device. The method includes: acquiring a human body image; inputting the human body image into a human pose network model to obtain three-dimensional 3D rotation values ​​of at least one joint of the human body, wherein the human pose network model is trained using a first dataset through machine learning, and the data in the first dataset includes: the human body image and the 3D rotation values ​​of each joint of the human body corresponding to the human body image; and generating a virtual human based on the three-dimensional 3D rotation values ​​of at least one joint of the human body. This invention solves the technical problem of inaccurate movements of virtual humans generated from human body images in related technologies.

Description

Technical Field

[0001] This invention relates to the field of image processing, and more specifically, to an image processing method, apparatus, storage medium, and computer device. Background Technology

[0002] By capturing and recognizing human movements, expressions, and gestures in images, virtual humans can be driven to perform the same movements, expressions, and gestures as the humans in the images in real time. Applying this technology to scenarios such as live streaming enables interaction between real people and cartoon virtual humans, achieving excellent interactive effects.

[0003] In related technologies, when using keypoint-based schemes to achieve interaction between real and virtual humans, keypoints are easily lost in the captured images containing real people, often leading to inaccurate control of virtual human movements. Furthermore, keypoint-based schemes, due to their high computational cost, result in a low frame rate (fps) for the generated virtual human, making it difficult to meet the requirement of real-time replication of virtual human movements.

[0004] There is currently no effective solution to the above problems. Summary of the Invention

[0005] This invention provides an image processing method, apparatus, storage medium, and computer device to at least solve the technical problem of inaccurate virtual human movements generated from human images in related technologies.

[0006] According to one aspect of the present invention, an image processing method is provided, comprising: acquiring a human body image; inputting the human body image into a human pose network model to obtain three-dimensional 3D rotation values ​​of at least one joint of the human body, wherein the human pose network model is trained by machine learning using a first dataset, the data in the first dataset including: a human body image and 3D rotation values ​​of each joint of the human body corresponding to the human body image; and generating the virtual human body based on the three-dimensional 3D rotation values ​​of at least one joint of the human body.

[0007] Optionally, acquiring a human image includes: receiving an input image; using a detection network model to detect the input image and detect the human image from the input image, wherein the detection network model is trained using a second dataset through machine learning, and the data in the second dataset includes: the input image and the human image in the input image.

[0008] Optionally, the method further includes: using the detection network model to detect the input image and detect a face image from the input image, wherein the detection network model is further trained using a third dataset through machine learning, the data in the third dataset including: the input image and the face image in the input image; using a face parameter model to extract parameters from the face image to obtain the face expression parameters of the face image, wherein the face parameter model is trained using a fourth dataset through machine learning, the data in the fourth dataset including: the face image and the face expression parameters of the face image; wherein generating the virtual human based on the three-dimensional 3D rotation values ​​of at least one joint of the human body includes: generating the virtual human based on the three-dimensional 3D rotation values ​​of at least one joint of the human body and the face expression parameters.

[0009] Optionally, the method further includes: using the detection network model to detect the input image and detect gesture images from the input image, wherein the detection network model is further trained using a fifth dataset through machine learning, the data in the fifth dataset including: the input image and the gesture images in the input image; using a gesture classification model to classify the gesture images to obtain the gesture categories of the hands in the gesture images, wherein the gesture classification model is trained using a sixth dataset through machine learning, the data in the sixth dataset including: the gesture images and the gesture categories of the hands in the gesture images; wherein generating the virtual human based on the three-dimensional 3D rotation values ​​of at least one joint of the human body includes: generating the virtual human based on the three-dimensional 3D rotation values ​​of at least one joint of the human body, the facial expression parameters, and the gesture categories.

[0010] Optionally, the method further includes: using the detection network model to detect the input image and detect foot images from the input image, wherein the detection network model is further trained using a seventh dataset through machine learning, the data in the seventh dataset including: the input image and the foot images in the input image; using a foot recognition model to perform action recognition on the foot images to obtain the foot actions in the foot images, wherein the foot recognition model is trained using an eighth dataset through machine learning, the data in the eighth dataset including: foot images and the foot actions in the foot images; wherein generating the virtual human based on the three-dimensional 3D rotation values ​​of at least one joint of the human body includes: generating the virtual human based on the three-dimensional 3D rotation values ​​of at least one joint of the human body, the facial expression parameters, the gesture category, and the foot actions.

[0011] Optionally, when the input image is a multi-frame video image, generating the virtual human based on the 3D rotation value of at least one joint of the human body, the facial expression parameters, the gesture category, and the foot movement includes: inputting the 3D rotation value of at least one joint of the human body corresponding to each frame of the multi-frame video image, the facial expression parameters, the gesture category, and the foot movement into a predetermined driving engine; and generating an animation of the virtual human through the driving engine.

[0012] Optionally, the multi-frame video images include at least one of the following: multi-frame video images in a recorded video, and multi-frame video images in a live video.

[0013] Optionally, the method further includes: receiving an interaction instruction, wherein the interaction instruction is used to interact with the virtual human; obtaining an input image corresponding to the interaction instruction based on the interaction instruction; and adjusting the virtual human according to the input image to generate a new virtual human.

[0014] According to another aspect of the present invention, an image processing method is also provided, comprising: displaying a virtual human on a display interface, wherein the virtual human is generated based on the three-dimensional 3D rotation values ​​of at least one joint of a human body; receiving an interaction instruction, wherein the interaction instruction is used to interact with the virtual human; and displaying an interaction result corresponding to the interaction instruction on the display interface.

[0015] Optionally, the method further includes: displaying a human body image on the display interface; displaying a three-dimensional 3D rotation value of at least one joint of the human body on the display interface, wherein the three-dimensional 3D rotation value of at least one joint of the human body is obtained by processing the human body image using a human pose network model.

[0016] Optionally, displaying the interaction result corresponding to the interaction command on the display interface includes: receiving an input image based on the interaction command; using a detection network model to detect the input image, and detecting at least one of the following images from the input image: a human body image, a face image, a gesture image, and a foot image, wherein the detection network model is trained using a second dataset through machine learning, and the data in the second dataset includes: the input image and at least one of the following images in the input image: a human body image, a face image, a gesture image, and a foot image; performing at least one of the following operations: obtaining a 3D rotation value of at least one joint of the human body based on the human body image, obtaining facial expression parameters based on the face image, obtaining a gesture classification based on the gesture image, and obtaining a foot movement based on the foot image; generating an interaction result corresponding to the input image based on at least one of the following: the 3D rotation value of at least one joint of the human body, the facial expression parameters, the gesture classification, and the foot movement; and displaying the interaction result.

[0017] According to another aspect of the present invention, an image processing apparatus is also provided, comprising: an acquisition module for acquiring a human body image; a recognition module for inputting the human body image into a human posture network model to obtain three-dimensional 3D rotation values ​​of at least one joint of the human body, wherein the human posture network model is trained using a first dataset through machine learning, and the data in the first dataset includes: a human body image and 3D rotation values ​​of each joint of the human body corresponding to the human body image; and a generation module for generating the virtual human body based on the three-dimensional 3D rotation values ​​of at least one joint of the human body.

[0018] According to another aspect of the present invention, an image processing apparatus is also provided, comprising: a first display module for displaying a virtual human on a display interface, wherein the virtual human is generated based on the three-dimensional 3D rotation values ​​of at least one joint of a human body; a receiving module for receiving an interaction instruction, wherein the interaction instruction is used to interact with the virtual human; and a second display module for displaying an interaction result corresponding to the interaction instruction on the display interface.

[0019] According to another aspect of the present invention, a storage medium is also provided, the storage medium including a stored program, wherein, when the program is executed, the device where the storage medium is located controls the execution of the image processing method described in any one of the above embodiments.

[0020] According to another aspect of the present invention, a computer device is also provided, comprising: a memory and a processor, the memory storing a computer program; the processor being configured to execute the computer program stored in the memory, wherein the computer program, when executed, causes the processor to perform the image processing method described in any of the preceding embodiments.

[0021] According to another aspect of the present invention, an image processing method is also provided, comprising: acquiring a sample dataset, wherein the data of the sample dataset includes: a target object image and true 3D rotation values ​​of each joint of the target object corresponding to the target object image; inputting the target object image in the sample dataset into a target object network model to obtain output 3D rotation values ​​of each joint of the target object; comparing the output 3D rotation values ​​with the true 3D rotation values ​​to obtain a difference; and optimizing the model parameters of the target object network model based on the difference to obtain a training result of the target object network model.

[0022] According to another aspect of the present invention, an image processing method is also provided, comprising: acquiring an image of a target object; inputting the image of the target object into a target object network model to obtain three-dimensional 3D rotation values ​​of at least one joint of the target object, wherein the target object network model is trained by machine learning using a ninth dataset, the data in the ninth dataset including the target object image and the 3D rotation values ​​of each joint of the target object corresponding to the target object image; and generating a virtual image of the target object based on the three-dimensional 3D rotation values ​​of at least one joint of the target object.

[0023] Optionally, the target object includes at least one of the following: an animal, a machine-moving object.

[0024] According to another aspect of the present invention, an image processing method is also provided, comprising: acquiring a target image; identifying a three-dimensional rotation value of at least one joint of a target object from the target image; and generating and displaying a virtual object of the target object based on the three-dimensional rotation value.

[0025] In this embodiment of the invention, a human posture network model trained by machine learning is used. By acquiring a human image, the human image is input into the human posture network model to obtain the three-dimensional 3D rotation value of at least one joint of the human body. A virtual human is generated based on the three-dimensional 3D rotation value of at least one joint of the human body. This achieves the goal of obtaining the 3D rotation value of at least one joint of the human body from the human image and generating the corresponding virtual human. This realizes the technical effect of accurately and quickly calculating the corresponding virtual human from the human image, thereby solving the technical problem of inaccurate virtual human movements generated from human images in related technologies. Attached Figure Description

[0026] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0027] Figure 1 This is a hardware structure block diagram of a computer terminal for implementing an image processing method according to an embodiment of the present invention;

[0028] Figure 2 This is a flowchart of an image processing method according to Embodiment 1 of the present invention;

[0029] Figure 3 This is a flowchart of image processing method two according to Embodiment 1 of the present invention;

[0030] Figure 4 This is a flowchart of image processing method three according to Embodiment 1 of the present invention;

[0031] Figure 5 This is a flowchart of image processing method four according to Embodiment 1 of the present invention;

[0032] Figure 6 This is a flowchart of image processing method five according to Embodiment 1 of the present invention;

[0033] Figure 7 This is a schematic diagram of a method for generating a simulated human according to an optional embodiment of the present invention;

[0034] Figure 8 This is a structural block diagram of an image processing apparatus according to Embodiment 2 of the present invention;

[0035] Figure 9 This is a structural block diagram of the image processing apparatus 2 according to Embodiment 2 of the present invention;

[0036] Figure 10 This is a structural block diagram of the image processing apparatus three according to Embodiment 2 of the present invention;

[0037] Figure 11 This is a structural block diagram of the image processing apparatus four according to Embodiment 2 of the present invention;

[0038] Figure 12 This is a structural block diagram of the image processing apparatus five according to Embodiment 2 of the present invention;

[0039] Figure 13 This is a structural block diagram of a computer terminal according to an embodiment of the present invention. Detailed Implementation

[0040] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0041] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0042] Example 1

[0043] According to an embodiment of the present invention, an embodiment of an image processing method is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0044] The method embodiment provided in Embodiment 1 of this application can be executed in a mobile terminal, computer terminal or similar computing device. Figure 1 A hardware block diagram of a computer terminal (or mobile device) for implementing an image processing method is shown. Figure 1 As shown, the computer terminal 10 (or mobile device) may include one or more processors 102 (shown as 102a, 102b, ..., 102n in the figure) 102 (processor 102 may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 104 for storing data, and a transmission device for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of a BUS bus), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0045] It should be noted that the aforementioned one or more processors 102 and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the computer terminal 10 (or mobile device). As involved in the embodiments of this application, the data processing circuits serve as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).

[0046] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the image processing method in this embodiment of the invention. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby implementing the above-mentioned application vulnerability detection method. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0047] The transmission device is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of computer terminal 10. In one example, the transmission device includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0048] The display can be, for example, a touchscreen liquid crystal display (LCD) that allows the user to interact with the user interface of the computer terminal 10 (or mobile device).

[0049] Under the aforementioned operating environment, this application provides the following: Figure 2 The image processing method shown. Figure 2 This is a flowchart of an image processing method according to Embodiment 1 of the present invention, as follows: Figure 2 As shown, the method includes the following steps:

[0050] Step S202: Obtain a human body image;

[0051] Step S204: Input the human body image into the human body pose network model to obtain the three-dimensional 3D rotation value of at least one joint of the human body. The human body pose network model is trained by machine learning using the first dataset. The data in the first dataset includes: human body image and 3D rotation values ​​of each joint of the human body corresponding to the human body image.

[0052] Step S206: Generate a virtual human based on the three-dimensional 3D rotation value of at least one joint of the human body.

[0053] Through the above steps, the goal of obtaining the 3D rotation value of at least one joint in the human body based on the human body image and generating the corresponding virtual human is achieved. This realizes the technical effect of accurately and quickly generating the corresponding virtual human based on the human body image, and solves the technical problem of inaccurate virtual human movements generated based on human body images in related technologies.

[0054] According to the above embodiments, the human pose network model can identify the 3D rotation value of at least one joint of the human body in a human image. The human pose network model can be a neural network model, obtained through machine learning using a sample dataset including a human image and the 3D rotation values ​​of each joint corresponding to that human image. By processing the human body into multiple parts connected by multiple joints, the positional relationships between the various parts of the human body can be accurately determined based on the 3D rotation value of at least one joint. Based on the determined positional relationships between the various parts of the human body, a virtual human can be generated quickly. This method avoids the problem of inaccurate virtual human generation caused by missing key points when generating virtual humans based on key points, thus improving the accuracy and efficiency of generating virtual humans from human images.

[0055] As an optional embodiment, the at least one joint of the human body mentioned above can be a part of the joint of the human body or all of the joints of the human body, and can be flexibly selected as needed.

[0056] As an optional embodiment, human images can be obtained by receiving an input image and then using a detection network model to detect the human image from the input image. The detection network model can be trained using a second dataset via machine learning. The second dataset includes the input image and the human image within that input image. The human image is the basis for generating a virtual human. To obtain the human image conveniently and quickly, various methods can be used. For example, an input image containing a labeled human image can be used. By recognizing the input image and its labels, the human image in the input image can be obtained. Alternatively, a detection network model can be used to directly detect the input image and extract the human image directly. The aforementioned detection network model can be a neural network model trained using a sample dataset via deep learning. The samples in the sample dataset can be multiple input images, each containing a human image and its labels, used to train the detection network model. Using a detection network model to detect human images offers advantages over other detection methods, as the detection network model is trained with a large amount of training sample data, resulting in greater intelligence, accuracy, and efficiency.

[0057] It should be noted that the input image can be an image captured by a camera device or a video image from a live broadcast or other similar scenario. All of these images can serve as sources of human body images, and this embodiment does not impose any limitations on them.

[0058] As an optional embodiment, facial expressions can also be generated for the virtual human based on the face image in the input image. Generating facial expressions for the virtual human can be achieved through the following method: A detection network model is used to detect the face image in the input image. The detection network model is trained using a third dataset via machine learning. The data in the third dataset includes the input image and the face image within it. A face parameter model is used to extract parameters from the face image to obtain the facial expression parameters. The face parameter model is trained using a fourth dataset via machine learning. The data in the fourth dataset includes the face image and the facial expression parameters. The virtual human is then generated based on the 3D rotation values ​​of at least one joint of the human body, including generating the virtual human based on the 3D rotation values ​​of at least one joint and the facial expression parameters. Integrating the network models for detecting human and face images into a single detection network model effectively reduces the computational load of the model and simplifies the complexity of the virtual human generation system.

[0059] As an alternative implementation, the method of training a detection network model to detect face images from input images can be the same as the method of training a detection network model to detect human images from input images described above. That is, a detection network model, such as a neural network model, is used for training and detection to obtain the identified face images.

[0060] Furthermore, a face parameter model can be used to extract parameters from the recognized face images, extracting features from the face images and generating virtual expressions consistent with the corresponding face images on the facial regions of the generated virtual human based on these features. The face parameter model can be a deep learning model trained through machine learning. Through the above processing, the pose and expression in the image are replicated on the virtual human, and the recognition of human pose and expression in the input image is integrated into the same detection network model, simplifying the image recognition steps and accelerating the image recognition speed.

[0061] As an optional embodiment, gesture images can also be generated for the virtual human based on gesture images in the input image. Generating gesture images for the virtual human can be achieved through the following steps: A detection network model is used to detect gesture images from the input image. This detection network model is trained using a fifth dataset via machine learning. The fifth dataset includes the input image and the gesture images within it. A gesture classification model is used to classify the gestures in the gesture images, obtaining the hand gesture categories. This gesture classification model is trained using a sixth dataset via machine learning. The sixth dataset includes the gesture images and the hand gesture categories within them. Finally, the virtual human is generated based on the 3D rotation values ​​of at least one joint of the human body, including: generating the virtual human based on the 3D rotation values ​​of at least one joint of the human body, facial expression parameters, and gesture categories. Integrating the network models for detecting human body images, face images, and gesture categories into a single detection network model can further effectively reduce the computational load of the model and simplify the complexity of the virtual human generation system.

[0062] As an alternative implementation, the method of training a detection network model to detect gesture images from an input image can be the same as the method of training a detection network model to detect human body images and face images from an input image. By using sample images that include gesture images and labels of gesture images, a detection network model, such as a neural network model, is trained to obtain a detection network model that can recognize gesture images from an input image.

[0063] Furthermore, a gesture classification model can be used to classify the gesture images recognized by the detection network model, and based on the classification results, a virtual gesture image corresponding to the classification result can be generated on the virtual human's hand. The gesture classification model can be a deep learning model trained through machine learning, or other mature classifiers; no limitation is made here. Through the above processing, the posture, expression, and gesture in the image are replicated on the virtual human, and the recognition of human posture, expression, and gesture in the input image is integrated into the same detection network model, simplifying the image recognition steps and accelerating the image recognition speed.

[0064] As an optional embodiment, foot images can also be generated for the virtual human based on foot images in the input image. Generating foot images for the virtual human can be achieved through the following steps: A detection network model is used to detect the input image and extract foot images from it. This detection network model is trained using a seventh dataset via machine learning, which includes the input image and the foot images within it. A foot recognition model is then used to perform motion recognition on the foot images to obtain the foot motions within them. This foot recognition model is trained using an eighth dataset via machine learning, which includes the foot images and the foot motions within them. Finally, the virtual human is generated based on the 3D rotation values ​​of at least one joint of the human body, including facial expression parameters, gesture categories, and foot motions. Integrating the network models for detecting human images, facial images, gesture categories, and foot motions into a single detection network model can further reduce the computational load of the model and simplify the complexity of the virtual human generation system.

[0065] As an alternative implementation, the method of training a detection network model to detect foot images from an input image can be the same as the method of training a detection network model to detect human body images, face images, and gesture images from an input image. By using sample images that include foot images and labeled foot images, a detection network model, such as a neural network model, is trained to obtain a detection network model that can identify foot images from an input image.

[0066] Furthermore, a foot recognition model can be used to perform foot motion recognition on foot images obtained from the detection network model. Based on the foot motion recognition results, virtual foot movements corresponding to the foot motion recognition results can be generated on the virtual human's feet. The foot recognition model can be a deep learning model trained through machine learning, or other mature classifiers; no limitation is made here. Through the above processing, the posture, expression, gesture, and foot movements in the image are replicated on the virtual human. Moreover, the recognition of human posture, expression, gesture, and foot movements in the input image is integrated into the same detection network model, simplifying the image recognition steps and accelerating the image recognition speed.

[0067] As an optional embodiment, when the input image is a multi-frame video image, an animation of a virtual human can be generated. The generation of the virtual human animation can be achieved as follows: Having obtained the 3D rotation values ​​of at least one joint of the human body, facial expression parameters, gesture category, and foot movements, the 3D rotation values ​​of at least one joint of the human body corresponding to each frame of the multi-frame video image are input into a predetermined driving engine; the animation of the virtual human is formed by driving the driving engine.

[0068] As an optional implementation, when an input image consisting of multiple video frames is obtained, the input image can be processed to obtain the 3D rotation values ​​of at least one joint of the human body corresponding to each frame, facial expression parameters, gesture category, and foot movements, and then input into a predetermined driving engine. The driving engine can process the 3D rotation values ​​of at least one joint of the human body corresponding to each frame of video image separately, thereby determining the virtual human corresponding to that frame of video image. After obtaining multiple virtual humans corresponding to multiple frames of video images, an animation of the virtual human can be formed by driving the driving engine.

[0069] As an optional embodiment, the multi-frame video images may include at least one of the following video types: multi-frame video images from a recorded video, and multi-frame video images from a live stream. When the multi-frame video images used are from a recorded video, an animation of a virtual human based on postures, expressions, gestures, and foot movements in the recorded video can be obtained according to the above optional embodiment; when the multi-frame video images used are from a live stream, an animation of a virtual human based on postures, expressions, gestures, and foot movements in the live stream can be obtained according to the above optional embodiment.

[0070] As an optional embodiment, the virtual human can be adjusted and a new virtual human can be generated based on the received interaction instructions. For example, an interaction instruction for interacting with the virtual human can be received, an input image corresponding to the interaction instruction can be obtained based on the interaction instruction, and the virtual human can be adjusted according to the input image to generate a new virtual human. Through this optional embodiment, the purpose of adjusting the input image on which the virtual human is based for generating the virtual human according to the interaction instructions can be achieved, and the virtual human's posture, expression, gestures, or foot movements can be adjusted.

[0071] Figure 3 This is a flowchart of the second image processing method provided in Embodiment 1 of the present invention, as follows: Figure 3 As shown, the method includes the following steps:

[0072] Step S302: Display a virtual human on the display interface, wherein the virtual human is generated based on the three-dimensional 3D rotation value of at least one joint of the human body;

[0073] Step S304: Receive interaction instructions, wherein the interaction instructions are used to interact with the virtual human;

[0074] Step S306: Display the interaction result corresponding to the interaction command on the display interface.

[0075] Through the above steps, the goal of interacting with the virtual human displayed on the screen according to the interaction instructions and displaying the interaction results is achieved. This realizes the technical effect of accurately and quickly displaying the interaction results obtained after interacting with the virtual human on the screen according to the interaction instructions, thereby solving the technical problems of slow response speed and inaccurate interaction results when interacting with the virtual human displayed on the screen in related technologies.

[0076] As an optional embodiment, before displaying the virtual human on the display interface, a human body image can also be displayed on the display interface; the display interface displays the 3D rotation values ​​of at least one joint of the human body, wherein the 3D rotation values ​​of at least one joint of the human body are obtained by processing the human body image using a human pose network model. For example, the display interface receives a human body image uploaded by the user for generating the virtual human. This human body image can be a photograph taken with a camera or a frame extracted from a video. Then, the human body image is input into the human pose network model, which outputs the 3D rotation values ​​of at least one joint of the human body in the human body image. The human pose network model can be trained using a first dataset through machine learning. The data in the first dataset includes: a human body image and the 3D rotation values ​​of each joint of the human body corresponding to the human body image.

[0077] As an optional embodiment, displaying the interaction result corresponding to the interaction command on the display interface can be done in the following manner: Based on the interaction command, receive an input image; use a detection network model to detect the input image, and detect at least one of the following images from the input image: human body image, face image, gesture image, foot image, wherein the detection network model is trained by machine learning using a second dataset, and the data in the second dataset includes: the input image and at least one of the following images in the input image: human body image, face image, gesture image, foot image; perform at least one of the following operations: obtain the 3D rotation value of at least one joint of the human body based on the human body image, obtain facial expression parameters based on the face image, obtain gesture classification based on the gesture image, and obtain foot movement based on the foot image; generate the interaction result corresponding to the input image based on at least one of the following: 3D rotation value of at least one joint of the human body, facial expression parameters, gesture classification, and foot movement; display the interaction result.

[0078] Figure 4 This is a flowchart of the image processing method three provided in Embodiment 1 of the present invention, as follows: Figure 4 As shown, the method includes the following steps:

[0079] Step S402: Obtain a sample dataset, wherein the data in the sample dataset includes: a target object image and the actual 3D rotation values ​​of each joint of the target object corresponding to the target object image;

[0080] Step S404: Input the target object image from the sample dataset into the target object network model to obtain the output 3D rotation values ​​of each joint of the target object;

[0081] Step S406: Compare the output 3D rotation value with the actual 3D rotation value to obtain the difference;

[0082] Step S408: Optimize the model parameters of the target object network model based on the difference to obtain the training result of the target object network model.

[0083] Through the above steps, the output 3D rotation values ​​of each joint of the target object obtained from the sample dataset are compared with the true 3D rotation values ​​to obtain the difference. This achieves the goal of optimizing the model parameters of the target object network based on the difference to obtain the training result of the target object network model. Thus, the target object network model is trained, and based on the trained target object network model, the 3D rotation values ​​of each joint of the target object in the image can be obtained quickly and accurately. This solves the technical problem of imperfect model optimization in related technologies.

[0084] Figure 5 This is a flowchart of the image processing method four provided in Embodiment 1 of the present invention, as follows: Figure 5 As shown, the method includes the following steps:

[0085] Step S502: Obtain the image of the target object;

[0086] Step S504: Input the target object image into the target object network model to obtain the three-dimensional 3D rotation value of at least one joint of the target object. The target object network model is trained by machine learning using the ninth dataset. The data in the ninth dataset includes the target object image and the 3D rotation values ​​of each joint of the target object corresponding to the target object image.

[0087] Step S506: Generate a virtual image of the target object based on the three-dimensional 3D rotation value of at least one joint of the target object.

[0088] Through the above steps, the goal of obtaining the 3D rotation value of at least one joint of the target object in the target object image using the target object network model and generating the corresponding virtual image is achieved. This realizes the technical effect of accurately and quickly generating the corresponding virtual image based on the target object image, and solves the technical problem of inaccurate animation of the virtual image generated based on the target object image in related technologies.

[0089] As an optional embodiment, the target object includes at least one of the following: an animal, a moving machine object. For example, animals may include cats, dogs, etc. Moving machines may include movable robots, movable simulation models, etc.

[0090] Figure 6 This is a flowchart of the image processing method five provided in Embodiment 1 of the present invention, as follows: Figure 6 As shown, the method includes the following steps:

[0091] Step S602: Obtain the target image;

[0092] Step S604: Identify the three-dimensional rotation value of at least one joint of the target object from the target image;

[0093] Step S606: Generate and display a virtual object of the target object based on the three-dimensional rotation value.

[0094] Through the above steps, the goal of obtaining the 3D rotation value of at least one joint in the target object based on the target image and generating the corresponding virtual object is achieved. This realizes the technical effect of accurately and quickly generating the corresponding virtual object based on the target image, thereby solving the technical problem of inaccurate animation of virtual objects generated based on the target image in related technologies.

[0095] Figure 7 This is a schematic diagram of a virtual human generation method according to an optional embodiment of the present invention. Figure 7As shown, generating a humanoid simulator can be achieved through the following steps:

[0096] S1, use a detection network to detect the input image. Specifically, the detection network can detect faces, bodies, and hands in the image.

[0097] S2, based on the detection results of the detection network, segment the input image to obtain face image, body image, and hand image;

[0098] S3, input the human body image into the human pose estimation network to obtain the 3D rotation vector of at least one joint of the human body, that is, to obtain the pose (3D pose);

[0099] S4, input the face image into the facial expression parameter regression network to obtain the facial expression parameters;

[0100] S5. Input the image of the human hand into the gesture classification model to obtain the hand gesture category;

[0101] S6 obtains the facial expression parameters, human posture, and gesture categories of multiple frames of images, inputs them into the driving engine, and drives the generation of virtual human animation based on the multiple frames of images.

[0102] According to the above optional implementation, it is not necessary to estimate the positions of 2D or 3D keypoints in the human body image. The detection network can directly output the 3D rotation of the human joints, avoiding the problem of inaccurate pose estimation caused by the loss of keypoints during image detection. Because the above implementation uses a fused detection network model, the detection network model can detect pose, expression, gesture, feet, etc., reducing redundant algorithm computation and improving computing speed. In related technologies, a refresh rate of 5fps is typically achievable. Using the above optional implementation, a virtual human animation refresh rate of at least 30fps can be achieved, thus enabling the replication of the entire human body, including face, gestures, and human movements, on the device side.

[0103] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, because according to the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to the present invention.

[0104] Through the above description of the embodiments, those skilled in the art can clearly understand that the image processing method according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platform. Of course, it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of the present invention.

[0105] Example 2

[0106] According to an embodiment of the present invention, an apparatus for implementing the above-described image processing method is also provided. Figure 8 This is a structural block diagram of an image processing apparatus according to Embodiment 2 of the present invention, as shown below. Figure 8 As shown, the device includes: a first acquisition module 82, a first identification module 84, and a first generation module 86. The device will now be described in detail.

[0107] The first acquisition module 82 is used to acquire human body images;

[0108] The first recognition module 84 is connected to the first acquisition module 82 and is used to input the human body image into the human body posture network model to obtain the three-dimensional 3D rotation value of at least one joint of the human body. The human body posture network model is trained by machine learning using the first dataset. The data in the first dataset includes: human body image and 3D rotation values ​​of each joint of the human body corresponding to the human body image.

[0109] The first generation module 86, connected to the first recognition module 84, is used to generate a virtual human based on the three-dimensional 3D rotation value of at least one joint of the human body.

[0110] It should be noted that the first acquisition module 82, the first identification module 84, and the first generation module 86 mentioned above correspond to steps S202 to S206 in Embodiment 1. The three modules and their corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in Embodiment 1. It should also be noted that the above modules, as part of the device, can run on the computer terminal 10 provided in Embodiment 1.

[0111] Example 3

[0112] According to an embodiment of the present invention, an apparatus for implementing the above-described image processing method two is also provided. Figure 9 This is a structural block diagram of the image processing apparatus 2 according to Embodiment 3 of the present invention, as follows: Figure 9As shown, the device includes: a first display module 92, a receiving module 94, and a second display module 96. The device will now be described in detail.

[0113] The first display module 92 is used to display a virtual human on the display interface, wherein the virtual human is generated based on the three-dimensional 3D rotation value of at least one joint of the human body; the receiving module 94 is connected to the first display module 92 and is used to receive interaction commands, wherein the interaction commands are used to interact with the virtual human; the second display module 96 is connected to the receiving module 94 and is used to display the interaction result corresponding to the interaction command on the display interface.

[0114] It should be noted that the first display module 92, the receiving module 94, and the second display module 96 mentioned above correspond to steps S302 to S306 in Embodiment 1. The three modules and their corresponding steps implement the same examples and application scenarios, but are not limited to the content disclosed in Embodiment 1. It should also be noted that the above modules, as part of the device, can run in the computer terminal 10 provided in Embodiment 1.

[0115] Example 4

[0116] According to an embodiment of the present invention, an apparatus for implementing the above-described image processing method three is also provided. Figure 10 This is a structural block diagram of the image processing apparatus three according to Embodiment 4 of the present invention, as shown below. Figure 10 As shown, the device includes: a second acquisition module 1002, a second identification module 1004, a comparison module 1006, and a second generation module 1008. The device will now be described in detail.

[0117] The second acquisition module 1002 is used to acquire a sample dataset, wherein the data in the sample dataset includes: a target object image and the actual 3D rotation values ​​of each joint of the target object corresponding to the target object image;

[0118] The second recognition module 1004 is connected to the second acquisition module 1002 and is used to input the target object image in the sample dataset into the target object network model to obtain the output 3D rotation values ​​of each joint of the target object.

[0119] The comparison module 1006 is connected to the second recognition module 1004 and is used to compare the output 3D rotation value with the real 3D rotation value to obtain the difference.

[0120] The second generation module 1008, connected to the comparison module 1006, is used to optimize the model parameters of the target object network model based on the difference, and obtain the training result of the target object network model.

[0121] It should be noted that the second acquisition module 1002, the second identification module 1004, the comparison module 1006, and the second generation module 1008 mentioned above correspond to steps S402 to S408 in Embodiment 1. The four modules and their corresponding steps implement the same examples and application scenarios, but are not limited to the content disclosed in Embodiment 1. It should be noted that the above modules, as part of the device, can run in the computer terminal 10 provided in Embodiment 1.

[0122] Example 5

[0123] According to an embodiment of the present invention, an apparatus for implementing the above-described image processing method four is also provided. Figure 11 This is a structural block diagram of the image processing apparatus four according to Embodiment 5 of the present invention, as follows: Figure 11 As shown, the device includes: a third acquisition module 1102, a third identification module 1104, and a third generation module 1106. The device will now be described in detail.

[0124] The third acquisition module 1102 is used to acquire the image of the target object;

[0125] The third recognition module 1104 is connected to the third acquisition module 1102 and is used to input the target object image into the target object network model to obtain the three-dimensional 3D rotation value of at least one joint of the target object. The target object network model is trained by machine learning using the ninth dataset. The data in the ninth dataset includes the target object image and the 3D rotation values ​​of each joint of the target object corresponding to the target object image.

[0126] The third generation module 1106, connected to the third recognition module 1104, is used to generate a virtual image of the target object based on the three-dimensional 3D rotation value of at least one joint of the target object.

[0127] It should be noted that the third acquisition module 1102, the third identification module 1104, and the third generation module 1106 mentioned above correspond to steps S502 to S506 in Embodiment 1. The three modules and their corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in Embodiment 1. It should be noted that the above modules, as part of the device, can run in the computer terminal 10 provided in Embodiment 1.

[0128] Example 6

[0129] According to an embodiment of the present invention, an apparatus for implementing the above-described image processing method five is also provided. Figure 12 This is a structural block diagram of the image processing apparatus five according to Embodiment 6 of the present invention, as follows: Figure 12As shown, the device includes: a fourth acquisition module 1202, a fourth identification module 1204, and a fourth generation module 1206. The device will now be described in detail.

[0130] The fourth acquisition module 1202 is used to acquire the target image;

[0131] The fourth identification module 1204 is connected to the fourth acquisition module 1202 and is used to identify the three-dimensional rotation value of at least one joint of the target object from the target image.

[0132] The fourth generation module 1206, connected to the fourth recognition module 1204, is used to generate and display a virtual object of the target object based on the three-dimensional rotation value.

[0133] It should be noted that the fourth acquisition module 1202, the fourth identification module 1204, and the fourth generation module 1206 mentioned above correspond to steps S602 to S606 in Embodiment 1. The three modules and their corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in Embodiment 1. It should be noted that the above modules, as part of the device, can run in the computer terminal 10 provided in Embodiment 1.

[0134] Example 7

[0135] Embodiments of the present invention can provide a computer terminal, which can be any computer terminal device in a group of computer terminals. Optionally, in this embodiment, the computer terminal can also be replaced by a mobile terminal or other terminal device.

[0136] Optionally, in this embodiment, the computer terminal may be located in at least one of a plurality of network devices in a computer network.

[0137] In this embodiment, the computer terminal can execute the program code of the following steps in the image processing method of the application: acquiring a human body image; inputting the human body image into a human body pose network model to obtain the three-dimensional 3D rotation value of at least one joint of the human body, wherein the human body pose network model is obtained by machine learning training using a first dataset, and the data in the first dataset includes: a human body image and the 3D rotation value of each joint of the human body corresponding to the human body image; generating a virtual human based on the three-dimensional 3D rotation value of at least one joint of the human body.

[0138] Optionally, Figure 13 This is a structural block diagram of a computer terminal according to an embodiment of the present invention. Figure 13 As shown, the computer terminal may include one or more (only one is shown in the figure) processors 1302, memory 1304, etc.

[0139] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the image processing method and apparatus in this embodiment of the invention. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby realizing the aforementioned image processing method. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to a computer terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0140] The processor can access the information and application programs stored in the memory via the transmission device to perform the following steps: acquiring a human body image; inputting the human body image into a human pose network model to obtain the three-dimensional 3D rotation values ​​of at least one joint of the human body, wherein the human pose network model is trained using a first dataset through machine learning, and the data in the first dataset includes: a human body image and the 3D rotation values ​​of each joint of the human body corresponding to the human body image; and generating a virtual human based on the three-dimensional 3D rotation values ​​of at least one joint of the human body.

[0141] Optionally, the processor may also execute program code for the following steps: acquiring a human image, including: receiving an input image; using a detection network model to detect the input image and detect the human image from the input image, wherein the detection network model is trained using a second dataset through machine learning, and the data in the second dataset includes: the input image and the human image in the input image.

[0142] Optionally, the processor may also execute program code for the following steps: using a detection network model to detect the input image and extract a face image from it, wherein the detection network model is trained using a third dataset via machine learning, the data in the third dataset including the input image and the face image within the input image; using a face parameter model to extract parameters from the face image to obtain facial expression parameters, wherein the face parameter model is trained using a fourth dataset via machine learning, the data in the fourth dataset including the face image and the facial expression parameters of the face image; and generating a virtual human based on the 3D rotation values ​​of at least one joint of the human body, including generating a virtual human based on the 3D rotation values ​​of at least one joint of the human body and the facial expression parameters.

[0143] Optionally, the processor may also execute program code for the following steps: using a detection network model to detect the input image and extract gesture images from it, wherein the detection network model is trained using a fifth dataset via machine learning, the data in the fifth dataset including the input image and the gesture images within it; using a gesture classification model to classify the gesture images and obtain the hand gesture categories within them, wherein the gesture classification model is trained using a sixth dataset via machine learning, the data in the sixth dataset including the gesture images and the hand gesture categories within them; and generating a virtual human based on the 3D rotation values ​​of at least one joint of the human body, including generating the virtual human based on the 3D rotation values ​​of at least one joint of the human body, facial expression parameters, and gesture categories.

[0144] Optionally, the processor may also execute program code for the following steps: using a detection network model to detect the input image and detect foot images from the input image, wherein the detection network model is trained using a seventh dataset through machine learning, the data in the seventh dataset including: the input image and the foot images in the input image; using a foot recognition model to perform action recognition on the foot images to obtain the foot actions in the foot images, wherein the foot recognition model is trained using an eighth dataset through machine learning, the data in the eighth dataset including: foot images and the foot actions in the foot images; and generating a virtual human based on the three-dimensional 3D rotation values ​​of at least one joint of the human body, including: generating a virtual human based on the three-dimensional 3D rotation values ​​of at least one joint of the human body, facial expression parameters, gesture categories, and foot actions.

[0145] Optionally, the processor may also execute program code for the following steps: when the input image is a multi-frame video image, generate a virtual human based on the three-dimensional 3D rotation value of at least one joint of the human body, facial expression parameters, gesture category, and foot movement, including: inputting the three-dimensional 3D rotation value of at least one joint of the human body, facial expression parameters, gesture category, and foot movement corresponding to each frame of the multi-frame video image into a predetermined drive engine; and forming an animation of the virtual human through the drive engine.

[0146] Optionally, the processor may also execute program code that includes at least one of the following steps: multi-frame video images include multi-frame video images in a recorded video, or multi-frame video images in a live video.

[0147] Optionally, the processor may also execute program code that performs the following steps: receiving an interaction instruction, wherein the interaction instruction is used to interact with the virtual human; obtaining an input image corresponding to the interaction instruction based on the interaction instruction; and adjusting the virtual human according to the input image to generate a new virtual human.

[0148] The processor can also invoke information and applications stored in memory via a transmission device to perform the following steps: receiving interaction instructions, wherein the interaction instructions are used to interact with the virtual human; and displaying the interaction results corresponding to the interaction instructions on a display interface.

[0149] Optionally, the processor may also execute program code that performs the following steps: displaying a human body image on a display interface; displaying the three-dimensional 3D rotation value of at least one joint of the human body on the display interface, wherein the three-dimensional 3D rotation value of at least one joint of the human body is obtained by processing the human body image using a human pose network model.

[0150] Optionally, the processor may also execute program code that performs the following steps: displaying the interaction result corresponding to the interaction command on the display interface, including: receiving an input image based on the interaction command; using a detection network model to detect the input image, and detecting at least one of the following images from the input image: human body image, face image, gesture image, foot image, wherein the detection network model is trained by machine learning using a second dataset, and the data in the second dataset includes: the input image and at least one of the following images in the input image: human body image, face image, gesture image, foot image; performing at least one of the following operations: obtaining the three-dimensional 3D rotation value of at least one joint of the human body based on the human body image, obtaining facial expression parameters based on the face image, obtaining gesture classification based on the gesture image, and obtaining foot movement based on the foot image; generating the interaction result corresponding to the input image based on at least one of the following: the three-dimensional 3D rotation value of at least one joint of the human body, facial expression parameters, gesture classification, and foot movement; and displaying the interaction result.

[0151] The processor can also access information and applications stored in the memory via a transmission device to perform the following steps: acquiring a sample dataset, wherein the sample dataset includes: a target object image and the true 3D rotation values ​​of each joint of the target object corresponding to the target object image; inputting the target object image from the sample dataset into the target object network model to obtain the output 3D rotation values ​​of each joint of the target object; comparing the output 3D rotation values ​​with the true 3D rotation values ​​to obtain the difference; optimizing the model parameters of the target object network model based on the difference to obtain the training result of the target object network model.

[0152] The processor can also call the information and application programs stored in the memory through the transmission device to perform the following steps: acquiring a target object image; inputting the target object image into a target object network model to obtain the three-dimensional 3D rotation values ​​of at least one joint of the target object, wherein the target object network model is trained by machine learning using a ninth dataset, and the data in the ninth dataset includes the target object image and the 3D rotation values ​​of each joint of the target object corresponding to the target object image; and generating a virtual image of the target object based on the three-dimensional 3D rotation values ​​of at least one joint of the target object.

[0153] Optionally, the processor may also execute program code that includes at least one of the following: an animal, a moving machine.

[0154] The processor can also call the information and application programs stored in the memory through the transmission device to perform the following steps: acquire a target image; identify the three-dimensional rotation value of at least one joint of the target object from the target image; and generate and display a virtual object of the target object based on the three-dimensional rotation value.

[0155] Those skilled in the art will understand that Figure 13 The structure shown is for illustrative purposes only. The computer terminal can also be a smartphone (such as an Android phone, an iOS phone, etc.), a tablet computer, a mobile internet device (MID), a PAD, and other terminal devices. Figure 13 This does not limit the structure of the aforementioned electronic devices. For example, a computer terminal may also include components that are more... Figure 13 The more or fewer components shown (such as network interfaces, display devices, etc.), or having the same Figure 13 The different configurations shown.

[0156] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0157] Example 8

[0158] Embodiments of the present invention also provide a storage medium. Optionally, in this embodiment, the storage medium can be used to store the program code executed by the image processing method provided in Embodiment 1.

[0159] Optionally, in this embodiment, the storage medium may be located in any computer terminal in a group of computer terminals in a computer network, or in any mobile terminal in a group of mobile terminals.

[0160] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps: acquiring a human body image; inputting the human body image into a human pose network model to obtain the three-dimensional 3D rotation values ​​of at least one joint of the human body, wherein the human pose network model is trained by machine learning using a first dataset, and the data in the first dataset includes: a human body image and the 3D rotation values ​​of each joint of the human body corresponding to the human body image; generating a virtual human based on the three-dimensional 3D rotation values ​​of at least one joint of the human body.

[0161] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps: acquiring a human image, including: receiving an input image; using a detection network model to detect the input image and detect the human image from the input image, wherein the detection network model is trained using a second dataset through machine learning, and the data in the second dataset includes: the input image and the human image in the input image.

[0162] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps: the method further includes: using a detection network model to detect an input image and detect a face image from the input image, wherein the detection network model is also trained using a third dataset through machine learning, the data in the third dataset including: the input image and the face image in the input image; using a face parameter model to extract parameters from the face image to obtain facial expression parameters of the face image, wherein the face parameter model is trained using a fourth dataset through machine learning, the data in the fourth dataset including: the face image and the facial expression parameters of the face image; wherein generating a virtual human based on the three-dimensional 3D rotation values ​​of at least one joint of the human body includes: generating a virtual human based on the three-dimensional 3D rotation values ​​of at least one joint of the human body and the facial expression parameters.

[0163] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps: the method further includes: using a detection network model to detect the input image and detect gesture images from the input image, wherein the detection network model is also trained by machine learning using a fifth dataset, the data in the fifth dataset including: the input image and the gesture images in the input image; using a gesture classification model to classify the gesture images to obtain the gesture categories of the hands in the gesture images, wherein the gesture classification model is trained by machine learning using a sixth dataset, the data in the sixth dataset including: the gesture images and the gesture categories of the hands in the gesture images; wherein generating a virtual human based on the three-dimensional 3D rotation values ​​of at least one joint of the human body includes: generating a virtual human based on the three-dimensional 3D rotation values ​​of at least one joint of the human body, facial expression parameters, and gesture categories.

[0164] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps: the method further includes: using a detection network model to detect an input image and detect foot images from the input image, wherein the detection network model is also trained using a seventh dataset through machine learning, the data in the seventh dataset including: the input image and the foot images in the input image; using a foot recognition model to perform action recognition on the foot images to obtain foot actions in the foot images, wherein the foot recognition model is trained using an eighth dataset through machine learning, the data in the eighth dataset including: foot images and the foot actions in the foot images; wherein generating a virtual human based on the three-dimensional 3D rotation values ​​of at least one joint of the human body includes: generating a virtual human based on the three-dimensional 3D rotation values ​​of at least one joint of the human body, facial expression parameters, gesture categories, and foot actions.

[0165] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps: when the input image is a multi-frame video image, generating a virtual human based on the three-dimensional 3D rotation values ​​of at least one joint of the human body, facial expression parameters, gesture category, and foot movements, including: inputting the three-dimensional 3D rotation values ​​of at least one joint of the human body, facial expression parameters, gesture category, and foot movements corresponding to each frame of the multi-frame video image into a predetermined drive engine; and forming an animation of the virtual human through the drive engine.

[0166] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps: the multi-frame video images include at least one of the following: multi-frame video images in a recorded video, multi-frame video images in a live video.

[0167] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps: receiving an interaction instruction, wherein the interaction instruction is used to interact with the virtual human; obtaining an input image corresponding to the interaction instruction based on the interaction instruction; and adjusting the virtual human according to the input image to generate a new virtual human.

[0168] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps: displaying a virtual human on a display interface, wherein the virtual human is generated based on the three-dimensional 3D rotation values ​​of at least one joint of the human body; receiving an interaction instruction, wherein the interaction instruction is used to interact with the virtual human; and displaying the interaction result corresponding to the interaction instruction on the display interface.

[0169] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps: displaying a human body image on a display interface; displaying the three-dimensional 3D rotation value of at least one joint of the human body on the display interface, wherein the three-dimensional 3D rotation value of at least one joint of the human body is obtained by processing the human body image using a human pose network model.

[0170] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps: displaying the interaction result corresponding to the interaction command on the display interface, including: receiving an input image based on the interaction command; using a detection network model to detect the input image, detecting at least one of the following images from the input image: human body image, face image, gesture image, foot image, wherein the detection network model is trained by machine learning using a second dataset, and the data in the second dataset includes: the input image and at least one of the following images in the input image: human body image, face image, gesture image, foot image; performing at least one of the following operations: obtaining the three-dimensional 3D rotation value of at least one joint of the human body based on the human body image, obtaining facial expression parameters based on the face image, obtaining gesture classification based on the gesture image, and obtaining foot movement based on the foot image; generating the interaction result corresponding to the input image based on at least one of the following: the three-dimensional 3D rotation value of at least one joint of the human body, facial expression parameters, gesture classification, and foot movement; and displaying the interaction result.

[0171] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps: obtaining a sample dataset, wherein the data in the sample dataset includes: a target object image and the true 3D rotation values ​​of each joint of the target object corresponding to the target object image; inputting the target object image in the sample dataset into a target object network model to obtain the output 3D rotation values ​​of each joint of the target object; comparing the output 3D rotation values ​​with the true 3D rotation values ​​to obtain the difference; optimizing the model parameters of the target object network model based on the difference to obtain the training result of the target object network model.

[0172] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps: acquiring a target object image; inputting the target object image into a target object network model to obtain the three-dimensional 3D rotation values ​​of at least one joint of the target object, wherein the target object network model is trained using a ninth dataset through machine learning, and the data in the ninth dataset includes the target object image and the 3D rotation values ​​of each joint of the target object corresponding to the target object image; and generating a virtual image of the target object based on the three-dimensional 3D rotation values ​​of at least one joint of the target object.

[0173] Optionally, in this embodiment, the storage medium is also configured to store program code for performing the following steps: the target object includes at least one of the following: an animal, a machine-like moving object.

[0174] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps: acquiring a target image; identifying the three-dimensional rotation value of at least one joint of the target object from the target image; and generating and displaying a virtual object of the target object based on the three-dimensional rotation value.

[0175] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0176] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0177] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection of units or modules may be electrical or other forms.

[0178] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0179] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0180] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0181] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. An image processing method, characterized in that, include: Acquire human body images; The human body image is input into the human body pose network model to obtain the three-dimensional 3D rotation value of at least one joint of the human body. The human body pose network model is trained by machine learning using a first dataset. The data in the first dataset includes: human body image and 3D rotation values ​​of each joint of the human body corresponding to the human body image. A virtual human is generated based on the three-dimensional 3D rotation value of at least one joint of the human body; The method further includes: displaying the virtual human in a display interface and receiving interaction instructions, wherein the interaction instructions are used to interact with the virtual human.

2. The method according to claim 1, characterized in that, Acquiring the human body image includes: Receive input image; A detection network model is used to detect the input image and detect the human body image from the input image. The detection network model is trained using a second dataset through machine learning. The data in the second dataset includes the input image and the human body image in the input image.

3. The method according to claim 2, characterized in that, The method further includes: The detection network model is used to detect the input image and detect the face image from the input image. The detection network model is also trained by machine learning using a third dataset, which includes the input image and the face image in the input image. A face parameter model is used to extract parameters from the face image to obtain the facial expression parameters of the face image. The face parameter model is trained by machine learning using a fourth dataset, and the data in the fourth dataset includes: face image and facial expression parameters of the face image. The virtual human is generated based on the three-dimensional 3D rotation values ​​of at least one joint of the human body, including: The virtual human is generated based on the 3D rotation values ​​of at least one joint of the human body and the facial expression parameters.

4. The method according to claim 3, characterized in that, The method further includes: The detection network model is used to detect the input image and detect the gesture image from the input image. The detection network model is also trained by machine learning using a fifth dataset, which includes the input image and the gesture image in the input image. A gesture classification model is used to classify the gesture images to obtain the gesture categories of the hands in the gesture images. The gesture classification model is trained using a sixth dataset through machine learning. The data in the sixth dataset includes: gesture images and the gesture categories of the hands in the gesture images. The virtual human is generated based on the three-dimensional 3D rotation values ​​of at least one joint of the human body, including: The virtual human is generated based on the three-dimensional 3D rotation value of at least one joint of the human body, the facial expression parameters, and the gesture category.

5. The method according to claim 4, characterized in that, The method further includes: The detection network model is used to detect the input image and detect the foot image from the input image. The detection network model is also trained by machine learning using a seventh dataset, which includes the input image and the foot image in the input image. A foot recognition model is used to perform action recognition on the foot image to obtain the foot action in the foot image. The foot recognition model is trained by machine learning using an eighth dataset, which includes: foot images and foot actions in the foot images. The virtual human is generated based on the three-dimensional 3D rotation values ​​of at least one joint of the human body, including: The virtual human is generated based on the three-dimensional 3D rotation value of at least one joint of the human body, the facial expression parameters, the gesture category, and the foot movement.

6. The method according to claim 5, characterized in that, When the input image is a multi-frame video image, the virtual human is generated based on the 3D rotation value of at least one joint of the human body, the facial expression parameters, the gesture category, and the foot movement, including: The three-dimensional 3D rotation value of at least one joint of the human body corresponding to each video image in the multi-frame video images, the facial expression parameters, the gesture category and the foot movement are input into the predetermined drive engine; The animation of the virtual human is generated by the drive engine.

7. The method according to claim 6, characterized in that, The multi-frame video images include at least one of the following: Multiple frames of video images in a recorded video, or multiple frames of video images in a live stream.

8. The method according to any one of claims 1 to 7, characterized in that, After receiving the interaction command, the method further includes: Based on the interaction command, obtain the input image corresponding to the interaction command; Based on the input image, the virtual human is adjusted to generate a new virtual human.

9. An image processing method, characterized in that, include: A virtual human is displayed on the display interface, wherein the virtual human is generated based on the three-dimensional 3D rotation value of at least one joint of the human body; Receive interaction instructions, wherein the interaction instructions are used to interact with the virtual human; The interaction results corresponding to the interaction commands are displayed on the display interface.

10. The method according to claim 9, characterized in that, Also includes: The display interface shows a human body image; The display interface shows the three-dimensional 3D rotation value of at least one joint of the human body, wherein the three-dimensional 3D rotation value of at least one joint of the human body is obtained by processing the human body image using a human pose network model.

11. The method according to claim 9, characterized in that, The display interface shows the interaction results corresponding to the interaction command, including: Receive input images based on interactive commands; A detection network model is used to detect the input image, and at least one of the following images is detected from the input image: human body image, face image, gesture image, and foot image. The detection network model is trained by machine learning using a second dataset. The data in the second dataset includes: the input image and at least one of the following images in the input image: human body image, face image, gesture image, and foot image. Perform at least one of the following operations: obtain a three-dimensional 3D rotation value of at least one joint of the human body based on the human body image, obtain facial expression parameters based on the face image, obtain gesture classification based on the gesture image, and obtain foot movements based on the foot image; Generate an interactive result corresponding to the input image based on at least one of the following: the three-dimensional 3D rotation value of at least one joint of the human body, the facial expression parameter, the gesture classification, and the foot movement; The interaction results are displayed.

12. An image processing apparatus, characterized in that, include: The acquisition module is used to acquire human body images; The recognition module is used to input the human body image into the human body posture network model to obtain the three-dimensional 3D rotation value of at least one joint of the human body. The human body posture network model is trained by machine learning using a first dataset. The data in the first dataset includes: human body image and 3D rotation values ​​of each joint of the human body corresponding to the human body image. A generation module is used to generate a virtual human based on the three-dimensional 3D rotation values ​​of at least one joint of the human body; The device is further configured to: display the virtual human on a display interface and receive interaction instructions, wherein the interaction instructions are used to interact with the virtual human.

13. An image processing apparatus, characterized in that, include: The first display module is used to display a virtual human on a display interface, wherein the virtual human is generated based on the three-dimensional 3D rotation values ​​of at least one joint of the human body; A receiving module is used to receive interaction instructions, wherein the interaction instructions are used to interact with the virtual human; The second display module is used to display the interaction result corresponding to the interaction command on the display interface.

14. A storage medium, characterized in that, The storage medium includes a stored program, wherein, when the program is executed, it controls the device where the storage medium is located to perform the image processing method according to any one of claims 1 to 11.

15. A computer device, characterized in that, include: Memory and processor The memory stores computer programs; The processor is configured to execute a computer program stored in the memory, wherein when the computer program is executed, the processor performs the image processing method according to any one of claims 1 to 11.

16. An image processing method, characterized in that, include: Obtain a sample dataset, wherein the data in the sample dataset includes: a target object image and the actual 3D rotation values ​​of each joint of the target object corresponding to the target object image; Input the target object image from the sample dataset into the target object network model to obtain the output 3D rotation values ​​of each joint of the target object; The difference is obtained by comparing the output 3D rotation value with the actual 3D rotation value; The model parameters of the target object network model are optimized based on the difference to obtain the training result of the target object network model; The virtual image of the target object is displayed on the display interface, and the user interacts with the virtual image based on the received interaction commands.

17. An image processing method, characterized in that, include: Obtain the image of the target object; The target object image is input into the target object network model to obtain the three-dimensional 3D rotation value of at least one joint of the target object. The target object network model is trained by machine learning using the ninth dataset. The data in the ninth dataset includes the target object image and the 3D rotation values ​​of each joint of the target object corresponding to the target object image. A virtual image of the target object is generated based on the three-dimensional 3D rotation value of at least one joint of the target object. The method further includes: displaying the virtual image on a display interface and receiving an interaction command, wherein the interaction command is used to interact with the virtual image.

18. The method according to claim 17, characterized in that, The target object includes at least one of the following: an animal, a moving machine.

19. An image processing method, characterized in that, include: Acquire the target image; The target image is input into the human pose mesh model to identify the three-dimensional rotation value of at least one joint of the target object. The human pose mesh model is trained by machine learning using a first dataset. The data in the first dataset includes: the target image and the 3D rotation values ​​of each joint of the human body corresponding to the target image. Based on the three-dimensional rotation values, a virtual object of the target object is generated and displayed; The method further includes: displaying the virtual object in a display interface and receiving an interaction command, wherein the interaction command is used to interact with the virtual object.

Images