AIGC-based digital human and holographic display method, system and storage medium

Abstract

The present invention relates to the field of image communication technology, and more specifically to a digital human and holographic display method, system and storage medium based on AIGC. The method comprises: step S1: obtaining a multi-angle operation video, establishing a coordinate system with the center position of the operating table as the coordinate origin, and obtaining the initial position coordinates of the target object, the operating object and the operating table; step S2: obtaining the skeletal coordinate change trajectory of each sub-action and the motion coordinate change trajectory of each operating object; step S3: placing the digital human, the virtual operating table and the virtual operating object in the virtual space; step S4: generating a first virtual animation corresponding to each pronunciation through a voice model, and obtaining a second virtual animation corresponding to the explanation content based on the first virtual animation; step S5: generating a virtual operation video, and holographically displaying the virtual operation video. The present invention solves the problem of poor authenticity of digital humans and improves the authenticity of digital humans.

Description

Technical Field

[0001] The present invention relates to the field of image communication technology, and more specifically to a digital human and holographic display method, system and storage medium based on AIGC. Background Art

[0002] In recent years, new display technologies such as naked-eye 3D and holographic display have gradually been applied to the display of digital humans.

[0003] These technologies can provide users with a more realistic and immersive experience, allowing users to interact with digital humans without wearing any equipment. For example, through holographic display devices, digital humans can present a three-dimensional image in the real world, and the audience can interact with them without wearing any equipment. Digital humans are widely used in vocational training, which can simulate real work scenes and processes and help trainees improve their skills. Similar prior art includes a Chinese patent with publication number CN117540051A, which proposes a business display method based on virtual digital humans, the method comprising: obtaining a first user voice input by a user for a target virtual human; determining a target sub-process to be executed from a created business display process according to the first user voice, the business display process including multiple sub-processes, and different sub-processes are used to generate display data for different businesses; generating display data of corresponding businesses according to the target sub-process to obtain target display data; and driving the target virtual human to display corresponding businesses according to the target display data, so that the user can flexibly drive the target virtual human to select the required business for display by interacting with the target virtual human, without manual operation, simplifying the process of business display. In addition, similar prior art includes a Japanese patent with publication number JP2022545851A, which discloses a virtual object control method, device, apparatus and computer-readable storage medium. The method includes collecting multiple frames of interactive images of interactive objects in real scenes and display images of target display objects in real scenes respectively; determining the posture information of virtual objects according to the state information of interactive targets; determining the virtual effect data of target exhibition objects according to the exhibition images; rendering using the posture information and virtual effect data to obtain virtual effect images, and presenting augmented reality effects including virtual effect images. Both of the above documents solve the problem of displaying virtual objects, but the realism of virtual objects is not high enough, resulting in a relatively poor user experience. Summary of the invention

[0004] In order to better solve the above problems, the present invention provides a digital human and holographic display method based on AIGC, and the method comprises the following steps:

[0005] Step S1: acquiring a multi-angle operation video through a camera unit, establishing a coordinate system with the center position of the operation platform in the operation video as the coordinate origin, and acquiring the initial position coordinates of the target object, the operation object and the operation platform respectively;

[0006] Step S2: extracting the operation action of the target object and the motion trajectory of the operation object based on the operation video, dividing the operation action into multiple sub-actions, and obtaining the bone coordinate change trajectory corresponding to each sub-action and the motion coordinate change trajectory corresponding to each sub-action of the operation object;

[0007] Step S3: The virtual operation table, the digital human and the virtual operation object are obtained by the virtual unit based on the AIGC technology, and the appearance of the digital human corresponding to the target object is selected according to the virtual content library to create a virtual space, and the digital human, the virtual operation table and the virtual operation object are placed in the virtual space according to the initial position coordinates of the target object, the operation table and the operation object in the coordinate system;

[0008] Step S4: extracting explanation contents from the explanation voice corresponding to each sub-action in the operation video, generating a first virtual animation corresponding to each pronunciation through a voice model based on the lip type, facial image and explanation contents of the digital human, and acquiring a second virtual animation corresponding to the explanation contents based on the first virtual animation;

[0009] Step S5: Generate a virtual operation video based on the digital human and the skeletal coordinate change trajectory corresponding to each of the sub-actions, the virtual operating table, the virtual operation object, the motion coordinate change trajectory of the operation object, the second virtual animation and the virtual voice, and display the virtual operation video holographically.

[0010] As a preferred technical solution of the present invention, step S1 includes the following steps:

[0011] Step S11: shooting the operation video by using the cameras of the camera unit at multiple angles, wherein the operation video is a video of the target object demonstrating the standard operation of setting the production process;

[0012] Step S12: Taking the center position of the operating table in the operation video as the coordinate origin of the coordinate system, obtaining the boundary coordinates of the operating table, and using the boundary coordinates as the initial position coordinates of the operating table, and also obtaining the initial position coordinates of the target object and the operation object.

[0013] As a preferred technical solution of the present invention, step S2 includes the following steps:

[0014] Step S21: extracting the operation actions at multiple angles of the target object and the motion trajectories at multiple angles of the operation object from the operation video;

[0015] Step S22: Divide the operation action into multiple sub-actions according to the operation process, and obtain the bone coordinate change trajectory of each sub-action and the motion coordinate change trajectory of the operation object corresponding to the sub-action based on the operation action and the bone extraction unit, wherein the bone coordinate change trajectory of the sub-action and the motion coordinate change trajectory of the operation object are both three-dimensional trajectories.

[0016] As a preferred technical solution of the present invention, the generation of the speech model in step S4 includes the following steps:

[0017] Step S411: extracting speech content, lip type and facial image of speech object from multiple speech videos in the database, and extracting lip and teeth change animation and neck and mandible change animation corresponding to each word pronunciation in the speech content from the speech video, and taking the lip type of speech object in each speech video, each word pronunciation in the speech content and the lip and teeth change animation and neck and mandible change animation corresponding to each word pronunciation as a training data set;

[0018] Step S412: creating the speech model, and training the speech model using a plurality of the training data sets and a machine learning algorithm;

[0019] The character pronunciation is the pinyin of each character, and the lip and teeth change animation is the animation of the change of the lip shape and the change of the area and number of teeth during the pronunciation.

[0020] As a preferred technical solution of the present invention, step S4 includes:

[0021] Extracting the explanation voice corresponding to each sub-action from the operation video, and obtaining the explanation content from the explanation voice, inputting the pronunciation of each word in the explanation content, the lip type and facial image of the digital human into the voice model, generating and using the virtual animation of the lips and teeth and the virtual animation of the jaw and neck of the digital human corresponding to each word pronunciation as the first virtual animation;

[0022] The first virtual animations corresponding to the pronunciation of each character in the explanation content are spliced ​​to obtain the second virtual animation corresponding to the explanation content.

[0023] As a preferred technical solution of the present invention, step S5 includes:

[0024] Step S51: generating a virtual sub-action corresponding to the digital human based on the appearance of the digital human and the skeletal coordinate change trajectory corresponding to the sub-action, and acquiring the virtual operation object trajectory based on the virtual operation object and the motion coordinate change trajectory of the operation object corresponding to the sub-action;

[0025] Step S52: adjusting the second virtual animation corresponding to the explanation content of the sub-action in real time according to the facial angle change and speech speed change of the target object corresponding to the sub-action, and superimposing the facial expression of the target object corresponding to the sub-action on the basis of the adjusted second virtual animation, and obtaining a third virtual animation;

[0026] Step S53: Generate the virtual operation video based on the virtual operating console in the virtual space, each virtual sub-action and the virtual operation object trajectory corresponding to each virtual sub-action, the third virtual animation and the virtual voice, and display the virtual operation video holographically.

[0027] As a preferred technical solution of the present invention, the step S5 also includes: calculating the first vector and the second vector between the action trajectory of the virtual sub-action and the virtual operating table and the trajectory of the virtual operation object respectively, and also calculating the third vector and the fourth vector between the action trajectory of the sub-action and the motion coordinate change trajectory of the operating table and the operation object respectively, obtaining the first vector difference between the first vector and the third vector, and the second vector difference between the second vector and the fourth vector, and when the first vector difference is greater than or equal to the first set difference or the second vector difference is greater than or equal to the second set difference, adjusting the action trajectory of the virtual sub-action and the trajectory of the virtual operation object so that the first vector difference and the second vector difference are respectively smaller than the first set difference and the second set difference.

[0028] As a preferred technical solution of the present invention, the step S5 further includes step S6:

[0029] The user's actual operation video is obtained through the camera unit, and the actual operation object motion trajectory corresponding to each actual operation sub-action of the user is extracted from the actual operation video, and the actual operation object motion trajectory is compared with the virtual operation object trajectory, and the trajectory difference is obtained. When the trajectory difference is greater than the set trajectory difference, the virtual sub-action corresponding to the actual operation sub-action and the virtual operation object trajectory are amplified and displayed in multiple dimensions, and an early warning is issued, wherein the actual operation object motion trajectory and the virtual operation object trajectory respectively include the motion trajectory and deformation trajectory of the actual operation object and the virtual operation object.

[0030] The present invention also provides a digital human and holographic display system based on AIGC, the system is used to implement the above method, and the system includes:

[0031] A camera unit, used to obtain multi-angle operation videos, establish a coordinate system with the center position of the operation platform in the operation video as the coordinate origin, and obtain the initial position coordinates of the target object, the operation object and the operation platform respectively;

[0032] An extraction unit, used to extract the operation action of the target object and the motion trajectory of the operation object based on the operation video, divide the operation action into multiple sub-actions, and obtain the bone coordinate change trajectory corresponding to each sub-action and the motion coordinate change trajectory of the operation object corresponding to each sub-action;

[0033] A virtual unit, for acquiring a virtual operation table, a digital human and a virtual operation object based on the AIGC technology, and selecting the appearance of the digital human corresponding to the target object according to the virtual content library, creating a virtual space, and placing the digital human, the virtual operation table and the virtual operation object in the virtual space according to the initial position coordinates of the target object, the operation table and the operation object in the coordinate system;

[0034] A model creation unit, configured to extract explanation contents from the explanation voice corresponding to each sub-action in the operation video, generate a first virtual animation corresponding to each pronunciation through a voice model based on the lip type, facial image and explanation contents of the digital human, and obtain a second virtual animation corresponding to the explanation contents based on the first virtual animation;

[0035] The video generation unit is used to generate a virtual operation video based on the digital human and the skeletal coordinate change trajectory corresponding to each of the sub-actions, the virtual operation table, the virtual operation object, the motion coordinate change trajectory of the operation object, the second virtual animation and the virtual voice, and to holographically display the virtual operation video.

[0036] The present invention also provides a computer storage medium, wherein the storage medium stores program instructions, wherein when the program instructions are executed, the device where the storage medium is located is controlled to execute the above method.

[0037] Compared with the prior art, the beneficial effects of the present invention are at least as follows:

[0038] The present invention can obtain the skeletal coordinate change trajectory of each sub-action in the above operation video and the motion coordinate change trajectory of the operation object through the multi-angle operation video, extract the explanation voice corresponding to each sub-action from the above operation video, and convert the above explanation voice into the above explanation content, wherein the above explanation content is text information, by inputting each word pronunciation in each of the above explanation contents, that is, the pinyin of the word, the lip type of the above digital human and the facial image of the above digital human into the above voice model, the above facial image includes the image of the face and neck, generate the above first virtual animation of the digital human corresponding to the above word pronunciation, and splice the above word pronunciation according to the order in the above explanation content to obtain the above second virtual animation, generate the virtual sub-action corresponding to the above digital human according to the above skeletal coordinate change trajectory corresponding to the above sub-action, based on the above motion The coordinate change trajectory generates the above-mentioned virtual operation object trajectory, and the above-mentioned second virtual animation is adjusted in real time through the facial angle change of the above-mentioned target object corresponding to the above-mentioned sub-action and the above-mentioned voice speed change, so that the above-mentioned second virtual animation matches the above-mentioned explanation content, and the above-mentioned sub-action is superimposed on the facial expression of the above-mentioned target object on the basis of the above-mentioned second virtual animation, and the above-mentioned third virtual animation is obtained, and the operation action video is generated according to the above-mentioned virtual operating table in the above-mentioned virtual space, each of the above-mentioned virtual sub-actions and the above-mentioned virtual operation object trajectory corresponding to the above-mentioned virtual sub-action, and then the above-mentioned virtual operation video is generated in combination with the above-mentioned third virtual animation and virtual voice. Through the above-mentioned technical solution, a virtual operation video with high fidelity is obtained, and the above-mentioned virtual operation video is displayed in a holographic display manner, so that the user can interact with the digital human without wearing any equipment. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 This is a flow chart of the digital human and holographic display method based on AIGC of the present invention;

[0040] Figure 2 This is a structural diagram of the digital human and holographic display system based on AIGC of the present invention. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0042] The present invention provides a digital human and holographic display method based on AIGC, such as Figure 1 As shown, the method comprises the following steps:

[0043] Step S1: acquiring a multi-angle operation video through a camera unit, establishing a coordinate system with the center position of the operation platform in the operation video as the coordinate origin, and acquiring the initial position coordinates of the target object, the operation object and the operation platform respectively;

[0044] Specifically, the target object is the operator of the operation video, the operation object is the product or semi-finished product corresponding to the set production process, the operating table is the workbench for implementing the operation, the operation video is used for training new employees on the production line for the set production process, and the standard operation video of the set production process is shot by the multi-angle cameras of the camera unit. The three-dimensional motion trajectory of each sub-action in the operation video can be obtained through the multi-angle operation video. Since the standard operation is implemented on the operating table and the operating table is stationary, the operating table is used as a reference, and the center of the operating table is used as the coordinate origin to create a coordinate system, and the boundary coordinates of the operating table are used as the initial position coordinates of the operating table, and the position coordinates of the operation object and the target object before the start of the operation video are used as the initial position coordinates. Through the technical solution, a basis is provided for the relative positions of the virtual operating table, the virtual target object and the virtual operation object, so that the obtained virtual operation video is more realistic.

[0045] Step S2: extracting the operation action of the target object and the motion trajectory of the operation object based on the operation video, dividing the operation action into multiple sub-actions, and obtaining the bone coordinate change trajectory corresponding to each sub-action and the motion coordinate change trajectory corresponding to each sub-action of the operation object;

[0046] Specifically, the operation actions of the target object at multiple angles and the motion trajectory corresponding to the operation object can be extracted through the operation video. The operation action is divided into multiple sub-actions according to the operation process. The skeletal coordinate change trajectory of the sub-action and the motion coordinate change trajectory of the sub-action corresponding to the operation object are obtained according to the motion trajectory of the sub-action and the skeleton extraction unit. Since the skeletal coordinate change trajectory of the sub-action and the motion coordinate change trajectory of the operation object are both three-dimensional, the technical solution can not only three-dimensionalize and quantify the sub-action trajectory and the motion trajectory of the operation object, but also improve the accuracy and authenticity of the digital human and the virtual operation video.

[0047] Step S3: The virtual operation table, the digital human and the virtual operation object are obtained by the virtual unit based on the AIGC technology, and the appearance of the digital human corresponding to the target object is selected according to the virtual content library to create a virtual space, and the digital human, the virtual operation table and the virtual operation object are placed in the virtual space according to the initial position coordinates of the target object, the operation table and the operation object in the coordinate system;

[0048] Specifically, the virtual operating console corresponding to the operating console, the digital human corresponding to the target object, and the virtual operation object corresponding to the operation object are obtained through the virtual unit based on the AIGC technology, and the virtual space is created. The digital human, the virtual operating console, and the virtual operation object are placed in the virtual space according to the initial position coordinates. Through the technical solution, the environment, target object, operating console, and operation object of the operation video are virtualized to provide a virtual environment and initial position coordinates for further obtaining the virtual operation video.

[0049] Step S4: extracting explanation contents from the explanation voice corresponding to each sub-action in the operation video, generating a first virtual animation corresponding to each pronunciation through a voice model based on the lip type, facial image and explanation contents of the digital human, and acquiring a second virtual animation corresponding to the explanation contents based on the first virtual animation;

[0050] Specifically, the explanation voice corresponding to each of the above-mentioned sub-actions is extracted from the above-mentioned operation video, and the above-mentioned explanation voice is converted into the above-mentioned explanation content, wherein the above-mentioned explanation content is text information, by inputting each character pronunciation in each of the above-mentioned explanation contents, i.e. the pinyin of the characters, the lip type of the above-mentioned digital human and the facial image of the above-mentioned digital human into the above-mentioned voice model, the above-mentioned facial image includes the image of the face and neck, and the above-mentioned first virtual animation of the digital human corresponding to the above-mentioned character pronunciation is generated, wherein the appearance of the above-mentioned digital human includes the facial image, and the above-mentioned first virtual animation includes the above-mentioned lip and teeth virtual animation and the above-mentioned mandible and neck virtual animation, the above-mentioned first virtual animation corresponding to each of the above-mentioned character pronunciation is obtained, and the above-mentioned character pronunciations are spliced ​​according to the order of the sequence in the above-mentioned explanation content to obtain the above-mentioned second virtual animation. Through the above-mentioned technical solution, the above-mentioned second virtual animation corresponding to the explanation content of each of the above-mentioned sub-actions can be obtained, laying a foundation for obtaining a virtual voice animation matching the above-mentioned digital human.

[0051] Step S5: Generate a virtual operation video based on the digital human and the skeletal coordinate change trajectory corresponding to each of the sub-actions, the virtual operating table, the virtual operation object, the motion coordinate change trajectory of the operation object, the second virtual animation and the virtual voice, and display the virtual operation video holographically.

[0052] Specifically, based on the appearance of the digital human, the skeleton of the sub-action is replaced with the corresponding part of the digital human, and the virtual sub-action corresponding to the digital human is generated according to the change track of the skeleton coordinates corresponding to the sub-action. The virtual sub-action includes head action and limb and body action. The virtual operation object is replaced by the virtual operation object, and the track of the virtual operation object is generated based on the change track of the motion coordinates. The second virtual animation is adjusted in real time according to the facial angle change of the target object corresponding to the sub-action and the change of the voice speed, so that the second virtual animation matches the explanation content. The facial expression of the target object corresponding to the sub-action is superimposed on the second virtual animation, and the third virtual animation is obtained. The operation action video is generated according to the virtual operation table in the virtual space, each of the virtual sub-actions and the track of the virtual operation object corresponding to the virtual sub-action, and the virtual operation video is generated in combination with the third virtual animation and the virtual voice. Through the technical solution, a virtual operation video with high fidelity is obtained, and the virtual operation video is displayed in a holographic display manner, so that the user can interact with the digital human without wearing any equipment.

[0053] Specifically, step S1 includes the following steps:

[0054] Step S11: shooting the operation video by using the cameras of the camera unit at multiple angles, wherein the operation video is a video of the target object demonstrating the standard operation of setting the production process;

[0055] Step S12: Taking the center position of the operating table in the operation video as the coordinate origin of the coordinate system, obtaining the boundary coordinates of the operating table, and using the boundary coordinates as the initial position coordinates of the operating table, and also obtaining the initial position coordinates of the target object and the operation object.

[0056] Specifically, the standard operation video of the set production process is shot by the multi-angle cameras of the camera unit, and the three-dimensional motion trajectory of each sub-action in the operation video can be obtained through the multi-angle operation video. Since the standard operation is implemented on the operation table, and the operation table is stationary, the operation table is used as a reference, and the center of the operation table is used as the coordinate origin to create a coordinate system, and the boundary coordinates of the operation table are used as the initial position coordinates of the operation table, and the position coordinates of the operation object and the target object before the start of the operation video are used as the initial position coordinates. Through the above technical solution, a basis is provided for the relative positions of the virtual operation table, the virtual target object and the virtual operation object, so that the obtained virtual operation video is more realistic.

[0057] Furthermore, the step S2 comprises the following steps:

[0058] Step S21: extracting the operation actions at multiple angles of the target object and the motion trajectories at multiple angles of the operation object from the operation video;

[0059] Step S22: Divide the operation action into multiple sub-actions according to the operation process, and obtain the bone coordinate change trajectory of each sub-action and the motion coordinate change trajectory of the operation object corresponding to the sub-action based on the operation action and the bone extraction unit, wherein the bone coordinate change trajectory of the sub-action and the motion coordinate change trajectory of the operation object are both three-dimensional trajectories.

[0060] Specifically, the operation actions of the target object at multiple angles and the motion trajectory corresponding to the operation object can be extracted through the above-mentioned operation video, and the above-mentioned operation actions are divided into multiple sub-actions according to the above-mentioned operation process. Since the appearance of the digital human can be set when the target object is virtualized into a digital human, the appearance and even the gender of the digital human may not be the same as those of the target object. In order to remove the appearance characteristics of the target object and retain the action characteristics, the bone coordinate change trajectory of the sub-action and the motion coordinate change trajectory of the sub-action corresponding to the operation object are obtained according to the above-mentioned sub-action and the above-mentioned bone extraction unit. Since the bone coordinate change trajectory of the sub-action and the motion coordinate change trajectory of the operation object are both three-dimensional, through the above-mentioned technical solution, not only the sub-action trajectory and the motion trajectory of the operation object are three-dimensionalized and quantified, but also the accuracy and authenticity of the digital human and the virtual operation video are improved.

[0061] Furthermore, the generation of the speech model in step S4 includes the following steps:

[0062] Step S411: extracting speech content, lip type and facial image of speech object from multiple speech videos in the database, and extracting lip and teeth change animation and neck and mandible change animation corresponding to each word pronunciation in the speech content from the speech video, and taking the lip type of speech object in each speech video, each word pronunciation in the speech content and the lip and teeth change animation and neck and mandible change animation corresponding to each word pronunciation as a training data set;

[0063] Step S412: creating the speech model, and training the speech model using a plurality of the training data sets and a machine learning algorithm;

[0064] The character pronunciation is the pinyin of each character, and the lip and teeth change animation is the animation of the change of the lip shape and the change of the area and number of teeth during the pronunciation.

[0065] Specifically, in the process of virtualizing the target object into a digital human, the problem of mismatch between lip shape and explanation content often occurs, resulting in a poor user experience and reducing the training effect. The multiple voice videos are read from a database storing multiple voice videos, and the lip type of the voice object is extracted from the voice video, wherein the lip type is divided according to the shape of the lips, for example: M lips, peach blossom lips, etc. The lip and teeth change animation and the neck and jaw change animation corresponding to each word pronunciation are also extracted from the voice video, wherein the lip and teeth change animation is composed of multiple lip and teeth change images, and the neck and jaw change animation is composed of multiple neck and jaw change images. The word pronunciation, the lip type of the voice object, the lip and teeth change animation and the neck and jaw change animation are used as the training data group, and the voice model is trained. Through the technical solution, the lip and teeth change animation and the jaw and neck change animation corresponding to each word pronunciation can be obtained according to the voice content, lip type and facial image of the digital human, so that the lip shape of the explanation content of each sub-action of the digital human is matched, thereby improving the user experience.

[0066] Furthermore, the step S4 comprises:

[0067] Extracting the explanation voice corresponding to each sub-action from the operation video, and obtaining the explanation content from the explanation voice, inputting the pronunciation of each word in the explanation content, the lip type and facial image of the digital human into the voice model, generating and using the virtual animation of the lips and teeth and the virtual animation of the jaw and neck of the digital human corresponding to each word pronunciation as the first virtual animation;

[0068] The first virtual animations corresponding to the pronunciation of each character in the explanation content are spliced ​​to obtain the second virtual animation corresponding to the explanation content.

[0069] Specifically, the explanation voice corresponding to each of the above-mentioned sub-actions is extracted from the above-mentioned operation video, and the above-mentioned explanation voice is converted into the above-mentioned explanation content, wherein the above-mentioned explanation content is text information, by inputting each character pronunciation in each of the above-mentioned explanation contents, i.e. the pinyin of the characters, the lip type of the above-mentioned digital human and the facial image of the above-mentioned digital human into the above-mentioned voice model, the above-mentioned facial image includes the image of the face and neck, and the above-mentioned first virtual animation of the digital human corresponding to the above-mentioned character pronunciation is generated, wherein the above-mentioned first virtual animation includes the above-mentioned lip and teeth virtual animation and the above-mentioned mandible and neck virtual animation, repeatedly obtaining the above-mentioned first virtual animation corresponding to each of the above-mentioned character pronunciations, and splicing the above-mentioned character pronunciations according to the order of precedence in the above-mentioned explanation content to obtain the above-mentioned second virtual animation, through the above-mentioned technical scheme, the above-mentioned second virtual animation corresponding to the explanation content of each of the above-mentioned sub-actions can be obtained, laying a foundation for obtaining the virtual voice animation matching the above-mentioned digital human.

[0070] Furthermore, the step S5 comprises:

[0071] Step S51: generating a virtual sub-action corresponding to the digital human based on the appearance of the digital human and the skeletal coordinate change trajectory corresponding to the sub-action, and acquiring the virtual operation object trajectory based on the virtual operation object and the motion coordinate change trajectory of the operation object corresponding to the sub-action;

[0072] Specifically, the skeleton of the sub-action is replaced with the corresponding part of the digital human based on the appearance of the digital human, the appearance includes the body image and the head image, and the virtual sub-action corresponding to the digital human is generated according to the change trajectory of the skeleton coordinates corresponding to the sub-action, the virtual sub-action includes the head movement and the limbs and the body movement, and the virtual operation object is replaced with the operation object, and the virtual operation object trajectory is generated based on the movement coordinate change trajectory. Through the technical solution, the virtual sub-action and virtual operation object trajectory corresponding to each sub-action are obtained, laying a foundation for obtaining accurate virtual operation videos.

[0073] Step S52: adjusting the second virtual animation corresponding to the explanation content of the sub-action in real time according to the facial angle change and speech speed change of the target object corresponding to the sub-action, and superimposing the facial expression of the target object corresponding to the sub-action on the basis of the adjusted second virtual animation, and obtaining a third virtual animation;

[0074] Specifically, since the second virtual animation is not adjusted according to the speech speed and angle, it does not match the virtual sub-action. Therefore, the second virtual animation is adjusted in real time according to the facial angle change of the target object and the speech speed change corresponding to the sub-action, so that the second virtual animation matches the explanation content. In order to make the virtual operation video more realistic, the facial expression of the target object corresponding to the sub-action is superimposed on the second virtual animation, and the third virtual animation is obtained. Through the technical solution, the third virtual animation that not only matches the explanation content but also is more realistic can be obtained, thereby improving the authenticity of the virtual operation video.

[0075] Step S53: Generate the virtual operation video based on the virtual operating console in the virtual space, each virtual sub-action and the virtual operation object trajectory corresponding to each virtual sub-action, the third virtual animation and the virtual voice, and display the virtual operation video holographically.

[0076] Specifically, an operation action video is generated according to the virtual operating console in the virtual space, each of the virtual sub-actions and the virtual operation object trajectory corresponding to the virtual sub-actions, and then the virtual operation video is generated in conjunction with the third virtual animation and virtual voice, wherein the virtual voice is the explanation content corresponding to the virtual sub-actions, and the generation of virtual voice is a prior art and will not be repeated here. The virtual operation video is a three-dimensional video, and the display angle is adjusted by user operation. Through the above technical solution, a virtual operation video with high realism is obtained, and the virtual operation video is displayed in a holographic display manner, so that the user can interact with the digital human without wearing any equipment.

[0077] Furthermore, the step S5 also includes: calculating the first vector and the second vector between the motion trajectory of the virtual sub-action and the trajectory of the virtual operating table and the virtual operation object respectively, and also calculating the third vector and the fourth vector between the motion trajectory of the sub-action and the motion coordinate change trajectory of the operating table and the operation object respectively, obtaining the first vector difference between the first vector and the third vector, and the second vector difference between the second vector and the fourth vector, and when the first vector difference is greater than or equal to the first set difference or the second vector difference is greater than or equal to the second set difference, adjusting the motion trajectory of the virtual sub-action and the trajectory of the virtual operation object so that the first vector difference and the second vector difference are respectively smaller than the first set difference and the second set difference.

[0078] Specifically, although the above virtual sub-action is generated by the skeletal coordinate change trajectory of the sub-action of the above target object, since the digital human is the virtual incarnation of the above target object, there are still some subtle differences between the two. The fit between the digital human and the virtual operating table and the operating object may not be very good. Even if there is a small difference in the fit, the realism of the virtual operation video will be reduced. Therefore, by comparing the above first vector with the above third vector, wherein the first vector and the third vector are respectively the distance vector between the virtual sub-action and the virtual operating table and the distance vector between the sub-action of the target object and the operating table at the same time, the above virtual sub-action and the target object are generated by the skeletal coordinate change trajectory of the sub-action of the target object. However, since the digital human is the virtual incarnation of the above target object, there are still some subtle differences between the two. The fit between the digital human and the virtual operating table and the operating object may not be very good. Even if there is a small difference in the fit, the realism of the virtual operation video will be reduced. The second vector and the fourth vector are respectively the distance vector between the virtual sub-action and the trajectory of the virtual operation object and the distance vector between the sub-action of the target object and the trajectory of the motion coordinate change of the operation object at the same time. When the first vector difference is greater than or equal to the first set difference or the second vector difference is greater than or equal to the second set difference, that is, the fit between the virtual sub-action and the virtual operating table is not good or the fit between the virtual sub-action and the virtual operation object is not good. Therefore, by adjusting the virtual sub-action and the trajectory of the virtual operation object, the fit meets the requirement. Through the above technical solution, the realism of the virtual operation video can be further improved.

[0079] Furthermore, the step S5 further includes a step S6:

[0080] The user's actual operation video is obtained through the camera unit, and the actual operation object motion trajectory corresponding to each actual operation sub-action of the user is extracted from the actual operation video, and the actual operation object motion trajectory is compared with the virtual operation object trajectory, and the trajectory difference is obtained. When the trajectory difference is greater than the set trajectory difference, the virtual sub-action corresponding to the actual operation sub-action and the virtual operation object trajectory are amplified and displayed in multiple dimensions, and an early warning is issued, wherein the actual operation object motion trajectory and the virtual operation object trajectory respectively include the motion trajectory and deformation trajectory of the actual operation object and the virtual operation object.

[0081] Specifically, when employees are trained through the above-mentioned virtual operation video, the actual operation video of the employee, i.e. the user, is obtained, and the motion trajectory of the actual operation object corresponding to each actual operation sub-action of the above-mentioned user is obtained therefrom, and the motion trajectory of the above-mentioned actual operation object is compared with the above-mentioned virtual operation object trajectory, and the trajectory difference is obtained. The above-mentioned trajectory difference includes the deformation trajectory and motion trajectory of the actual operation object. Therefore, the operation action difference and the deformation difference can be obtained through the above-mentioned trajectory difference. Both of the above-mentioned differences may cause product quality problems. Therefore, when the above-mentioned trajectory difference is greater than the above-mentioned set trajectory difference, the virtual sub-actions corresponding to the above-mentioned actual operation sub-actions and the above-mentioned virtual operation object trajectories are magnified and displayed in multiple dimensions, so that the display effect is better and the training effect is further improved.

[0082] The present invention also provides a digital human and holographic display system based on AIGC, the system is used to implement the above method, such as Figure 2 As shown, the system comprises:

[0083] A camera unit, used to obtain multi-angle operation videos, establish a coordinate system with the center position of the operation platform in the operation video as the coordinate origin, and obtain the initial position coordinates of the target object, the operation object and the operation platform respectively;

[0084] An extraction unit, used to extract the operation action of the target object and the motion trajectory of the operation object based on the operation video, divide the operation action into multiple sub-actions, and obtain the bone coordinate change trajectory corresponding to each sub-action and the motion coordinate change trajectory of the operation object corresponding to each sub-action;

[0085] A virtual unit, for acquiring a virtual operation table, a digital human and a virtual operation object based on the AIGC technology, and selecting the appearance of the digital human corresponding to the target object according to the virtual content library, creating a virtual space, and placing the digital human, the virtual operation table and the virtual operation object in the virtual space according to the initial position coordinates of the target object, the operation table and the operation object in the coordinate system;

[0086] A model creation unit, configured to extract explanation contents from the explanation voice corresponding to each sub-action in the operation video, generate a first virtual animation corresponding to each pronunciation through a voice model based on the lip type, facial image and explanation contents of the digital human, and obtain a second virtual animation corresponding to the explanation contents based on the first virtual animation;

[0087] The video generation unit is used to generate a virtual operation video based on the digital human and the skeletal coordinate change trajectory corresponding to each of the sub-actions, the virtual operation table, the virtual operation object, the motion coordinate change trajectory of the operation object, the second virtual animation and the virtual voice, and to holographically display the virtual operation video.

[0088] The present invention also provides a computer storage medium, wherein the storage medium stores program instructions, wherein when the program instructions are executed, the device where the storage medium is located is controlled to execute the above method.

[0089] In summary, the present invention can obtain the skeletal coordinate change trajectory of each sub-action and the motion coordinate change trajectory of the operation object in the above-mentioned operation video through multi-angle operation videos, extract the explanation voice corresponding to each of the above-mentioned sub-actions from the above-mentioned operation video, and convert the above-mentioned explanation voice into the above-mentioned explanation content, wherein the above-mentioned explanation content is text information, by inputting each word pronunciation in each of the above-mentioned explanation content, that is, the pinyin of the word, the lip type of the above-mentioned digital human and the facial image of the above-mentioned digital human into the above-mentioned voice model, the above-mentioned facial image includes the image of the face and neck, generate the above-mentioned first virtual animation of the digital human corresponding to the above-mentioned word pronunciation, and splice the above-mentioned word pronunciation according to the order in the above-mentioned explanation content to obtain the above-mentioned second virtual animation, generate the virtual sub-action corresponding to the above-mentioned digital human according to the above-mentioned skeletal coordinate change trajectory corresponding to the above-mentioned sub-action, based on the above The motion coordinate change trajectory generates the virtual operation object trajectory, and the second virtual animation is adjusted in real time through the facial angle change of the target object corresponding to the sub-action and the voice speed change, so that the second virtual animation matches the explanation content, and the facial expression of the target object corresponding to the sub-action is superimposed on the second virtual animation, and the third virtual animation is obtained, and the operation action video is generated according to the virtual operating table in the virtual space, each of the virtual sub-actions and the virtual operation object trajectory corresponding to the virtual sub-action, and the virtual operation video is generated in combination with the third virtual animation and the virtual voice. Through the above technical solution, a virtual operation video with high fidelity is obtained, and the virtual operation video is displayed in a holographic display manner, so that the user can interact with the digital human without wearing any equipment.

[0090] The technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above-mentioned embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0091] The above-mentioned embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

[0092] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A digital human and holographic display method based on AIGC, characterized in that: The method comprises the following steps: Step S1: acquiring a multi-angle operation video through a camera unit, establishing a coordinate system with the center position of the operation platform in the operation video as the coordinate origin, and acquiring the initial position coordinates of the target object, the operation object and the operation platform respectively; Step S2: extracting the operation action of the target object and the motion trajectory of the operation object based on the operation video, dividing the operation action into multiple sub-actions, and obtaining the bone coordinate change trajectory corresponding to each sub-action and the motion coordinate change trajectory corresponding to each sub-action of the operation object; Step S3: The virtual operation table, the digital human and the virtual operation object are obtained by the virtual unit based on the AIGC technology, and the appearance of the digital human corresponding to the target object is selected according to the virtual content library to create a virtual space, and the digital human, the virtual operation table and the virtual operation object are placed in the virtual space according to the initial position coordinates of the target object, the operation table and the operation object in the coordinate system; Step S4: extracting explanation contents from the explanation voice corresponding to each sub-action in the operation video, generating a first virtual animation corresponding to each pronunciation through a voice model based on the lip type, facial image and explanation contents of the digital human, and acquiring a second virtual animation corresponding to the explanation contents based on the first virtual animation; Step S5: Generate a virtual operation video based on the digital human and the skeletal coordinate change trajectory corresponding to each of the sub-actions, the virtual operating table, the virtual operation object, the motion coordinate change trajectory of the operation object, the second virtual animation and the virtual voice, and perform a holographic display of the virtual operation video.

2. The method according to claim 1, characterized in that The step S1 comprises the following steps: Step S11: shooting the operation video by using the cameras of the camera unit at multiple angles, wherein the operation video is a video of the target object demonstrating the standard operation of setting the production process; Step S12: Taking the center position of the operating table in the operation video as the coordinate origin of the coordinate system, obtaining the boundary coordinates of the operating table, and using the boundary coordinates as the initial position coordinates of the operating table, and also obtaining the initial position coordinates of the target object and the operation object.

3. The method according to claim 1, characterized in that The step S2 comprises the following steps: Step S21: extracting the operation actions at multiple angles of the target object and the motion trajectories at multiple angles of the operation object from the operation video; Step S22: Divide the operation action into multiple sub-actions according to the operation process, and obtain the bone coordinate change trajectory of each sub-action and the motion coordinate change trajectory of the operation object corresponding to the sub-action based on the operation action and the bone extraction unit, wherein the bone coordinate change trajectory of the sub-action and the motion coordinate change trajectory of the operation object are both three-dimensional trajectories.

4. The method according to claim 1, characterized in that The generation of the speech model in step S4 comprises the following steps: Step S411: extracting speech content, lip type and facial image of speech object from multiple speech videos in the database, and extracting lip and teeth change animation and neck and mandible change animation corresponding to each word pronunciation in the speech content from the speech video, and taking the lip type of speech object in each speech video, each word pronunciation in the speech content and the lip and teeth change animation and neck and mandible change animation corresponding to each word pronunciation as a training data set; Step S412: creating the speech model, and training the speech model using a plurality of the training data sets and a machine learning algorithm; The character pronunciation is the pinyin of each character, and the lip and teeth change animation is the animation of the change of the lip shape and the change of the area and number of teeth during the pronunciation.

5. The method according to claim 1, characterized in that The step S4 comprises: Extracting the explanation voice corresponding to each sub-action from the operation video, and obtaining the explanation content from the explanation voice, inputting the pronunciation of each word in the explanation content, the lip type and facial image of the digital human into the voice model, generating and using the virtual animation of the lips and teeth and the virtual animation of the jaw and neck of the digital human corresponding to each word pronunciation as the first virtual animation; The first virtual animations corresponding to the pronunciation of each character in the explanation content are spliced ​​to obtain the second virtual animation corresponding to the explanation content.

6. The method according to claim 1, characterized in that The step S5 comprises: Step S51: generating a virtual sub-action corresponding to the digital human based on the appearance of the digital human and the skeletal coordinate change trajectory corresponding to the sub-action, and acquiring the virtual operation object trajectory based on the virtual operation object and the motion coordinate change trajectory of the operation object corresponding to the sub-action; Step S52: adjusting the second virtual animation corresponding to the explanation content of the sub-action in real time according to the facial angle change and speech speed change of the target object corresponding to the sub-action, and superimposing the facial expression of the target object corresponding to the sub-action on the basis of the adjusted second virtual animation, and obtaining a third virtual animation; Step S53: Generate the virtual operation video based on the virtual operating console in the virtual space, each virtual sub-action and the virtual operation object trajectory corresponding to each virtual sub-action, the third virtual animation and the virtual voice, and display the virtual operation video holographically.

7. The method according to claim 6, characterized in that The step S5 also includes: calculating the first vector and the second vector between the motion trajectory of the virtual sub-action and the trajectory of the virtual operating table and the virtual operating object respectively, and also calculating the third vector and the fourth vector between the motion trajectory of the sub-action and the motion coordinate change trajectory of the operating table and the operating object respectively, obtaining the first vector difference between the first vector and the third vector, and the second vector difference between the second vector and the fourth vector, and when the first vector difference is greater than or equal to the first set difference or the second vector difference is greater than or equal to the second set difference, adjusting the motion trajectory of the virtual sub-action and the trajectory of the virtual operating object so that the first vector difference and the second vector difference are respectively smaller than the first set difference and the second set difference.

8. The method according to claim 1, characterized in that The step S5 further includes a step S6: The user's actual operation video is obtained through the camera unit, and the actual operation object motion trajectory corresponding to each actual operation sub-action of the user is extracted from the actual operation video, and the actual operation object motion trajectory is compared with the virtual operation object trajectory, and the trajectory difference is obtained. When the trajectory difference is greater than the set trajectory difference, the virtual sub-action corresponding to the actual operation sub-action and the virtual operation object trajectory are amplified and displayed in multiple dimensions, and an early warning is issued, wherein the actual operation object motion trajectory and the virtual operation object trajectory respectively include the motion trajectory and deformation trajectory of the actual operation object and the virtual operation object.

9. A digital human and holographic display system based on AIGC, the system is used to implement the method according to any one of claims 1 to 8, characterized in that: The system comprises: A camera unit, used to obtain multi-angle operation videos, establish a coordinate system with the center position of the operation platform in the operation video as the coordinate origin, and respectively obtain the initial position coordinates of the target object, the operation object and the operation platform; An extraction unit, used for extracting the operation action of the target object and the motion trajectory of the operation object based on the operation video, dividing the operation action into a plurality of sub-actions, and obtaining a skeletal coordinate change trajectory corresponding to each sub-action and a motion coordinate change trajectory corresponding to each sub-action of the operation object; A virtual unit, for acquiring a virtual operation table, a digital human and a virtual operation object based on the AIGC technology, and selecting the appearance of the digital human corresponding to the target object according to the virtual content library, creating a virtual space, and placing the digital human, the virtual operation table and the virtual operation object in the virtual space according to the initial position coordinates of the target object, the operation table and the operation object in the coordinate system; A model creation unit, configured to extract explanation contents from the explanation voice corresponding to each sub-action in the operation video, generate a first virtual animation corresponding to each pronunciation through a voice model based on the lip type, facial image and explanation contents of the digital human, and obtain a second virtual animation corresponding to the explanation contents based on the first virtual animation; The video generation unit is used to generate a virtual operation video based on the digital human and the skeletal coordinate change trajectory corresponding to each of the sub-actions, the virtual operation table, the virtual operation object, the motion coordinate change trajectory of the operation object, the second virtual animation and the virtual voice, and to holographically display the virtual operation video.

10. A computer storage medium, characterized in that: The storage medium stores program instructions, wherein when the program instructions are executed, the device where the storage medium is located is controlled to execute the method according to any one of claims 1 to 8.

Images