An image generation method, apparatus, electronic device, and storage medium

Abstract

This invention provides an image generation method, apparatus, electronic device, and storage medium, relating to the field of image processing technology. The method involves acquiring video frames with the same timestamp from various original videos of a target scene to form a video frame group. For each video frame in the video frame group, pose recognition is performed to obtain the two-dimensional coordinates of each joint of the target object within that video frame. For each joint of the target object, based on its two-dimensional coordinates in the video frames of the video frame group and the transformation relationships between the image coordinate system of the video frames in the video frame group and the three-dimensional coordinate system of the target scene, the target three-dimensional coordinates of the joint corresponding to that video frame group in the target scene are calculated. The three-dimensional coordinates of each joint of a virtual object in each preset image are adjusted according to the target three-dimensional coordinates of the joints of the target object in each video frame group to obtain the target video, thereby improving the efficiency of video generation.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to an image generation method, apparatus, electronic device, and storage medium. Background Technology

[0002] When creating videos containing virtual objects (e.g., virtual characters, virtual animals, etc.), a person wears motion capture equipment, such as optical capture equipment or inertial capture equipment, to collect posture data of the person performing a specified action, and to capture video images of the person performing the specified action. Then, based on the captured images and the person's posture data, the posture of the virtual object in the preset images is adjusted to obtain the target video containing the virtual object performing the specified action.

[0003] However, the equipment used for motion capture is expensive, and capturing human posture data requires repeated shooting. As can be seen, the above process incurs significant time and labor costs, resulting in low video generation efficiency in related technologies. Summary of the Invention

[0004] The purpose of this invention is to provide an image generation method, apparatus, electronic device, and storage medium to improve the efficiency of video generation. The specific technical solution is as follows:

[0005] In a first aspect of the present invention, an image generation method is provided, the method comprising:

[0006] The original videos of the target scene from multiple perspectives are obtained, and multiple video frames with the same timestamp are extracted from each original video to form a video frame group; wherein, the timestamp of a video frame indicates the position of the video frame in the original video to which it belongs;

[0007] For each video frame in the video frame group, pose recognition is performed on the video frame to obtain the two-dimensional coordinates of each joint of the target object in the video frame.

[0008] For each joint of the target object, based on the two-dimensional coordinates of the joint in each video frame of the video frame group, and the transformation relationships between the image coordinate system of each video frame in the video frame group and the three-dimensional coordinate system of the target scene, the three-dimensional coordinates of the joint corresponding to the video frame group in the target scene are calculated and used as the target three-dimensional coordinates.

[0009] Based on the target three-dimensional coordinates of each joint of the target object corresponding to each video frame group in the target scene, adjust the three-dimensional coordinates of each joint of the virtual object in each preset image to obtain a target video in which the virtual object has the same action as the target object.

[0010] Optionally, before calculating the three-dimensional coordinates of the joint point corresponding to the video frame group in the target scene for each joint point of the target object, based on the two-dimensional coordinates of the joint point in each video frame of the video frame group and the transformation relationships between the image coordinate system of each video frame in the video frame group and the three-dimensional coordinate system of the target scene, and using these as the target three-dimensional coordinates, the method further includes:

[0011] Based on the two-dimensional coordinates of the key points of multiple target objects in each video frame of the video frame group, the same target objects in each video frame of the video frame group are determined.

[0012] Optionally, determining the same target object in each video frame of the video frame group based on the two-dimensional coordinates of the joints of multiple target objects in each video frame of the video frame group includes:

[0013] For each target object, calculate the epipolar distance from each joint of the target object to the corresponding epipolar plane, and calculate the average of the epipolar distances corresponding to each joint of the target object to obtain the average distance for the target object; wherein, the epipolar plane corresponding to a joint point represents the plane in which the joint point is located in the target scene;

[0014] For each pair of video frames in the video frame group, the similarity between the two target objects is calculated based on the average distance between each pair of target objects in the two video frames, resulting in a first similarity matrix; wherein, an element in the first similarity matrix represents the probability that the two target objects in the two video frames are the same.

[0015] The two video frames are input into a pre-trained object matching model to obtain the similarity between every two target objects in the two video frames, resulting in a second similarity matrix; where each element in the second similarity matrix represents the probability that the two corresponding target objects in the two video frames are the same.

[0016] The first similarity matrix and the second similarity matrix are fused to obtain the target similarity matrix;

[0017] Based on the target similarity matrix, the same target objects in each video frame of the video frame group are identified.

[0018] Optionally, for each joint of the target object, calculating the three-dimensional coordinates of the joint in the target scene based on the two-dimensional coordinates of the joint in each video frame of the video frame group and the transformation relationships between the image coordinate system of each video frame in the video frame group and the three-dimensional coordinate system of the target scene, as the target three-dimensional coordinates, includes:

[0019] For each joint of the target object, based on the two-dimensional coordinates of the joint in every two video frames in the video frame group, and the transformation relationships between the image coordinate system of the two video frames and the three-dimensional coordinate system of the target scene, the three-dimensional coordinates of the joint in the target scene are calculated as the three-dimensional coordinates to be processed; the average value of each three-dimensional coordinate to be processed is calculated as the initial three-dimensional coordinates of the joint in the target scene corresponding to the video frame group.

[0020] Based on the initial three-dimensional coordinates of the joint point corresponding to the video frame group in the target scene, calculate the target three-dimensional coordinates of the joint point corresponding to the video frame group in the target scene.

[0021] Optionally, the step of performing pose recognition on each video frame in the video frame group to obtain the two-dimensional coordinates of each joint of the target object in the video frame includes:

[0022] For each video frame in the video frame group, the video frame is input into a pre-trained pose recognition model to obtain the two-dimensional coordinates and corresponding confidence scores of each joint of the target object in the video frame; wherein, the confidence score corresponding to the two-dimensional coordinates of a joint represents the probability that the joint is located at the position represented by the two-dimensional coordinates in the video frame.

[0023] The step of calculating the target 3D coordinates of the joint point corresponding to the video frame group in the target scene based on the initial 3D coordinates of the joint point in the target scene includes:

[0024] Based on the confidence level of the two-dimensional coordinates of the key point in each video frame of the video frame group, and the initial three-dimensional coordinates of the key point in the target scene corresponding to the video frame group, the target three-dimensional coordinates of the key point in the target scene corresponding to the video frame group are calculated.

[0025] Optionally, calculating the target three-dimensional coordinates of the joint in the target scene based on the confidence level of the two-dimensional coordinates of the joint in each video frame of the video frame group and the initial three-dimensional coordinates of the joint in the target scene of the video frame group includes:

[0026] Select one joint from the joints of the target object as the parent joint;

[0027] For each child joint of the parent joint, the average confidence score of the two-dimensional coordinates of the child joint in each video frame of the video frame group is calculated as the first average confidence score; wherein, the child joints of the parent joint include: the joints of the target object that are connected to the parent joint.

[0028] If the first average confidence level is less than a preset threshold, the offset value of the target three-dimensional coordinates of the sub-joint point corresponding to the first video frame group relative to the target three-dimensional coordinates of the parent joint point corresponding to the first video frame group is calculated as the first offset value; wherein, the first video frame group includes: each video frame in each original video that is located before each video frame in the video frame group and has the same position; the target three-dimensional coordinates of the joint point corresponding to a video frame group are: determined based on the two-dimensional coordinates of the joint point in each video in the video frame group;

[0029] Calculate the offset value of the target three-dimensional coordinates of the sub-joint point corresponding to the second video frame group relative to the target three-dimensional coordinates of the parent joint point corresponding to the second video frame group, and use it as the second offset value; wherein, the second video frame group includes: each video frame in each original video that is located after each video frame in the video frame group and has the same position;

[0030] Based on the first offset value, the second offset value, and the target three-dimensional coordinates of the parent joint corresponding to the video frame group, calculate the target three-dimensional coordinates of the child joint corresponding to the video frame group in the target scene.

[0031] If the first average confidence level is not less than the preset threshold, the initial three-dimensional coordinates of the sub-joint point corresponding to the video frame group in the target scene are obtained, and the target three-dimensional coordinates of the sub-joint point corresponding to the video frame group in the target scene are obtained.

[0032] Optionally, after calculating the target 3D coordinates of the child joint corresponding to the video frame group in the target scene based on the first offset value, the second offset value, and the target 3D coordinates of the parent joint corresponding to the video frame group, the method further includes:

[0033] Take the child joint of the target object as the parent joint, and return to execute the step of calculating the average confidence of the two-dimensional coordinates of the child joint in each video frame of the video frame group for each child joint of the parent joint, and using it as the first average confidence, until the target three-dimensional coordinates of each joint of the target object in the target scene corresponding to the video frame group are obtained.

[0034] Optionally, calculating the target 3D coordinates of the child joint corresponding to the video frame group in the target scene based on the first offset value, the second offset value, and the target 3D coordinates of the parent joint corresponding to the video frame group includes:

[0035] Calculate the average of the first offset value and the second offset value, and use it as the average offset value;

[0036] The sum of the average offset value and the target 3D coordinates of the parent joint corresponding to the video frame group is calculated to obtain the target 3D coordinates of the child joint corresponding to the video frame group in the target scene.

[0037] Optionally, before selecting a joint point from the joint points of the target object as a parent joint point, the method further includes:

[0038] From the joints of the target object, determine the designated center joint;

[0039] The mean confidence score of the two-dimensional coordinates of the central joint point in each video frame of the video frame group is calculated as the second average confidence score.

[0040] If the second average confidence level is less than a preset threshold, the average of the target 3D coordinates of the center joint point corresponding to the third video frame group and the target 3D coordinates of the center joint point corresponding to the fourth video frame is calculated to obtain the target 3D coordinates of the center node corresponding to the video frame group in the target scene; wherein, the third video frame group includes: each video frame in each original video that is located before each video frame contained in the video frame group and has the same position; the fourth video frame group includes: each video frame in each original video that is located after each video frame contained in the video frame group and has the same position;

[0041] If the second average confidence level is not less than the preset threshold, the initial three-dimensional coordinates of the central joint point corresponding to the video frame group are obtained, and the target three-dimensional coordinates of the central joint point corresponding to the video frame group in the target scene are obtained.

[0042] The step of selecting a joint point from the joint points of the target object as a parent joint point includes:

[0043] From the joints of the target object, determine the central joint as the parent joint.

[0044] In a second aspect of the invention, an image generation apparatus is also provided, the apparatus comprising:

[0045] The acquisition module is used to acquire the original video of the target scene from multiple perspectives, and to extract multiple video frames with the same timestamp from each original video to obtain a video frame group; wherein, the timestamp of a video frame indicates the position of the video frame in the original video to which it belongs;

[0046] The recognition module is used to perform pose recognition on each video frame in the video frame group to obtain the two-dimensional coordinates of each joint of the target object in the video frame.

[0047] The first determining module is used to calculate the three-dimensional coordinates of the joint point corresponding to the video frame group in the target scene for each joint point of the target object, based on the two-dimensional coordinates of the joint point in each video frame in the video frame group and the transformation relationships between the image coordinate system of each video frame in the video frame group and the three-dimensional coordinate system of the target scene, and use them as the target three-dimensional coordinates.

[0048] The generation module is used to adjust the three-dimensional coordinates of each joint of the virtual object in each preset image according to the target three-dimensional coordinates of each joint of the target object corresponding to each video frame group in the target scene, so as to obtain a target video in which the virtual object has the same action as the target object.

[0049] Optionally, the device further includes:

[0050] The matching module is used to perform the following steps in the first determining module: for each joint of the target object, based on the two-dimensional coordinates of the joint in each video frame of the video frame group and the transformation relationships between the image coordinate system of each video frame in the video frame group and the three-dimensional coordinate system of the target scene, calculate the three-dimensional coordinates of the joint corresponding to the video frame group in the target scene and use them as the target three-dimensional coordinates. Before this, the matching module performs the following steps in the first determining module: based on the two-dimensional coordinates of the joints of multiple target objects in each video frame of the video frame group, determine the same target objects in each video frame of the video frame group.

[0051] Optionally, the matching module is specifically used to calculate the epipolar distance from each joint of the target object to the corresponding epipolar plane for each target object, and to calculate the average value of each epipolar distance corresponding to each joint of the target object, so as to obtain the average distance corresponding to the target object; wherein, the epipolar plane corresponding to a joint point represents the plane in which the joint point is located in the target scene;

[0052] For each pair of video frames in the video frame group, the similarity between the two target objects is calculated based on the average distance between each pair of target objects in the two video frames, resulting in a first similarity matrix; wherein, an element in the first similarity matrix represents the probability that the two target objects in the two video frames are the same.

[0053] The two video frames are input into a pre-trained object matching model to obtain the similarity between every two target objects in the two video frames, resulting in a second similarity matrix; where each element in the second similarity matrix represents the probability that the two corresponding target objects in the two video frames are the same.

[0054] The first similarity matrix and the second similarity matrix are fused to obtain the target similarity matrix;

[0055] Based on the target similarity matrix, the same target objects in each video frame of the video frame group are identified.

[0056] Optionally, the first determining module is specifically used to, for each joint of the target object, calculate the three-dimensional coordinates of the joint in the target scene based on the two-dimensional coordinates of the joint in every two video frames in the video frame group, and the transformation relationships between the image coordinate system of the two video frames and the three-dimensional coordinate system of the target scene, as the three-dimensional coordinates to be processed; and calculate the average value of each three-dimensional coordinate to be processed as the initial three-dimensional coordinates of the joint corresponding to the video frame group in the target scene.

[0057] Based on the initial three-dimensional coordinates of the joint point corresponding to the video frame group in the target scene, calculate the target three-dimensional coordinates of the joint point corresponding to the video frame group in the target scene.

[0058] Optionally, the recognition module is specifically used to input each video frame in the video frame group into a pre-trained pose recognition model to obtain the two-dimensional coordinates and corresponding confidence scores of each joint of the target object in the video frame; wherein, the confidence score corresponding to the two-dimensional coordinates of a joint represents the probability that the joint is located at the position represented by the two-dimensional coordinates in the video frame.

[0059] The first determining module is specifically used to calculate the target three-dimensional coordinates of the joint point in the target scene based on the confidence level of the two-dimensional coordinates of the joint point in each video frame of the video frame group and the initial three-dimensional coordinates of the joint point in the target scene.

[0060] Optionally, the first determining module is specifically used to select one joint point from the joint points of the target object as the parent joint point;

[0061] For each child joint of the parent joint, the average confidence score of the two-dimensional coordinates of the child joint in each video frame of the video frame group is calculated as the first average confidence score; wherein, the child joints of the parent joint include: the joints of the target object that are connected to the parent joint.

[0062] If the first average confidence level is less than a preset threshold, the offset value of the target 3D coordinates of the sub-joint point corresponding to the first video frame group relative to the target 3D coordinates of the parent joint point corresponding to the first video frame group is calculated as the first offset value; wherein, the first video frame group includes: each original video frame located before each video frame in the video frame group and at the same position.

[0063] Each video frame; the target three-dimensional coordinates of the key point corresponding to a video frame group are: determined based on the two-dimensional coordinates of the key point in each video in the video frame group;

[0064] Calculate the offset value of the target three-dimensional coordinates of the sub-joint point corresponding to the second video frame group relative to the target three-dimensional coordinates of the parent joint point corresponding to the second video frame group, and use it as the second offset value; wherein, the second video frame group includes: each video frame in each original video that is located after each video frame in the video frame group and has the same position;

[0065] Based on the first offset value, the second offset value, and the target three-dimensional coordinates of the parent joint corresponding to the video frame group, calculate the target three-dimensional coordinates of the child joint corresponding to the video frame group in the target scene.

[0066] If the first average confidence level is not less than the preset threshold, the initial three-dimensional coordinates of the sub-joint point corresponding to the video frame group in the target scene are obtained, and the target three-dimensional coordinates of the sub-joint point corresponding to the video frame group in the target scene are obtained.

[0067] Optionally, the device further includes:

[0068] The processing module is configured to, after the first determining module performs the calculation of the target three-dimensional coordinates of the child joint corresponding to the video frame group in the target scene based on the first offset value, the second offset value, and the target three-dimensional coordinates of the parent joint corresponding to the video frame group, perform the step of taking the child joint of the target object as the parent joint and triggering the first determining module to perform the step of calculating the average confidence of the two-dimensional coordinates of each child joint in each video frame of the video frame group as the first average confidence for each child joint of the parent joint, until the target three-dimensional coordinates of each joint of the target object corresponding to the video frame group in the target scene are obtained.

[0069] Optionally, the first determining module is specifically used to calculate the average of the first offset value and the second offset value as the average offset value;

[0070] The sum of the average offset value and the target 3D coordinates of the parent joint corresponding to the video frame group is calculated to obtain the target 3D coordinates of the child joint corresponding to the video frame group in the target scene.

[0071] Optionally, the device further includes:

[0072] The second determining module is used to determine a specified center joint point from each joint point of the target object before the first determining module performs the operation of selecting a joint point from each joint point of the target object as a parent joint point.

[0073] The third determining module is used to calculate the average confidence level of the two-dimensional coordinates of the central joint point in each video frame of the video frame group, as the second average confidence level.

[0074] The fourth determining module is used to calculate the average of the target three-dimensional coordinates of the center joint point corresponding to the third video frame group and the target three-dimensional coordinates of the center joint point corresponding to the fourth video frame if the second average confidence level is less than a preset threshold, so as to obtain the target three-dimensional coordinates of the center node corresponding to the video frame group in the target scene; wherein, the third video frame group includes: each video frame in each original video that is located before each video frame included in the video frame group and has the same position; the fourth video frame group includes: each video frame in each original video that is located after each video frame included in the video frame group and has the same position.

[0075] The fifth determining module is used to obtain the initial three-dimensional coordinates of the central joint point corresponding to the video frame group if the second average confidence level is not less than the preset threshold, and to obtain the target three-dimensional coordinates of the central joint point corresponding to the video frame group in the target scene.

[0076] The first determining module specifically determines the central joint point from each joint point of the target object, and uses it as the parent joint point.

[0077] In another aspect of the present invention, an electronic device is also provided, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus.

[0078] Memory, used to store computer programs;

[0079] The processor, when executing a program stored in memory, implements any of the image generation method steps described above.

[0080] In another aspect of the present invention, a computer-readable storage medium is also provided, wherein a computer program is stored therein, and when the computer program is executed by a processor, it implements the steps of any of the above-described image generation methods.

[0081] In another aspect of the present invention, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute any of the image generation methods described above.

[0082] This invention provides an image generation method that acquires original videos of a target scene from multiple perspectives, and extracts multiple video frames with the same timestamp from each original video to form a video frame group. The timestamp of a video frame indicates its position in the original video to which it belongs. For each video frame in the video frame group, pose recognition is performed to obtain the two-dimensional coordinates of each joint of the target object in the video frame. For each joint of the target object, based on the two-dimensional coordinates of the joint in each video frame in the video frame group and the transformation relationships between the image coordinate system of each video frame in the video frame group and the three-dimensional coordinate system of the target scene, the three-dimensional coordinates of the joint corresponding to the video frame group in the target scene are calculated as the target three-dimensional coordinates. According to the target three-dimensional coordinates of each joint of the target object in the target scene corresponding to each video frame group, the three-dimensional coordinates of each joint of the virtual object in each preset image are adjusted to obtain a target video in which the virtual object has the same action as the target object in each original video.

[0083] Based on the above processing, the 3D coordinates of each joint of the target object in the target scene can be determined by using the 2D coordinates of the target object in each video frame from multiple perspectives, and the transformation relationships between the image coordinate system of each video frame and the 3D coordinate system of the target scene. The 3D coordinates of each joint of a target object in the target scene can represent the 3D pose of the target object, meaning that the 3D pose of the target object can be determined without the target object wearing motion capture equipment. Furthermore, generating target videos based on the 3D pose of the target object can reduce the time and labor costs of video generation, and improve the efficiency of video generation. Attached Figure Description

[0084] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below.

[0085] Figure 1 This is a flowchart of an image generation method provided in an embodiment of the present invention;

[0086] Figure 2 This is a schematic diagram illustrating the principle of camera calibration provided in an embodiment of the present invention;

[0087] Figure 3(a) is a schematic diagram of the joints of a target object provided in an embodiment of the present invention;

[0088] Figure 3(b) is a schematic diagram of the joints of another target object provided in an embodiment of the present invention;

[0089] Figure 4(a) is a schematic diagram of a video frame containing a target scene provided in an embodiment of the present invention;

[0090] Figure 4(b) is a schematic diagram of another video frame containing a target scene provided in an embodiment of the present invention;

[0091] Figure 5 This is a flowchart of another image generation method provided in an embodiment of the present invention;

[0092] Figure 6 This is a flowchart of another image generation method provided in an embodiment of the present invention;

[0093] Figure 7 A schematic diagram of a target similarity matrix provided in an embodiment of the present invention;

[0094] Figure 8 This is a flowchart of another image generation method provided in an embodiment of the present invention;

[0095] Figure 9 This is a flowchart of another image generation method provided in an embodiment of the present invention;

[0096] Figure 10 This is a flowchart of another image generation method provided in an embodiment of the present invention;

[0097] Figure 11(a) is a schematic diagram of the joints of a target object provided in an embodiment of the present invention;

[0098] Figure 11(b) is a schematic diagram of the joints of another target object provided in an embodiment of the present invention;

[0099] Figure 11(c) is a schematic diagram of the joints of another target object provided in an embodiment of the present invention;

[0100] Figure 11(d) is a schematic diagram of the joints of another target object provided in an embodiment of the present invention;

[0101] Figure 12 This is a flowchart of a method for determining the three-dimensional coordinates of a target object provided in an embodiment of the present invention;

[0102] Figure 13 This is a flowchart of another method for determining the three-dimensional coordinates of a target object provided in an embodiment of the present invention;

[0103] Figure 14 This is a structural diagram of an image generation device provided in an embodiment of the present invention;

[0104] Figure 15 This is a structural diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0105] The technical solutions of the present invention will now be described with reference to the accompanying drawings in the embodiments of the present invention.

[0106] In related technologies, when creating videos containing virtual objects (e.g., virtual characters, virtual animals, etc.), a person wears motion capture equipment to collect posture data of the person performing a specified action, and also captures video images of the person performing the specified action. Then, based on the captured images and the person's posture data, the posture of the virtual object in a preset image is adjusted to obtain a target video of the virtual object performing the specified action. Because the equipment used for motion capture is expensive, and capturing the person's posture data requires repeated shooting, the above process requires significant time and labor costs, resulting in low video generation efficiency in these technologies.

[0107] To address the aforementioned problems, this invention provides an image generation method applied to an electronic device, which can be a server. The electronic device can acquire original videos of a target scene from multiple perspectives. Based on the two-dimensional coordinates of the target object's joints within each video frame of the original videos, the three-dimensional coordinates of each joint of the target object in the target scene are determined. These three-dimensional coordinates represent the target object's three-dimensional pose, meaning the target object's three-dimensional pose can be determined without requiring motion capture equipment. Furthermore, generating a target video based on the target object's three-dimensional pose reduces the time and labor costs of video generation, improving efficiency.

[0108] See Figure 1 , Figure 1 A flowchart of an image generation method provided in an embodiment of the present invention, the method may include the following steps:

[0109] S101: Obtain the original video of the target scene from multiple perspectives, and extract multiple video frames with the same timestamp from each original video to obtain a video frame group.

[0110] The timestamp of a video frame indicates its position within the original video.

[0111] S102: For each video frame in the video frame group, perform pose recognition on the video frame to obtain the two-dimensional coordinates of each joint of the target object in the video frame.

[0112] S103: For each joint of the target object, based on the two-dimensional coordinates of the joint in each video frame of the video frame group, and the transformation relationships between the image coordinate system of each video frame in the video frame group and the three-dimensional coordinate system of the target scene, calculate the three-dimensional coordinates of the joint in the target scene corresponding to the video frame group, and use them as the target three-dimensional coordinates.

[0113] S104: Adjust the three-dimensional coordinates of each joint of the virtual object in each preset image according to the target three-dimensional coordinates of each joint of the target object in the target scene corresponding to each video frame group, so as to obtain the target video containing the virtual object with the same action as the target object.

[0114] Based on the image generation method provided in this invention, the target 3D coordinates of each joint of the target object in the target scene can be determined by using the 2D coordinates of the target object in each video frame from multiple perspectives, and the transformation relationships between the image coordinate system of each video frame and the 3D coordinate system of the target scene. The target 3D coordinates of each joint of a target object in the target scene can represent the 3D pose of the target object, meaning that the 3D pose of the target object can be determined without the target object wearing motion capture equipment. Furthermore, generating target videos based on the 3D pose of the target object can reduce the time and labor costs of video generation, and improve the efficiency of video generation.

[0115] For step S101, the target scene can be any scene containing the target object, which can be a person, etc. For example, the target scene can be a stage where people are performing, etc.

[0116] In this embodiment of the invention, multiple cameras can be set up at different locations in the target scene, and these multiple cameras can capture images of the target objects in the target scene. These multiple cameras can be RGB (Red, Green, Blue) cameras. For example, when the target scene is a stage, multiple cameras can be set up around the stage, ensuring that most cameras can capture images of the people on the stage.

[0117] For example, see Figure 2 The target scene is a stage. A checkerboard calibration board can be laid flat at the center of the stage. This board can be used to calibrate the intrinsic and extrinsic parameters of each camera around the stage. Consequently, each camera can clearly capture the people on the stage. For example, Figure 2 The images shown are of the target scene from different perspectives. Figure 2 Each of the images shown contains a clear and complete image of the checkerboard calibration board.

[0118] Furthermore, by simultaneously capturing images of the target scene from multiple perspectives using multiple cameras, multiple videos (i.e., original videos) are obtained. Since each camera has the same acquisition rate—meaning each camera captures one frame of the target scene at the same time—the number of video frames in each original video is consistent. For example, if each camera's acquisition rate is 30 FPS (Frames Per Second), meaning each camera captures 30 frames per second, and the acquisition time for each camera is 2 seconds, multiple original videos containing 60 frames each can be obtained.

[0119] Correspondingly, the electronic device acquires the original videos of the target scene from multiple perspectives, and extracts multiple video frames with the same timestamp from each original video to obtain a video frame group.

[0120] A video frame's timestamp indicates its position within the original video. For example, if the original video has a frame rate of 25 FPS, and a video frame lasts for 40ms within the original video, then the timestamp of the first video frame in the original video is 40ms, the timestamp of the second video frame is 80ms, the timestamp of the third video frame is 120ms, and so on.

[0121] For multiple video frames with the same timestamp, the position of each video frame in its original video is the same as the position of the other video frames in their original video.

[0122] For example, each original video includes: original videos of the target scene from three perspectives, namely: original video 1, original video 2, and original video 3, each of which contains 50 video frames. The electronic device acquires the first frame of original video 1, the first frame of original video 2, and the first frame of original video 3 to obtain the first video frame group; the electronic device acquires the second frame of original video 1, the second frame of original video 2, and the second frame of original video 3 to obtain the second video frame group, and so on, to obtain 50 video frame groups.

[0123] Regarding step S102, the joints of the target object include the joints of the human skeleton. For example, referring to Figure 3(a), the joints of the target object include 25 joints from joint 0 to joint 24. Furthermore, in addition to the joints of the human skeleton shown in Figure 3(a), the joints of the target object also include joints of designated parts of the human body. For example, the 21 joints of one hand of a person shown in Figure 3(b).

[0124] For each video frame in a video frame group, the electronic device can perform self-pose recognition on that video frame to obtain the two-dimensional coordinates of each joint of the target object within that video frame. For example, the electronic device can input the video frame into a pre-trained pose recognition model to obtain the two-dimensional coordinates of each joint of the target object within that video frame, output by the pose recognition model. The pose recognition model can be a 2D pose recognition model provided by OpenPose.

[0125] For example, referring to Figures 4(a) and 4(b), Figures 4(a) and 4(b) show four video frames in a video frame group, which are images of the target scene from different perspectives. Each video frame contains three target objects, and each rectangular region contains one target object. The four video frames are input into the pose recognition model to obtain the two-dimensional coordinates of each joint of the multiple target objects in the four video frames.

[0126] For step S103, for each video frame group, if each video frame in the video frame group contains a target object, the electronic device calculates the target three-dimensional coordinates of each joint of the target object in the target scene based on the two-dimensional coordinates of each joint of the target object in each video frame of the video frame group, and the transformation relationships between the image coordinate system of each video frame in the video frame group and the three-dimensional coordinate system of the target scene. The target three-dimensional coordinates of each joint of the target object corresponding to a video frame group are: the three-dimensional coordinates of each joint of the target object in the target scene at the time when each video frame in the video frame group is acquired.

[0127] If each video frame in the video frame group contains multiple target objects, the electronic device matches the multiple target objects in each video frame to obtain the same target objects in each video frame of the video frame group. Then, for each target object, the electronic device can determine the target's three-dimensional coordinates in the target scene for each key point of the target object corresponding to the video frame group.

[0128] Accordingly, in some embodiments, in Figure 1 Based on this, see Figure 5 Before step S103, the method may further include the following steps:

[0129] S105: Based on the two-dimensional coordinates of the key points of multiple target objects in each video frame of the video frame group, determine the same target objects in each video frame of the video frame group.

[0130] For each group of video frames, the electronic device can determine the same target object in each video frame of that group in the following manner.

[0131] Method 1

[0132] For each target object, the epipolar distance from each joint of the target object to the corresponding epipolar plane is calculated, and the mean of each epipolar distance corresponding to each joint of the target object is calculated to obtain the mean distance for the target object. For every two video frames in the video frame group, based on the mean distances corresponding to every two target objects in the two video frames, the similarity between the two target objects is calculated to obtain the first similarity matrix.

[0133] Here, the epipolar plane corresponding to a keypoint represents the plane in which that keypoint is located in the target scene. An element in the first similarity matrix represents the probability that two corresponding target objects in the two video frames are identical.

[0134] For each video frame group, each video frame in the video frame group is an image of the target scene captured by multiple cameras from different perspectives. In other words, each video frame in the video frame group contains images of the target object from different perspectives.

[0135] According to the multi-view geometry principle in computer vision, the same joint point of the same target object lies within the same epipolar plane in the target scene from different viewpoints. Therefore, from different viewpoints, the same joint point of each target object corresponds to an epipolar plane, and the epipolar distance from the joint point to the epipolar plane indicates whether the joint point lies within the epipolar plane.

[0136] Accordingly, for each video frame in the video frame group, the electronic device selects a target object from that video frame. For each joint of the target object, the electronic device calculates the straight line between the joint and the optical center of the camera that acquired the video frame based on the two-dimensional coordinates of the joint in the video frame and the intrinsic and extrinsic parameters of the camera that acquired the video frame, thus obtaining the ray corresponding to the joint.

[0137] Furthermore, the electronic device can calculate the intersection point of the rays corresponding to the joint point of each pair of target objects in the target scene, obtaining multiple intersection points. Then, based on the three-dimensional coordinates of each intersection point in the target scene, the plane equation of the epipolar plane corresponding to the joint point in the target scene is calculated.

[0138] For each video frame in the video frame group, the electronic device selects a target object from that video frame. For each joint of the target object, the electronic device can calculate the epipolar distance from each joint of the target object to the epipolar plane corresponding to that joint, based on the two-dimensional coordinates of the joint in the video frame, the transformation relationship between the image coordinate system of the video frame and the three-dimensional coordinate system of the target scene, and the plane equation of the epipolar plane corresponding to the joint in the target scene.

[0139] Then, the electronic device calculates the average distance of each epipolar line corresponding to each key point of the target object, thus obtaining the average distance of the target object. Furthermore, for every two video frames in the video frame group, the sum of the average distances corresponding to each pair of target objects in those two video frames is calculated to obtain the similarity between the two target objects.

[0140] An element in the first similarity matrix represents the probability that two corresponding target objects in two video frames are the same. Accordingly, the electronic device determines the same target objects in each video frame within the video frame group based on the first similarity matrix.

[0141] For example, the electronic device can cluster multiple target objects in each video frame of the video frame group based on the first similarity matrix to obtain the same target objects in each video frame of the video frame group. Alternatively, the electronic device can calculate the same target objects in each video frame of the video frame group based on the Hungarian algorithm and the first similarity matrix.

[0142] Method 2

[0143] For each pair of video frames in the video frame group, the electronic device can input the two video frames into a pre-trained object matching model to obtain the similarity between each pair of target objects in the two video frames, thus obtaining a second similarity matrix.

[0144] Among them, the object matching model can be a network model based on ReID (Person Re-Identification) technology.

[0145] An element in the second similarity matrix represents the probability that two corresponding target objects in the two video frames are the same. Accordingly, the electronic device uses the second similarity matrix to determine the same target objects in each video frame within the video frame group.

[0146] The way in which an electronic device determines the same target object in each video frame of a video frame group based on a second similarity matrix is ​​similar to the way in which an electronic device determines the same target object in each video frame of a video frame group based on a first similarity matrix, and can be referred to the relevant description in the foregoing embodiments.

[0147] Method 3

[0148] To improve the accuracy of identifying the same target object in each video frame within the identified video frame group, Figure 5 Based on this, see Figure 6 Step S105 may include the following steps:

[0149] S1051: For each target object, calculate the epipolar distance from each joint of the target object to the corresponding epipolar plane, and calculate the average value of each epipolar distance corresponding to each joint of the target object to obtain the average distance corresponding to the target object.

[0150] In this context, the epipolar plane corresponding to a joint point represents the plane in which that joint point is located in the target scene.

[0151] S1052: For every two video frames in the video frame group, calculate the similarity between the two target objects based on the average distance between the two target objects in the two video frames, and obtain the first similarity matrix.

[0152] In the first similarity matrix, an element represents the probability that the two target objects in the two video frames are the same.

[0153] S1053: Input the two video frames into the pre-trained object matching model to obtain the similarity between each pair of target objects in the two video frames, and obtain the second similarity matrix.

[0154] In the second similarity matrix, an element represents the probability that the two target objects in the two video frames are the same.

[0155] S1054: Merge the first similarity matrix and the second similarity matrix to obtain the target similarity matrix.

[0156] S1055: Based on the target similarity matrix, identify the same target objects in each video frame of the video frame group.

[0157] The method by which the electronic device obtains the first similarity matrix and the second similarity matrix can be referred to the relevant description in the foregoing embodiments.

[0158] After obtaining the first similarity matrix and the second similarity matrix, the electronic device can fuse the first similarity matrix and the second similarity matrix to obtain the target similarity matrix.

[0159] In one implementation, the electronic device can calculate the weighted sum of each element in the first similarity matrix and the corresponding element in the second similarity matrix to obtain the target similarity matrix.

[0160] In another implementation, the electronic device can fuse each element in the first similarity matrix with the corresponding element in the second similarity matrix according to the following formula (1) to obtain the target similarity matrix.

[0161]

[0162] A i,jThis represents the element in the i-th row and j-th column of the target similarity matrix; a i,j b represents the element in the i-th row and j-th column of the first similarity matrix; i,j w1 represents the element in the i-th row and j-th column of the second similarity matrix; w2 represents the weight of the element in the i-th row and j-th column of the first similarity matrix; w3 represents the weight of the element in the i-th row and j-th column of the second similarity matrix.

[0163] For example, see Figure 7 , Figure 7 This is a schematic diagram of a target similarity matrix provided in an embodiment of the present invention. Figure 7 The target similarity matrix shown corresponds to 11 target objects horizontally and 11 target objects vertically. The rectangular area corresponding to each pair of target objects represents the similarity between the two target objects. The darker the color of the rectangular area, the higher the similarity between the two target objects. Furthermore, based on the target similarity matrix, the electronic device identifies the same target objects in each video frame within the video frame group.

[0164] The way in which an electronic device determines the same target object in each video frame of a video frame group based on a target similarity matrix is ​​similar to the way in which an electronic device determines the same target object in each video frame of a video frame group based on a first similarity matrix. For details, please refer to the relevant descriptions in the foregoing embodiments.

[0165] In some embodiments, Figure 1 Based on this, see Figure 8 Step S103 may include the following steps:

[0166] S1031: For each joint of the target object, based on the two-dimensional coordinates of the joint in every two video frames in the video frame group, and the transformation relationships between the image coordinate system of the two video frames and the three-dimensional coordinate system of the target scene, calculate the three-dimensional coordinates of the joint in the target scene as the three-dimensional coordinates to be processed; calculate the average value of each three-dimensional coordinate to be processed as the initial three-dimensional coordinates of the joint in the target scene corresponding to the video frame group.

[0167] S1032: Based on the initial three-dimensional coordinates of the joint point corresponding to the video frame group in the target scene, calculate the target three-dimensional coordinates of the joint point corresponding to the video frame group in the target scene.

[0168] For each joint of the target object, the electronic device calculates the three-dimensional coordinates of the joint in the target scene based on the two-dimensional coordinates of the joint in every two video frames in the video frame group and the following formula (2).

[0169]

[0170] (um v m () represents the two-dimensional coordinates of the m-th joint of the target object in a video frame, K represents the intrinsic parameters of the camera that acquired the video frame, [R|t] represents the extrinsic parameters of the camera that acquired the video frame, (X m Y m Z m ) represents the three-dimensional coordinates of the m-th joint of the target object in the target scene.

[0171] The intrinsic and extrinsic parameters of the camera that acquired the video frame represent the transformation relationship between the image coordinate system of the video frame and the three-dimensional coordinate system of the target scene.

[0172] The electronic device uses the two-dimensional coordinates of the target object's joint point in every two video frames of the video frame group as (u) in the above formula (2). m v m ), and by taking the intrinsic and extrinsic parameters of the camera that captured the two video frames as K[R|t] in the above formula (2), we can obtain two parameters about (X). m Y m Z m By solving the system of equations, we can obtain the three-dimensional coordinates of the m-th joint in the target scene.

[0173] Based on the two-dimensional coordinates of the joint point in every two video frames of the video frame group, multiple three-dimensional coordinates of the joint point to be processed can be obtained. Then, the electronic device calculates the average value of each three-dimensional coordinate to be processed to obtain the initial three-dimensional coordinates of the joint point in the target scene corresponding to the video frame group.

[0174] In one implementation, for each joint of the target object, the electronic device can directly use the initial three-dimensional coordinates of the joint corresponding to the video frame group in the target scene as the target three-dimensional coordinates of the joint corresponding to the video frame group in the target scene.

[0175] In another implementation, Figure 8 Based on this, see Figure 9 Step S102 may include the following steps:

[0176] S1021: For each video frame in the video frame group, input the video frame into the pre-trained pose recognition model to obtain the two-dimensional coordinates of each joint of the target object in the video frame and the corresponding confidence score.

[0177] The confidence level corresponding to the two-dimensional coordinates of a key point represents the probability that the key point is located at the position represented by the two-dimensional coordinates in the video frame.

[0178] Accordingly, step S1032 may include the following steps:

[0179] S10321: Based on the confidence level of the two-dimensional coordinates of the joint point in each video frame of the video frame group, and the initial three-dimensional coordinates of the joint point in the target scene corresponding to the video frame group, calculate the target three-dimensional coordinates of the joint point in the target scene corresponding to the video frame group.

[0180] For each video frame in the video frame group, the electronic device inputs the video frame into a pre-trained pose recognition model to obtain the two-dimensional coordinates and corresponding confidence scores of each key point of the target object in that video frame. The confidence score corresponding to the two-dimensional coordinates of a key point represents the probability that the key point is located at the position represented by that two-dimensional coordinate in the video frame.

[0181] If the confidence level of the two-dimensional coordinates of a key point of a target object in the video frame is low, it means that the accuracy of the two-dimensional coordinates of the key point of the target object in the video frame is low. Consequently, the accuracy of the target three-dimensional coordinates of the key point of the target object determined based on the two-dimensional coordinates of the key point of the target object in the video frame is also low.

[0182] To improve the accuracy of the target 3D coordinates of the joint point of the identified target object, the electronic device can calculate the target 3D coordinates of the joint point in the target scene corresponding to the video frame group in the following manner.

[0183] Method 1,

[0184] For each joint of the target object, the average confidence score of the two-dimensional coordinates of that joint in each video frame of the video frame group is calculated as the third confidence score. If the third confidence score is not less than a preset threshold, it indicates that the accuracy of the initial three-dimensional coordinates of the joint corresponding to the video frame group is high, and the electronic device can directly use the initial three-dimensional coordinates of the joint corresponding to the video frame group as the target three-dimensional coordinates of the joint in the target scene.

[0185] The preset threshold can be set by technicians based on experience. For example, the preset threshold can be 0.2 or 0.3, but it is not limited to this.

[0186] If the third confidence level is less than the preset threshold, it indicates that the accuracy of the initial three-dimensional coordinates of the joint point corresponding to the video frame group is low. The electronic device can calculate the average of the target three-dimensional coordinates of the joint point corresponding to the first video frame group and the target three-dimensional coordinates of the joint point corresponding to the second video frame group as the target three-dimensional coordinates of the joint point corresponding to the video frame group in the target scene.

[0187] The first video frame group includes all video frames in the original video that are located before each video frame in the video frame group and are in the same position. That is, for each video frame in the video frame group, the video frames in the first video frame group that belong to the same original video as that video frame are located before that video frame in their respective original videos. Furthermore, all video frames in the first video frame group have the same timestamp.

[0188] The second video frame group comprises all video frames in the original video that are located after each video frame in the original video and are positioned at the same location. In other words, for each video frame in the original video group, the video frames in the second video frame group that belong to the same original video are located after that video frame in their respective original videos. Furthermore, all video frames in the second video frame group have the same timestamp.

[0189] For example, each video frame group includes: video frame group 1 containing each first frame of each original video, video frame group 2 containing each second frame of each original video, video frame group 3 containing each third frame of each original video, video frame group 4 containing each fourth frame of each original video, and video frame group 5 containing each fifth frame of each original video.

[0190] For each joint of the target object, if the third confidence level of the joint corresponding to video frame group 3 is less than a preset threshold, if the target three-dimensional coordinates of the joint corresponding to video frame group 2 have been determined, then video frame group 2 is determined as the first video frame group; if the target three-dimensional coordinates of the joint corresponding to video frame group 2 have not been determined, but the target three-dimensional coordinates of the joint corresponding to video frame group 1 have been determined, then video frame group 1 is determined as the first video frame group.

[0191] If the target 3D coordinates of the joint point corresponding to video frame group 4 have been determined, then video frame group 4 is determined to be the second video frame group. If the target 3D coordinates of the joint point corresponding to video frame group 4 have not been determined, but the target 3D coordinates of the joint point corresponding to video frame group 5 have been determined, then video frame group 5 is determined to be the second video frame group.

[0192] Then, the average of the target 3D coordinates of the joint point corresponding to the first video frame group and the target 3D coordinates of the joint point corresponding to the second video frame group is calculated to obtain the target 3D coordinates of the joint point corresponding to the video frame group in the target scene.

[0193] In some embodiments, if each video frame in each video frame group contains multiple target objects, for each pair of target objects in two adjacent video frame groups, the electronic device can determine whether the two target objects are the same target object based on the initial three-dimensional coordinates of the joints of the two target objects corresponding to the two adjacent video frame groups, so as to realize object tracking between different video frame groups.

[0194] The electronic device can calculate the difference in the initial three-dimensional coordinates of each joint point of the two target objects corresponding to two adjacent video frame groups, and calculate the mean of the differences to obtain the joint point error between the two target objects. Accordingly, for each target object, it is determined that the target object with the smallest joint point error between the target objects is the same.

[0195] For example, video frame group 1 corresponds to object 1, object 2, and object 3, while video frame group 2 corresponds to object A, object B, and object C. The electronic device calculates the difference between the initial 3D coordinates of the first joint of object 1 corresponding to video frame group 1 and the initial 3D coordinates of the first joint of object A corresponding to video frame group 2. Then, it calculates the difference between the initial 3D coordinates of the second joint of object 1 corresponding to video frame group 1 and the initial 3D coordinates of the second joint of object A corresponding to video frame group 2, and so on, to obtain multiple differences.

[0196] Then, the average of each difference is calculated to obtain the joint error between object 1 and object A. Similarly, the joint error between object 1 and object B, and the joint error between object 1 and object C can be calculated. Furthermore, if the joint error between object A and object 1 is the smallest among object A, object B, and object C, then object 1 and object A can be determined to be the same object.

[0197] Method 2,

[0198] If the target 3D coordinates of the joint point corresponding to the video frame group differ significantly from those of the joint point corresponding to the first video frame group and the joint point corresponding to the second video frame group, then calculating the average of the target 3D coordinates of the joint point corresponding to the first video frame group and the joint point corresponding to the second video frame group will result in a lower accuracy of the target 3D coordinates of the joint point corresponding to the video frame group.

[0199] To improve the accuracy of the target 3D coordinates of the joint point corresponding to the determined video frame group, in Figure 9 Based on this, see Figure 10 Step S10321 may include the following steps:

[0200] S103211: Select one joint from the joints of the target object as the parent joint.

[0201] S103212: For each child node of the parent node, calculate the average confidence level of the two-dimensional coordinates of the child node in each video frame of the video frame group, and use it as the first average confidence level.

[0202] Among them, the child joints of the parent joint include the joints of the target object that are connected to the parent joint.

[0203] S103213: If the first average confidence level is less than a preset threshold, calculate the offset value of the target three-dimensional coordinates of the sub-joint point corresponding to the first video frame group relative to the target three-dimensional coordinates of the parent joint point corresponding to the first video frame group, and use it as the first offset value.

[0204] The first video frame group includes: each video frame in the original video that is located before each video frame in the video frame group and is in the same position; the target three-dimensional coordinates of the joint point corresponding to a video frame group are: determined based on the two-dimensional coordinates of the joint point in each video in the video frame group.

[0205] S103214: Calculate the offset value of the target three-dimensional coordinates of the sub-joint point corresponding to the second video frame group relative to the target three-dimensional coordinates of the parent joint point corresponding to the second video frame group, and use it as the second offset value.

[0206] The second video frame group includes: each video frame in the original video that is located after each video frame in the video frame group and is in the same position.

[0207] S103215: Based on the first offset value, the second offset value, and the target 3D coordinates of the parent joint corresponding to the video frame group, calculate the target 3D coordinates of the child joint corresponding to the video frame group in the target scene.

[0208] S103216: If the first average confidence level is not less than a preset threshold, obtain the initial three-dimensional coordinates of the sub-joint point corresponding to the video frame group in the target scene, and obtain the target three-dimensional coordinates of the sub-joint point corresponding to the video frame group in the target scene.

[0209] In one implementation, the electronic device can select any one of the joints of the target object as the current parent joint, and determine the joints connected to the current parent joint to obtain the child joints of the current parent joint.

[0210] For example, in the embodiment of Figure 3(a), the electronic device can select joint 1 as the current parent joint, then the child joints of the current parent joint include: joint 2, joint 0, joint 5, and joint 8. Alternatively, the electronic device can select joint 0 as the current parent joint, then the child joints of the current parent joint include: joint 1, joint 15, and joint 16.

[0211] In another implementation, before step S103211, the method may further include the following steps:

[0212] Step 1: Determine the specified center joint from among the joints of the target object.

[0213] Step 2: Calculate the mean confidence level of the two-dimensional coordinates of the central joint point in each video frame of the video frame group, and use it as the second average confidence level.

[0214] Step 3: If the second average confidence level is less than the preset threshold, calculate the average of the target 3D coordinates of the center joint point corresponding to the third video frame group and the target 3D coordinates of the center joint point corresponding to the fourth video frame group to obtain the target 3D coordinates of the center node corresponding to the video frame group in the target scene.

[0215] The third video frame group includes: each video frame in each original video that is located before each video frame contained in the video frame group and is in the same position; the fourth video frame group includes: each video frame in each original video that is located after each video frame contained in the video frame group and is in the same position.

[0216] Step 4: If the second average confidence level is not less than the preset threshold, obtain the initial three-dimensional coordinates of the center joint corresponding to the video frame group, and obtain the target three-dimensional coordinates of the center joint corresponding to the video frame group in the target scene.

[0217] Accordingly, step S103211 may include the following steps: determining the central joint point from each joint point of the target object as the parent joint point.

[0218] The third video frame group includes all video frames in the original video that are located before each video frame included in the video frame group and are in the same position. In other words, for each video frame in the video frame group, the video frames in the third video frame group that belong to the same original video as that video frame are located before that video frame in their respective original videos. Furthermore, all video frames in the third video frame group have the same timestamp.

[0219] The fourth video frame group comprises all video frames in the original video that are located after each video frame included in the group and are in the same position. In other words, for each video frame in the original video group, the video frames in the fourth video frame group that belong to the same original video as that video frame are located after it in their respective original videos. Furthermore, all video frames in the fourth video frame group have the same timestamp.

[0220] The electronic device determines a designated center joint from among the joints of the target object. For example, in the embodiment of Figure 3(a), the electronic device can determine the hip joint (i.e., joint number 8) as the center joint. Then, the electronic device calculates the average confidence level of the two-dimensional coordinates of the center joint in each video frame of the video frame group, obtaining a second average confidence level. The second average confidence level can represent the probability that the center joint is located at the position represented by the initial three-dimensional coordinates in the target scene.

[0221] If the second average confidence level is less than the preset threshold, it indicates that the accuracy of the initial three-dimensional coordinates of the central node is low. The electronic device obtains the target three-dimensional coordinates of the central node corresponding to the third video frame group and the target three-dimensional coordinates of the central node corresponding to the fourth video frame group, and calculates the average of the target three-dimensional coordinates of the central node corresponding to the third video frame group and the target three-dimensional coordinates of the central node corresponding to the fourth video frame group to obtain the target three-dimensional coordinates of the central node corresponding to the video frame group in the target scene.

[0222] If the second average confidence level is not less than the preset threshold, it indicates that the accuracy of the initial three-dimensional coordinates of the central joint is high. Then, the electronic device can directly obtain the initial three-dimensional coordinates of the central joint corresponding to the video frame group as the target three-dimensional coordinates of the central joint corresponding to the video frame group in the target scene.

[0223] Correspondingly, the electronic device can select the central joint as the current parent joint from the joints of the target object, and determine the joints connected to the current parent joint to obtain the child joints of the current parent joint.

[0224] For each child node of the current parent node, the electronic device obtains the confidence level of the two-dimensional coordinates of that child node in each video frame of the video frame group, and calculates the average of the obtained confidence levels to obtain the first average confidence level. The first average confidence level can represent the probability that the node is located at the position represented by the initial three-dimensional coordinates in the target scene.

[0225] If the first average confidence level is less than a preset threshold, it indicates that the accuracy of the initial three-dimensional coordinates of the joint point corresponding to the video frame group is low. The electronic device calculates the difference between the target three-dimensional coordinates of the sub-joint point corresponding to the first video frame group and the target three-dimensional coordinates of the parent joint point corresponding to the first video frame group, and obtains the offset value of the target three-dimensional coordinates of the sub-joint point corresponding to the first video frame group relative to the target three-dimensional coordinates of the parent joint point corresponding to the first video frame group, which is used as the first offset value.

[0226] The electronic device calculates the difference between the target 3D coordinates of the sub-joint point corresponding to the second video frame group and the target 3D coordinates of the parent joint point corresponding to the second video frame group, and obtains the offset value of the target 3D coordinates of the sub-joint point corresponding to the second video frame group relative to the target 3D coordinates of the parent joint point corresponding to the second video frame group, which is used as the second offset value.

[0227] In one implementation, step S103215 may include the following steps: calculating the average of the first offset value and the second offset value as the average offset value; calculating the sum of the average offset value and the target three-dimensional coordinates of the parent joint corresponding to the video frame group to obtain the target three-dimensional coordinates of the child joint corresponding to the video frame group in the target scene.

[0228] In another implementation, the electronic device can calculate the weighted sum of the first offset value and the second offset value, and calculate the sum of the sum and the target three-dimensional coordinates of the parent joint corresponding to the video frame group, so as to obtain the target three-dimensional coordinates of the child joint corresponding to the video frame group in the target scene.

[0229] If the first average confidence level is not less than a preset threshold, it indicates that the accuracy of the initial three-dimensional coordinates of the key point corresponding to the video frame group is high. Then, the electronic device can directly use the initial three-dimensional coordinates of the sub-key point corresponding to the video frame group in the target scene as the target three-dimensional coordinates of the sub-key point corresponding to the video frame group in the target scene.

[0230] In some embodiments, after step S103215, the method may further include the following steps:

[0231] Take the child joint of the target object as the parent joint, and return to perform the following steps for each child joint of the parent joint: calculate the average confidence of the two-dimensional coordinates of the child joint in each video frame of the video frame group as the first average confidence, until the target three-dimensional coordinates of each joint of the target object in the target scene are obtained.

[0232] After calculating the target 3D coordinates of the child joints of the current parent joint in the target scene, for each child joint of the current parent joint, the electronic device can take the child joint as the current parent joint and recalculate the average confidence of the 2D coordinates of each child joint of the current parent joint in each video frame of the video frame group as the first average confidence. Based on the first average confidence, the target 3D coordinates of the child joint in the target scene corresponding to the video frame group are calculated. This process is repeated until the target 3D coordinates of each joint of the target object corresponding to the video frame group in the target scene are obtained.

[0233] For example, referring to Figure 11(a), Figure 11(a) is a schematic diagram of the joints of a target object provided by an embodiment of the present invention. From left to right, the images in Figure 11(a) are schematic diagrams of the joints of the target object corresponding to each video frame group. The target three-dimensional coordinates of the missing joints in the second, third, and fourth images from left to right in Figure 11(a) can be determined according to the method provided by the embodiment of the present invention, resulting in the schematic diagram of the joints of the target object shown in Figure 11(d).

[0234] For example, for the image in the middle of Figure 11(a), a current parent joint can be selected from the joints of the target object, and the target three-dimensional coordinates of each child joint of the current parent joint can be determined according to the method provided in the embodiment of the present invention, so as to obtain a schematic diagram of the joints of the target object as shown in Figure 11(b).

[0235] Then, the child joints of the current parent joint are taken as the current parent joint, and the target three-dimensional coordinates of each child joint of the current parent joint are determined according to the method provided in the embodiment of the present invention. Thus, a schematic diagram of the joints of the target object as shown in Figure 11(c) can be obtained. This process is repeated until the target three-dimensional coordinates of each joint of the target object are obtained. Thus, a schematic diagram of the joints of the target object as shown in Figure 11(d) can be obtained.

[0236] In some embodiments, after calculating the target three-dimensional coordinates of each joint of the target object in the target scene corresponding to each video frame group, the electronic device can also use Savgol filtering to smooth the target three-dimensional coordinates of each joint of the determined target object, so as to improve the accuracy of the determined target three-dimensional coordinates.

[0237] Based on the above processing, the first offset value is the offset of the target 3D coordinates of the sub-joint point corresponding to the first video frame relative to the target 3D coordinates of the sub-joint point corresponding to the first video frame group, and the second offset value is the offset of the target 3D coordinates of the sub-joint point corresponding to the second video frame group relative to the target 3D coordinates of the sub-joint point corresponding to the second video frame. Since the parent joint point and the child joint point of the target object are connected, it means that the offset value of the child joint point relative to the parent joint point is fixed. Accordingly, based on the first offset value, the second offset value, and the target 3D coordinates of the parent joint point corresponding to the video frame group, the target 3D coordinates of the sub-joint point corresponding to the video frame group in the target scene can be calculated, which can improve the accuracy of the calculated target 3D coordinates.

[0238] Regarding step S104, the electronic device stores a preset image containing a virtual object that has not performed any actions. After calculating the target 3D coordinates of each joint of the target object in the target scene corresponding to each video frame group, the electronic device can adjust the 3D coordinates of each joint of the virtual object in the preset image according to the target 3D coordinates of each joint of the target object corresponding to each video frame group, thereby obtaining a target video in which the virtual object's actions are the same as those of the target object.

[0239] In some embodiments, after calculating the target 3D coordinates of each joint of the target object in the target scene corresponding to each video frame group, the electronic device can also generate a 3D (3D) character mesh model in SMSLX (Skinned Multi-Person Linear Model) format based on the target 3D coordinates of each joint of the target object. Then, the electronic device can use the Blender tool to convert the 3D character mesh model into a BVH (a common human feature animation file) format file and save the generated file.

[0240] See Figure 12 , Figure 12 A flowchart illustrating a method for determining the three-dimensional coordinates of a target object, as provided in an embodiment of the present invention.

[0241] S1201: Multi-camera calibration.

[0242] The electronic device calibrates multiple cameras at different locations in the target scene to determine the intrinsic and extrinsic parameters of the multiple cameras, and then uses these multiple cameras to acquire original videos of the target scene from different perspectives.

[0243] S1202: 2D (two-dimensional) pose recognition.

[0244] The electronic device acquires multiple video frames with the same timestamp from various original videos of the target scene from multiple perspectives, resulting in multiple video frame groups. For each video frame group, the electronic device performs 2D pose recognition on each video frame in the video frame group to obtain the two-dimensional coordinates of each key point of the target object in that video frame.

[0245] S1203: Multi-view character selection.

[0246] For each video frame group, if each video frame in the video frame group contains multiple target objects, the electronic device performs multi-view character selection, that is, the electronic device matches multiple target objects in each video frame to obtain the same target object in each video frame in the video frame group.

[0247] S1204: 3D pose generation.

[0248] For each target object, the electronic device performs 3D pose generation on the target object. That is, based on the two-dimensional coordinates of each joint of the target object in each video frame of the video frame group, and the transformation relationship between the image coordinate system of each video frame in the video frame group and the three-dimensional coordinate system of the target scene, the initial three-dimensional coordinates of each joint of the target object in the target scene corresponding to the video frame group are calculated.

[0249] S1205: Frame interpolation smoothing.

[0250] After determining the initial 3D coordinates of each joint of the target object in the target scene corresponding to each video frame group, the electronic device determines the target 3D coordinates of each joint of the target object based on the initial 3D coordinates of each joint of the target object. Then, the electronic device uses savgol filtering to smooth the determined target 3D coordinates of each joint of the target object, to obtain the final target 3D coordinates of each joint of the target object.

[0251] S1206: Save action.

[0252] After determining the target object's 3D coordinates in the target scene for each video frame group, the electronic device can generate a 3D character mesh model in SMSLX format based on these coordinates. Then, the electronic device can use a blender tool to convert the 3D character mesh model into a BVH file and save the generated file.

[0253] Based on the above processing, the 3D coordinates of each joint of the target object in the target scene can be determined by using the 2D coordinates of the target object in each video frame from multiple perspectives, and the transformation relationships between the image coordinate system of each video frame and the 3D coordinate system of the target scene. The 3D coordinates of each joint of a target object in the target scene can represent the 3D pose of the target object, meaning that the 3D pose of the target object can be determined without the target object wearing motion capture equipment. Furthermore, generating target videos based on the 3D pose of the target object can reduce the time and labor costs of video generation, and improve the efficiency of video generation.

[0254] See Figure 13 , Figure 13 A flowchart of another method for determining the three-dimensional coordinates of a target object provided in an embodiment of the present invention.

[0255] S1301: Average interpolation of the central node.

[0256] The central node, also known as the central joint point in this embodiment of the invention, involves the electronic device performing central node average interpolation. This means the electronic device determines the specified central joint point from among the joint points of the target object. For each video frame group, the electronic device obtains the second average confidence level corresponding to the central joint point of that video frame group. If the second average confidence level is less than a preset threshold, the average of the target 3D coordinates of the central joint point corresponding to the third video frame group and the target 3D coordinates of the central joint point corresponding to the fourth video frame group is calculated to obtain the target 3D coordinates of the central node corresponding to that video frame group in the target scene. If the second average confidence level is not less than the preset threshold, the electronic device obtains the initial 3D coordinates of the central joint point corresponding to that video frame group, thus obtaining the target 3D coordinates of the central joint point corresponding to that video frame group in the target scene.

[0257] S1302: Calculate the relative position offset between the unknown node and its parent node.

[0258] The electronic device determines the central joint point as the current parent joint point from among the joint points of the target object. Unknown nodes are the child joint points of the current parent joint point. For each child joint point of the current parent joint point, the electronic device calculates a first average confidence score corresponding to the two-dimensional coordinates of that child joint point in each video frame of the video frame group. If the first average confidence score is less than a preset threshold, a first offset value is calculated between the target three-dimensional coordinates of the child joint point corresponding to the first video frame group and the target three-dimensional coordinates of the parent joint point corresponding to the first video frame group, and a second offset value is calculated between the target three-dimensional coordinates of the child joint point corresponding to the second video frame group and the target three-dimensional coordinates of the parent joint point corresponding to the second video frame group.

[0259] S1303: Calculate the current node position.

[0260] The current node is also the child node of the current parent node. Based on the first offset value, the second offset value, and the target 3D coordinates of the parent node corresponding to the video frame group, the electronic device calculates the target 3D coordinates of the child node in the target scene.

[0261] Then, the electronic device takes the current node as the parent node and continues to calculate the relationship between its child nodes and the parent node. That is, the electronic device takes the child node of the target object as the current parent node and recalculates the first average confidence of the two-dimensional coordinates of each child node of the current parent node in each video frame of the video frame group. Based on the current first average confidence, the target three-dimensional coordinates of each child node of the current parent node are calculated until the target three-dimensional coordinates of each node of the target object corresponding to the video frame group in the target scene are obtained.

[0262] S1304: Savgol filter.

[0263] After calculating the target 3D coordinates of each joint of the target object in the target scene corresponding to each video frame group, the electronic device uses Savgol filtering to smooth the target 3D coordinates of each joint of the target object to improve the accuracy of the determined target 3D coordinates.

[0264] S1305: Calculation results.

[0265] The electronic device uses Savgol filtering to smooth the target three-dimensional coordinates of each joint of the identified target object, and obtains the final calculation result, which is the target three-dimensional coordinates of each joint of the target object corresponding to each video frame group.

[0266] Based on the above processing, the first offset value is the offset of the target 3D coordinates of the sub-joint point corresponding to the first video frame group relative to the target 3D coordinates of the sub-joint point corresponding to the first video frame group, and the second offset value is the offset of the target 3D coordinates of the sub-joint point corresponding to the second video frame group relative to the target 3D coordinates of the sub-joint point corresponding to the second video frame group. Since the parent joint point and the child joint point of the target object are connected, it means that the offset value of the child joint point relative to the parent joint point is fixed. Accordingly, based on the first offset value, the second offset value, and the target 3D coordinates of the parent joint point corresponding to the video frame group, the target 3D coordinates of the sub-joint point corresponding to the video frame group in the target scene can be calculated, which can improve the accuracy of the calculated target 3D coordinates.

[0267] and Figure 1 For the corresponding method implementation examples, see [link to relevant documentation]. Figure 14 , Figure 14 This is a structural diagram of an image generation apparatus provided in an embodiment of the present invention. The apparatus includes:

[0268] The acquisition module 1401 is used to acquire the original video of the target scene from multiple perspectives, and to acquire multiple video frames with the same timestamp from each original video to obtain a video frame group; wherein, the timestamp of a video frame indicates the position of the video frame in the original video to which it belongs;

[0269] The recognition module 1402 is used to perform pose recognition on each video frame in the video frame group to obtain the two-dimensional coordinates of each joint of the target object in the video frame.

[0270] The first determining module 1403 is used to calculate the three-dimensional coordinates of the joint point corresponding to the video frame group in the target scene for each joint point of the target object, based on the two-dimensional coordinates of the joint point in each video frame in the video frame group and the transformation relationships between the image coordinate system of each video frame in the video frame group and the three-dimensional coordinate system of the target scene, and use them as the target three-dimensional coordinates.

[0271] The generation module 1404 is used to adjust the three-dimensional coordinates of each joint of the virtual object in each preset image according to the target three-dimensional coordinates of each joint of the target object corresponding to each video frame group in the target scene, so as to obtain a target video in which the virtual object has the same action as the target object.

[0272] Optionally, the device further includes:

[0273] The matching module is used to perform the following steps in the first determining module 1403: for each joint of the target object, based on the two-dimensional coordinates of the joint in each video frame of the video frame group and the transformation relationships between the image coordinate system of each video frame in the video frame group and the three-dimensional coordinate system of the target scene, calculate the three-dimensional coordinates of the joint corresponding to the video frame group in the target scene and use them as the target three-dimensional coordinates. Before this, the matching module performs the following steps: based on the two-dimensional coordinates of the joints of multiple target objects in each video frame of the video frame group, determine the same target objects in each video frame of the video frame group.

[0274] Optionally, the matching module is specifically used to calculate the epipolar distance from each joint of the target object to the corresponding epipolar plane for each target object, and to calculate the average value of each epipolar distance corresponding to each joint of the target object, so as to obtain the average distance corresponding to the target object; wherein, the epipolar plane corresponding to a joint point represents the plane in which the joint point is located in the target scene;

[0275] For each pair of video frames in the video frame group, the similarity between the two target objects is calculated based on the average distance between each pair of target objects in the two video frames, resulting in a first similarity matrix; wherein, an element in the first similarity matrix represents the probability that the two target objects in the two video frames are the same.

[0276] The two video frames are input into a pre-trained object matching model to obtain the similarity between every two target objects in the two video frames, resulting in a second similarity matrix; where each element in the second similarity matrix represents the probability that the two corresponding target objects in the two video frames are the same.

[0277] The first similarity matrix and the second similarity matrix are fused to obtain the target similarity matrix;

[0278] Based on the target similarity matrix, the same target objects in each video frame of the video frame group are identified.

[0279] Optionally, the first determining module 1403 is specifically used to, for each joint of the target object, calculate the three-dimensional coordinates of the joint in the target scene based on the two-dimensional coordinates of the joint in every two video frames in the video frame group, and the transformation relationships between the image coordinate system of the two video frames and the three-dimensional coordinate system of the target scene, as the three-dimensional coordinates to be processed; and calculate the average value of each three-dimensional coordinate to be processed as the initial three-dimensional coordinates of the joint corresponding to the video frame group in the target scene.

[0280] Based on the initial three-dimensional coordinates of the joint point corresponding to the video frame group in the target scene, calculate the target three-dimensional coordinates of the joint point corresponding to the video frame group in the target scene.

[0281] Optionally, the recognition module 1402 is specifically used to input each video frame in the video frame group into a pre-trained pose recognition model to obtain the two-dimensional coordinates and corresponding confidence scores of each joint of the target object in the video frame; wherein, the confidence score corresponding to the two-dimensional coordinates of a joint represents the probability that the joint is located at the position represented by the two-dimensional coordinates in the video frame.

[0282] The first determining module 1403 is specifically used to calculate the target three-dimensional coordinates of the joint point in the target scene based on the confidence level of the two-dimensional coordinates of the joint point in each video frame of the video frame group and the initial three-dimensional coordinates of the joint point in the target scene.

[0283] Optionally, the first determining module 1403 is specifically used to select one joint point from the joint points of the target object as the parent joint point;

[0284] For each child joint of the parent joint, the average confidence score of the two-dimensional coordinates of the child joint in each video frame of the video frame group is calculated as the first average confidence score; wherein, the child joints of the parent joint include: the joints of the target object that are connected to the parent joint.

[0285] If the first average confidence level is less than a preset threshold, the offset value of the target three-dimensional coordinates of the sub-joint point corresponding to the first video frame group relative to the target three-dimensional coordinates of the parent joint point corresponding to the first video frame group is calculated as the first offset value; wherein, the first video frame group includes: each video frame in each original video that is located before each video frame in the video frame group and has the same position; the target three-dimensional coordinates of the joint point corresponding to a video frame group are: determined based on the two-dimensional coordinates of the joint point in each video in the video frame group;

[0286] Calculate the offset value of the target three-dimensional coordinates of the sub-joint point corresponding to the second video frame group relative to the target three-dimensional coordinates of the parent joint point corresponding to the second video frame group, and use it as the second offset value; wherein, the second video frame group includes: each video frame in each original video that is located after each video frame in the video frame group and has the same position;

[0287] Based on the first offset value, the second offset value, and the target three-dimensional coordinates of the parent joint corresponding to the video frame group, calculate the target three-dimensional coordinates of the child joint corresponding to the video frame group in the target scene.

[0288] If the first average confidence level is not less than the preset threshold, the initial three-dimensional coordinates of the sub-joint point corresponding to the video frame group in the target scene are obtained, and the target three-dimensional coordinates of the sub-joint point corresponding to the video frame group in the target scene are obtained.

[0289] Optionally, the device further includes:

[0290] The processing module is configured to, after the first determining module 1403 performs the calculation of the target three-dimensional coordinates of the child joint corresponding to the video frame group in the target scene based on the first offset value, the second offset value, and the target three-dimensional coordinates of the parent joint corresponding to the video frame group, perform the following steps: taking the child joint of the target object as the parent joint, and triggering the first determining module to perform the calculation of the average confidence of the two-dimensional coordinates of each child joint in each video frame of the video frame group as the first average confidence, for each child joint of the parent joint, until the target three-dimensional coordinates of each joint of the target object corresponding to the video frame group in the target scene are obtained.

[0291] Optionally, the first determining module 1403 is specifically used to calculate the average of the first offset value and the second offset value as the average offset value;

[0292] The sum of the average offset value and the target 3D coordinates of the parent joint corresponding to the video frame group is calculated to obtain the target 3D coordinates of the child joint corresponding to the video frame group in the target scene.

[0293] Optionally, the device further includes:

[0294] The second determining module is used to determine a specified center joint point from each joint point of the target object before the first determining module 1403 performs the operation of selecting a joint point from each joint point of the target object as a parent joint point.

[0295] The third determining module is used to calculate the average confidence level of the two-dimensional coordinates of the central joint point in each video frame of the video frame group, as the second average confidence level.

[0296] The fourth determining module is used to calculate the average of the target three-dimensional coordinates of the center joint point corresponding to the third video frame group and the target three-dimensional coordinates of the center joint point corresponding to the fourth video frame if the second average confidence level is less than a preset threshold, so as to obtain the target three-dimensional coordinates of the center node corresponding to the video frame group in the target scene; wherein, the third video frame group includes: each video frame in each original video that is located before each video frame included in the video frame group and has the same position; the fourth video frame group includes: each video frame in each original video that is located after each video frame included in the video frame group and has the same position.

[0297] The fifth determining module is used to obtain the initial three-dimensional coordinates of the central joint point corresponding to the video frame group if the second average confidence level is not less than the preset threshold, and to obtain the target three-dimensional coordinates of the central joint point corresponding to the video frame group in the target scene.

[0298] The first determining module specifically determines the central joint point from each joint point of the target object, and uses it as the parent joint point.

[0299] Based on the image generation apparatus provided in this invention, the target three-dimensional coordinates of each joint of the target object in the target scene can be determined by using the two-dimensional coordinates of the target object in each video frame from multiple perspectives, and the transformation relationships between the image coordinate system of each video frame and the three-dimensional coordinate system of the target scene. The target three-dimensional coordinates of each joint of a target object in the target scene can represent the three-dimensional pose of the target object, meaning that the three-dimensional pose of the target object can be determined without the target object wearing motion capture equipment. Furthermore, generating target video based on the three-dimensional pose of the target object can reduce the time and labor costs of video generation and improve the efficiency of video generation.

[0300] This invention also provides an electronic device, such as... Figure 15As shown, it includes a processor 1501, a communication interface 1502, a memory 1503, and a communication bus 1504. The processor 1501, communication interface 1502, and memory 1503 communicate with each other via the communication bus 1504.

[0301] Memory 1503 is used to store computer programs;

[0302] When the processor 1501 executes the program stored in the memory 1503, it implements the image generation method steps described in any of the above embodiments.

[0303] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not indicate that there is only one bus or one type of bus.

[0304] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0305] The memory may include random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0306] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0307] In another embodiment of the present invention, a computer-readable storage medium is also provided, wherein a computer program is stored therein, and when the computer program is executed by a processor, it implements any of the image generation methods described in the above embodiments.

[0308] In another embodiment of the present invention, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to perform any of the image generation methods described in the above embodiments.

[0309] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)).

[0310] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0311] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments of apparatus, electronic devices, computer-readable storage media, and computer program products are basically similar to the method embodiments, and therefore the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0312] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. An image generation method characterized by, The method includes: The original videos of the target scene from multiple perspectives are obtained, and multiple video frames with the same timestamp are extracted from each original video to form a video frame group; wherein, the timestamp of a video frame indicates the position of the video frame in the original video to which it belongs; For each video frame in the video frame group, pose recognition is performed on the video frame to obtain the two-dimensional coordinates of each joint of the target object in the video frame. For each joint of the target object, based on the two-dimensional coordinates of the joint in each video frame of the video frame group, and the transformation relationships between the image coordinate system of each video frame in the video frame group and the three-dimensional coordinate system of the target scene, the three-dimensional coordinates of the joint corresponding to the video frame group in the target scene are calculated and used as the target three-dimensional coordinates. According to the target three-dimensional coordinates of each joint of the target object corresponding to each video frame group in the target scene, adjust the three-dimensional coordinates of each joint of the virtual object in each preset image to obtain a target video in which the virtual object has the same action as the target object; For each joint of the target object, based on the two-dimensional coordinates of the joint in each video frame of the video frame group, and the transformation relationships between the image coordinate system of each video frame in the video frame group and the three-dimensional coordinate system of the target scene, the three-dimensional coordinates of the joint corresponding to the video frame group in the target scene are calculated as the target three-dimensional coordinates, including: For each joint of the target object, based on the two-dimensional coordinates of the joint in every two video frames in the video frame group, and the transformation relationships between the image coordinate system of the two video frames and the three-dimensional coordinate system of the target scene, the three-dimensional coordinates of the joint in the target scene are calculated as the three-dimensional coordinates to be processed; the average value of each three-dimensional coordinate to be processed is calculated as the initial three-dimensional coordinates of the joint in the target scene corresponding to the video frame group. Based on the initial three-dimensional coordinates of the joint point corresponding to the video frame group in the target scene, calculate the target three-dimensional coordinates of the joint point corresponding to the video frame group in the target scene. The step of performing pose recognition on each video frame in the video frame group to obtain the two-dimensional coordinates of each joint of the target object in the video frame includes: For each video frame in the video frame group, the video frame is input into a pre-trained pose recognition model to obtain the two-dimensional coordinates and corresponding confidence scores of each joint of the target object in the video frame; wherein, the confidence score corresponding to the two-dimensional coordinates of a joint represents the probability that the joint is located at the position represented by the two-dimensional coordinates in the video frame. The step of calculating the target 3D coordinates of the joint point corresponding to the video frame group in the target scene based on the initial 3D coordinates of the joint point in the target scene includes: Based on the confidence level of the two-dimensional coordinates of the key point in each video frame of the video frame group, and the initial three-dimensional coordinates of the key point in the target scene corresponding to the video frame group, the target three-dimensional coordinates of the key point in the target scene corresponding to the video frame group are calculated. The step of calculating the target 3D coordinates of the joint in the target scene based on the confidence level of the 2D coordinates of the joint in each video frame of the video frame group and the initial 3D coordinates of the joint in the target scene of the video frame group includes: Select one joint from the joints of the target object as the parent joint; For each child joint of the parent joint, the average confidence score of the two-dimensional coordinates of the child joint in each video frame of the video frame group is calculated as the first average confidence score; wherein, the child joints of the parent joint include: the joints of the target object that are connected to the parent joint. If the first average confidence level is less than a preset threshold, the offset value of the target three-dimensional coordinates of the sub-joint point corresponding to the first video frame group relative to the target three-dimensional coordinates of the parent joint point corresponding to the first video frame group is calculated as the first offset value; wherein, the first video frame group includes: each video frame in each original video that is located before each video frame in the video frame group and has the same position; the target three-dimensional coordinates of the joint point corresponding to a video frame group are: determined based on the two-dimensional coordinates of the joint point in each video in the video frame group; Calculate the offset value of the target three-dimensional coordinates of the sub-joint point corresponding to the second video frame group relative to the target three-dimensional coordinates of the parent joint point corresponding to the second video frame group, and use it as the second offset value; wherein, the second video frame group includes: each video frame in each original video that is located after each video frame in the video frame group and has the same position; Based on the first offset value, the second offset value, and the target three-dimensional coordinates of the parent joint corresponding to the video frame group, calculate the target three-dimensional coordinates of the child joint corresponding to the video frame group in the target scene. If the first average confidence level is not less than the preset threshold, the initial three-dimensional coordinates of the sub-joint point corresponding to the video frame group in the target scene are obtained, and the target three-dimensional coordinates of the sub-joint point corresponding to the video frame group in the target scene are obtained.

2. The method of claim 1, wherein, Before calculating the three-dimensional coordinates of the joint point corresponding to the video frame group in the target scene for each joint point of the target object, based on the two-dimensional coordinates of the joint point in each video frame of the video frame group and the transformation relationships between the image coordinate system of each video frame in the video frame group and the three-dimensional coordinate system of the target scene, and using these as the target three-dimensional coordinates, the method further includes: Based on the two-dimensional coordinates of the key points of multiple target objects in each video frame of the video frame group, the same target objects in each video frame of the video frame group are determined.

3. The method of claim 2, wherein, The method of determining the same target object in each video frame of the video frame group based on the two-dimensional coordinates of the joints of multiple target objects in each video frame includes: For each target object, calculate the epipolar distance from each joint of the target object to the corresponding epipolar plane, and calculate the average of the epipolar distances corresponding to each joint of the target object to obtain the average distance for the target object; wherein, the epipolar plane corresponding to a joint point represents the plane in which the joint point is located in the target scene; For every two video frames in the video frame group, the similarity between the two target objects is calculated based on the average distance between them, resulting in a first similarity matrix. An element in the first similarity matrix represents the probability that the two target objects in the two video frames are the same. The two video frames are input into a pre-trained object matching model to obtain the similarity between every two target objects in the two video frames, resulting in a second similarity matrix; where each element in the second similarity matrix represents the probability that the two corresponding target objects in the two video frames are the same. The first similarity matrix and the second similarity matrix are fused to obtain the target similarity matrix; Based on the target similarity matrix, the same target objects in each video frame of the video frame group are identified.

4. The method according to claim 1, characterized in that, After calculating the target 3D coordinates of the child joint corresponding to the video frame group in the target scene based on the first offset value, the second offset value, and the target 3D coordinates of the parent joint corresponding to the video frame group, the method further includes: Take the child joint of the target object as the parent joint, and return to execute the step of calculating the average confidence of the two-dimensional coordinates of the child joint in each video frame of the video frame group for each child joint of the parent joint, and using it as the first average confidence, until the target three-dimensional coordinates of each joint of the target object in the target scene corresponding to the video frame group are obtained.

5. The method of claim 1, wherein, The step of calculating the target 3D coordinates of the child joint corresponding to the video frame group in the target scene based on the first offset value, the second offset value, and the target 3D coordinates of the parent joint corresponding to the video frame group includes: Calculate the average of the first offset value and the second offset value, and use it as the average offset value; The sum of the average offset value and the target 3D coordinates of the parent joint corresponding to the video frame group is calculated to obtain the target 3D coordinates of the child joint corresponding to the video frame group in the target scene.

6. The method of claim 1, wherein, Before selecting a joint point from the joint points of the target object as a parent joint point, the method further includes: From the joints of the target object, determine the designated center joint; The mean confidence score of the two-dimensional coordinates of the central joint point in each video frame of the video frame group is calculated as the second average confidence score. If the second average confidence level is less than a preset threshold, the average of the target 3D coordinates of the center joint point corresponding to the third video frame group and the target 3D coordinates of the center joint point corresponding to the fourth video frame is calculated to obtain the target 3D coordinates of the center joint point corresponding to the video frame group in the target scene; wherein, the third video frame group includes: each video frame in each original video that is located before each video frame included in the video frame group and has the same position; the fourth video frame group includes: each video frame in each original video that is located after each video frame included in the video frame group and has the same position. If the second average confidence level is not less than the preset threshold, the initial three-dimensional coordinates of the central joint point corresponding to the video frame group are obtained, and the target three-dimensional coordinates of the central joint point corresponding to the video frame group in the target scene are obtained. The step of selecting a joint point from the joint points of the target object as a parent joint point includes: From the joints of the target object, determine the central joint as the parent joint.

7. An image generation apparatus characterized by comprising: The device includes: The acquisition module is used to acquire the original video of the target scene from multiple perspectives, and to extract multiple video frames with the same timestamp from each original video to obtain a video frame group; wherein, the timestamp of a video frame indicates the position of the video frame in the original video to which it belongs; The recognition module is used to perform pose recognition on each video frame in the video frame group to obtain the two-dimensional coordinates of each joint of the target object in the video frame. The first determining module is used to calculate the three-dimensional coordinates of the joint point corresponding to the video frame group in the target scene for each joint point of the target object, based on the two-dimensional coordinates of the joint point in each video frame in the video frame group and the transformation relationships between the image coordinate system of each video frame in the video frame group and the three-dimensional coordinate system of the target scene, and use them as the target three-dimensional coordinates. The generation module is used to adjust the three-dimensional coordinates of each joint of the virtual object in each preset image according to the target three-dimensional coordinates of each joint of the target object corresponding to each video frame group in the target scene, so as to obtain a target video in which the virtual object has the same action as the target object. The first determining module is specifically used to, for each joint of the target object, calculate the three-dimensional coordinates of the joint in the target scene based on the two-dimensional coordinates of the joint in every two video frames in the video frame group, and the transformation relationships between the image coordinate system of the two video frames and the three-dimensional coordinate system of the target scene, as the three-dimensional coordinates to be processed; and calculate the average value of each three-dimensional coordinate to be processed as the initial three-dimensional coordinates of the joint corresponding to the video frame group in the target scene. Based on the initial three-dimensional coordinates of the joint point corresponding to the video frame group in the target scene, calculate the target three-dimensional coordinates of the joint point corresponding to the video frame group in the target scene. The recognition module is specifically used to input each video frame in the video frame group into a pre-trained pose recognition model to obtain the two-dimensional coordinates and corresponding confidence scores of each joint of the target object in the video frame; wherein, the confidence score corresponding to the two-dimensional coordinates of a joint represents the probability that the joint is located at the position represented by the two-dimensional coordinates in the video frame. The first determining module is specifically used to calculate the target three-dimensional coordinates of the joint point in the target scene based on the confidence level of the two-dimensional coordinates of the joint point in each video frame of the video frame group and the initial three-dimensional coordinates of the joint point in the target scene corresponding to the video frame group. The first determining module is specifically used to select one joint point from the joint points of the target object as the parent joint point; For each child joint of the parent joint, the average confidence score of the two-dimensional coordinates of the child joint in each video frame of the video frame group is calculated as the first average confidence score; wherein, the child joints of the parent joint include: the joints of the target object that are connected to the parent joint. If the first average confidence level is less than a preset threshold, the offset value of the target three-dimensional coordinates of the sub-joint point corresponding to the first video frame group relative to the target three-dimensional coordinates of the parent joint point corresponding to the first video frame group is calculated as the first offset value; wherein, the first video frame group includes: each video frame in each original video that is located before each video frame in the video frame group and has the same position; the target three-dimensional coordinates of the joint point corresponding to a video frame group are: determined based on the two-dimensional coordinates of the joint point in each video in the video frame group; Calculate the offset value of the target three-dimensional coordinates of the sub-joint point corresponding to the second video frame group relative to the target three-dimensional coordinates of the parent joint point corresponding to the second video frame group, and use it as the second offset value; wherein, the second video frame group includes: each video frame in each original video that is located after each video frame in the video frame group and has the same position; Based on the first offset value, the second offset value, and the target three-dimensional coordinates of the parent joint corresponding to the video frame group, calculate the target three-dimensional coordinates of the child joint corresponding to the video frame group in the target scene. If the first average confidence level is not less than the preset threshold, the initial three-dimensional coordinates of the sub-joint point corresponding to the video frame group in the target scene are obtained, and the target three-dimensional coordinates of the sub-joint point corresponding to the video frame group in the target scene are obtained.

8. An electronic device, comprising: It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the steps of the method described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method described in any one of claims 1-6.

Images