Rendering processing method and electronic device

Abstract

Embodiments of the present application provide a rendering processing method and an electronic device, and relate to the technical field of terminals. The method comprises: determining a first human body model corresponding to each of at least one image frame in a first video. According to inertial measurement unit (IMU) data of a terminal device when the image frame is collected, a rotation matrix corresponding to the image frame is obtained, the rotation matrix being used to indicate rotation of a ground reference coordinate system relative to a device coordinate system of the terminal device. According to the rotation matrix, a first coordinate axis corresponding to a gravity direction in an image coordinate system of the image frame is subjected to rotation processing to obtain a mapped coordinate axis after rotation. Then, the first human body model is subjected to multi-angle rendering processing around the mapped coordinate axis to obtain a rendering result corresponding to the image frame. In this way, the multi-view rendering result of the three-dimensional human body model is rotated around the correct gravity direction, so as to avoid the rendering result of the three-dimensional human body model from being inclined.

Description

Technical Field

[0001] This application relates to the field of terminal technology, and in particular to rendering processing methods and electronic devices. Background Technology

[0002] In the field of computer vision, it is possible to realize image-based 3D human body reconstruction, which refers to the process of reconstructing a 3D human body model from one or more 2D images containing the human body.

[0003] After reconstructing the 3D human body model, freeview rendering can be performed on it, allowing users to observe the reconstructed 3D human body model from multiple angles. Currently, when performing freeview rendering on the 3D human body model, the viewpoint is usually rotated around a fixed coordinate axis by default to obtain the rendering results from various viewpoints.

[0004] However, if the reconstructed 3D human body model is tilted, the rendering result obtained after rotating the viewpoint based on a fixed coordinate axis will also be tilted, resulting in poor visual effects in the final rendering result. Summary of the Invention

[0005] This application provides a rendering processing method and an electronic device, applied in the field of terminal technology, to avoid skewing in the rendering results of three-dimensional human body models, thereby improving the visual effect of three-dimensional human body model rendering.

[0006] Firstly, embodiments of this application propose a rendering processing method. The method includes:

[0007] A first human body model is determined for each of at least one image frame in the first video. The first human body model is obtained by reconstructing a 3D human body from the human body in the image frame, and can also be referred to as a 3D human body model. Then, based on the inertial measurement unit (IMU) data of the terminal device when the image frame was acquired, the rotation matrix corresponding to the image frame is obtained. The rotation matrix is ​​used to indicate the rotation of the ground reference coordinate system relative to the device coordinate system of the terminal device.

[0008] Next, based on the rotation matrix, the first coordinate axis corresponding to the direction of gravity in the image coordinate system of the image frame is rotated to obtain the rotated mapped coordinate axis. Then, multi-angle rendering processing is performed on the first human body model around the mapped coordinate axis to obtain the rendering result corresponding to the image frame. This multi-angle rendering processing can also be called freeview rendering processing. Accordingly, the rendering result corresponding to the image frame includes the rendering results from multiple perspectives obtained from rendering the 3D human body model.

[0009] In this implementation, the first coordinate axis in the image coordinate system is rotated based on a rotation matrix. This allows the rotation of the ground reference coordinate system relative to the device coordinate system to be synchronously applied to the first coordinate axis, resulting in a rotated mapped coordinate system. The rotated mapped coordinate system aligns with the actual gravity direction of the human body in the image. Then, free-view rendering is performed on the first human body model around this mapped coordinate system, accurately and effectively obtaining multi-view rendering results of the 3D human body model. Since the rotation of the 3D human body model is performed around the correct gravity direction, it prevents the 3D human body model from tilting.

[0010] In one possible implementation, the rotation matrix corresponding to the image frame is obtained based on the inertial measurement unit (IMU) data of the terminal device during image frame acquisition, including:

[0011] Based on the IMU data of the terminal device when acquiring image frames, the attitude information of the terminal device when acquiring image frames is obtained. The attitude information is used to indicate the attitude of the terminal device in the ground reference coordinate system.

[0012] Based on the attitude information of the terminal device, obtain the rotation matrix corresponding to the image frame.

[0013] In this way, by analyzing IMU data, the attitude information of the terminal device can be determined first, and then the rotation matrix corresponding to the image frame can be obtained simply and accurately based on the attitude information.

[0014] In one possible implementation, the rotation matrix corresponding to the image frame is obtained based on the pose information of the terminal device, including:

[0015] Based on the attitude information of the terminal device, an initial rotation matrix is ​​calculated. This initial rotation matrix indicates the rotation of the terminal device's coordinate system relative to the ground reference coordinate system. Then, the initial rotation matrix is ​​inverted to obtain the rotation matrix corresponding to the image frame.

[0016] In this way, by first calculating the initial rotation matrix and then inverting the initial rotation matrix, a rotation matrix that can indicate the rotation of the ground reference coordinate system relative to the device coordinate system can be obtained, thus enabling the required rotation matrix to be obtained quickly and accurately.

[0017] In one possible implementation, the first coordinate axis of the ground reference coordinate system is parallel to the direction of gravity and points to the sky in the positive direction, the second coordinate axis of the ground reference coordinate system is parallel to the ground and points to the east in the positive direction, and the third coordinate axis of the ground reference coordinate system is parallel to the ground and points to the south in the positive direction.

[0018] The first coordinate axis in the device coordinate system is parallel to the screen of the terminal device and points upward in the positive direction. The second coordinate axis in the device coordinate system is parallel to the screen and points to the right in the positive direction. The third coordinate system in the device coordinate system is perpendicular to the screen of the terminal device and points from the inside of the screen to the outside of the screen in the positive direction.

[0019] Attitude information includes pitch angle, roll angle, and roll angle.

[0020] In one possible implementation, the first coordinate axis corresponding to the direction of gravity in the image coordinate system of the image frame is rotated according to the rotation matrix to obtain the rotated mapped coordinate axis:

[0021] Obtain the first unit vector in the positive direction of the first coordinate axis in the image coordinate system;

[0022] The second unit vector is obtained by multiplying the first unit vector and the rotation matrix. The second unit vector is used to represent the rotated mapped coordinate axes.

[0023] In this way, by processing the first unit vector, the rotation matrix can be applied to the rotation of the first coordinate axis in a simple and effective manner to obtain the second unit vector used to represent the rotated mapped coordinate axis.

[0024] In one possible implementation, before obtaining the rotation matrix corresponding to the image frame based on the inertial measurement unit (IMU) data of the terminal device during image frame acquisition, the method further includes:

[0025] Acquire IMU data from the terminal device during the first time period, which is the period during which the first video is captured;

[0026] Obtain the first timestamp of each IMU data and the second timestamp of each image frame within the first time period;

[0027] Alignment is performed based on the first and second timestamps to obtain the IMU data corresponding to the image frame.

[0028] In this way, by performing timestamp alignment processing, the IMU data corresponding to the image frame can be effectively obtained, so as to ensure the correctness of the rotation matrix obtained by subsequent analysis based on the IMU data, and thus ensure that the mapped coordinate axis after rotation can be effectively aligned with the gravity direction of the human body in the image.

[0029] In one possible implementation, determining the first human model corresponding to at least one image frame in the first video includes:

[0030] Three-dimensional human body reconstruction is performed based on at least one image frame included in the first video to obtain the first human body model corresponding to each image frame.

[0031] In this way, the reconstruction of three-dimensional human body models can be achieved quickly and efficiently using electronic devices.

[0032] In one possible implementation, determining the first human model corresponding to at least one image frame in the first video includes:

[0033] Send the first video to the cloud device. Receive the first human model corresponding to each image frame from the cloud device.

[0034] In this way, because cloud devices have strong computing power, the quality of the reconstructed 3D human body model can be effectively improved.

[0035] Secondly, embodiments of this application provide a rendering processing apparatus, which may be an electronic device, or a chip or chip system within an electronic device. The rendering processing apparatus may include a display unit and a processing unit.

[0036] When the rendering processing apparatus is an electronic device, the display unit therein can be a display screen. The display unit is used to perform the display steps so that the electronic device implements a rendering processing method described in the first aspect or any possible implementation of the first aspect.

[0037] When the rendering processing apparatus is an electronic device, the processing unit may be a processor. The rendering processing apparatus may further include a storage unit, which may be a memory. The storage unit stores instructions, and the processing unit executes the instructions stored in the storage unit to cause the electronic device to implement a rendering processing method described in the first aspect or any possible implementation of the first aspect. When the rendering processing apparatus is a chip or chip system within an electronic device, the processing unit may be a processor. The processing unit executes the instructions stored in the storage unit to cause the electronic device to implement a rendering processing method described in the first aspect or any possible implementation of the first aspect. The storage unit may be a storage unit within the chip (e.g., a register, cache, etc.) or a storage unit located outside the chip within the electronic device (e.g., read-only memory, random access memory, etc.).

[0038] For example, a display unit is used to display the rendering result of a three-dimensional human body model. A processing unit is used to execute the methods described in the first aspect or any possible implementation thereof.

[0039] Thirdly, embodiments of this application provide an electronic device including a processor and a memory, the memory for storing code instructions, and the processor for running the code instructions to perform the methods described in the first aspect or any possible implementation of the first aspect.

[0040] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program or instructions that, when executed on a computer, cause the computer to perform the methods described in the first aspect or any possible implementation thereof.

[0041] Fifthly, embodiments of this application provide a computer program product including a computer program, which, when run on a computer, causes the computer to perform the methods described in the first aspect or any possible implementation thereof.

[0042] Sixthly, this application provides a chip or chip system including at least one processor and a communication interface. The communication interface and the at least one processor are interconnected via a circuit. The at least one processor is used to run computer programs or instructions to perform the methods described in the first aspect or any possible implementation of the first aspect. The communication interface in the chip can be an input / output interface, pins, or circuits, etc.

[0043] In one possible implementation, the chip or chip system described above in this application further includes at least one memory storing instructions. The memory can be an internal storage unit of the chip, such as a register or cache, or it can be a storage unit of the chip itself (e.g., read-only memory, random access memory, etc.).

[0044] It should be understood that the second to sixth aspects of this application correspond to the technical solutions of the first aspect of this application, and the beneficial effects achieved by each aspect and the corresponding feasible implementation are similar, and will not be repeated here. Attached Figure Description

[0045] Figure 1 A schematic diagram illustrating the implementation of Euler angles provided in an embodiment of this application;

[0046] Figure 2 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application;

[0047] Figure 3 This application provides a scenario illustration of video capture using an electronic device. Figure 1 ;

[0048] Figure 4 Implementation illustration of the rendering results provided in the embodiments of this application Figure 1 ;

[0049] Figure 5 This application provides a scenario illustration of video capture using an electronic device. Figure 2 ;

[0050] Figure 6 Implementation illustration of the rendering results provided in the embodiments of this application Figure 2 ;

[0051] Figure 7 This application provides a scenario illustration of video capture using an electronic device. Figure 3 ;

[0052] Figure 8 Implementation illustration of the rendering results provided in the embodiments of this application Figure 3 ;

[0053] Figure 9 This is a schematic diagram of the rendering process provided in the embodiments of this application;

[0054] Figure 10 A schematic diagram illustrating the process of adjusting coordinate axes provided in an embodiment of this application;

[0055] Figure 11 Implementation illustration of the rendering results provided in the embodiments of this application Figure 4 ;

[0056] Figure 12 Implementation illustration of the rendering results provided in the embodiments of this application Figure 5 ;

[0057] Figure 13 This is a schematic block diagram of a rendering processing apparatus provided in an embodiment of this application. Detailed Implementation

[0058] To facilitate a clear description of the technical solutions in the embodiments of this application, some terms and technologies involved in the embodiments of this application will be briefly introduced below:

[0059] 1. Euler angles

[0060] Euler angles are a mathematical representation used to describe the rotation of a rigid body. They represent the rotational state of an object in three-dimensional space using three angle values. These three angles are usually called yaw (heading angle, usually denoted as α), pitch (pitch angle, usually denoted as β), and roll (roll angle, usually denoted as γ).

[0061] For example, you can refer to Figure 1 To understand the three perspectives mentioned above, we can consider the motion of the aircraft's fuselage. Figure 1 This is a schematic diagram illustrating the implementation of Euler angles in an embodiment of this application.

[0062] exist Figure 1The diagram illustrates the aircraft's body coordinate system, which includes three axes, denoted as a, b, and c. It's understandable that typical coordinate systems are XYZ coordinate systems, but in practice... Figure 1 The body coordinate system shown in the figure can also be understood as the XYZ coordinate system. However, this embodiment does not specifically limit the correspondence between the three coordinate axes a, b, and c and the three coordinate axes X, Y, and Z. In other words, the positions of the three coordinate axes X, Y, and Z can be set arbitrarily according to actual needs.

[0063] Reference Figure 1 The angle at which the aircraft rotates around coordinate axis a can be understood as the heading angle. The heading angle affects the direction of the aircraft's nose in the horizontal direction. A change in the heading angle will cause the aircraft's nose to shift to the left or right, similar to the effect of turning the nose left and right.

[0064] Furthermore, the angle at which the aircraft rotates around coordinate axis b can be understood as the pitch angle, which affects the angle between the aircraft's nose and the horizontal plane. A change in the pitch angle can be understood as causing the aircraft's nose to pitch up or down.

[0065] Furthermore, the angle at which the aircraft rotates around coordinate axis c can be understood as the roll angle, which affects the angle at which the aircraft fuselage rolls. A change in the roll angle can be understood as causing the aircraft to roll at a corresponding angle.

[0066] Furthermore, the definition of Euler angles also depends on the order of rotation; different orders lead to different results. That is, Euler angle rotations can be performed along different axes; common orders include:

[0067] ZYX sequence: That is to say, first rotate around the Z-axis by a certain angle, then rotate around the new Y-axis by a certain angle, and finally rotate around the new X-axis by a certain angle.

[0068] XYZ sequence: That is to say, first rotate around the X-axis by a certain angle, then rotate around the new Y-axis by a certain angle, and finally rotate around the new Z-axis by a certain angle.

[0069] 2. Rotation Matrix

[0070] A rotation matrix is ​​a 3x3 matrix used to describe the rotation of an object in three-dimensional space. Rotation can be achieved by transforming a vector from one coordinate system to another through matrix multiplication. Each element of a rotation matrix is ​​defined by trigonometric functions such as sine and cosine.

[0071] It should be noted that there is a corresponding conversion relationship between Euler angles and rotation matrices. That is, rotation matrices can be obtained from Euler angles, and Euler angles can be derived from rotation matrices. The specific conversion relationship between Euler angles and rotation matrices can be found in the descriptions in related technologies, and will not be described in detail in this application.

[0072] 3. Bullet Time

[0073] Bullet time is a photographic technique used in movies, television commercials, or computer games to simulate speed-up effects, such as enhanced slow motion or time freeze.

[0074] 4. IMU

[0075] An IMU (inertial measurement unit) is a device that measures the three-axis attitude angles (or angular rates) and acceleration of an object. In one implementation, an IMU may contain three single-axis accelerometers and three single-axis gyroscopes, where the accelerometers are used to detect the acceleration signals of the object along the three independent axes of the carrier coordinate system, and the gyroscopes are used to detect the angular velocity signals of the object relative to the navigation coordinate system.

[0076] Therefore, an IMU can measure the angular velocity and acceleration of an object in three-dimensional space and use this to calculate the object's attitude. IMU data can include, for example, acceleration data and gyroscope data.

[0077] 5. Other terms

[0078] In the embodiments of this application, terms such as "first" and "second" are used to distinguish identical or similar items with substantially the same function and purpose. For example, "first chip" and "second chip" are used only to distinguish different chips and do not limit their order of execution. Those skilled in the art will understand that terms such as "first" and "second" do not limit the quantity or execution order, and that "first" and "second" do not necessarily imply that they are different.

[0079] It should be noted that, in the embodiments of this application, the terms "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design scheme described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0080] In this application embodiment, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, a--c, bc, or abc, where a, b, and c can be single or multiple.

[0081] 6. Electronic equipment

[0082] The electronic devices in this application embodiment may include handheld devices, vehicle-mounted devices, etc., with data processing and image processing functions. For example, some electronic devices include: mobile phones, tablets, PDAs, laptops, mobile internet devices (MIDs), wearable devices, virtual reality (VR) devices, augmented reality (AR) devices, wireless terminals in industrial control, wireless terminals in self-driving, wireless terminals in remote medical surgery, wireless terminals in smart grids, wireless terminals in transportation safety, wireless terminals in smart cities, wireless terminals in smart homes, cellular phones, cordless phones, session initiation protocol (SIP) phones, wireless local loop (WLL) stations, personal digital assistants (PDAs), handheld devices with wireless communication capabilities, computing devices or other processing devices connected to a wireless modem, in-vehicle devices, wearable devices, terminal devices in 5G networks, or future evolution of public land mobile communication networks. Terminal devices in a network (PLMN), etc., are not limited to this in the embodiments of this application.

[0083] By way of example and not limitation, in this embodiment, the electronic device can also be a wearable device. Wearable devices, also known as wearable smart devices, are a general term for devices that utilize wearable technology to intelligently design and develop everyday wearables, such as glasses, gloves, watches, clothing, and shoes. Wearable devices are portable devices that are worn directly on the body or integrated into the user's clothing or accessories. Wearable devices are not merely hardware devices, but also achieve powerful functions through software support, data interaction, and cloud interaction. Broadly speaking, wearable smart devices include those that are feature-rich, large in size, and can achieve complete or partial functions without relying on a smartphone, such as smartwatches or smart glasses, as well as those that focus on a specific type of application function and require the use of other devices such as smartphones, such as various smart bracelets and smart jewelry for vital sign monitoring.

[0084] Furthermore, in this embodiment of the application, the electronic device can also be a terminal device in the Internet of Things (IoT) system. IoT is an important part of the future development of information technology. Its main technical feature is to connect objects to the network through communication technology, thereby realizing an intelligent network of human-machine interconnection and object-to-object interconnection.

[0085] The electronic devices in the embodiments of this application may also be referred to as: terminal equipment, user equipment (UE), mobile station (MS), mobile terminal (MT), access terminal, user unit, user station, mobile station, mobile station, remote station, remote terminal, mobile device, user terminal, terminal, wireless communication equipment, user agent, or user device, etc.

[0086] In this embodiment, the electronic device or various network devices include a hardware layer, an operating system layer running on top of the hardware layer, and an application layer running on top of the operating system layer. The hardware layer includes hardware such as a central processing unit (CPU), a memory management unit (MMU), and memory (also called main memory). The operating system can be any one or more computer operating systems that implement business processing through processes, such as Linux, Unix, Android, iOS, or Windows. The application layer includes applications such as browsers, address books, word processing software, and instant messaging software.

[0087] The hardware structure of the above-mentioned electronic devices can be as follows: Figure 2 As shown. Figure 2A schematic diagram of the structure of an electronic device 100 applicable to this application is shown.

[0088] Electronic device 100 may include processor 110, external memory interface 120, internal memory 121, universal serial bus (USB) interface 130, charging management module 140, power management module 141, battery 142, antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, audio module 170, sensor module 180, button 190, motor 191, indicator 192, camera 193, display screen 194, and subscriber identification module (SIM) card interface 195, etc.

[0089] It should be noted that, Figure 2 The structure shown does not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include... Figure 2 The components shown may include more or fewer components, or the electronic device 100 may include... Figure 2 The components shown may be a combination of certain components, or the electronic device 100 may include... Figure 2 The components shown are sub-components of certain components. For example, Figure 2 The proximity sensor 180G shown is optional. Figure 2 The components shown can be implemented in hardware, software, or a combination of software and hardware.

[0090] Processor 110 may include one or more processing units. For example, processor 110 may include at least one of the following processing units: application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, video codec, digital signal processor (DSP), baseband processor, and neural network processing unit (NPU). These different processing units may be independent devices or integrated devices. In this application, processor 110 is used to execute the steps included in the rendering processing method provided in this application.

[0091] Furthermore, the electronic device 100 can perform the shooting function through an ISP, camera 193, GPU, display screen 194, and application processor. In this application, the electronic device 100 can, for example, use these devices to achieve video capture, and then the electronic device 100 can provide the captured video to the processor 110 so that the processor 110 can execute the corresponding rendering processing method.

[0092] The ISP is used to process data fed back by the camera 193. The camera 193 is used to capture still images or videos. For example, when taking a picture, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, and the camera's photosensitive element transmits the electrical signal to the ISP for processing, converting it into an image visible to the naked eye. In some embodiments, the ISP may be located within the camera 193.

[0093] In some embodiments, the electronic device 100 may include one or N cameras 193, where N is a positive integer greater than 1.

[0094] The electronic device 100 can also implement display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. The processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0095] The display screen 194 can be used to display the reconstructed 3D human body model, as well as the multi-view rendering results of the 3D human body model.

[0096] It should be understood that Figure 2 The connection relationships between the modules shown are merely illustrative and do not constitute a limitation on the connection relationships between the modules of the electronic device 100. Optionally, the modules of the electronic device 100 may also adopt a combination of various connection methods described in the above embodiments.

[0097] Based on the above introduction, the relevant technologies involved in this application will be further described in detail below.

[0098] 3D human body reconstruction (or human body 3D reconstruction) is a technique that uses computer vision, image processing and deep learning technologies to generate a 3D human body model (or human body 3D model) from one or more 2D human body images (or video data).

[0099] Its core lies in analyzing and processing two-dimensional images from different perspectives to infer information such as the shape, posture, and texture of the human body in three-dimensional space. 3D human body reconstruction technology is widely used in film and television production, virtual reality, augmented reality, medical diagnosis, motion analysis, and game development. By providing high-precision 3D models, it can achieve more realistic visual effects and interactive experiences.

[0100] In recent years, with the rapid development of computer vision and deep learning technologies, 3D human reconstruction technology has also made significant progress. In particular, the emergence of new methods such as Nerf (Neural Radiance Fields) has greatly improved the accuracy and efficiency of 3D human reconstruction. Understandably, traditional 3D human reconstruction typically relies on multiple cameras, specialized operation, and long processing times, while technological advancements and new methods enable the use of simpler equipment to achieve the same or even better results.

[0101] In one application scenario, a user can use an electronic device to capture monocular video. The electronic device can then use the captured monocular video to perform the 3D human body reconstruction described above and display the reconstructed 3D human body model on the electronic device.

[0102] Taking the creation of a "bullet time" effect as an example, traditional methods require numerous cameras and specialized operations to achieve the desired effect. However, current methods only require electronic devices to capture video and perform 3D human reconstruction, after which the electronic devices can output the bullet time image effect. Furthermore, the electronic devices can perform the aforementioned processing on each frame of the video individually, thus obtaining the corresponding bullet time image effect for each frame.

[0103] Typically, after reconstructing the 3D human body model, it is necessary to perform 360° freeview rendering to allow the reconstructed 3D human body model to be presented from multiple angles. In the bullet moment scenario described above, only after performing freeview rendering can the image effect corresponding to the bullet moment be obtained for the image frame.

[0104] The following is combined with Figure 3 and Figure 4 This section explains the rendering process for free-viewpoint perspectives. Figure 3 This application provides a scenario illustration of video capture using an electronic device. Figure 1 , Figure 4 Implementation illustration of the rendering results provided in the embodiments of this application Figure 1 .

[0105] exist Figure 3 The diagram illustrates two coordinate systems: the ground reference coordinate system and the electronic device coordinate system. These two coordinate systems will be introduced separately below.

[0106] First, we will introduce the device coordinate system of electronic devices. Figure 3 The electronic device is illustrated in the form of a mobile phone. The coordinate system formed by the three coordinate axes X_ue, Y_ue, and Z_ue is the device coordinate system of the electronic device.

[0107] The origin of the device coordinate system can be the center point of the electronic device's screen, or it can be the top-left corner of the screen; this embodiment does not impose any restrictions on this. Furthermore, the first coordinate axis in the device coordinate system (… Figure 3 The Y-axis in the device coordinate system is parallel to the screen of the electronic device and points upwards in the positive direction. The second coordinate axis in the device coordinate system ( Figure 3 The X_ue axis in the device coordinate system is parallel to the screen of the electronic device and points to the right in the positive direction. The Z_ue axis in the device coordinate system is the third coordinate system. Figure 3 The Z_ue axis in the diagram is perpendicular to the screen of the electronic device and its positive direction points from the inside of the screen to the outside of the screen.

[0108] However, it should be noted that the above-described first coordinate axis as the Y-axis, second coordinate axis as the X-axis, and third coordinate axis as the Z-axis are only one possible implementation. For example, the first coordinate axis can also be set as the X-axis, the second coordinate axis as the Y-axis, and the third coordinate axis as the Z-axis. It can be understood that the positions of these three coordinate axes are fixed, but their correspondence with the X, Y, and Z axes can be set according to actual needs. The specific correspondence between the first, second, and third coordinate axes and the three specific X, Y, and Z coordinate axes can be arbitrarily set according to actual needs, and this embodiment does not impose any restrictions on this.

[0109] Next, we will introduce the device coordinate system of electronic devices. Figure 3 The circle in the diagram can be understood as the Earth, and the corresponding coordinate system composed of the three coordinate axes X_gro, Y_gro, and Z_gro is the ground reference coordinate system. Here, gro means ground.

[0110] The origin of the ground reference coordinate system can be the ground location corresponding to the electronic device; for example, it can be the ground location where the user holding the electronic device stands, or the ground location where the electronic device is placed. And, the first coordinate axis in the ground reference coordinate system (i.e., Figure 3 The Y_gro axis in the ground reference coordinate system is parallel to the direction of gravity and points positively toward the sky. The second coordinate axis in the ground reference coordinate system (i.e., Figure 3The X_gro axis in the ground reference coordinate system is parallel to the ground and points eastward in its positive direction. The third coordinate axis in the ground reference coordinate system (i.e., Figure 3 The Z_gro axis in the graph is parallel to the ground and points south.

[0111] Similar to the above description, the specific correspondence between the first, second, and third coordinate axes and the three specific coordinate axes X, Y, and Z can be arbitrarily set according to actual needs, and this embodiment does not impose any restrictions on this.

[0112] It's understandable that in scenarios where photography is done using electronic devices, the subject is usually standing on the ground, and the electronic device should also be in a vertical position to capture the image. (Refer to...) Figure 3 When the electronic device is in a vertical position and not tilted, the Y_ue axis of the device coordinate system and the Y_gro axis of the ground reference coordinate system are parallel.

[0113] Furthermore, taking the scenario of a user holding an electronic device to take a picture as an example, the user can actually rotate the electronic device arbitrarily on the horizontal plane when holding it. Therefore, the electronic device may be facing any direction to take a picture. The device coordinate system of the electronic device will change its position in the ground reference coordinate system as the position of the electronic device changes, while the ground reference coordinate system remains unchanged.

[0114] Therefore, the X_ue axis of the device coordinate system and the X_gro axis of the ground reference coordinate system can have any possible angle, and the Z_ue axis of the device coordinate system and the Z_gro axis of the ground reference coordinate system can have any possible angle. However, as long as the electronic device is shooting in a vertical position and there is no tilt, the Y_ue axis of the device coordinate system and the Y_gro axis of the ground reference coordinate system will be parallel.

[0115] exist Figure 3 In the example, the electronic device captures video or images of the subject 30 in a vertical position, thereby acquiring image frames containing the subject 30. The acquired image frames can be, for example, […]. Figure 4 Image frame 401 is shown in the diagram.

[0116] as well as Figure 4 The diagram also illustrates the coordinate system corresponding to image frame 401, which includes three axes: X_img, Y_img, and Z_img. "img" stands for image. The definition of the coordinate system for an image frame is similar to that of the device coordinate system, and will not be repeated here.

[0117] It is understandable that when video is being captured, the coordinate system corresponding to the image frame coincides with the device coordinate system of the electronic device. Therefore, the coordinate system corresponding to the image frame and the device coordinate system of the electronic device when capturing the image frame can be considered to be the same.

[0118] Based on the above introduction, when the electronic device is in a vertical position and does not tilt, the Y_ue axis of the device coordinate system and the Y_gro axis of the ground reference coordinate system are parallel. The coordinate system corresponding to the image frame and the device coordinate system of the electronic device when the image frame is acquired can be understood as the same. Therefore, when the electronic device is not tilted, the Y_img axis in the coordinate system corresponding to the image frame and the Y_gro axis of the ground reference coordinate system also remain parallel.

[0119] Referring to image frame 401, it can be understood that the photographed object 30 contained in image frame 401 is presented in a state perpendicular to the ground, which can be seen as a representation that the Y_img axis and Y_gro axis are parallel.

[0120] Then, the electronic device can, for example, construct a three-dimensional human body model corresponding to image frame 401, and surround it. Figure 4 The Y_img axis shown is used for free-view rendering of the constructed 3D human model. Here, the Y_img axis is the fixed coordinate axis described above. It should also be noted that this embodiment does not limit which specific coordinate axis the "fixed coordinate axis" is; as long as a certain fixed coordinate axis is selected by default, and then that coordinate axis is used as the coordinate axis around which the viewpoint is rotated during free-view rendering.

[0121] The rendered result can be referenced Figure 4 Understand (a) to (d) in the text.

[0122] exist Figure 4 The diagram illustrates the effect of displaying the 3D human body model corresponding to image frame 401 from four different perspectives, as well as the reference... Figure 4 It can be confirmed that these multiple different perspectives are actually perspectives presented by rotating around the Y_img axis.

[0123] For example, suppose that for Figure 4 The 3D human body model shown in (a) is rotated around the Y_img axis by a certain angle according to the arrow shown in the figure. For example, we can obtain... Figure 4 The effect of the three-dimensional human body model presented in (b) is shown.

[0124] And the assumptions for Figure 4 The 3D human body model shown in (b) is rotated around the Y_img axis by a certain angle according to the arrow shown in the figure. For example, we can obtain... Figure 4The effect of the three-dimensional human body model presented in (c) is shown.

[0125] And the assumptions for Figure 4 The 3D human body model shown in (c) is rotated around the Y_img axis by a certain angle according to the arrow shown in the figure. For example, we can obtain... Figure 4 The effect of the three-dimensional human body model presented in (d) is shown in the figure.

[0126] And the assumptions for Figure 4 The 3D human body model shown in (d) is rotated around the Y_img axis by a certain angle according to the arrow shown in the figure. For example, we can obtain... Figure 4 The effect of the three-dimensional human body model presented in (a) is shown in the image.

[0127] In other words, after free-view rendering, the rendering results of the three-dimensional human body model can be obtained from various perspectives within a 360° range around the Y_img axis.

[0128] In one implementation, after obtaining the rendering results from the free-viewpoint perspective, the electronic device can automatically rotate the 3D human body model to sequentially present the rendering results of the 3D human body model from each viewpoint. Alternatively, the electronic device can also respond to a viewpoint switching operation on the 3D human body model, thereby presenting the rendering result of the 3D human body model from the adjusted viewpoint indicated by the viewpoint switching operation.

[0129] The perspective switching operation can be a touch operation on the screen, specifically a swipe operation. Alternatively, the perspective switching operation can also be a keyboard input operation or a mouse input operation; this embodiment does not limit this.

[0130] Regardless of the method used to switch perspectives, electronic devices can display the rendered results of a 3D human body model from various viewpoints rotated around the Y_img axis. It's important to understand that free-viewpoint rendering based on the Y_img axis essentially aims to allow the 3D human body model to rotate around the Y_gro axis to various viewpoints, thus presenting the corresponding rendering results. This ensures the rendered 3D human body model is upright from all perspectives.

[0131] When the electronic device is held vertically and not tilted for shooting, the effect described above, "the 3D human model can be rotated around the Y_gro axis to various viewpoints," can be achieved because the Y_img axis and Y_gro axis are parallel.

[0132] However, when the electronic device is in a vertical position but tilted at a certain angle, the Y_img axis and Y_gro axis are no longer parallel. If the rendering process is still based on the Y_img axis, the final rendering result of the 3D human body model will not be a rotation around the Y_gro axis, resulting in the 3D human body model being tilted.

[0133] The following can be combined Figures 5 to 8 This section provides a detailed explanation of situations where a 3D human body model tilts. Figure 5 This application provides a scenario illustration of video capture using an electronic device. Figure 2 , Figure 6 Implementation illustration of the rendering results provided in the embodiments of this application Figure 2 . Figure 7 This application provides a scenario illustration of video capture using an electronic device. Figure 3 , Figure 8 Implementation illustration of the rendering results provided in the embodiments of this application Figure 3 .

[0134] First, combine Figure 5 and Figure 6 An explanation of one type of tilt. Figure 5 The diagram also illustrates the equipment coordinate system and the ground reference coordinate system. The implementation of the coordinate system is similar to that described in the above embodiments, and will not be repeated here.

[0135] Reference Figure 5 It can be determined that when the electronic device rotates around its Z_ue axis, the electronic device exhibits [a certain behavior / effect]. Figure 5 As shown in the tilt diagram, when the electronic device captures video or images of the subject 30 based on the current state, the acquired image frames can be, for example, […]. Figure 6 Image frame 601 is shown in the diagram.

[0136] Figure 6 The coordinate system corresponding to image frame 601 is also illustrated, and it is similar to that described in the above embodiments, so it will not be repeated here. (Comparison) Figure 5 and Figure 6 It is understandable that the coordinate system corresponding to the image frame is actually the same as the device coordinate system of the electronic device when the image frame is acquired.

[0137] In this current shooting scenario, because the electronic device is tilted, and the device coordinate system is fixed relative to the electronic device, the device coordinate system also tilts accordingly. Figure 5It can be confirmed that, in the current situation, the Y_ue axis and the Y_gro axis are no longer parallel. Similarly, the Y_img axis and the Y_gro axis are also no longer parallel.

[0138] Then, the electronic device can construct a three-dimensional human body model corresponding to image frame 601, and surround it. Figure 6 The Y_img axis shown in the image is used for free-view rendering of the constructed 3D human body model. The rendered result can be referenced. Figure 6 Understand (a) to (d) in the text.

[0139] exist Figure 6 The diagram illustrates the effect of displaying the 3D human body model corresponding to image frame 601 from four different perspectives, and its understanding is the same as described above. Figure 4 The descriptions are similar, so I won't repeat them here.

[0140] and reference Figure 6 It is certain that these multiple different perspectives are the perspectives presented by rotating around the Y_img axis. However, because the Y_img axis and the Y_gro axis are no longer parallel in the current situation, therefore... Figure 6 The multiple perspectives of the 3D human body model presented in the image do not show a rotation around the Y_gro axis. Instead, the 3D human body model appears to be rotating around the Y_img axis, resulting in a skewed rendering effect.

[0141] Secondly, combine Figure 7 and Figure 8 Let's illustrate another case of tilt. Figure 7 The diagram also illustrates the equipment coordinate system and the ground reference coordinate system. The implementation of the coordinate system is similar to that described in the above embodiments, and will not be repeated here.

[0142] Reference Figure 7 It can be determined that when the electronic device rotates around its X_ue axis, the electronic device exhibits [a certain behavior / effect]. Figure 7 As shown in the tilt diagram, when the electronic device captures video or images of the subject 30 based on the current state, the acquired image frames can be, for example, […]. Figure 8 Image frame 801 is shown in the diagram.

[0143] Figure 8 The coordinate system corresponding to image frame 801 is also illustrated, and it is similar to that described in the above embodiments, so it will not be repeated here. (Comparison) Figure 7 and Figure 8 It is understandable that the coordinate system corresponding to the image frame is actually the same as the device coordinate system of the electronic device when the image frame is acquired.

[0144] In this current shooting scenario, because the electronic device is tilted, and the device coordinate system is fixed relative to the electronic device, the device coordinate system also tilts accordingly. Figure 7 It can be confirmed that, in the current situation, the Y_ue axis and the Y_gro axis are no longer parallel. Similarly, the Y_img axis and the Y_gro axis are also no longer parallel.

[0145] Then, the electronic device can construct a three-dimensional human body model corresponding to image frame 801, and surround it. Figure 8 The Y_img axis shown in the image is used for free-view rendering of the constructed 3D human body model. The rendered result can be referenced. Figure 8 Understand (a) to (d) in the text.

[0146] exist Figure 8 The diagram illustrates the effect of displaying the 3D human body model corresponding to image frame 801 from four different perspectives, and its understanding is the same as described above. Figure 4 The descriptions are similar, so I won't repeat them here.

[0147] and reference Figure 8 It is certain that these multiple different perspectives are the perspectives presented by rotating around the Y_img axis. However, because the Y_img axis and the Y_gro axis are no longer parallel in the current situation, therefore... Figure 8 The multiple perspectives of the 3D human body model presented in the image do not show a rotation around the Y_gro axis. Instead, the 3D human body model appears to be rotating around the Y_img axis, resulting in a skewed rendering effect.

[0148] The above combination Figure 5 and Figure 6 The paper describes the tilting effect caused by the rotation of an electronic device around its Z_ue axis, and combines this with... Figure 7 and Figure 8 This paper introduces the tilting situation caused by the rotation of an electronic device around its X_ue axis. In actual implementation, these two tilting situations may occur simultaneously. That is, the electronic device rotates around both its Z_ue axis and its X_ue axis, resulting in the corresponding tilting situation.

[0149] In real-world shooting scenarios, whether the user holds the electronic device or places it in a fixed position, the device is very prone to tilting. The tilt angle may be so small that the user cannot visually detect it, or the user may notice the tilt but be unable to precisely adjust the device to a tilt-free state. In this embodiment, the tilt can refer to a state where the Y_ue axis and Y_gro axis are not parallel.

[0150] Based on the above introduction, it can be determined that when the electronic device is tilted during the shooting process, the rendering result obtained after the free-view rendering process of the 3D human body model will also be tilted. In the end, the multiple perspectives of the 3D human body model will not be rotated around the Y_gro axis, but will be tilted and rotated relative to the Y_gro axis. This will result in poor visual effect of the final rendering result.

[0151] To address the aforementioned technical problems, this application proposes the following technical concept: A rotation matrix of the ground reference coordinate system relative to the device coordinate system can be determined based on the ground reference coordinate system and the device coordinate system. Then, the Y_img axis in the coordinate system corresponding to the image frame is rotated accordingly based on the rotation matrix, thereby adjusting the Y_img axis to be parallel to the Y_gro axis. Finally, free-view rendering processing is performed based on the Y_img axis, thus accurately obtaining multi-view rendering results of a 3D human body model without tilting.

[0152] The rendering processing method provided in this application will be described in detail below with reference to specific embodiments. The rendering processing method of this application can be executed by an electronic device equipped with a camera and a processor, or by a chip, chip system, or processor that supports the implementation of the rendering processing method by the electronic device, or by a logic module or software that can implement all or part of the functions of the electronic device. This application does not impose specific limitations in this regard. The rendering processing method of this application will be described in detail below using an electronic device as the execution subject as an example.

[0153] First, combine Figure 9 This section explains the overall workflow for free-viewpoint rendering based on video. Figure 9 This is a schematic diagram of the rendering process provided in an embodiment of this application.

[0154] like Figure 9 As shown, the rendering process includes:

[0155] 1. Enter the first video

[0156] The first video is a video containing a human image captured by the electronic device through its camera, and the first video may include multiple image frames.

[0157] In one implementation, the first video can be used as the basis for subsequent processing steps to output a 3D human body model from image frames. The first video includes multiple image frames, thus providing richer depth information and multi-angle human body information. Alternatively, image frames from the first video can also be used as the basis for subsequent processing steps to output a 3D human body model from those image frames. Current technical solutions can support the reconstruction of 3D human body models based on a single image.

[0158] 2. Camera parameter estimation

[0159] The purpose of camera parameter estimation is to determine the parameters of the camera, which may include, for example, focal length, optical axis position, rotation, and displacement. These parameters are used to provide a reference when converting two-dimensional image frames into three-dimensional human body models.

[0160] For example, camera parameter estimation can be achieved through methods such as checkerboard calibration and structured cursor calibration. This embodiment does not limit the specific implementation method of camera parameter estimation.

[0161] 3. Human posture estimation

[0162] This step is used to identify and determine the human body's pose in each image frame. In one implementation, human pose can be represented by posture parameters and shape parameters. The posture parameters describe the rotational information of each joint in the human body, representing the human body's pose in the form of joint angles. The shape parameters describe the body's shape features, thus effectively reflecting the position and angle of the joints corresponding to the human body's pose.

[0163] For example, image frames can be processed using a deep learning model to output estimated human pose. The deep learning model can be, for example, an SMPL (Skinned Multi-Person Linear) model, or other models can be used to achieve human pose estimation; this embodiment does not limit this.

[0164] 4. Human body segmentation

[0165] This step is used to separate the human body from the image frame in order to reconstruct the human body model more accurately. For example, semantic segmentation techniques, such as deep learning models like U-Net and Mask R-CNN, can be used to achieve the purpose of human body segmentation. This embodiment does not limit the specific implementation of human body segmentation.

[0166] 5. Human Reconstruction

[0167] This step involves reconstructing a 3D human model based on the human pose and segmentation results, thereby obtaining the 3D human model corresponding to the image frame. The specific implementation of the 3D human model reconstruction can be found in related technologies, and will not be elaborated upon in this embodiment.

[0168] Regarding the reconstruction process of the three-dimensional human body model described in steps 1 to 5 above, in one implementation, for example, steps 1 to 5 above can be executed by an electronic device to perform three-dimensional human body reconstruction based on the image frames included in the first video, thereby obtaining the three-dimensional human body model corresponding to the image frames.

[0169] Alternatively, in another implementation, steps 1 to 5 described above can be performed by a cloud device (such as a cloud server). For example, an electronic device can send a first video to a cloud device, which then performs the steps of the 3D human body model described above based on the first video, thereby outputting the 3D human body model corresponding to the image frame. The cloud device then sends the reconstructed 3D human body model back to the electronic device, enabling the electronic device to perform subsequent processing based on the 3D human body model.

[0170] Alternatively, steps 1 through 5 described above can be executed separately by electronic devices and cloud devices. For example, electronic devices can perform some of the steps described above, while cloud devices can perform the other parts. The specific steps performed by the electronic devices and cloud devices can be configured according to actual needs.

[0171] 6. Trajectory Correction

[0172] In traditional implementations, after obtaining a 3D human body model, free-view rendering can be performed on the model. However, in the implementation of this application, after obtaining the 3D human body model, a trajectory correction step can be added. The purpose of trajectory correction is to correct the specific coordinate axis around which the image frame rotates during free-view rendering, so that the specific coordinate axis is consistent with the actual gravity direction of the human figure in the image.

[0173] The specific implementation method of trajectory correction will be described in detail in subsequent embodiments, and will not be elaborated here.

[0174] 7. Free-view rendering

[0175] After performing the aforementioned correction process on a specific coordinate system within the image coordinate system, free-view rendering of the 3D human body model can be performed around the corrected coordinate system. This allows for the generation of rendering results from multiple perspectives corresponding to the 3D human body model. For example, refer to the above... Figure 4 Understand the rendering results from multiple perspectives.

[0176] The following is combined with Figures 10 to 12 The specific implementation of step 6, namely trajectory correction, described above, will be explained in detail. Figure 10 This is a schematic diagram of the process for adjusting the coordinate axes provided in an embodiment of this application. Figure 11 Implementation illustration of the rendering results provided in the embodiments of this application Figure 4 , Figure 12 Implementation illustration of the rendering results provided in the embodiments of this application Figure 5 .

[0177] like Figure 10 As shown, the adjustment steps include:

[0178] 1. Acquire IMU data from electronic devices

[0179] This embodiment describes free-view rendering of a 3D human body model for image frames. Therefore, trajectory correction is performed on specific coordinate axes of specific image frames. Thus, this embodiment can acquire IMU data of the electronic device when acquiring image frames and perform subsequent processing based on this.

[0180] As can be understood, the first video contains multiple image frames, and the processing method for each image frame is actually similar. Therefore, the following introduction will take any image frame as an example.

[0181] In one implementation, the IMU data of the electronic device during image frame acquisition can be obtained in the following way:

[0182] The IMU data of the electronic device is acquired within a first time period, which is the period during which the first video is captured. The first time period may include IMU data from multiple moments, each with its own timestamp. In this embodiment, the timestamp of the IMU data is referred to as the first timestamp. Similarly, each image frame also has its own timestamp; in this embodiment, the timestamp of the image frame is referred to as the second timestamp.

[0183] Then, alignment processing can be performed based on the first timestamp of each IMU in the first time period and the second timestamp of the image frame to obtain the IMU data corresponding to the image frame.

[0184] For example, IMU data with the same first and second timestamps can be identified as IMU data acquired during image frame acquisition. Alternatively, IMU data with the smallest difference between the first and second timestamps can be identified as IMU data acquired during image frame acquisition.

[0185] 2. Determine the attitude information of the electronic device based on the IMU data of the electronic device.

[0186] The attitude information is used to indicate the attitude of the electronic device in the ground reference coordinate system. In one implementation, the attitude information of the electronic device may include pitch angle, roll angle, and yaw angle; that is, the attitude information can be the Euler angles described above.

[0187] IMU data may include, for example, acceleration data and gyroscope data from an electronic device, and may also include, for example, magnetometer data. For instance, the Euler angles of the electronic device can be calculated by fusing at least one of the acceleration data, gyroscope data, and magnetometer data. This fusing process may involve substituting the relevant data into a specified formula to calculate the Euler angles of the electronic device.

[0188] This embodiment does not provide a detailed description of the specific implementation of calculating Euler angles based on IMU data; however, it can be understood by referring to the descriptions in related technologies.

[0189] 3. Determine the rotation matrix corresponding to the image frame based on the attitude information of the electronic device.

[0190] It is understandable that there is a transformation relationship between Euler angles and rotation matrices. Therefore, for example, based on the attitude information of the electronic device and the corresponding transformation relationship, the initial rotation matrix can be calculated first. It is called the initial rotation matrix because it is calculated based on the Euler angles of the electronic device. Therefore, the initial rotation matrix indicates the rotation of the terminal device's coordinate system relative to the ground reference coordinate system.

[0191] In this embodiment, a rotation matrix is ​​needed to indicate the rotation of the ground reference coordinate system relative to the device coordinate system. Therefore, after obtaining the initial rotation matrix, the initial rotation matrix can be inverted to obtain the rotation matrix corresponding to the image frame required by this application.

[0192] By first calculating the initial rotation matrix and then inverting it, a rotation matrix that indicates the rotation of the ground reference coordinate system relative to the device coordinate system can be obtained, thus enabling the quick and accurate acquisition of the required rotation matrix.

[0193] 4. Based on the rotation matrix, rotate the first coordinate axis in the image coordinate system of the image frame to obtain the rotated mapped coordinate system.

[0194] In this embodiment, the first coordinate axis corresponding to the direction of gravity in the image coordinate system of the image frame is adjusted. This is because, under normal circumstances, the direction of gravity corresponding to the human figure in the image should be consistent with the Y_img axis of the image coordinate system. Therefore, in one implementation, the Y_img axis in the image coordinate system can be regarded as the first coordinate axis corresponding to the direction of gravity.

[0195] Specifically, the Y_img axis in the image coordinate system can be described as a vertical coordinate axis in the image, with its positive direction pointing upwards. Similarly, the X_img axis can be described as a horizontal coordinate axis in the image, with its positive direction pointing to the right. Finally, the Z_img axis can be described as a coordinate axis perpendicular to the image, with its positive direction pointing outwards.

[0196] Without tilt, the Y_img axis in the image coordinate system aligns with the direction of gravity corresponding to the person in the image. However, with tilt, the Y_img axis in the image coordinate system will have an angle with the direction of gravity corresponding to the person in the image. The current purpose of adjustment based on the rotation matrix is ​​to align the first coordinate axis in the image coordinate system with the direction of gravity corresponding to the person in the image.

[0197] In one implementation, for example, a first unit vector can be obtained along the positive direction of the first coordinate axis in the image coordinate system. This first unit vector can be represented as (0,1,0). Then, the first unit vector and the rotation matrix can be multiplied to obtain a second unit vector, which is used to represent the rotated first coordinate axis. To distinguish the concepts, the rotated first coordinate axis is referred to as the mapped coordinate axis in this embodiment.

[0198] This can be combined Figure 11 and Figure 12 The effect of rotation on the first coordinate axis is illustrated in the diagram.

[0199] First, it should be noted that this embodiment describes the rotation of the ground reference coordinate system relative to the device coordinate system. That is, the device coordinate system is used as a reference, and the rotation of the ground reference coordinate system relative to this reference is determined. Furthermore, without rotation, the ground reference coordinate system and the device coordinate system should coincide; the current ground reference coordinate system is obtained after rotation. Therefore, it can be understood that rotating the device coordinate system (i.e., the position of the ground reference coordinate system before rotation) by a certain angle yields the current ground reference coordinate system. This "certain angle rotation" is precisely the rotation of the ground reference coordinate system relative to the device coordinate system.

[0200] exist Figure 5 In the tilted scenario described, the ground reference coordinate system can be obtained by rotating the device reference coordinate system counterclockwise around its Z_ue axis by a certain angle (e.g., δ). Therefore, the rotation of the ground reference coordinate system relative to the device coordinate system can be understood as a counterclockwise rotation of the Z_gro axis by an angle δ.

[0201] Applying this rotation to the rotation of the first coordinate system in the image coordinate system—that is, rotating the Y_img axis counterclockwise by an angle δ around the Z_img axis—results in the rotated mapped coordinate system Y_img`. (Refer to...) Figure 11 It's understandable that the rotated mapped coordinate system Y_img` coincides with the direction of gravity corresponding to the human body in the image. Then, by performing free-view rendering based on the rotated mapped coordinate system Y_img`, we can effectively obtain multi-view rendering results around the direction of gravity corresponding to the human body, i.e., rendering results without tilting.

[0202] And, in Figure 7 In the tilted scenario described, the ground reference coordinate system can be obtained by rotating the device coordinate system clockwise around its X_ue axis by a certain angle (e.g., ε). Therefore, the rotation of the ground reference coordinate system relative to the device coordinate system can be understood as a clockwise rotation of the X_gro axis by an angle ε.

[0203] Applying this rotation to the rotation of the first coordinate system in the image coordinate system—that is, rotating the Y_img axis clockwise by an angle ε around the X_img axis—will yield the rotated mapped coordinate system Y_img`. (Refer to...) Figure 12 It's understandable that the rotated mapped coordinate system Y_img` coincides with the direction of gravity corresponding to the human body in the image. Then, by performing free-view rendering based on the rotated mapped coordinate system Y_img`, we can effectively obtain multi-view rendering results around the direction of gravity corresponding to the human body, i.e., rendering results without tilting.

[0204] In summary, the rendering method provided in this application rotates the first coordinate axis in the image coordinate system based on a rotation matrix. This allows the rotation of the ground reference coordinate system relative to the device coordinate system to be synchronously applied to the first coordinate axis, resulting in a rotated mapped coordinate system. The rotated mapped coordinate system aligns with the actual gravity direction of the human body in the image. Then, free-view rendering is performed on the first human body model around the mapped coordinate system, thus accurately and effectively obtaining multi-view rendering results of the 3D human body model. Since the rotation of the 3D human body model is performed around the correct gravity direction, the 3D human body model remains tilted.

[0205] The rendering processing method of the embodiments of this application has been described above. The apparatus for performing the above method provided in the embodiments of this application is described below. Those skilled in the art will understand that the methods and apparatus can be combined and referenced with each other, and the related apparatus provided in the embodiments of this application can perform the steps in the above list sorting method.

[0206] Figure 13 This is a schematic block diagram of a rendering processing apparatus 1300 provided in an embodiment of this application. The apparatus 1300 includes a processor 1301, a communication interface 1302, and a memory 1303. The processor 1301, communication interface 1302, and memory 1303 communicate with each other via internal connection paths. The memory 1303 is used to store instructions, and the processor 1301 is used to execute the instructions stored in the memory 1303. The communication interface 1302 can be used to send signals to other devices (e.g., the processor 1301 or a touchscreen of an electronic device) and to receive signals from other devices (e.g., the memory 1303). Exemplarily, the communication interface 1302 reads instructions stored in the memory 1303 and sends the instructions to the processor 1301.

[0207] It should be understood that the device 1300 may specifically be an electronic device as described in the above embodiments, and may be used to perform the various steps and / or processes corresponding to the electronic device in the above method embodiments. Optionally, the memory 1303 may include read-only memory and random access memory, and provide instructions and data to the processor. A portion of the memory may also include non-volatile random access memory. For example, the memory may also store device type information. The processor 1301 may be used to execute instructions stored in the memory, and when the processor 1301 executes instructions stored in the memory, the processor 1301 is used to perform the various steps and / or processes of the above method embodiments.

[0208] It should be understood that, in the embodiments of this application, the processor may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0209] In implementation, each step of the above method can be completed by integrated logic circuits in the processor's hardware or by instructions in software. The steps of the method disclosed in the embodiments of this application can be directly manifested as execution by a hardware processor, or as a combination of hardware and software modules within the processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory, and the processor executes the instructions in the memory, combining them with its hardware to complete the steps of the above method. To avoid repetition, detailed descriptions are omitted here.

[0210] The rendering processing method provided in this application can be applied to electronic devices with communication functions. Electronic devices include terminal devices, and the specific device form of the terminal device can be referred to the above-mentioned descriptions, which will not be repeated here.

[0211] This application provides a terminal device, which includes a processor and a memory; the memory stores computer execution instructions; the processor executes the computer execution instructions stored in the memory, causing the terminal device to perform the above-described method.

[0212] This application provides a chip. The chip includes a processor, which is used to call a computer program in memory to execute the technical solutions in the above embodiments. Its implementation principle and technical effects are similar to those in the related embodiments described above, and will not be repeated here.

[0213] This application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program. When the computer program is executed by a processor, it implements the methods described above. The methods described in the above embodiments can be implemented wholly or partially by software, hardware, firmware, or any combination thereof. If implemented in software, the functionality can be stored as one or more instructions or code on or transmitted over the computer-readable medium. The computer-readable medium can include computer storage media and communication media, and can also include any medium that can transfer a computer program from one place to another. The storage medium can be any target medium accessible by a computer.

[0214] In one possible implementation, a computer-readable medium may include RAM, ROM, compact disc read-only memory (CD-ROM) or other optical disc storage, disk storage or other magnetic storage devices, or any other medium targeted to carry or to store the required program code in the form of instructions or data structures, and accessible by a computer. Furthermore, any connection is appropriately referred to as a computer-readable medium. For example, if software is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave, then coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium. As used herein, disks and optical discs include optical discs, laser discs, optical discs, Digital Versatile Discs (DVDs), floppy disks, and Blu-ray discs, where disks typically reproduce data magnetically, while optical discs optically reproduce data using lasers. Combinations of the above should also be included within the scope of computer-readable media.

[0215] This application provides a computer program product, which includes a computer program that, when run, causes a computer to perform the above-described method.

[0216] This application describes embodiments of methods, apparatus (systems), and computer program products according to embodiments of this application with reference to flowchart illustrations and / or block diagrams. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processing unit of a general-purpose computer, special-purpose computer, embedded processor, or other programmable device to produce a machine, such that the instructions, which execute via the processing unit of the computer or other programmable data processing device, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0217] The above specific embodiments further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made on the basis of the technical solution of the present invention should be included within the scope of protection of the present invention.

Claims

1. A rendering processing method, characterized in that, include: Determine a first human body model corresponding to at least one image frame in the first video, wherein the first human body model is obtained by three-dimensional human body reconstruction of the human body in the image frame; Based on the inertial measurement unit (IMU) data of the terminal device when acquiring the image frame, the rotation matrix corresponding to the image frame is obtained. The rotation matrix is ​​used to indicate the rotation of the ground reference coordinate system relative to the device coordinate system of the terminal device. Obtain the first unit vector in the positive direction of the first coordinate axis in the image coordinate system of the image frame; the first coordinate axis in the image coordinate system corresponds to the direction of gravity. The first unit vector and the rotation matrix are multiplied to obtain the second unit vector, which is used to represent the rotated mapped coordinate axes. The first human body model is rendered from multiple angles around the mapped coordinate axes to obtain the rendering result corresponding to the image frame.

2. The method according to claim 1, characterized in that, The step of obtaining the rotation matrix corresponding to the image frame based on the inertial measurement unit (IMU) data of the terminal device during the acquisition of the image frame includes: Based on the IMU data of the terminal device when the image frame was acquired, the attitude information of the terminal device when the image frame was acquired is obtained, and the attitude information is used to indicate the attitude of the terminal device in the ground reference coordinate system. The rotation matrix corresponding to the image frame is obtained based on the posture information of the terminal device.

3. The method according to claim 2, characterized in that, Based on the posture information of the terminal device, the rotation matrix corresponding to the image frame is obtained, including: Based on the attitude information of the terminal device, an initial rotation matrix is ​​calculated, which is used to indicate the rotation of the device coordinate system of the terminal device relative to the ground reference coordinate system; The initial rotation matrix is ​​inverted to obtain the rotation matrix corresponding to the image frame.

4. The method according to claim 2 or 3, characterized in that, The first coordinate axis of the ground reference coordinate system is parallel to the direction of gravity and points to the sky in the positive direction; the second coordinate axis of the ground reference coordinate system is parallel to the ground and points to the east in the positive direction; and the third coordinate axis of the ground reference coordinate system is parallel to the ground and points to the south in the positive direction. The first coordinate axis in the device coordinate system is parallel to the screen of the terminal device and points upward in the positive direction; the second coordinate axis in the device coordinate system is parallel to the screen and points to the right in the positive direction; the third coordinate system in the device coordinate system is perpendicular to the screen of the terminal device and points from the inside of the screen to the outside of the screen in the positive direction. The attitude information includes pitch angle, roll angle, and roll angle.

5. The method according to any one of claims 1-3, characterized in that, Before obtaining the rotation matrix corresponding to the image frame based on the inertial measurement unit (IMU) data of the terminal device when acquiring the image frame, the method further includes: Acquire the IMU data of the terminal device during a first time period, where the first time period is the period during which the first video is captured; Obtain the first timestamp of each IMU data within the first time period and the second timestamp of the image frame; Alignment processing is performed based on the first timestamp and the second timestamp to obtain the IMU data corresponding to the image frame.

6. The method according to any one of claims 1-3, characterized in that, Determining the first human model corresponding to at least one image frame in the first video includes: Three-dimensional human body reconstruction processing is performed based on at least one image frame included in the first video to obtain a first human body model corresponding to each image frame.

7. The method according to any one of claims 1-3, characterized in that, Determining the first human model corresponding to at least one image frame in the first video includes: Send the first video to the cloud device; Receive the first human body model corresponding to each of the image frames from the cloud device.

8. An electronic device, characterized in that, The electronic device includes: one or more processors and a memory; the memory is coupled to the one or more processors, the memory being used to store computer program code, the computer program code including computer instructions, and the one or more processors invoking the computer instructions to cause the electronic device to perform the method as described in any one of claims 1 to 7.

9. A chip system, characterized in that, The chip system is applied to an electronic device, the chip system including one or more processors, the one or more processors being used to invoke computer instructions to cause the electronic device to perform the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes computer instructions that, when executed on an electronic device, cause the electronic device to perform the method as described in any one of claims 1 to 7.

11. A computer program product, characterized in that, The computer program product includes computer program code that, when run on an electronic device, causes the electronic device to perform the method as described in any one of claims 1 to 7.

Images