Three-dimensional rendering method, device and computer equipment for mobile terminal

Abstract

The disclosure provides a three-dimensional rendering method, device and computer equipment for a mobile terminal, relates to the technical field of image processing, especially to the fields of 3D rendering, naked-eye 3D, and the implementation scheme is: acquiring a heterogeneous data source, including first to third input data; performing data fusion based on space-time synchronization on the data to obtain an instant view angle control data set; determining first to third scores of each pixel in a to-be-rendered image on the first to third levels respectively; determining a target weight distribution ratio of the first to third levels on each pixel based on the first input data; determining an algorithm power distribution coefficient assigned to each pixel based on the first to third scores and the target weight distribution ratio; and performing three-dimensional rendering based on the first input data, the instant view angle control data set and the algorithm power distribution coefficient to display a three-dimensional rendering result on the mobile terminal, so that dynamic naked-eye 3D rendering sensitive to a user's view angle can be realized.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of image processing, in particular to the fields of three-dimensional (3D) rendering, naked-eye 3D, and the like, and more particularly to a three-dimensional rendering method and device for a mobile terminal, a computer device, a computer readable storage medium, and a computer program product. BACKGROUND

[0002] 3D rendering technology is generally implemented through 2D to 3D conversion, and can generally include hardware-dependent schemes and pure software rendering schemes. Common hardware-dependent schemes include 3D raster screens, frequency division screens, 3D glasses, and the like, and pure software rendering schemes include depth maps, paraxial projection, and the like.

[0003] However, on a mobile terminal such as a mobile phone or a tablet computer, 3D rendering not only needs to consider the rendering effect, but also needs to take into account the computing power condition of the mobile terminal itself, which forms a great challenge for implementing 3D rendering on a mobile terminal and puts higher requirements on the methods used in engineering. In particular, with the development of mobile technology, the frequency of users using mobile phones or tablet computers has increased significantly, and 3D rendering can be applied in practical applications such as games, videos, live broadcasts, and the like to improve the visual experience of users, and therefore there is an urgent need for a more effective solution to 3D rendering on a mobile terminal. SUMMARY

[0004] The present disclosure provides a three-dimensional rendering method and device for a mobile terminal, a computer device, a computer readable storage medium, and a computer program product.

[0005] According to an aspect of the present disclosure, a three-dimensional rendering method for a mobile terminal is provided, including: obtaining heterogeneous data sources for performing three-dimensional rendering, including first input data associated with an image to be rendered, second input data associated with a gaze feature of a user gazing at the mobile terminal, and third input data associated with a pose of the mobile terminal; performing spatio-temporal synchronization-based data fusion on the first input data, the second input data, and the third input data to obtain an instant view control data set, the instant view control data set including instant view parameters of multiple degrees of freedom, the multiple degrees of freedom including degrees of freedom for representing rotation and degrees of freedom for representing translation; determining a first score of each pixel in the image to be rendered on a first level, a second score on a second level, and a third score on a third level, wherein the first level is associated with a motion feature and the first score is determined based on sparse optical flow detection, the second level is associated with semantic content and the second score is determined based on semantic recognition, and the third level is associated with visual saliency and the third score is determined based on visual saliency detection with the gaze feature as a reference; determining a target weight allocation ratio of the first level, the second level, and the third level assigned to each pixel based on the first input data; determining an algorithmic power allocation coefficient assigned to each pixel based on the first score to the third score and the target weight allocation ratio; and performing three-dimensional rendering based on the first input data, the instant view control data set, and the algorithmic power allocation coefficient to display a three-dimensional rendering result on the mobile terminal.

[0006] In some embodiments, the spatio-temporal synchronization-based data fusion on the first input data, the second input data, and the third input data to obtain the instant view control data set includes: obtaining spatio-temporally aligned first input data, second input data, and third input data by aligning timestamps of the first input data, the second input data, and the third input data, and projecting the first input data, the second input data, and the third input data into a same coordinate system; and calculating the instant view parameters of the multiple degrees of freedom based on the spatio-temporally aligned first input data, the second input data, and the third input data.

[0007] In some embodiments, the calculation of the instant view parameters of the multiple degrees of freedom based on the spatio-temporally aligned first input data, the second input data, and the third input data includes: calculating a plurality of rotation vectors of the mobile terminal based on the third input data, the plurality of rotation vectors being associated with the degrees of freedom for representing rotation; and calculating a plurality of translation vectors of the gaze feature based on the first input data and the second input data, the plurality of translation vectors being associated with the degrees of freedom for representing translation.

[0008] In some embodiments, based on the first input data, determining the target weight distribution ratio of the first level, the second level and the third level assigned on each pixel comprises: determining a target scene type corresponding to the first input data from a plurality of preset scene types, wherein each preset scene type indicates a content type of the image to be rendered and has a preset weight distribution ratio, so that the sum of the first weight, the second weight and the third weight corresponding to the first level, the second level and the third level respectively is 1; and determining the target weight distribution ratio of the first weight, the second weight and the third weight based on the target scene type and the preset weight distribution ratio.

[0009] In some embodiments, the method further comprises: before performing the three-dimensional rendering, selecting a target rendering pipeline for the three-dimensional rendering from a plurality of preset rendering pipelines based on the state of the mobile terminal, wherein each preset rendering pipeline indicates an algorithm scheduling for the three-dimensional rendering, so that the algorithms corresponding to the plurality of preset rendering pipelines are sequentially decreasing in algorithm power.

[0010] In some embodiments, the algorithms corresponding to the plurality of preset rendering pipelines respectively comprise a four-dimensional Gaussian sputtering algorithm, a three-dimensional Gaussian sputtering algorithm and a two-dimensional parallax algorithm.

[0011] According to an aspect of the present disclosure, there is provided a three-dimensional rendering apparatus for a mobile terminal, comprising: a data source acquisition module configured to acquire heterogeneous data sources for performing three-dimensional rendering, including first input data associated with an image to be rendered, second input data associated with a gaze feature of a user gazing at the mobile terminal, and third input data associated with a pose of the mobile terminal; a space-time synchronization module configured to perform space-time synchronization-based data fusion on the first input data, the second input data, and the third input data to obtain an instant view control data set, the instant view control data set including instant view parameters of a plurality of degrees of freedom, the plurality of degrees of freedom including degrees of freedom for representing rotation and degrees of freedom for representing translation; a scene analysis module configured to determine a first score of each pixel in the image to be rendered on a first level, a second score on a second level, and a third score on a third level, wherein the first level is associated with a motion feature and the first score is determined based on sparse optical flow detection, the second level is associated with semantic content and the second score is determined based on semantic recognition, and the third level is associated with visual saliency and the third score is determined based on visual saliency detection based on the gaze feature; a weight allocation module configured to determine a target weight allocation ratio of the first level, the second level, and the third level assigned to each pixel based on the first input data; a computing power allocation module configured to determine a computing power allocation coefficient assigned to each pixel based on the first score to the third score and the target weight allocation ratio; and a rendering execution module configured to perform three-dimensional rendering based on the first input data, the instant view control data set, and the computing power allocation coefficient to display a three-dimensional rendering result on the mobile terminal.

[0012] According to another aspect of the present disclosure, there is provided a computer device, comprising: at least one processor; and a memory having a computer program stored thereon, the computer program, when executed by the at least one processor, causing the at least one processor to perform the method provided by the present disclosure.

[0013] According to another aspect of the present disclosure, there is provided a computer-readable storage medium having a computer program stored thereon, the computer program, when executed by a processor, causing the processor to perform the method provided by the present disclosure.

[0014] According to another aspect of the present disclosure, there is provided a computer program product comprising a computer program, the computer program, when executed by a processor, causing the processor to perform the method provided by the present disclosure.

[0015] According to one or more embodiments of the present disclosure, a dynamic naked-eye 3D rendering effect sensitive to a user's view angle can be achieved, and a computing power self-adaptive mechanism for a mobile terminal can be provided in a rendering process to optimize power consumption, thereby bringing good visual and use experience to the user.

[0016] These and other aspects of this disclosure will be apparent from the embodiments described below, and will be elucidated with reference to the embodiments described below. Attached Figure Description

[0017] The accompanying drawings exemplify embodiments and form part of the specification, serving together with the textual description to explain exemplary implementations of the embodiments. The illustrated embodiments are for illustrative purposes only and do not limit the scope of this disclosure. Throughout the drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0018] Figure 1 This is a schematic diagram illustrating an example system in which various methods described herein may be implemented according to exemplary embodiments.

[0019] Figure 2 This is a flowchart illustrating a 3D rendering method for a mobile terminal according to an exemplary embodiment.

[0020] Figure 3 This is a schematic diagram illustrating the scene resolution process according to an exemplary embodiment.

[0021] Figure 4 This is a schematic diagram illustrating the algorithm scheduling process according to an exemplary embodiment.

[0022] Figure 5 This is a schematic diagram illustrating the control flow according to an exemplary embodiment.

[0023] Figure 6 This is a schematic block diagram illustrating a 3D rendering apparatus for a mobile terminal according to an exemplary embodiment.

[0024] Figure 7 This is a block diagram illustrating an exemplary computer device that can be applied to an exemplary embodiment. Detailed Implementation

[0025] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0026] In this disclosure, unless otherwise stated, the use of terms such as "first," "second," etc., to describe various elements is not intended to limit the positional, temporal, or importance relationships of these elements; such terms are merely used to distinguish one element from another. In some examples, the first element and the second element may refer to the same instance of that element, while in other cases, based on the context, they may refer to different instances.

[0027] The terminology used in the description of the various examples described in this disclosure is for the purpose of describing particular examples only and is not intended to be limiting. Unless the context explicitly indicates otherwise, an element may be one or more unless the number of elements is specifically limited. As used herein, the term "multiple" means two or more, and the term "based on" should be interpreted as "at least partially based on". Furthermore, the terms "and / or" and "at least one of..." cover any one of the listed items and all possible combinations thereof.

[0028] While mobile applications such as games, videos, and live streaming are increasingly prevalent on mobile devices, current 3D rendering technologies on mobile devices still rely on traditional hardware-dependent and pure software rendering solutions. These traditional methods consistently present various limitations for 3D rendering on mobile devices. On the one hand, hardware-dependent solutions require specialized hardware (such as raster screens and frequency dividers), leading to high equipment costs and low adoption rates. On the other hand, pure software rendering solutions cannot yet achieve dynamic 3D rendering that flexibly matches the user's perspective, significantly impacting the user's visual experience. Simultaneously, there is an imbalance between computing power and rendering quality on mobile devices, for which no effective solutions have yet been developed.

[0029] Therefore, embodiments of this disclosure provide a 3D rendering method for mobile terminals that can overcome traditional hardware limitations. It can not only achieve dynamic naked-eye 3D rendering effects that are sensitive to the user's viewing angle, but also provide a computing power adaptive mechanism for mobile terminals to optimize power consumption during the rendering process, thereby bringing users a good visual and user experience.

[0030] Exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. Before describing in detail the 3D rendering method for a mobile terminal according to embodiments of the present disclosure, an example system in which the method may be implemented will first be described.

[0031] Figure 1 This is a schematic diagram illustrating an example system 100 in which various methods described herein may be implemented according to exemplary embodiments.

[0032] like Figure 1As shown, the system 100 includes a mobile terminal 110, a server 120, and a network 130 that communicatively couples the mobile terminal 110 and the server 120.

[0033] Mobile terminal 110 may include screen 114. Client application (APP) 112 may be installed on mobile terminal 110. Client application 112 may be an application or a mini-program as a lightweight application. In some embodiments, mobile terminal 110 may be a mobile phone or tablet computer, etc. User 102 can use mobile terminal 110 to access services such as games, videos, or live streaming via client application 112, and the 3D rendering effects achieved in this process can be displayed on screen 114.

[0034] Server 120 may include a single server, a cluster of multiple servers, a distributed system, or a cloud server providing basic cloud services (such as cloud databases, cloud computing, cloud storage, and cloud communications). Network 130 may include a local area network (LAN), a wide area network (WAN), a personal area network (PAN), and / or a combination of communication networks such as the Internet. Network 130 may be wired or wireless.

[0035] Figure 2 This is a flowchart illustrating a 3D rendering method for a mobile terminal according to an exemplary embodiment.

[0036] like Figure 2 As shown, the 3D rendering method for mobile terminals includes steps S201, S202, S203, S204, S205 and S206.

[0037] In step S201, a heterogeneous data source for performing 3D rendering is obtained. This heterogeneous data source includes first input data associated with the image to be rendered, second input data associated with the gaze characteristics of the user gazing at the mobile terminal, and third input data associated with the pose of the mobile terminal.

[0038] In the example, the content to be rendered may include a sequence of images, which may include multiple images to be rendered, i.e., multiple image frames, managed by a frame buffer manager. The resolution of the images to be rendered may be the same as the screen resolution of the mobile terminal, such as 1920×1080 pixels. Accordingly, the first input data may refer to the image data of the images to be rendered. This image data may include information about multiple pixels. The information for each pixel may include information about the red, green, and blue color channels, each of which can be quantized using an 8-bit unsigned integer, representing 256 levels of intensity, thereby defining the color and brightness of each pixel.

[0039] In the example, the second input data can be obtained using eye-tracking technology. Eye-tracking technology, also known as gaze tracking, uses the front-facing camera of a mobile terminal to continuously capture a video stream containing the user's face, calculating information associated with the user's gaze point, such as the 3D coordinates of the gaze point (e.g., in the camera coordinate system) and the duration of the gaze—that is, eye-tracking data. Accordingly, the second input data can refer to this eye-tracking data. Therefore, the user's gaze characteristics when looking at the mobile terminal can be characterized by this gaze point or fixation point information. That is, the gaze characteristics can reflect where the user is looking and for how long.

[0040] In the example, the third input data could be 3D attitude data acquired from the inertial measurement unit (IMU) built into the mobile terminal, which can be used to reflect the attitude of the mobile terminal. This 3D attitude data can be represented in the form of quaternions to describe the rotational state of the mobile terminal relative to a fixed coordinate system (such as the world coordinate system).

[0041] In step S202, the first input data, the second input data, and the third input data are fused based on spatiotemporal synchronization to obtain an instantaneous viewpoint control dataset. This instantaneous viewpoint control dataset includes instantaneous viewpoint parameters with multiple degrees of freedom. These multiple degrees of freedom include degrees of freedom representing rotation and degrees of freedom representing translation.

[0042] In the example, spatiotemporal synchronization can include both time synchronization and spatial synchronization. Data fusion based on spatiotemporal synchronization can include two parts: data alignment and data fusion. In data alignment, the first, second, and third input data are synchronized both temporally and spatially. In data fusion, the synchronized first, second, and third input data are processed to obtain real-time viewpoint parameters with multiple degrees of freedom (such as 6DOF). These real-time viewpoint parameters can include pitch, roll, and yaw angles, which may be related to rotation; they can also include offsets in the x, y, and z directions, which may be related to translation. Therefore, the real-time viewpoint control dataset can reflect how the user is looking at the mobile terminal and how the mobile terminal is tilted through multi-dimensional vectors. Such a real-time viewpoint control dataset can provide data support for achieving viewpoint-sensitive 3D rendering.

[0043] In step S203, a first score, a second score, and a third score are determined for each pixel in the image to be rendered at a first level, a second score at a second level, and a third score at a third level. The first level is associated with motion features and the first score is determined based on sparse optical flow detection; the second level is associated with semantic content and the second score is determined based on semantic recognition; and the third level is associated with visual saliency and the third score is determined based on visual saliency detection based on gaze features.

[0044] In the embodiments of this disclosure, the division into first, second, and third layers is not a division of different regions in the image to be rendered, but rather an attempt to quantify the score of each pixel in these three dimensions. Therefore, each pixel simultaneously has corresponding scores in all three dimensions, namely, the first score to the third score.

[0045] In the example, sparse optical flow detection can determine the differences in motion features among individual pixels in the image to be rendered and assign them different scores. The scores can be selected from a numerical range of 0 to 1. For example, pixels involving interactive motion (such as two armies fighting, boxers, etc.) can be assigned a high score, such as 0.8; pixels involving independent motion (such as swaying grass, fluttering flags, flowing water, etc.) can be assigned a medium score, such as 0.5; and pixels involving completely static motion (such as mountains, buildings, indoor environments, etc.) can be assigned a low score, such as 0.2.

[0046] In the example, the YOLO model can be used to perform semantic recognition on the image to be rendered, determining the differences in semantic content among the pixels and assigning them different scores. The scores can also be selected from a numerical range of 0 to 1. For example, pixels involving important elements in a game or video (such as characters, vehicles, airplanes, etc.) can be assigned a high score, such as 0.8; pixels involving secondary elements in a game or video (such as objects or devices that interact with the main character) can be assigned a medium score, such as 0.5; and pixels involving objects such as mountains, flowing water, or buildings can be assigned a low score, such as 0.2.

[0047] In the example, visual saliency detection may include defining a region centered on the fixation point; therefore, the visual saliency detection is based on fixation features. Visual saliency detection can also be obtained using eye-tracking data. Similar to the previous embodiments, the score can also be selected from a numerical range of 0 to 1. The score can gradually decrease as the distance from the center point increases, for example, from 1 to 0, and the proportional relationship between the increase in distance and the decrease in score can be set differently depending on the specific circumstances.

[0048] In step S204, based on the first input data, the target weight allocation ratios for the first, second, and third layers are determined for each pixel.

[0049] Since each pixel has already been scored and quantized in three dimensions in step S203, the purpose of step S204 is to modulate these three dimensions with a target weight allocation ratio to adapt to the unique image content attributes of the first input data. For example, for games or videos with many motion elements, a higher weight can be assigned to the first level. For games or videos with many plot elements, a higher weight can be assigned to the second or third level. Therefore, the target weight allocation ratio can be expressed as a: b: c, where a, b, and c are the weights assigned to the first, second, and third levels, respectively, such as 40%: 20%: 40%, 50%: 20%: 30%, etc.

[0050] In step S205, based on the first score to the third score and the target weight allocation ratio, the computing power allocation coefficient assigned to each pixel is determined.

[0051] In the example, assuming that in step S203 the first to third scores of pixel i are determined to be 0.8, 0.6 and 0.9 respectively, and the first to third scores of pixel j are determined to be 0.2, 0.3 and 0.1 respectively, and in step S204 the target weight allocation ratio is determined to be 40%: 20%: 40%, then the computing power allocation coefficient assigned to pixel i can be determined as 0.8×0.4 + 0.6×0.2 + 0.9×0.4 = 0.80, and the computing power allocation coefficient assigned to pixel j can be determined as 0.2×0.4 + 0.3×0.2 + 0.1×0.4 = 0.18.

[0052] In step S206, 3D rendering is performed based on the first input data, the real-time viewpoint control dataset, and the computing power allocation coefficient to display the 3D rendering result on the mobile terminal.

[0053] In the example, after determining the computing power allocation coefficient corresponding to each pixel in step S205, when a specific computing power is used for 3D rendering, the optimal computing power resource can be accurately determined from the pixel granularity. In the embodiments of this disclosure, the first input data is associated with the image to be rendered, and therefore serves as the rendering source during the 3D rendering process. Based on this, on the one hand, the real-time viewpoint control dataset is used to control the sensitivity of the 3D rendering result to the user's viewpoint, so as to flexibly match the user's viewpoint; on the other hand, the computing power allocation coefficient is used to optimize the allocation of computing power at the pixel granularity, so as to more effectively adapt to the application scenarios of mobile terminals.

[0054] Therefore, in the 3D rendering method for mobile terminals according to embodiments of this disclosure, by performing spatiotemporal synchronization-based data fusion on heterogeneous data sources, an instantaneous viewpoint control dataset can be obtained to provide data support for achieving viewpoint-sensitive 3D rendering; simultaneously, by quantifying computing power requirements at the pixel level, a customized computing power allocation mechanism is provided. Thus, not only can a dynamic naked-eye 3D rendering effect sensitive to the user's viewpoint be achieved, but a computing power adaptive mechanism for mobile terminals can also be provided during the rendering process to optimize power consumption, thereby bringing users a good visual and user experience.

[0055] In some embodiments, as information from the image sequence containing the image to be rendered accumulates, the 3D rendering results, such as the 3D model, can be dynamically improved. During the rendering process, asynchronous pipelines for rendering and modeling, as well as inter-frame buffer reuse mechanisms, can also be configured. Furthermore, regarding the output of the 3D rendering results, optical adaptive dispersion simulation can be performed to achieve radial blur of varying intensities, making it more physiologically accurate. Closed-loop temperature and brightness control strategies can also be implemented to prevent overheating that could lead to frequency throttling or uncontrolled power consumption.

[0056] In some embodiments, such as in combination Figure 2 The step S202, which involves performing spatiotemporally synchronized data fusion on the first input data, the second input data, and the third input data to obtain an instantaneous viewpoint control dataset, may include: aligning the timestamps of the first input data, the second input data, and the third input data, and projecting the first input data, the second input data, and the third input data onto the same coordinate system to obtain spatiotemporally aligned first input data, second input data, and third input data; and calculating instantaneous viewpoint parameters for multiple degrees of freedom based on the spatiotemporally aligned first input data, second input data, and third input data.

[0057] In the example, based on the properties of these heterogeneous data sources, the third input data is typically high-frame-rate data (such as 100~2000Hz), which is better able to handle fast, sudden motion, while the first and second input data are typically low-frame-rate data. Therefore, the low-frame-rate first and second input data can be aligned with the third input data in terms of timestamps. For example, assuming the latest timestamp of the third input data is t0, interpolation or extrapolation can be performed on the existing timestamps t1 and t2 of the first and second input data to calculate the data value corresponding to t0.

[0058] In the example, based on the sources of these heterogeneous data sources, the first input data can correspond to the screen coordinate system, the second input data can correspond to the camera coordinate system, and the third input data can correspond to the device (i.e., mobile terminal) coordinate system. In some embodiments, the three types of data can be projected into the device coordinate system or into the screen coordinate system. Taking the latter as an example, considering that the screen coordinate system of the first input data defines the spatial position of the rendering result and the gaze point, the second and third input data can be transformed together into this screen coordinate system. For example, since the intrinsic parameters of the camera, such as focal length and principal point, are known, and the transformation coefficients (rotation and translation matrices, i.e., extrinsic parameters) from the camera coordinate system to the screen coordinate system can be obtained from the internal information of the mobile terminal (such as firmware) and are also known, the transformation from the camera coordinate system to the screen coordinate system can be achieved. This process does not require the intermediate world coordinate system. Based on a similar transformation principle, the transformation from the device coordinate system to the screen coordinate system can also be achieved.

[0059] By employing the above method, after obtaining the spatiotemporally aligned first, second, and third input data, real-time viewpoint parameters with multiple degrees of freedom (such as 6DOF) can be calculated through the fusion of multi-source data. Therefore, spatiotemporally synchronized data fusion can construct a unified geometric model based on heterogeneous data sources, enabling the reflection of how the user views the mobile terminal and how the mobile terminal is tilted through multi-dimensional vectors. This allows the calculated real-time viewpoint control dataset to provide data support for achieving viewpoint-sensitive 3D rendering.

[0060] In some embodiments, the step of calculating real-time viewpoint parameters of multiple degrees of freedom based on spatiotemporally aligned first input data, second input data, and third input data may include: calculating multiple rotation vectors of the mobile terminal based on the third input data, the multiple rotation vectors being associated with degrees of freedom for characterizing rotation; and calculating multiple translation vectors of gaze features based on the first and second input data, the multiple translation vectors being associated with degrees of freedom for characterizing translation.

[0061] In the example, pitch, roll, and yaw angles can be calculated from third input data. For instance, IMU data includes a three-axis gyroscope (measuring angular velocity), a three-axis accelerometer (measuring linear acceleration), and a three-axis magnetometer (measuring magnetic field direction). The data is processed using sensor fusion algorithms (such as complementary filters, Kalman filters, etc.) to filter out noise and compensate for sensor drift, resulting in the device's precise orientation in space, which is characterized by pitch, roll, and yaw angles.

[0062] In the example, the offsets in the x, y, and z directions can be calculated using the first and second input data. The offset in the x direction can refer to the horizontal offset of the left and right eyes, the offset in the y direction can refer to the vertical offset of the eyes, and the offset in the z direction can refer to the depth offset of the eyes. These offsets in the x, y, and z directions can be used to implement stereo parallax during rendering. For example, when using a frame buffer manager, the depth information generated by the previous frame rendering can be cached with the image sequence, while eye-tracking data can provide 2D positional information, thus allowing the calculation of offsets in the x, y, and z directions.

[0063] The above method can effectively utilize heterogeneous data sources to construct a unified geometric model, thereby enabling the calculated real-time viewpoint parameters of multiple degrees of freedom to reflect the user's gaze position and device posture in the same coordinate system in real time, without the need for external calibration or world coordinate system transformation, so as to directly drive the subsequent 3D rendering process.

[0064] In some embodiments, such as in combination Figure 2 The step S204, which determines the target weight allocation ratio for the first, second, and third layers on each pixel based on the first input data, may include: determining the target scene type corresponding to the first input data among a plurality of preset scene types, wherein each preset scene type indicates the content type of the image to be rendered and has a preset weight allocation ratio, such that the sum of the first weight, second weight, and third weight corresponding to the first, second, and third layers is 1; and determining the target weight allocation ratio of the first weight, second weight, and third weight based on the target scene type and the preset weight allocation ratio.

[0065] In the example, multiple preset scene types can include movie watching, role-playing, racing, and shooting, etc. That is, different preset scene types are based on content type. For example, for movie watching, the preset weight allocation ratio can be 30%: 20%: 50%; for role-playing, the ratio can be 20%: 30%: 50%; for racing, 50%: 20%: 30%; and for shooting, 40%: 20%: 40%. Therefore, assuming the target scene type is determined to be one of the example scene types above, such as shooting, the target weight allocation ratio can be determined to be 40%: 20%: 40% according to the above correspondence. As described above, the target weight allocation ratio can be expressed as a: b: c, where a, b, and c are the weights assigned to the first, second, and third levels, respectively, i.e., the first weight, the second weight, and the third weight, such that the sum of the three is 1.

[0066] Therefore, by pre-setting the weight allocation ratios of the first, second, and third levels on each pixel based on various possible scene types, and selecting the matching target weight allocation ratio according to the actual scene type, it is possible to select the optimal computing power allocation strategy for different video or game types, thereby obtaining the optimal combination of 3D rendering quality and performance.

[0067] Figure 3 This is a schematic diagram illustrating the scene resolution process according to an exemplary embodiment.

[0068] like Figure 3 As shown, for each pixel in the image to be rendered, there exists a first level associated with motion features, a second level associated with semantic content, and a third level associated with visual saliency. A hierarchical decision engine can be set up to configure corresponding weights for different levels for different scene types. That is, based on the first input data, the step of determining the target weight allocation ratio of the first, second, and third weights of the first, second, and third levels for each pixel (such as combining...) Figure 2 Step S204 can be executed via this hierarchical decision engine. In the example described above, assuming that pixel i corresponds to scores of 0.8, 0.6, and 0.9 for the first to third levels respectively, and the target weight allocation ratio for the first, second, and third weights is 40%:20%:40%, then the computational power allocation coefficient assigned to pixel i can be 0.8×0.4 + 0.6×0.2 + 0.9×0.4 = 0.80. In this way, the computational power requirement can be quantified at the pixel level for each pixel in the image to be rendered, thereby providing a customized computational power allocation mechanism.

[0069] In some embodiments, the 3D rendering method for a mobile terminal may further include: before performing 3D rendering, selecting a target rendering pipeline for the 3D rendering from a plurality of preset rendering pipelines based on the state of the mobile terminal. Each preset rendering pipeline indicates the algorithm scheduling for 3D rendering, such that the computational power of the algorithms corresponding to the plurality of preset rendering pipelines decreases sequentially.

[0070] In the example, it can be combined as follows. Figure 2Before step S206, the state of the mobile terminal is determined. The state of the mobile terminal can be determined using information such as battery level and / or temperature. For example, when the mobile terminal is plugged in, it means that the mobile terminal's battery retention capability is likely good, so the rendering algorithm with the highest or higher computing power requirement can be used in the preset rendering pipeline corresponding to the plugged-in state. When the mobile terminal is in a normal state, such as when the battery level is above 50% and the temperature is below 35 degrees Celsius, it means that both the battery level and temperature conditions are likely good, so the rendering algorithm with a higher, medium, or lower computing power requirement can be used in the preset rendering pipeline corresponding to the normal state. When the mobile terminal is in an extreme state, such as when the battery level is below 20% or the temperature is above 60 degrees Celsius, it means that the mobile terminal is currently unable to support good 3D rendering effects, so the rendering algorithm with a lower or lower computing power requirement can be used in the preset rendering pipeline corresponding to the extreme state.

[0071] In some embodiments, the algorithms corresponding to each of the plurality of preset rendering pipelines may include a four-dimensional Gaussian sputtering algorithm, a three-dimensional Gaussian sputtering algorithm, and a two-dimensional parallax algorithm, respectively.

[0072] In the examples, the 4D-GS algorithm is a real-time neural rendering technique for dynamic scenes, extending traditional 3D Gaussian point clouds into a deformable model with 3D space and one-dimensional time. The 3D-GS algorithm is a lightweight real-time neural rendering technique that represents a static scene as a set of differentiable 3D Gaussian ellipsoids, directly projecting the point cloud onto any viewpoint. The 2D parallax algorithm is a minimalist pseudo-3D technique that uses only a single frame of RGB and its depth map to generate 2D parallax maps with lateral pixel offsets for the left and right eyes, then synthesizes the left and right views through radial blurring or simple offset.

[0073] In the example, for 3D-GS or 4D-GS algorithms, the mobile terminal can adaptively manage and render Gaussian point clouds, achieving dynamic adjustment of quantity, accuracy, and density.

[0074] Therefore, by pre-setting different levels of preset rendering pipelines and their corresponding matching algorithms, the optimal algorithm allocation scheme can be determined based on the latest real-time status of the mobile terminal, thereby further optimizing the computing power optimization mechanism for 3D rendering on the mobile terminal and achieving flexible and seamless switching between different algorithms.

[0075] Figure 4 This is a schematic diagram illustrating the algorithm scheduling process according to an exemplary embodiment.

[0076] like Figure 4As shown, this example describes a three-level rendering pipeline. The first-level rendering pipeline is suitable for mobile devices when plugged in, the second-level pipeline for normal operation, and the third-level pipeline for extreme conditions. 4D-GS, 3D-GS, and 2D parallax algorithms can be used in the first, second, and third levels of the rendering pipeline, respectively. This is because a flexible and seamless switching mechanism between different rendering pipelines is provided, such as... Figure 4 As shown, when the state of the mobile terminal changes in any rendering pipeline, that is, when the conditions change, the algorithm can be rescheduled for real-time updates.

[0077] In some embodiments, the 3D rendering method for a mobile terminal according to this disclosure can be implemented through an architecture of an input layer, a processing engine, a rendering engine, and an output layer. (In combination) Figure 2 Taking steps S201 to S206 as an example, the operation of the input layer may involve step S201, the operation of the processing engine may involve steps S202 to S205, and the operation of the rendering engine and the output layer may involve step S206.

[0078] Figure 5 This is a schematic diagram illustrating the control flow according to an exemplary embodiment.

[0079] like Figure 5 As shown, the input layer can send the raw data stream to the processing engine for processing to obtain a structured scene description. This structured scene description can contain two aspects of information: one is to indicate how the user is looking at the mobile terminal and how the mobile terminal is tilted; the other is to indicate how to allocate computing power. When the rendering engine receives this structured scene description, it can provide the rendering result to the output layer for display. The output layer can also provide quality feedback to the processing engine for rendering optimization, thereby further outputting optimized rendering results.

[0080] The control flow implemented through the above architecture can achieve dynamic naked-eye 3D rendering effects that are sensitive to the user's viewing angle. Moreover, it can provide a computing power adaptive mechanism for mobile terminals during the rendering process to optimize power consumption, thereby bringing users a good visual and user experience. For example, when the user moves their head slightly to the left or focuses their gaze on different characters in the picture, they can see corresponding parallax changes and obtain an immersive visual experience.

[0081] Embodiments of this disclosure also provide a three-dimensional rendering apparatus for a mobile terminal.

[0082] Figure 6 This is a schematic block diagram illustrating a 3D rendering apparatus for a mobile terminal according to an exemplary embodiment.

[0083] likeFigure 6 As shown, in some embodiments, the 3D rendering device 600 for a mobile terminal includes a data source acquisition module 601, a spatiotemporal synchronization module 602, a scene parsing module 603, a weight allocation module 604, a computing power allocation module 605, and a rendering execution module 606.

[0084] The data source acquisition module 601 is configured to acquire heterogeneous data sources for performing 3D rendering, including first input data associated with the image to be rendered, second input data associated with the gaze characteristics of the user's gaze on the mobile terminal, and third input data associated with the pose of the mobile terminal.

[0085] The spatiotemporal synchronization module 602 is configured to perform spatiotemporal synchronization-based data fusion on the first input data, the second input data, and the third input data to obtain an instantaneous viewpoint control dataset. The instantaneous viewpoint control dataset includes instantaneous viewpoint parameters with multiple degrees of freedom, including degrees of freedom representing rotation and degrees of freedom representing translation.

[0086] The scene parsing module 603 is configured to determine a first score for each pixel in the image to be rendered at a first level, a second score at a second level, and a third score at a third level. The first level is associated with motion features and the first score is determined based on sparse optical flow detection; the second level is associated with semantic content and the second score is determined based on semantic recognition; and the third level is associated with visual saliency and the third score is determined based on visual saliency detection based on gaze features.

[0087] The weight allocation module 604 is configured to determine the target weight allocation ratios for the first, second, and third layers on each pixel based on the first input data.

[0088] The computing power allocation module 605 is configured to determine the computing power allocation coefficient assigned to each pixel based on the first score to the third score and the target weight allocation ratio.

[0089] The rendering execution module 606 is configured to perform 3D rendering based on the first input data, the real-time view control dataset, and the computing power allocation coefficients to display the 3D rendering results on a mobile terminal.

[0090] The operations of the aforementioned data source acquisition module 601, spatiotemporal synchronization module 602, scene parsing module 603, weight allocation module 604, computing power allocation module 605, and rendering execution module 606 can be combined. Figure 2 The operations of steps S201 to S206 are the same, so the details of each aspect will not be repeated here.

[0091] In some embodiments, the 3D rendering apparatus 600 for a mobile terminal may further include a rendering pipeline determination module 607, configured to select a target rendering pipeline for 3D rendering from a plurality of preset rendering pipelines based on the state of the mobile terminal before performing 3D rendering. Each preset rendering pipeline indicates an algorithm scheduling for 3D rendering, such that the computational power of the algorithms corresponding to the plurality of preset rendering pipelines decreases sequentially.

[0092] In some embodiments, the spatiotemporal synchronization module 602 may include: a data alignment module 6021, configured to align the timestamps of the first input data, the second input data, and the third input data, and to project the first input data, the second input data, and the third input data onto the same coordinate system to obtain spatiotemporally aligned first input data, second input data, and third input data; and a data fusion module 6022, which calculates real-time viewpoint parameters of multiple degrees of freedom based on the spatiotemporally aligned first input data, second input data, and third input data.

[0093] In some embodiments, the data fusion module 6022 may include: a rotation determination module 6022a, configured to calculate a plurality of rotation vectors of the mobile terminal based on third input data, the plurality of rotation vectors being associated with degrees of freedom for characterizing rotation; and a translation determination module 6022b, configured to calculate a plurality of translation vectors of gaze features based on first input data and second input data, the plurality of translation vectors being associated with degrees of freedom for characterizing translation.

[0094] In some embodiments, the weight allocation module 604 may include: a scene determination module 6041, configured to determine a target scene type corresponding to the first input data among a plurality of preset scene types, wherein each preset scene type indicates the content type of the image to be rendered and has a preset weight allocation ratio such that the sum of the first weight, the second weight and the third weight corresponding to the first level, the second level and the third level is 1 respectively; and a weight determination module 6042, configured to determine the target weight allocation ratio of the first weight, the second weight and the third weight based on the target scene type and the preset weight allocation ratio.

[0095] While specific functions have been discussed above with reference to specific modules, it should be noted that the functions of the modules discussed herein can be divided into multiple modules, and / or at least some functions of multiple modules can be combined into a single module. The specific actions performed by the modules discussed herein include the specific module itself performing the action, or alternatively, the specific module calling or otherwise accessing another component or module that performs the action (or performs the action in conjunction with the specific module). Therefore, a specific module performing an action can include the specific module performing the action itself and / or another module that performs the action, called or otherwise accessed by the specific module.

[0096] It should also be understood that the above regarding Figure 6 The described modules can be implemented in hardware or in hardware in combination with software and / or firmware. For example, these modules can be implemented as computer program code / instructions configured to execute in one or more processors and stored in a computer-readable storage medium. Alternatively, these modules can be implemented as hardware logic / circuit. For example, these modules can be implemented together in a System on Chip (SoC). An SoC may include an integrated circuit chip (which includes one or more components in a processor (e.g., a Central Processing Unit (CPU), microcontroller, microprocessor, digital signal processor (DSP), etc.), memory, one or more communication interfaces, and / or other circuitry) and may optionally execute received program code and / or include embedded firmware to perform functions.

[0097] According to one aspect of this disclosure, a computer device is also provided, including a memory, a processor, and a computer program stored in the memory. The processor is configured to execute the computer program to implement the steps of any of the method embodiments described above.

[0098] According to one aspect of this disclosure, a non-transitory computer-readable storage medium is also provided, on which a computer program is stored, which, when executed by a processor, implements the steps of any of the method embodiments described above.

[0099] According to one aspect of this disclosure, a computer program product is also provided, which includes a computer program that, when executed by a processor, implements the steps of any of the method embodiments described above.

[0100] In the following text, combined with Figure 7 Illustrative examples describing such computer devices, non-transitory computer-readable storage media, and computer program products.

[0101] Figure 7 An example configuration of a computer device 700 that can be used to implement the methods described herein is shown.

[0102] Computer device 700 can be a variety of different types of devices. Examples of computer device 700 include, but are not limited to: desktop computers, server computers, laptop or netbook computers, mobile devices (e.g., tablet computers, cellular or other wireless phones (e.g., smartphones), notebook computers, mobile stations), wearable devices (e.g., glasses, watches), entertainment devices (e.g., entertainment appliances, set-top boxes communicatively coupled to a display device, game consoles), televisions or other display devices, automotive computers, and so on.

[0103] Computer device 700 may include at least one processor 702, memory 704, multiple communication interfaces 706, display device 708, other input / output (I / O) devices 710, and one or more mass storage devices 712 capable of communicating with each other, such as via system bus 714 or other suitable connections.

[0104] Processor 702 may be a single processing unit or multiple processing units, and all processing units may include single or multiple computing units or multiple cores. Processor 702 may be implemented as one or more microprocessors, microcomputers, microcontrollers, digital signal processors, central processing units, state machines, logic circuits, and / or any device that manipulates signals based on operating instructions. Among other capabilities, processor 702 may be configured to acquire and execute computer-readable instructions stored in memory 704, mass storage device 712, or other computer-readable media, such as program code of operating system 716, program code of application program 718, program code of other program 720, etc.

[0105] Memory 704 and mass storage device 712 are examples of computer-readable storage media for storing instructions that are executed by processor 702 to perform the various functions described above. For example, memory 704 can generally include both volatile and non-volatile memory (e.g., RAM, ROM, etc.). Furthermore, mass storage device 712 can generally include hard disk drives, solid-state drives, removable media, including external and removable drives, memory cards, flash memory, floppy disks, optical disks (e.g., CDs, DVDs), storage arrays, network-attached storage, storage area networks, etc. Both memory 704 and mass storage device 712 can be collectively referred to herein as memory or computer-readable storage media, and can be non-transitory media capable of storing computer-readable, processor-executable program instructions as computer program code, which can be executed by processor 702 as a specific machine configured to perform the operations and functions described in the examples herein.

[0106] Multiple programs may be stored on mass storage device 712. These programs include operating system 716, one or more application programs 718, other programs 720, and program data 722, and they may be loaded into memory 704 for execution. Examples of such application programs or program modules may include computer program logic (e.g., computer program code or instructions) of the methods described herein and / or other embodiments described herein.

[0107] Although Figure 7 The modules 716, 718, 720, and 722, or portions thereof, are illustrated as being stored in memory 704 of computer device 700; however, modules 716, 718, 720, and 722 may be implemented using any form of computer-readable medium accessible by computer device 700. As used herein, “computer-readable medium” includes at least two types of computer-readable media: computer-readable storage media and communication media.

[0108] Computer-readable storage media include volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer-readable storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, DVD, or other optical storage devices, magnetic cassettes, magnetic tapes, disk storage devices or other magnetic storage devices, or any other non-transmission medium that can be used to store information for access by computer devices. In contrast, communication media can embody computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms. Computer-readable storage media as defined herein do not include communication media.

[0109] One or more communication interfaces 706 are used for exchanging data with other devices, such as via a network, direct connection, etc. Such communication interfaces can be one or more of the following: any type of network interface (e.g., a network interface card (NIC)), wired or wireless (such as IEEE 802.11 Wireless LAN (WLAN)) wireless interface, Wi-MAX interface, Ethernet interface, Universal Serial Bus (USB) interface, cellular network interface, Bluetooth. TM Interfaces include near field communication (NFC) interfaces. Communication interface 706 facilitates communication across various network and protocol types, including wired networks (e.g., LAN, cable, etc.) and wireless networks (e.g., WLAN, cellular, satellite, etc.), the Internet, etc. Communication interface 706 can also provide communication with external storage devices (not shown) such as storage arrays, network-attached storage, storage area networks, etc.

[0110] In some examples, a display device 708, such as a monitor, may be included for displaying information and images to the user. Other I / O devices 710 may be devices that receive various inputs from the user and provide various outputs to the user, and may include touch input devices, gesture input devices, cameras, keyboards, remote controls, mice, printers, audio input / output devices, and so on.

[0111] The technologies described herein can be supported by these various configurations of computer device 700, and are not limited to specific examples of the technologies described herein. For example, the functionality can also be implemented wholly or partially on a “cloud” using a distributed system. A cloud includes and / or represents a platform for resources. The platform abstracts the underlying functionality of the cloud’s hardware (e.g., servers) and software resources. Resources may include applications and / or data that can be used when performing computational processing on a server remote from computer device 700. Resources may also include services provided via the Internet and / or via subscriber networks such as cellular or Wi-Fi networks. The platform can abstract resources and functionality to connect computer device 700 to other computer devices. Therefore, the implementation of the functionality described herein can be distributed throughout the cloud. For example, the functionality can be implemented partly on computer device 700 and partly through a platform that abstracts the functionality of the cloud.

Claims

1. A three-dimensional rendering method for mobile terminals, characterized in that, The method includes: Obtain heterogeneous data sources for performing 3D rendering, including first input data associated with the image to be rendered, second input data associated with the user's gaze features on the mobile terminal, and third input data associated with the pose of the mobile terminal. The first input data, the second input data, and the third input data are fused based on spatiotemporal synchronization to obtain an instant view control dataset. The instant view control dataset includes instant view parameters with multiple degrees of freedom, including degrees of freedom for representing rotation and degrees of freedom for representing translation. Determine a first score, a second score, and a third score for each pixel in the image to be rendered at a first level, a second score at a second level, and a third score at a third level, wherein the first level is associated with motion features and the first score is determined based on sparse optical flow detection, the second level is associated with semantic content and the second score is determined based on semantic recognition, and the third level is associated with visual saliency and the third score is determined based on visual saliency detection based on the gaze features. Based on the first input data, determine the target weight allocation ratio for the first layer, the second layer, and the third layer on each pixel; Based on the first score to the third score and the target weight allocation ratio, a computing power allocation coefficient is determined for each pixel; and The 3D rendering is performed based on the first input data, the real-time view control dataset, and the computing power allocation coefficient to display the 3D rendering result on the mobile terminal.

2. The method according to claim 1, characterized in that, The step of performing spatiotemporal synchronization-based data fusion on the first input data, the second input data, and the third input data to obtain a real-time perspective control dataset includes: By aligning the timestamps of the first, second, and third input data, and projecting them onto the same coordinate system, spatiotemporally aligned first, second, and third input data are obtained; and The instantaneous viewpoint parameters of the multiple degrees of freedom are calculated based on the spatiotemporally aligned first, second, and third input data.

3. The method according to claim 2, characterized in that, The calculation of the instantaneous viewpoint parameters of the multiple degrees of freedom based on the spatiotemporally aligned first, second, and third input data includes: Based on the third input data, multiple rotation vectors of the mobile terminal are calculated, and these multiple rotation vectors are associated with the degrees of freedom used to characterize rotation; and Based on the first input data and the second input data, a plurality of translation vectors of the gaze feature are calculated, and the plurality of translation vectors are associated with the degrees of freedom used to characterize the translation.

4. The method according to any one of claims 1 to 3, characterized in that, The step of determining the target weight allocation ratio for the first layer, the second layer, and the third layer on each pixel based on the first input data includes: Determine the target scene type corresponding to the first input data from multiple preset scene types, wherein each preset scene type indicates the content type of the image to be rendered and has a preset weight allocation ratio, such that the sum of the first weight, second weight, and third weight corresponding to the first level, the second level, and the third level, respectively, is 1; and Based on the target scenario type and the preset weight allocation ratio, the target weight allocation ratio of the first weight, the second weight, and the third weight is determined.

5. The method according to any one of claims 1 to 3, characterized in that, The method further includes: Before performing the 3D rendering, based on the state of the mobile terminal, a target rendering pipeline for the 3D rendering is selected from multiple preset rendering pipelines. Each preset rendering pipeline indicates the algorithm scheduling for the 3D rendering, so that the computing power of the algorithms corresponding to the multiple preset rendering pipelines decreases sequentially.

6. The method according to claim 5, characterized in that, The algorithms corresponding to the multiple preset rendering pipelines include a four-dimensional Gaussian sputtering algorithm, a three-dimensional Gaussian sputtering algorithm, and a two-dimensional parallax algorithm.

7. A three-dimensional rendering device for a mobile terminal, characterized in that, The device includes: The data source acquisition module is configured to acquire a heterogeneous data source for performing 3D rendering, including first input data associated with the image to be rendered, second input data associated with the user's gaze features on the mobile terminal, and third input data associated with the pose of the mobile terminal. The spatiotemporal synchronization module is configured to perform spatiotemporal synchronization-based data fusion on the first input data, the second input data, and the third input data to obtain an instant view control dataset. The instant view control dataset includes instant view parameters with multiple degrees of freedom, including degrees of freedom for representing rotation and degrees of freedom for representing translation. The scene parsing module is configured to determine a first score at a first level, a second score at a second level, and a third score at a third level for each pixel in the image to be rendered, wherein the first level is associated with motion features and the first score is determined based on sparse optical flow detection, the second level is associated with semantic content and the second score is determined based on semantic recognition, and the third level is associated with visual saliency and the third score is determined based on visual saliency detection based on the gaze features. The weight allocation module is configured to determine, based on the first input data, the target weight allocation ratio for assigning the first layer, the second layer, and the third layer to each pixel; The computing power allocation module is configured to determine a computing power allocation coefficient for each pixel based on the first score to the third score and the target weight allocation ratio; and The rendering execution module is configured to perform the 3D rendering based on the first input data, the real-time view control dataset, and the computing power allocation coefficient, so as to display the 3D rendering result on the mobile terminal.

8. A computer device, characterized in that, The computer device includes: At least one processor; A memory having a computer program stored thereon, wherein, when executed by the at least one processor, the computer program causes the at least one processor to perform the method of any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, causes the processor to perform the method of any one of claims 1-6.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, causes the processor to perform the method of any one of claims 1-6.

Images