Method, apparatus, and vehicle for performing distortion rendering in a vehicle

Abstract

The application provides a method, device and vehicle for performing distortion rendering in a vehicle. The method comprises: identifying each layer of an original image to be rendered on a vehicle screen; for each layer: obtaining a layer transformation matrix corresponding to the layer, mapping a local coordinate space of the layer to a global coordinate space corresponding to the vehicle screen based on the layer transformation matrix corresponding to the layer, performing intersection detection on the layer mapped to the global coordinate space and a full-screen distortion grid parameter to extract local distortion grid vertex coordinates of the intersection between the layer and the full-screen distortion grid parameter, transforming the local distortion grid vertex coordinates to a layer texture space, obtaining texture coordinates of the local distortion grid vertex coordinates, and rendering the layer on the vehicle screen based on the texture coordinates. According to the embodiment of the application, the full-screen distortion grid parameter can be independently applied to each layer, and distortion rendering can be performed layer by layer, thereby reducing the memory read-write bandwidth overhead.

Description

Technical Field

[0001] This invention relates to the field of in-vehicle image display, and more specifically, to a method, apparatus, and vehicle for performing distortion rendering in a vehicle. Background Technology

[0002] With the development of smart cockpit technology, in-vehicle entertainment systems (IVI) are increasingly adopting large-size curved screens or irregularly shaped screens, such as the integrated dashboard and center console screen and the panoramic head-up display (PHUD). In order to adapt to the curved shape of the physical screen, the output image signal must be geometrically distorted to ensure that the image is not distorted when viewed by the human eye.

[0003] In the Android graphics subsystem, traditional image compositing mainly relies on the Hardware Composer (HWC). However, the HWC was originally designed to handle the translation, scaling, and 2D rotation of rectangular layers, and typically does not support non-linear geometric mesh transformations.

[0004] While some display driver chips (TCONs) or external field-programmable gate arrays (FPGAs) possess hardware distortion capabilities, hardware channel resources are extremely scarce in multi-display cockpit scenarios. For example, when a system needs to drive more than four screens, hardware channels often cannot cover all screens or support dynamically changing UI layouts. Therefore, it is necessary to introduce the general computing power of GPUs (OpenGL ES) to achieve distortion. Summary of the Invention

[0005] This invention provides a method for performing distortion rendering in a vehicle, comprising: identifying each layer of an original image to be rendered on an in-vehicle screen; for each layer: obtaining a layer transformation matrix corresponding to the layer; based on the layer transformation matrix corresponding to the layer, mapping the local coordinate space of the layer to a global coordinate space corresponding to the in-vehicle screen; performing intersection detection on the layer mapped to the global coordinate space and the full-screen distortion mesh parameters to extract the coordinates of the intersecting local distortion mesh vertices between the layer and the full-screen distortion mesh parameters; transforming the local distortion mesh vertex coordinates to a layer texture space to obtain the texture coordinates of the local distortion mesh vertex coordinates; and rendering the layer on the in-vehicle screen based on the texture coordinates.

[0006] In one example, rendering the layer on the vehicle screen based on the texture coordinates includes: sampling the texture data of the layer of the original image based on the texture coordinates; and rendering the layer on the vehicle screen based on the texture coordinates of the local distortion mesh vertex coordinates and the sampled texture data.

[0007] In one example, for each layer, mapping the local coordinate space corresponding to the layer to the global coordinate space corresponding to the vehicle screen includes: determining the axis-aligned bounding box of the layer in the global coordinate space based on the layer transformation matrix of the layer, and wherein extracting the coordinates of the intersecting local distortion mesh vertices between the layer and the full-screen distortion mesh parameters includes: determining whether the full-screen distortion mesh parameters intersect with the axis-aligned bounding box of the layer based on the sub-mesh of the full-screen distortion mesh parameters, and determining the vertex coordinates of the intersecting sub-mesh as the vertex coordinates of the local distortion mesh of the layer.

[0008] In one example, the subgrid of the full-screen distortion mesh parameter is subdivided, and determining whether the full-screen distortion mesh parameter intersects the axis-aligned bounding box of the layer includes: determining whether the full-screen distortion mesh parameter intersects the axis-aligned bounding box of the layer based on the subdivided subgrid.

[0009] In one example, rendering the layer on the vehicle screen based on the texture coordinates includes performing anti-aliasing on the edges of the layer.

[0010] In one example, the full-screen distortion mesh parameter indicates the distortion configuration of the in-vehicle screen.

[0011] In one example, the method further includes, before identifying each layer of the original image to be rendered on the vehicle screen: identifying the vehicle screen to which the original image is to be rendered; determining whether the vehicle screen has distortion; and based on the determination that the vehicle screen has distortion, marking all layers of the original image as enabling client-side compositing, wherein the original image includes picture and / or video images.

[0012] In one example, the in-vehicle screen includes a projection screen of a head-up display, and the full-screen distortion grid parameters are associated with the distortion configuration and optical reflection and refraction properties of the vehicle windshield.

[0013] This invention also provides an apparatus for performing distortion rendering in a vehicle, comprising: a layer composition management module configured to identify each layer of an original image to be rendered on a vehicle screen; a rendering engine module configured to acquire full-screen distortion mesh parameters of the vehicle screen; and a distortion mapping module configured to, for each layer, obtain a layer transformation matrix corresponding to that layer, map the local coordinate space of the layer to a global coordinate space corresponding to the vehicle screen based on the layer transformation matrix, perform intersection detection on the layer mapped to the global coordinate space and the full-screen distortion mesh parameters to extract the coordinates of the intersecting local distortion mesh vertices between the layer and the full-screen distortion mesh parameters, and transform the local distortion mesh vertex coordinates to a layer texture space to obtain the texture coordinates of the local distortion mesh vertex coordinates, wherein the rendering engine module is further configured to control the rendering of the layer on the vehicle screen based on the texture coordinates.

[0014] In one example, the rendering engine module is configured to load the full-screen distortion mesh parameters of the vehicle screen from the distortion mesh management module.

[0015] In one example, the rendering engine module is also configured to: sample the texture data of the layer of the original image based on the texture coordinates; and control the rendering of the layer on the vehicle screen based on the texture coordinates of the local distortion mesh vertex coordinates and the sampled texture data.

[0016] In one example, for each layer, the distortion mapping module maps the local coordinate space corresponding to the layer to the global coordinate space corresponding to the vehicle screen by: determining the axis-aligned bounding box of the layer in the global coordinate space based on the layer transformation matrix of the layer, and wherein the distortion mapping module extracts the coordinates of the intersecting local distortion mesh vertices between the layer and the full-screen distortion mesh parameters by: determining whether the full-screen distortion mesh parameters intersect with the axis-aligned bounding box of the layer based on the sub-mesh of the full-screen distortion mesh parameters, and determining the vertex coordinates of the intersecting sub-mesh as the vertex coordinates of the local distortion mesh of the layer.

[0017] In one example, the subgrid of the full-screen distortion mesh parameter is subdivided, and the distortion mapping module determines whether the full-screen distortion mesh parameter intersects the axis-aligned bounding box of the layer based on the subdivided subgrid.

[0018] In one example, the device also includes an anti-aliasing module configured to perform anti-aliasing on the edges of the layer.

[0019] In one example, the full-screen distortion mesh parameter indicates the distortion configuration of the in-vehicle screen.

[0020] In one example, the layer compositing management module is also configured to: identify the vehicle screen to which the original image is to be rendered; and, based on the determination that the vehicle screen has distortion, mark all layers of the original image as enabling client-side compositing, wherein the original image includes pictures and / or video images.

[0021] In one example, the device further includes a distortion strategy control module configured to determine whether the in-vehicle screen is distorted based on a distortion strategy, wherein the distortion strategy control module is also configured to store parameters for performing anti-aliasing on the edges of each layer.

[0022] In one example, the in-vehicle screen includes a projection screen of a head-up display, and the full-screen distortion grid parameters are associated with the distortion configuration and optical reflection and refraction properties of the vehicle windshield.

[0023] Another embodiment of the present invention provides a vehicle, including: a plurality of in-vehicle screens; and the device as described above.

[0024] According to embodiments of the present invention, full-screen distortion mesh parameters can be applied independently to each layer, distortion rendering can be performed layer by layer, and multiple layers can be drawn to the screen at once through instructions. This eliminates the need for write bandwidth to FBO, read bandwidth from FBO, and write bandwidth to FrameBuffer, greatly reducing memory bandwidth overhead. Attached Figure Description

[0025] The invention will be more readily understood from the following detailed description with reference to the accompanying drawings, wherein like reference numerals designate units of the same structure, and wherein:

[0026] Figure 1 This is a schematic flowchart illustrating a method for performing distortion rendering in a vehicle according to an embodiment of the present invention;

[0027] Figure 2 A schematic architecture for implementing a method for performing distortion rendering in a vehicle according to an embodiment of the present invention is shown;

[0028] Figure 3 This is a schematic flowchart illustrating a more detailed method for performing distortion rendering in a vehicle according to an embodiment of the present invention;

[0029] Figure 4 A schematic diagram illustrating an image comprising multiple layers according to an embodiment of the present invention is shown;

[0030] Figure 5 This illustrates the distortion rendering effect of a layer according to an embodiment of the present invention;

[0031] Figure 6 A schematic block diagram of an apparatus for performing distortion rendering in a vehicle according to an embodiment of the present invention is shown; and

[0032] Figure 7 A schematic block diagram of a vehicle according to an embodiment of the present invention is shown. Detailed Implementation

[0033] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0034] Furthermore, it should be noted that in this specification, A connected to B can mean "A is directly connected to B" or "A is connected to B via other middleware". A connected between B and C can mean "A is directly connected to both B and C" or "A is connected to both B and C via other middleware".

[0035] In the current design, distortion rendering of the image is performed using a post-processing approach based on Framebuffer Objects (FBOs) in OpenGL. For example, SurfaceFlinger renders all visible layers of the image (Layer 1, Layer 2, ..., Layer N) normally through the graphics processing unit (GPU) into a virtual video memory region (FBO Texture) created by glGenFramebuffers. At this point, the image is an undistorted planar image. Before the final display (SwapBuffers), this FBO Texture is used as a texture source to load the full-screen distortion mesh parameters. The texture is then applied to the mesh for a second rendering before being output to the physical screen.

[0036] However, this design suffers from a "bandwidth wall" problem. Specifically, on the one hand, it consumes twice the video memory bandwidth. Each frame needs to be written to the FBO (write bandwidth), then read from the FBO (read bandwidth), and finally written to the FrameBuffer (write bandwidth). At high resolutions (such as 4K cockpit screens), the video memory bandwidth is instantly saturated. On the other hand, in real-world performance testing, under high-load multi-window scenarios in automotive environments, the FBO solution causes the frame rate to plummet from a smooth 45.8 FPS to 27.6 FPS, resulting in severe UI stuttering and tearing, failing to meet automotive-grade smoothness requirements.

[0037] Figure 1This is a schematic flowchart illustrating a method 100 for performing distortion rendering in a vehicle according to an embodiment of the present invention.

[0038] like Figure 1 As shown, in method 100, each layer of the original image to be rendered on the vehicle screen can be identified.

[0039] Then, for each identified layer: at 120, the layer transformation matrix corresponding to that layer can be obtained; at 130, the local coordinate space of that layer can be mapped to the global coordinate space corresponding to the vehicle screen based on the layer transformation matrix corresponding to that layer; at 140, intersection detection can be performed on the layer mapped to the global coordinate space and the full-screen distortion mesh parameters to extract the coordinates of the local distortion mesh vertices that intersect between the layer and the full-screen distortion mesh parameters; at 150, the local distortion mesh vertex coordinates can be transformed to the layer texture space to obtain the texture coordinates of the local distortion mesh vertex coordinates; and at 160, the layer can be rendered on the vehicle screen based on the texture coordinates.

[0040] According to embodiments of the present invention, full-screen distortion mesh parameters can be applied independently to each layer, distortion rendering can be performed layer by layer, and multiple layers can be drawn to the screen at once through instructions. This eliminates the need for write bandwidth to FBO, read bandwidth from FBO, and write bandwidth to FrameBuffer, greatly reducing memory bandwidth overhead.

[0041] The method for performing distortion rendering in a vehicle according to embodiments of the present invention eliminates the FBO intermediate state and removes the traditional post-processing flow of "first compositing into an FBO texture → then performing mesh distortion on the texture", saving read and write bandwidth, improving frame rate, and enhancing user experience.

[0042] Furthermore, according to embodiments of the present invention, a global-local coordinate remapping algorithm is proposed, which can accurately map the full-screen distortion mesh parameters defined in the full-screen space to an independent layer of any position and size, achieving "what you see is what you get" layer-by-layer distortion.

[0043] Furthermore, according to embodiments of the present invention, vertex coordinate data can be generated only for the regions where the layer intersects with the full-screen distortion mesh parameters, significantly reducing the GPU load.

[0044] Hereinafter, embodiments of the invention will be specifically described in the context of OpenGL and / or OpenGL ES (OpenGL for Embedded Systems). However, those skilled in the art will understand that the embodiments of the invention are not limited thereto, and the scheme for performing distortion rendering in a vehicle according to the embodiments of the invention can also be applied to other designs and environments.

[0045] According to embodiments of the present invention, full-screen distortion mesh parameters can indicate the distortion configuration of an in-vehicle screen. When a vehicle is equipped with multiple in-vehicle screens, multiple full-screen distortion mesh parameters can exist, each corresponding to one of the multiple in-vehicle screens requiring distortion rendering. The full-screen distortion mesh parameters can quantify the degree of distortion of the corresponding in-vehicle screen and can have distortion parameters representing the degree of distortion.

[0046] Here, the screen requiring distortion rendering can refer to a screen whose shape is not a standard rectangle. For example, the screen can be trapezoidal, parallelogram-shaped, has at least one curved edge, or other irregularly shaped screens. In addition to two-dimensional distortion, the screen requiring distortion rendering can also be distorted on a three-dimensional scale, for example, it can be curved inward or outward.

[0047] According to an embodiment of the present invention, rendering the layer on a vehicle screen based on texture coordinates may include: sampling the texture data of the layer of the original image based on texture coordinates; and rendering the layer on the vehicle screen based on the texture coordinates of the vertex coordinates of the local distortion mesh and the sampled texture data.

[0048] According to embodiments of the present invention, the layer texture space can refer to the independent texture coordinate system possessed by each layer. For example, the layer texture space can be a two-dimensional coordinate system, such as the UV coordinate space.

[0049] According to an embodiment of the present invention, for each layer, mapping the local coordinate space corresponding to the layer to the global coordinate space corresponding to the vehicle screen may include: determining the axis-aligned bounding box of the layer in the global coordinate space based on the layer transformation matrix of the layer, and extracting the coordinates of the intersecting local distortion mesh vertices between the layer and the full-screen distortion mesh parameters may include: determining whether the full-screen distortion mesh parameters intersect with the axis-aligned bounding box of the layer based on the sub-mesh of the full-screen distortion mesh parameters, and determining the vertex coordinates of the intersecting sub-mesh as the vertex coordinates of the local distortion mesh of the layer.

[0050] According to an embodiment of the present invention, intersection detection can be performed on a sub-grid basis of the full-screen distortion grid parameters. For example, the size of the sub-grid of the full-screen distortion grid parameters can be 9 (pixels) × 21 (pixels).

[0051] According to an embodiment of the present invention, the sub-mesh of the full-screen distortion mesh parameter can be subdivided, and determining whether the full-screen distortion mesh parameter intersects with the axis-aligned bounding box of the layer can include: determining whether the full-screen distortion mesh parameter intersects with the axis-aligned bounding box of the layer based on the subdivided sub-mesh.

[0052] For example, the submesh of the full-screen distortion mesh parameter can be further subdivided into 4 subdivision submesh (2×2 subdivision), and intersection detection can be performed on a subdivision submesh basis to determine whether the full-screen distortion mesh parameter intersects with the axis-aligned bounding box of the layer.

[0053] According to an embodiment of the present invention, rendering the layer on the vehicle screen based on texture coordinates may further include: performing anti-aliasing processing on the edges of the layer.

[0054] According to an embodiment of the present invention, before identifying each layer of the original image to be rendered on the vehicle screen, the vehicle screen to which the original image is to be rendered can be identified, it can be determined whether the vehicle screen has distortion, and based on the determination that the vehicle screen has distortion, all layers of the original image are marked as enabling client compositing.

[0055] The original image may include pictures and / or video images.

[0056] According to embodiments of the present invention, the in-vehicle screen may include the projection screen of a head-up display device, and the full-screen distortion grid parameters may be associated with the distortion configuration, optical reflection properties, and optical refraction properties of the vehicle's windshield. Therefore, the solution according to embodiments of the present invention can be applied to a panoramic head-up display (PHUD).

[0057] From here on, refer to Figures 2 to 4 A scheme for performing distortion rendering in a vehicle according to an embodiment of the present invention is described in detail.

[0058] Figure 2 A schematic architecture 200 is shown for implementing a method for performing distortion rendering in a vehicle according to an embodiment of the present invention.

[0059] like Figure 2 As shown, architecture 200 can involve a SurfaceFlinger service layer, a RenderEngine rendering engine layer, and a hardware abstraction layer. For the sake of brevity in the illustration, Figure 2 Only the parts relevant to embodiments of the present invention are shown.

[0060] According to embodiments of the present invention, in an OpenGL context, full-screen distortion mesh parameters can be stored in a distortion mesh management module, for example, Figure 2The DistortionMeshManager module within the vehicle can pre-store the full-screen distortion mesh parameters corresponding to all in-vehicle screens. The screen index, identifier, and full-screen distortion mesh parameters can be stored in the form of a lookup table (LUT). However, this embodiment of the invention is not limited to this, and those skilled in the art will understand that the full-screen distortion mesh parameters for each screen can be stored in any other suitable manner.

[0061] In addition, distortion strategy control modules, for example, can be implemented in the distortion strategy control module. Figure 2 The DistortionPolicy module in the system pre-configures distortion policies. Distortion policies can include a whitelist of distorted screens, indicating which automotive screens require distortion rendering and / or pre-distortion. In other words, the whitelist of distorted screens can list automotive screens that exhibit distortion.

[0062] Figure 3 This is a schematic flowchart illustrating a more detailed method 300 for performing distortion rendering in a vehicle according to an embodiment of the present invention. Hereinafter, it will be combined with... Figure 2 and Figure 3 The embodiments of the present invention will be described in detail below.

[0063] like Figure 3 As shown, in step 302 of method 300, when a Vsync (vertical synchronization) signal is received, it can be considered that a new frame has been received, that is, the original image that needs to be displayed on the vehicle screen has been received.

[0064] Here, the original image can be a picture and / or a video image.

[0065] In 304, the layer composition management module, for example... Figure 2 The SurfaceFlinger main logic module can update the composition state (updateCompositionState).

[0066] During the update composition state at 306 or 304, it can be determined whether distortion is needed. Specifically, the SurfaceFlinger main logic module can control this via the distortion policy control module, i.e. Figure 2 The DistortionPolicy module determines whether the target vehicle screen to which the original image is to be rendered is in the distortion screen whitelist.

[0067] If the target vehicle screen is not in the distortion screen whitelist (No in 306), the process proceeds to 308. In 308, the general rendering process is performed, i.e., layer compositing is performed using the hardware compositor (HWC) and / or the GPU, and the original image is rendered onto the vehicle screen.

[0068] If the target vehicle screen is in the distorted screen whitelist (306 is), the process proceeds to step 310.

[0069] In version 310, all layers of the original image can be marked as having ForceClientComposition enabled, forcing the rendering of all layers to be performed via the GPU, i.e., the GLESRenderEngine. For example, the layer composition management module (SurfaceFlinger main logic module) can mark all layers of the original image as having ForceClientComposition enabled.

[0070] In 312, the CompositionEngine module in the SurfaceFlinger service layer can issue DrawLayers to the GLESRenderEngine module in the RenderEngine rendering engine layer. These DrawLayers can carry distortion requests.

[0071] In section 314, the rendering engine module GLESRenderEngine is initialized, and during initialization, the full-screen distortion mesh parameter V_grid for the vehicle screen is loaded. For example, the layer compositing management module (SurfaceFlinger main logic module) can... Figure 2 The full-screen distortion mesh parameter V_grid is loaded in the DistortionMeshManager module.

[0072] In 316, distortion mapping can be performed on each layer. For example, it can be done by a distortion mapping module, such as... Figure 2 The DistortionMapper module performs distortion mapping to generate and / or extract and / or determine and / or identify the local distortion mesh vertex coordinates for each layer.

[0073] Figure 4 A schematic diagram of an image 400 comprising multiple layers is shown according to an embodiment of the present invention.

[0074] like Figure 4 As shown, image 400 may include multiple layers, such as layer 1, layer 2, layer 3, and layer 4. Although Figure 4 The image 400 shown includes 4 layers, but those skilled in the art will understand that the image 400 may include N layers, where N is a natural number.

[0075] Multiple layers can completely overlap, partially overlap, or be separated from each other, and can have different sizes and shapes, and be located at different positions in the image 400.

[0076] Each layer has its own local coordinate space, such as a local coordinate space with the center, bottom left corner, or top left corner of the layer as the origin.

[0077] According to an embodiment of the present invention, the full-screen distortion grid parameter V_grid is located in the global coordinate space corresponding to the vehicle screen, such as in the Normalized Device Coordinates (NDC) space.

[0078] Therefore, the coordinate space of the layer (local coordinate space) is different from the coordinate space of the full-screen distortion mesh parameters (the global coordinate space corresponding to the vehicle screen). In order to perform distortion mapping in 316, the DistortionMapper module can map the local coordinate space of the layer to the global coordinate space corresponding to the vehicle screen.

[0079] Specifically, for each layer, the DistortionMapper module can obtain the layer transformation matrix corresponding to that layer, i.e., the Model matrix, which will be referred to hereafter as M_layerN, i.e., the Model matrix of layer N. N can be a natural number.

[0080] Return to reference Figure 1 In section 130, the DistortionMapper module can map the local coordinate space of the layer to the global coordinate space corresponding to the in-vehicle screen based on the layer's M_layerN. In one example, the mapping in section 130 may include the DistortionMapper module determining the layer's axis-aligned bounding box (AABB) Rect_global in the global coordinate space based on the layer's layer transformation matrix M_layerN. The axis-aligned bounding box is defined as the smallest hexahedron that contains the object being processed (e.g., the layer) and whose edges are parallel to the coordinate axes.

[0081] According to an embodiment of the present invention, the edges of the axis-aligned bounding box Rect_global are kept parallel to the coordinate axes of the global coordinate space, so that intersection detection can be performed.

[0082] continue Figure 3 In 318, intersection detection can be performed on the layer mapped to the global coordinate space (NDC) and the full-screen distortion mesh parameter V_grid to extract the coordinates of the local distortion mesh vertices that intersect between the layer and the full-screen distortion mesh parameter V_grid.

[0083] According to an embodiment of the present invention, the DistortionMapper module can determine whether the full-screen distortion grid parameter V_grid intersects the axis-aligned bounding box Rect_global of the layer based on the subgrids of the full-screen distortion grid parameter V_grid. The subgrids of the full-screen distortion grid parameter V_grid can be triangles. For example, the intersection of the full-screen distortion grid parameter V_grid with the axis-aligned bounding box Rect_global of the layer can be determined on a triangular subgrid basis.

[0084] According to embodiments of the present invention, the subgrids of the full-screen distortion grid parameters can be further subdivided. For example, each subgrid can be subdivided into 2×2 subgrids, resulting in four subdivided subgrids. Then, it can be determined whether the full-screen distortion grid parameter V_grid intersects the axis-aligned bounding box Rect_global of the layer based on the subdivided subgrids. For example, it can be determined whether the full-screen distortion grid parameter V_grid intersects the axis-aligned bounding box Rect_global of the layer on a unit of subdivided subgrids.

[0085] By further subdividing the subgrid of the full-screen distortion grid parameter V_grid, processing accuracy is improved, and pixel-level alignment issues during multi-layer stitching are resolved, resulting in visually seamless black borders and gaps. If no intersecting subgrids / subdivided subgrids exist (No in 318), the process proceeds to 320. In 320, the layer can be discarded.

[0086] If there are no intersecting subgrids / subdivided subgrids, the coordinates of this layer can be considered to be outside the screen, so the layer can be discarded.

[0087] Then, at step 330, the process checks whether all layers have been traversed.

[0088] Returning to 318, if intersecting sub-mesh exists (as in 318), the process proceeds to 324. In 324, the vertex coordinates of the local distortion mesh can be generated. In one example, the DistortionMapper module can determine the vertex coordinates of the intersecting sub-mesh as the vertex coordinates of the local distortion mesh for that layer. For each layer, the DistortionMapper module can determine one or more vertex coordinates of the local distortion mesh.

[0089] According to an embodiment of the present invention, the vertex coordinates of the locally distorted mesh are located in the global coordinate space (e.g., NDC space). Therefore, after extracting the vertex coordinates of the locally distorted mesh of the corresponding layer, the vertex coordinates of the locally distorted mesh can be transformed to the layer texture space to obtain the texture coordinates of the vertex coordinates of the locally distorted mesh. Figure 1 (of 150).

[0090] In one example, the vertex coordinates of a locally distorted mesh can be inversely mapped to the layer texture space using Equations 1 and 2.

[0091] Equation 1

[0092] Equation 2

[0093] Wherein, UV_local.x represents the x-coordinate of the local distortion mesh vertex coordinates in the layer texture space, V_grid.x represents the x-coordinate of the local distortion mesh vertex coordinates (i.e., the vertex coordinates of the sub-mesh that intersects the axis-aligned bounding box of the corresponding layer in the full-screen distortion mesh parameters) in the full-screen coordinate space, Rect_global.left represents the x-coordinate of the top-left corner of the axis-aligned bounding box of the layer in the full-screen coordinate space, and Rect_global.width represents the width of the axis-aligned bounding box of the layer.

[0094] Furthermore, UV_local.y represents the ordinate of the local distortion mesh vertex coordinates in the layer texture space, V_grid.y represents the ordinate of the local distortion mesh vertex coordinates (i.e., the vertex coordinates of the intersecting sub-mesh in the full-screen distortion mesh parameters) in the full-screen coordinate space, Rect_global.top represents the ordinate of the top-left corner of the layer's axis-aligned bounding box in the full-screen coordinate space, and Rect_global.height represents the height of the layer's axis-aligned bounding box.

[0095] Based on Equations 1 and 2, the coordinates of multiple local distortion mesh vertices of the layer can be transformed to the layer texture space, and the corresponding multiple texture coordinates can be obtained.

[0096] According to an embodiment of the present invention, the origin of the coordinate system is located at the upper left corner, and the Y-axis increases downwards. The starting point of the texture coordinates of the layer (UV=0) corresponds to the upper boundary (Top) of the layer. However, those skilled in the art should understand that the embodiments of the present invention are not limited thereto. When a coordinate system with other positions as the origin is used, Equations 1 and 2 can be modified accordingly without departing from the teachings of the present invention.

[0097] Then, in 326, the rendering engine module GLESRenderEngine can sample the texture data of that layer of the original image based on texture coordinates.

[0098] In 328, the rendering engine module GLESRenderEngine can draw the layer by sending drawing instructions to the GPU, such as glDraw. For example, the rendering engine module GLESRenderEngine can render the layer on the in-vehicle screen based on the texture coordinates of the vertex coordinates of the locally distorted mesh and sampled texture data.

[0099] Then, at 330, the rendering engine module GLESRenderEngine can determine whether all layers have been traversed. If there are remaining layers (no in 330), the process proceeds to 332, moves to the next layer, and returns to 316.

[0100] If all layers have been traversed (as in 330), the process proceeds to 334. In 334, the rendered results of all layers of the original image for that frame are output to the user at once, for example, by calling the eglSwapBuffers function.

[0101] Furthermore, according to embodiments of the present invention, before rendering each layer, at 326, the anti-aliasing module, for example... Figure 2 The DistortionShaderGen module can also perform anti-aliasing on the edges of the layer.

[0102] In one example, you can set horizontal anti-aliasing parameters aaWidthX and aaWidthY for each layer, and both parameters can be in pixels. For example, aaWidthX = C and aaWidthY = D, where C and D are natural numbers.

[0103] C and D can be equal or unequal. The specific values ​​of C and D can be set for each layer, each screen, etc., and the values ​​of C and D can be stored in the distortion strategy control module.

[0104] According to an embodiment of the present invention, the anti-aliasing module can perform alpha gradient processing on the edges of the layer based on the horizontal anti-aliasing parameters aaWidthX and aaWidthY to eliminate black or hard edges when the layers are stitched together, making the distorted rendering layers look softer and more natural.

[0105] According to an embodiment of the present invention, the DistortionShaderGen module can store relevant parameters, data, and instructions in [the relevant storage space]. Figure 2 It is stored in the shader factory (ProgramCache) module for later use.

[0106] Figure 5The distortion rendering effect of a layer according to an embodiment of the present invention is shown.

[0107] like Figure 5 As shown, this layer can be Figure 4 The image shows the distortion rendering effect of layer 1. According to an embodiment of the present invention, layer 1 after distortion rendering is adapted to the shape of the screen.

[0108] As can be seen, in Figure 5 In the image, both the graphics and text have undergone distorted rendering.

[0109] Although Figure 5 Only one layer is shown, but this is merely illustrative. Those skilled in the art will understand that although the scheme for layer-by-layer distortion rendering according to embodiments of the present invention renders layer by layer, the user can visually see the rendering result of the entire image.

[0110] Figure 6 A schematic block diagram of an apparatus 600 for performing distortion rendering in a vehicle according to an embodiment of the present invention is shown.

[0111] like Figure 6 As shown, the device 600 may include a layer composition management module 610, a rendering engine module 620, a distortion mapping module 630, and a distortion strategy control module 640.

[0112] The layer composition management module 610 can be configured to identify each layer of the original image to be rendered on the vehicle screen.

[0113] The rendering engine module 620 can be configured to acquire the full-screen distortion mesh parameters of the vehicle screen.

[0114] The distortion mapping module 630 can be configured to, for each layer, obtain the layer transformation matrix corresponding to that layer, map the local coordinate space of that layer to the global coordinate space corresponding to the vehicle screen based on the layer transformation matrix corresponding to that layer, perform intersection detection on the layer mapped to the global coordinate space and the full-screen distortion mesh parameters to extract the coordinates of the local distortion mesh vertices that intersect between the layer and the full-screen distortion mesh parameters, and transform the local distortion mesh vertex coordinates to the layer texture space to obtain the texture coordinates of the local distortion mesh vertex coordinates.

[0115] In addition, the rendering engine module 620 can also be configured to control the rendering of the layer on the vehicle screen based on texture coordinates.

[0116] According to an embodiment of the present invention, the full-screen distortion mesh parameters can indicate the distortion configuration of the in-vehicle screen, and the rendering engine module 620 can be configured to receive the distortion mesh from the distortion mesh management module ( Figure 6 (not shown) - for example Figure 2 The DistortionMeshManager module loads the full-screen distortion mesh parameters for the vehicle screen.

[0117] According to an embodiment of the present invention, the rendering engine module 620 may also be configured to: sample the texture data of the layer of the original image based on texture coordinates; and control the rendering of the layer on the vehicle screen based on the texture coordinates of the vertex coordinates of the local distortion mesh and the sampled texture data.

[0118] According to an embodiment of the present invention, for each layer, the distortion mapping module 630 may map the local coordinate space of the layer to the global coordinate space corresponding to the vehicle screen by: determining the axis-aligned bounding box of the layer in the global coordinate space based on the layer transformation matrix corresponding to the layer.

[0119] According to an embodiment of the present invention, the distortion mapping module 630 may extract the coordinates of the intersecting local distortion mesh vertices between the layer and the full-screen distortion mesh parameters by: determining whether the full-screen distortion mesh parameters intersect with the axis-aligned bounding box of the layer based on the sub-mesh of the full-screen distortion mesh parameters, and determining the vertex coordinates of the intersecting sub-mesh as the vertex coordinates of the local distortion mesh of the layer.

[0120] According to an embodiment of the present invention, the device 600 may further include: an anti-aliasing module 650, such as... Figure 6 As shown by the dashed line, it can be configured to perform anti-aliasing on the edges of this layer.

[0121] According to an embodiment of the present invention, the layer compositing management module 610 can also be configured to: identify the vehicle screen to which the original image is to be rendered; and, based on the determination that the vehicle screen has distortion, mark all layers of the original image as enabling client compositing, wherein the original image includes pictures and / or video images.

[0122] According to an embodiment of the present invention, the distortion strategy control module 640 can be configured to determine whether there is distortion in the vehicle screen based on the distortion strategy, and wherein the distortion strategy control module 640 can also be configured to store parameters for performing anti-aliasing processing on the edges of each layer.

[0123] According to embodiments of the present invention, full-screen distortion mesh parameters can be applied independently to each layer, distortion rendering can be performed layer by layer, and multiple layers can be drawn to the screen at once through instructions. This eliminates the need for write bandwidth to FBO, read bandwidth from FBO, and write bandwidth to FrameBuffer, greatly reducing memory bandwidth overhead.

[0124] The method for performing distortion rendering in a vehicle according to embodiments of the present invention abandons the FBO intermediate state, removes the traditional post-processing process of "first compositing into an FBO texture → then performing mesh distortion on the texture", saves read and write bandwidth, improves frame rate, can achieve near "zero loss" distortion, and improves user experience.

[0125] Furthermore, according to embodiments of the present invention, pixel-level alignment issues during multi-layer stitching can be resolved through mesh subdivision and anti-aliasing, resulting in visually seamless black borders and gaps. It also supports dynamically distributing distortion strategies (e.g., for screens of different car models) via JSON files without requiring recompiling the system image.

[0126] Figure 7 A schematic block diagram of a vehicle 700 according to an embodiment of the present invention is shown.

[0127] like Figure 7 As shown, vehicle 700 may include multiple in-vehicle screens 710 and the device 720 as described above.

[0128] Those skilled in the art will recognize that the units and algorithm steps of the various embodiments described in conjunction with the embodiments disclosed in this invention can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the composition and steps of each embodiment have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this invention.

[0129] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0130] In the embodiments provided by this invention, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0131] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0132] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0133] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0134] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for performing distortion rendering in a vehicle, comprising: Identify each layer of the original image to be rendered on the in-vehicle screen; For each layer: Obtain the layer transformation matrix corresponding to this layer. Based on the layer transformation matrix corresponding to the layer, the local coordinate space of the layer is mapped to the global coordinate space corresponding to the vehicle screen. Intersection detection is performed on the layer mapped to the global coordinate space and the full-screen distortion mesh parameters to extract the coordinates of the local distortion mesh vertices where the layer intersects with the full-screen distortion mesh parameters. The vertex coordinates of the locally distorted mesh are transformed to the layer texture space to obtain the texture coordinates of the locally distorted mesh vertex coordinates. The layer is rendered on the vehicle screen based on the texture coordinates.

2. The method according to claim 1, wherein, Rendering the layer on the in-vehicle screen based on the texture coordinates includes: The texture data of this layer of the original image is sampled based on the texture coordinates; and The layer is rendered on the vehicle screen based on the texture coordinates of the vertex coordinates of the locally distorted mesh and the sampled texture data.

3. The method according to claim 1, wherein, For each layer, mapping the local coordinate space corresponding to that layer to the global coordinate space corresponding to the vehicle screen includes: The layer transformation matrix of this layer determines the axis-aligned bounding box of this layer in the global coordinate space, and The extraction of the local distortion mesh vertex coordinates that intersect the parameters of this layer and the full-screen distortion mesh includes: The subgrids based on the full-screen distortion mesh parameters are used to determine whether the full-screen distortion mesh parameters intersect with the axis-aligned bounding box of the layer. The vertex coordinates of the intersecting sub-mesh are determined as the vertex coordinates of the local distortion mesh of the layer.

4. The method according to claim 3, wherein, The sub-mesh of the full-screen distortion mesh parameters is subdivided, and Determining whether the full-screen distortion mesh parameters intersect with the axis-aligned bounding box of the layer includes: The full-screen distortion mesh parameters are determined based on the subdivided sub-mesh to determine whether they intersect with the axis-aligned bounding box of the layer.

5. The method according to claim 1, wherein, Rendering the layer on the in-vehicle screen based on the texture coordinates includes: Apply anti-aliasing to the edges of this layer.

6. The method according to claim 1, wherein, The full-screen distortion mesh parameter indicates the distortion configuration of the in-vehicle screen.

7. The method of claim 1, further comprising, before identifying each layer of the original image to be rendered on the in-vehicle screen: Identify the in-vehicle screen whose original image is to be rendered; Determine if there is any distortion in the vehicle screen; as well as Based on the confirmed distortion on the in-vehicle screen, all layers of the original image were marked as enabling client-side compositing. The original images include pictures and / or video images.

8. The method according to claim 6, wherein, The in-vehicle screen includes the projection screen of the head-up display, and the full-screen distortion grid parameters are associated with the distortion configuration, optical reflection properties, and optical refraction properties of the vehicle windshield.

9. An apparatus for performing distortion rendering in a vehicle, comprising: The layer composition management module is configured to identify each layer of the original image to be rendered on the in-vehicle screen; The rendering engine module is configured to acquire the full-screen distortion mesh parameters of the in-vehicle screen; as well as The distortion mapping module is configured to, for each layer, Obtain the layer transformation matrix corresponding to this layer. Based on the layer transformation matrix corresponding to this layer, the local coordinate space of this layer is mapped to the global coordinate space corresponding to the vehicle screen, and Intersection detection is performed on the layer mapped to the global coordinate space and the full-screen distortion mesh parameters to extract the coordinates of the local distortion mesh vertices where the layer intersects with the full-screen distortion mesh parameters, and The vertex coordinates of the locally distorted mesh are transformed to the layer texture space to obtain the texture coordinates of the vertex coordinates of the locally distorted mesh. The rendering engine module is also configured to control the rendering of the layer on the vehicle screen based on the texture coordinates.

10. The apparatus according to claim 9, in, The rendering engine module is configured to load the full-screen distortion mesh parameters of the vehicle screen from the distortion mesh management module.

11. The apparatus according to claim 9, wherein, The rendering engine module is also configured as follows: The texture data of this layer of the original image is sampled based on the texture coordinates; and The rendering of this layer on the vehicle screen is controlled based on the texture coordinates of the vertex coordinates of the locally distorted mesh and the sampled texture data.

12. The apparatus according to claim 9, wherein, For each layer, the distortion mapping module maps the local coordinate space corresponding to that layer to the global coordinate space corresponding to the vehicle screen, including: The layer transformation matrix of this layer determines the axis-aligned bounding box of this layer in the global coordinate space, and Specifically, the distortion mapping module extracts the coordinates of the local distortion mesh vertices that intersect the layer with the full-screen distortion mesh parameters, including: The subgrids based on the full-screen distortion mesh parameters are used to determine whether the full-screen distortion mesh parameters intersect with the axis-aligned bounding box of the layer. The vertex coordinates of the intersecting sub-mesh are determined as the vertex coordinates of the local distortion mesh of the layer.

13. The apparatus according to claim 12, wherein, The sub-mesh of the full-screen distortion mesh parameters is subdivided, and The distortion mapping module determines whether the full-screen distortion grid parameters intersect with the axis-aligned bounding box of the layer based on the subdivided subgrid.

14. The apparatus according to claim 9, further comprising: The anti-aliasing module is configured to perform anti-aliasing on the edges of the layer.

15. The apparatus according to claim 9, wherein, The full-screen distortion mesh parameter indicates the distortion configuration of the in-vehicle screen.

16. The apparatus according to claim 9, wherein, The layer compositing management module is also configured as follows: Identify the in-vehicle screen to which the original image will be rendered; and Based on the confirmed distortion on the in-vehicle screen, all layers of the original image were marked as enabling client-side compositing. The original images include pictures and / or video images.

17. The apparatus of claim 14, further comprising: The distortion strategy control module is configured to determine whether distortion exists in the vehicle screen based on a distortion strategy, and The distortion strategy control module is also configured to store parameters for performing anti-aliasing on the edges of each layer.

18. The apparatus according to claim 9, wherein, The in-vehicle screen includes the projection screen of the head-up display, and the full-screen distortion grid parameters are associated with the distortion configuration, optical reflection properties, and optical refraction properties of the vehicle windshield.

19. A vehicle comprising: Multiple in-vehicle screens; as well as The apparatus as described in any one of claims 9-18.

Images