Vehicle, readable storage medium, and image processing method and system

Abstract

An image processing method. The method comprises: acquiring an original image to be displayed; performing meshing on the original image, so as to generate a target mesh; taking edge points of the target mesh as control points for stretching, so as to form the outer contour of a target image, and acquiring pixel coordinates corresponding to the edge points after the stretching is performed; generating a standard mesh corresponding to the target mesh, and on the basis of a matrix mapping relationship between the standard mesh and the target mesh after the stretching is performed, obtaining pixel coordinates of internal fill points; and on the basis of pixel coordinates of mesh points of the target mesh after the stretching is performed, deforming the texture of the original image, so as to acquire the texture of the target image.

Description

Vehicle, readable storage medium, image processing method and system

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of Chinese patent application No. 202410259272.7 filed on March 7, 2024, the contents of which are incorporated herein by reference. Technical Field

[0003] The present application relates to image processing, and in particular, to an image processing method, an image processing system, a vehicle, and a readable storage medium. Background Art

[0004] A Head-Up Display (HUD), a driving assistance device used in automobiles, is a comprehensive electronic display device consisting of electronic components, display components, and controllers. It projects information such as vehicle speed, navigation information, and warnings in the form of images and characters in front of the driver via optical components. The driver can view this information directly on the windshield above the hood. Compared to viewing driving information from the instrument cluster, the driver can keep their eyes on the road, requiring less head and eye movement and less time to adjust to road information. Therefore, a HUD increases safety, keeps the driver's eyes on the road, and improves driver comfort.

[0005] Traditional HUDs primarily include C-HUD (Combiner HUD) and W-HUD (Windshield HUD). While both address the potential safety hazard of blind driving caused by drivers needing to look down to check vehicle information, they are essentially a simple migration of central control and instrument panel information. Furthermore, due to limited projection size, they are unable to support the trend of intelligent development in modern vehicles. To fully display information and provide an immersive experience, AR-HUD (Augmented Reality Head-Up Display) has emerged. Based on W-HUD, AR-HUD combines AR technology to overlay the HUD display onto the real road surface, allowing for a realistic and vivid display of driving information within the driver's field of view, resulting in an AR-like real-world fit.

[0006] However, the overlay of 2D images with 3D reality requires deformation of the graphics to better fit the 2D image with the 3D world. OpenGL (Open Graphics Library) provides a texture stretching method that introduces a programmable rendering pipeline. When rendering graphics, images can be deformed in OpenGL's Vertex Shader (vertex shader). Programmers only need to input the correct vertex points and the UV coordinates of the texture corresponding to each point to render the texture graphics with the correct shape in the FrameBuffer. However, how to determine the correct vertex points and UV coordinates is a technical problem that needs to be solved by those skilled in the art.

[0007] Summary of the Invention

[0008] The technical problem to be solved in the first aspect of the present application is to provide an image processing method, which can ensure the internal texture filling after the image is deformed and obtain a correct texture graphic.

[0009] The technical problem to be solved in the second aspect of the present application is to provide an image processing system, which can ensure the internal texture filling after the image is deformed and obtain a correct texture graphic.

[0010] The technical problem to be solved in the third aspect of this application is to provide a vehicle that can make the image projected by the HUD have correct internal texture, thereby improving the sense of fit of the AR real scene.

[0011] The technical problem to be solved in the fourth aspect of this application is to provide a readable storage medium that can be executed by a computer processor to ensure that the processed image has the correct internal texture and improve the sense of fit of AR real scene.

[0012] In order to solve the above technical problems, the present application provides an image processing method, which includes:

[0013] Obtaining the original image to be displayed;

[0014] Dividing the original image into grids to generate a target grid;

[0015] Stretching the edge points of the target grid as control points to form the outer contour of the target image, and obtaining pixel coordinates corresponding to the edge points after stretching;

[0016] generating a standard grid corresponding to the target grid, parallelizing opposite sides of the standard grid to form a first standard side and a second standard side, and distributing grid points on the first standard side and the second standard side at equal distances;

[0017] deriving coordinates of grid points on another pair of sides of the standard grid based on the coordinates of points on the first standard side and the second standard side and a matrix mapping of edge points corresponding to the stretched target grid;

[0018] Traversing the internal filling points of the stretched target grid, obtaining coordinates of a grid point in the standard grid corresponding to the internal filling point, and obtaining pixel coordinates of the internal filling point based on a matrix mapping of the first standard edge and the second standard edge corresponding to the grid point;

[0019] The texture of the original image is deformed based on the pixel coordinates of each grid point of the stretched target grid to obtain the texture of the target image.

[0020] As a specific implementation of the present application, dividing the original image into grids to generate a target grid includes:

[0021] The original image is divided into M*N equidistant grids to generate a target grid.

[0022] As a specific implementation of the present application, generating a standard grid corresponding to the target grid, parallelizing opposite sides of the standard grid to form a first standard side and a second standard side, and distributing grid points on the first standard side and the second standard side at equal distances includes:

[0023] generating a standard grid corresponding to the target grid;

[0024] The edges of the first column and the Nth column of the standard grid are used as the first standard edge and the second standard edge respectively, and the first standard edge and the second standard edge are processed in parallel;

[0025] The grid points on the first standard side and the second standard side are distributed at equal distances.

[0026] As a specific implementation of the present application, the coordinates of the grid points on another pair of sides of the standard grid are obtained based on the coordinates of the points on the first standard side and the second standard side and the matrix mapping of the edge points corresponding to the stretched target grid, including:

[0027] Calculate the point a of the target grid after stretching 11 、Point a 21 、Point a 1N and point a 2N To point b of the standard grid 11 、Point b 21 、Point b 1N and point b 2N The first perspective transformation matrix Matrix_1 of the mapping is such that: b11 =Matrix_1*a 11 b 21 =Matrix_1*a 21 b 1N =Matrix_1*a 1N b 2N =Matrix_1*a 2N ;

[0028] Based on the first perspective transformation matrix Matrix_1, the coordinates of the remaining grid points in the first row of the standard grid are calculated as follows: 1k =Matrix_1*a 1k

[0029] Among them, b 1k is the coordinate of the grid point in the first row and the kth column of the standard grid, a 1k is the coordinate of the grid point in the first row and the kth column of the target grid after stretching, where k is any integer from 2 to (N-1);

[0030] Calculate the point a of the target grid after stretching M1 、Point a (M-1)1 、Point a MN and point a (M-1)N To point b of the standard grid M1 、Point b (M-1)1 、Point b MN and point b (M-1)N The second perspective transformation matrix Matrix_2 is mapped so that: b M1 =Matrix_2*a M1 b (M-1)1 =Matrix_2*a (M-1)1 b MN =Matrix_2*a MN b (M-1)N =Matrix_2*a (M-1)N ;

[0031] Based on the second perspective transformation matrix Matrix_2, the coordinates of the remaining grid points in the Mth row of the standard grid are calculated as follows: Mk =Matrix_2*a Mk

[0032] Among them, b Mk is the coordinate of the grid point in the Mth row and the kth column of the standard grid, a Mk is the coordinate of the grid point in the Mth row and the kth column of the target grid after stretching, where k is any integer from 2 to (N-1).

[0033] As a specific implementation of the present application, traversing the internal filling points of the target grid after stretching, obtaining the coordinates of the grid point corresponding to the internal filling point in the standard grid, and deriving the pixel coordinates of the internal filling point based on the matrix mapping of the first standard side and the second standard side corresponding to the grid point include:

[0034] Traversing the internal filling points of the target mesh after stretching;

[0035] Calculate the internal filling point a ij The pixel coordinates include:

[0036] Get point b of the standard grid 1j and point b Mj The coordinates of point b 1j and point b Mj Divide the lines evenly to obtain M equal points and get point b ij The coordinates of are as follows: b ij =(b Mj -b 1j )*(j-1) / (M-1)+b 1j ;

[0037] Calculate the point b of the standard grid (i-1)1 、Point b i1 、Point b (i-1)N and point b iN Point a of the target mesh after stretching (i-1)1 、Point a i1 、Point a (i-1)N and point a iN The third perspective transformation matrix Matrix_3 of the projection makes: a (i-1)1 =Matrix_3*b (i-1)1 a i1 =Matrix_3*b i1 a (i-1)N =Matrix_3*b (i-1)N a iN =Matrix_3*b iN ;

[0038] Calculate the point b of the standard grid (i+1)1 、Point b i1 、Point b (i+1)N and point b iN Point a of the target mesh after stretching (i+1)1 、Point a i1 、Point a (i+1)N and point a iN The fourth perspective transformation matrix Matrix_4 of the projection makes: a (i+1)1=Matrix_4*b (i+1)1 a i1 =Matrix_4*b i1 a (i+1)N =Matrix_4*b (i+1)N a iN =Matrix_4*b iN ;

[0039] Based on point b ij The coordinates of the third perspective transformation matrix Matrix_3 and the fourth perspective transformation matrix Matrix_4 are used to obtain the internal filling point a ij The pixel coordinates of a are as follows: ij =(Matrix_3*b ij +Matrix_4*b ij ) / 2.

[0040] As a specific implementation of the present application, deforming the texture of the original image based on the pixel coordinates of each grid point of the stretched target grid to obtain the texture of the target image includes:

[0041] triangulate the unstretched target mesh to triangulate the texture of the original image;

[0042] Based on the pixel coordinates of each grid point of the stretched target grid, the target image is grid-filled and triangulated;

[0043] According to the triangulation of the target image, the texture of the original image after the triangulation is deformed to obtain the texture of the target image.

[0044] As a specific embodiment of the present application, the image processing method is applied to a HUD, and the method further includes:

[0045] Acquire environmental parameters of a road surface in front of the vehicle and / or eye point position parameters of the driver, and adjust the deformed shape of the target image based on the environmental parameters and / or eye point position parameters.

[0046] A second aspect of the present application provides an image processing system, comprising:

[0047] An acquisition module, used for acquiring the original image to be displayed;

[0048] The image processing module includes a grid division unit, an edge stretching unit, a coordinate calculation unit and a texture filling unit, wherein:

[0049] The grid division unit is used to divide the original image into grids to generate a target grid;

[0050] The edge stretching unit is configured to stretch the edge points of the target grid as control points to form an outer contour of the target image, and obtain pixel coordinates corresponding to the edge points after stretching;

[0051] The coordinate calculation unit is configured to generate a standard grid corresponding to the target grid, perform parallel processing on opposite sides of the standard grid to form a first standard side and a second standard side, and distribute grid points on the first standard side and the second standard side at equal distances; obtain coordinates of grid points on the other pair of sides of the standard grid based on the coordinates of points on the first standard side and the second standard side and a matrix mapping of edge points corresponding to the stretched target grid; traverse internal filling points of the stretched target grid to obtain coordinates of grid points within the standard grid corresponding to the internal filling points, and obtain pixel coordinates of the internal filling points based on the matrix mapping of the first standard side and the second standard side corresponding to the grid points;

[0052] The texture filling unit is configured to deform the texture of the original image based on the pixel coordinates of each grid point of the stretched target grid to obtain the texture of the target image.

[0053] A third aspect of the present application provides a vehicle, comprising the image processing system described in the above technical solution, and executing the image processing method as described in any one of the above technical solutions, wherein the image processing method and the image processing system are applied to HUD.

[0054] A fourth aspect of the present application provides a readable storage medium, which stores executable instructions. The executable instructions are used to be read by a processor of a computer to execute the image processing method described in any one of the above technical solutions.

[0055] Through the above solution, the beneficial effects of this application are as follows:

[0056] The image processing method of the present application only needs to control the position of the edge points of the target image, and can accurately obtain the pixel coordinates of the insertion points inside the target image based on matrix mapping, so that the pixel coordinates, which are the coordinates of the Vertex point, can be used to automatically fill the internal texture of the target image when deforming the image in the Vertex Shader of OpenGL, rendering a texture graphic of the correct shape, thereby enhancing the sense of real-scene fit when using AR.

[0057] Other features and advantages of the present application will be described in detail in the subsequent detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] The accompanying drawings are used to provide a further understanding of the present application and constitute a part of the specification. Together with the following detailed description, they are used to explain the present application but do not constitute a limitation of the present application. In the accompanying drawings:

[0059] FIG1 is a flow chart of the image processing method of the present application;

[0060] FIG2 is a schematic diagram of dividing the original image into M*N grids;

[0061] FIG3 is a schematic diagram of mesh division of the target mesh after stretching;

[0062] FIG4 is a schematic diagram of a first standard side and a second standard side of a standard grid;

[0063] FIG5 is a schematic diagram of the positions of control points after standard grid mapping;

[0064] FIG6 is a schematic diagram showing the positions of the internal filling points of the target grid after stretching;

[0065] FIG7 is a schematic diagram of the positions of internal filling points after standard grid mapping;

[0066] FIG8 is a schematic diagram of dividing the original image into 6*6 grids;

[0067] FIG9 is a schematic diagram of triangulating the image shown in FIG6 ;

[0068] FIG10 is a schematic diagram of mesh division after the target mesh shown in FIG6 is stretched;

[0069] FIG11 is a schematic diagram of the first standard side and the second standard side after the standard grid is mapped to FIG10;

[0070] FIG12 is a schematic diagram of the positions of control points after the standard grid is mapped to FIG10;

[0071] FIG13 is a schematic diagram showing the positions of internal filling points after the standard grid is mapped to FIG10;

[0072] Figure 14 is the test texture;

[0073] FIG15 is the deformed texture of FIG14 obtained by calculating the matrix mapping algorithm;

[0074] FIG16 is a structural block diagram of the image processing system of the present application. DETAILED DESCRIPTION

[0075] The specific implementation methods of the present application are described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation methods described here are only used to illustrate and explain the present application, and the scope of protection of the present application is not limited to the specific implementation methods described below.

[0076] In the description of this application, it should be noted that, unless otherwise expressly specified or limited, the terms "formed," "provided with," "set," "connected," etc. should be understood in a broad sense. For example, the connection may be a direct connection or an indirect connection through an intermediate medium; it may be a fixed connection, a detachable connection, or an integral connection; it may be a direct connection or an indirect connection through an intermediate connector; it may be internal communication between two elements or an interaction relationship between two elements. For those skilled in the art, the specific meanings of the above terms in this application can be understood according to the specific circumstances.

[0077] It should be noted that the characters represented by the points in the following calculation formulas should be understood as the coordinates of the points. For example, the calculation formula "b 11 =Matrix_1*a 11 ", "b 11 " is point b 11 The coordinates of "a 11 " is a 11 's coordinates.

[0078] The present application provides an image processing method, as shown in FIG1 , which includes:

[0079] Step S110: Acquire the original image to be displayed.

[0080] Taking the application of HUD as an example, when the HUD display screen is displayed, driving information is selected according to the driver's needs, or the image to be displayed is automatically identified based on the road conditions, and the corresponding original image is called for display. In order to be able to superimpose the HUD display screen on the real road surface, the original image needs to be stretched and deformed so that it can reasonably and vividly superimpose the driving information in the driver's field of view, and have an AR real-scene fit.

[0081] Step S120: Divide the original image into grids to generate a target grid.

[0082] Referring to FIG2 , as a specific implementation, the original image is divided into M*N equidistant grids, that is, the grid lines are arranged in M ​​rows and N columns, and the spacing between adjacent horizontal grid lines is equal, and the spacing between adjacent vertical grid lines is equal, thereby generating M*N grid points.

[0083] Step S130: Use the edge points of the target grid as control points to stretch the outer contour of the target image and obtain the pixel coordinates corresponding to the edge points after stretching. Specifically, see Figures 2 and 3. The edge points include the four vertices A of the target grid. 11 (the point in row 1 and column 1), A 1N (the point at row 1 and column N), A M1(the point at row M and column 1) and A MN (the point at row M and column N) and the control points on the four edges, and obtain their corresponding UV coordinates. According to the shape of the desired target image, stretch the control points to the corresponding positions shown in Figure 3, such as point A. 11 Stretch to the edge point a of the target image 11 , point A 1N Stretch to the edge point a of the target image 1N , and obtain the pixel coordinates of the edge points of the target image corresponding to the above control points after stretching.

[0084] It should be noted that the shape of the target image can be pre-set, i.e., the pixel coordinates of each edge point of the target image are preset and correspond to the UV coordinates of the edge points of the target grid. Furthermore, during the AR-HUD display process, due to changes in environmental parameters of the road ahead of the vehicle, such as changes in road slope or road dividing lines, the shape of the target image needs to change to better fit the actual scene. Therefore, the shape of the target image can be adjusted based on these environmental parameters to adjust the deformed shape of the target image, thereby adjusting the outer contour of the target image and adjusting its internal texture deformation. Specifically, based on the detected environmental parameters of the road ahead of the vehicle, the pixel coordinates of the edge points of the target image corresponding to the stretched control points are adjusted to re-determine the outer contour of the target image. The environmental parameters of the road ahead of the vehicle can be acquired by sensors on the vehicle body, such as one or more of a camera, lidar, and millimeter-wave radar. The vehicle captures multi-angle video of the environment ahead of the vehicle and uploads the images to a processor for analysis and processing. This processor can be an onboard processor or a cloud-based processor transmitted via network communication.

[0085] It should also be noted that the image pictures observed from different perspectives are different. During the vehicle driving process, the driver's body parameters and driving posture are different, and the eye position parameters will also be different. In order to further enable the driver to view the display image of the AR-HUD to better fit the actual scene and increase the sense of fit with the real scene, the driver's eye position parameters can also be obtained. Based on the eye position parameters, the deformation shape of the target image can be adjusted. The pixel coordinates of the edge points of the target image corresponding to the control point after stretching can be adjusted in combination with the detected road environment parameters in front of the vehicle, so as to redefine the outer contour of the target image. Among them, the driver's eye position parameters can be obtained by capturing the driver's eye images through the driver monitoring camera in the car and uploading them to the processor for analysis and processing to obtain the eye position parameters. Alternatively, the driver's eye position parameters can be calculated by adjusting the seat posture parameters (seat front and rear adjustment parameters, left and right adjustment parameters, up and down adjustment parameters, backrest adjustment parameters, etc.) adjusted by the driver.

[0086] Step S140: After obtaining the pixel coordinates corresponding to the stretched edge points, generate a standard grid corresponding to the stretched target grid, and parallelize the opposite sides of the standard grid to form a first standard side and a second standard side, and distribute the grid points on the first standard side and the second standard side at equal distances. Specifically, referring to Figures 3 and 4, the side of the first column of the standard grid is used as the first standard side, and the side of the Nth column is used as the second standard side, and the first standard side and the second standard side are parallelized; the grid points on the first and second standard sides are distributed at equal distances, thereby obtaining the coordinates of each grid point on the first and second standard sides, such as point b in the first row and first column. 11 The UV coordinates are (0,0), point b in the Mth row and 1st column M1 The UV coordinates are (0,1), point b at row 1 and column N 1N The UV coordinates are (1,0), point b at row M and column N MN The UV coordinates are (1,1).

[0087] Step S150: based on the coordinates of the points on the first standard side and the second standard side and the matrix mapping of the edge points corresponding to the stretched target grid, the coordinates of the grid points on the other pair of sides of the standard grid are obtained.

[0088] Specifically, referring to FIG3-FIG5, the first standard edge and the second standard edge are the edge of the first column and the edge of the Nth column of the standard grid respectively. Therefore, the point a of the target grid after stretching is calculated. 11 、Point a 21 、Point a 1N and point a 2N To point b on the standard grid 11 、Point b 21 、Point b 1N and point b 2N The first perspective transformation matrix Matrix_1 of the mapping is such that: b 11 =Matrix_1*a 11 b 21 =Matrix_1*a 21 b 1N =Matrix_1*a 1N b 2N =Matrix_1*a 2N ;

[0089] Specifically, point a 11 、Point a 21 、Point a 1N and point a 2N and point b 11 、Point b 21 、Point b1N and point b 2N After substituting the coordinates into the above formula, Matrix_1 can be calculated.

[0090] Based on the first perspective transformation matrix Matrix_1, calculate the coordinates of the remaining grid points in the first row of the standard grid, as follows: 1k =Matrix_1*a 1k

[0091] Among them, b 1k is the coordinate of the grid point in the first row and the kth column of the standard grid, a 1k is the coordinate of the grid point in the first row and the kth column of the target grid after stretching, where k is any integer from 2 to (N-1);

[0092] Calculate the point a of the target grid after stretching M1 、Point a (M-1)1 、Point a MN and point a (M-1)N To point b of the standard grid M1 、Point b (M-1)1 、Point b MN and point b (M-1)N The second perspective transformation matrix Matrix_2 is mapped so that: b M1 =Matrix_2*a M1 b (M-1)1 =Matrix_2*a (M-1)1 b MN =Matrix_2*a MN b (M-1)N =Matrix_2*a (M-1)N ;

[0093] Specifically, point a M1 、Point a (M-1)1 、Point a MN and point a (M-1)N and point b M1 、Point b (M-1)1 、Point b MN and point b (M-1)N After substituting the coordinates into the above formula, Matrix_2 can be calculated.

[0094] Based on the second perspective transformation matrix Matrix_2, the coordinates of the remaining grid points in the Mth row of the standard grid are calculated as follows: Mk =Matrix_2*a Mk

[0095] Among them, b Mk is the coordinate of the grid point in the Mth row and the kth column of the standard grid, aMk is the coordinate of the grid point in the Mth row and the kth column of the target grid after stretching, where k is any integer from 2 to (N-1).

[0096] Step S160: After the coordinates of the grid points in the standard grid corresponding to the edge points in the stretched target grid are confirmed, the internal filling points of the stretched target grid are traversed to obtain the coordinates of the grid points in the standard grid corresponding to the internal filling points, and the pixel coordinates of the internal filling points are obtained based on the matrix mapping of the first standard edge and the second standard edge corresponding to the grid points.

[0097] Specifically, refer to Figures 6 and 7 to calculate the internal filling point a in the i-th row and j-th column (i is any integer from 2 to (M-1), and j is any integer from 2 to (N-1)) of the stretched target grid. ij As an example, get the pixel coordinates of point b of the standard grid 1j and point b Mj UV coordinates of point b 1j and point b Mj Divide the lines evenly to obtain M equal points and get point b ij The coordinates of are as follows: b ij =(b Mj -b 1j )*(j-1) / (M-1)+b 1j ;

[0098] Calculate the point b of the standard grid (i-1)1 、Point b i1 、Point b (i-1)N and point b iN Point a of the target mesh after stretching (i-1)1 、Point a i1 、Point a (i-1)N and point a iN The third perspective transformation matrix Matrix_3 of the projection makes: a (i-1)1 =Matrix_3*b (i-1)1 a i1 =Matrix_3*b i1 a (i-1)N =Matrix_3*b (i-1)N a iN =Matrix_3*b iN ;

[0099] Specifically, point b (i-1)1 、Point b i1 、Point b (i-1)N and point b iN and point a (i-1)1 、Point a i1 、Point a (i-1)N and point aiN After substituting the coordinates into the above formula, Matrix_3 can be calculated.

[0100] Calculate the point b of the standard grid (i+1)1 、Point b i1 、Point b (i+1)N and point b iN Point a of the target mesh after stretching (i+1)1 、Point a i1 、Point a (i+1)N and point a iN The fourth perspective transformation matrix Matrix_4 of the projection makes: a (i+1)1 =Matrix_4*b (i+1)1 a i1 =Matrix_4*b i1 a (i+1)N =Matrix_4*b (i+1)N a iN =Matrix_4*b iN ;

[0101] Specifically, point b (i+1)1 、Point b i1 、Point b (i+1)N and point b iN and a (i+1)1 、Point a i1 、Point a (i+1)N and point a iN After the coordinates are substituted into the above formula, Matrix_4 can be calculated.

[0102] Based on point b ij The coordinates of the third perspective transformation matrix Matrix_3 and the fourth perspective transformation matrix Matrix_4 are used to obtain the internal filling point a ij The pixel coordinates of a are as follows: ij =(Matrix_3*b ij +Matrix_4*b ij ) / 2.

[0103] To facilitate understanding of the above technical solution of the present application, taking FIG8 as an example, the original image is divided into a grid with 6 rows and 6 columns, thereby obtaining a total of 20 edge points numbered 1 to 20 as control points, which are stretched to the edge points of the outer contour of the target image. The edge points numbered 1 to 20 in FIG8 correspond one-to-one to the edge points numbered A to T in FIG10, thereby obtaining the target grid shown in FIG10, and correspondingly generating the standard grids shown in FIG11 and FIG12, wherein the points a, c, e, g, i and k of the first standard side of the standard grid correspond one-to-one to the points A, G, I, K, M and O shown in the figure, respectively, and the points b, d, f, h, j, l of the second standard side correspond one-to-one to the points F, H, J, L, N and T shown in the figure, wherein the coordinates next to each point shown in FIG11 are its UV coordinates in the standard grid, thereby calculating the first perspective transformation matrix Matrix_1 mapping A, F, G, H to the four points a, b, c, d, so that a=Matrix_1*A b=Matrix_1*F c=Matrix_1*G d=Matrix_1*H;

[0104] Calculate the second perspective transformation matrix Matrix_2 that maps M, N, O, and T to points i, j, k, and l, so that M = Matrix_2*i, N = Matrix_2*j, O = Matrix_2*k, and T = Matrix_2*l.

[0105] The coordinates of m, n, q, r, t, u, v, and w in FIG12 are calculated as follows: m=Matrix_1*B n=Matrix_1*C q=Matrix_1*D r=Matrix_1*E t=Matrix_2*P u=Matrix_2*Q v=Matrix_2*R w=Matrix_2*S;

[0106] As shown in Figure 13, connect points m, n, q, and r with points t, u, v, and w respectively, and divide them equally to obtain the equal-division points of the triangle. Among them, the equal-division point o corresponds to point U shown in Figure 10. Point o is obtained by connecting points q and v. Therefore, the coordinates of point o are calculated by the following formula:

[0107] Calculate the third perspective transformation matrix Matrix_3 that maps points a, b, c, d to points A, F, G, H, such that A = Matrix_3*a F = Matrix_3*b G = Matrix_3*c H = Matrix_3*d;

[0108] Calculate the fourth perspective transformation matrix Matrix_4 that maps points c, d, e, and f to G, H, I, and J, so that G = Matrix_4*c H = Matrix_4*d I = Matrix_4*e J = Matrix_4*f;

[0109] Then, the pixel coordinates of U are calculated by the following formula: U = (Matrix_3*o+Matrix_4*o) / 2

[0110] Point U is the grid point at the 2nd row and 4th column of the stretched target grid, and its UV coordinates corresponding to the grid point of the original image shown in FIG8 are (0.6, 0.2).

[0111] Step S170: After obtaining the pixel coordinates of all internal filling points of the stretched target grid, the texture of the original image is deformed based on the pixel coordinates of each grid point of the stretched target grid to obtain the texture of the target image. After meshing the original image, it can be triangulated as shown in FIG9 to triangulate the texture of the original image. Based on the obtained pixel coordinates of each grid point of the stretched target grid, the target image is meshed and triangulated as shown in FIG10 so that the triangular patches of the target image correspond one-to-one with the triangular patches of the original image. Based on the triangulation of the target image, the texture of the triangulated original image is deformed to obtain the texture of the target image, thereby forming a complete target image. The target image has a correct internal texture deformation effect, so that when the target image is projected onto the front windshield of a vehicle, it has a good sense of real-life fit. In addition, when achieving texture deformation within the image, it only needs to control the position of the texture edge points, and the internal changes in the texture can be automatically filled according to the matrix mapping algorithm.

[0112] To better illustrate the technical solution of the present application, the image deformation effect formed after the present application is applied on a computer is described below. The image is deformed in the OpenGL Vertex Shader. FIG14 is a test texture, and a grid is divided into 6 rows and 6 columns as shown in FIG8. A total of 20 edge points numbered 1 to 20 are used as control points. The UV coordinates of the 20 edge points and the pixel coordinates (Vertex point coordinates) of the corresponding edge points numbered A to T after stretching are obtained. The test data is as follows:

[0113] According to the above matrix mapping algorithm, the coordinates of each Vertex point are correctly calculated, so that the deformation effect shown in the figure can be rendered in the FrameBuffer. It can be clearly seen that the internal texture of the image after deformation is correctly filled, and the correct texture graphics can be obtained, ensuring that the 2D image projected by the HUD can be superimposed on the 3D reality and have a good sense of fit with the AR real scene. In addition, the image processing method of this application only needs to control the position of the texture edge point, that is, to determine the UV coordinates and pixel coordinates (Vertex point coordinates) corresponding to the edge point after the image is stretched, and it can automatically complete the filling of the internal changes of the texture. The calculation is simple and efficient, and the internal texture deformation of the image is correct.

[0114] Correspondingly, the second aspect of the present application provides an image processing system, as shown in FIG16 , which includes:

[0115] An acquisition module, used for acquiring the original image to be displayed;

[0116] The image processing module includes a grid division unit, an edge stretching unit, a coordinate calculation unit and a texture filling unit, wherein:

[0117] A grid division unit, used for dividing the original image into grids to generate a target grid;

[0118] An edge stretching unit is used to stretch the edge points of the target grid as control points to form the outer contour of the target image and obtain the pixel coordinates corresponding to the edge points after stretching;

[0119] a coordinate calculation unit configured to generate a standard grid corresponding to the target grid, parallelize opposite sides of the standard grid to form a first standard side and a second standard side, and distribute grid points on the first standard side and the second standard side at equal distances; obtain coordinates of grid points on the other pair of sides of the standard grid based on the coordinates of the points on the first standard side and the second standard side and a matrix mapping of edge points corresponding to the stretched target grid; traverse internal filling points of the stretched target grid to obtain coordinates of grid points within the standard grid corresponding to the internal filling points, and obtain pixel coordinates of the internal filling points based on the matrix mapping of the first standard side and the second standard side corresponding to the grid points;

[0120] The texture filling unit is used to deform the texture of the original image based on the pixel coordinates of each grid point of the stretched target grid to obtain the texture of the target image.

[0121] Therefore, once the deformed outer contour of the image to be projected is determined, that is, the position of the texture edge points is confirmed, the image processing module can automatically fill in the internal changes of the texture, obtain the correct texture image, and ensure that the 2D image projected by the HUD can be superimposed on the 3D reality, with a good sense of fit between the AR and the real scene.

[0122] The third aspect of the present application provides a vehicle, comprising the image processing system provided in the second aspect of the present application, and executing the image processing method provided in the first aspect of the present application. The image processing system and the image processing method are applied to the vehicle HUD.

[0123] In addition, the fourth aspect of the present application provides a readable storage medium, wherein the readable storage medium stores executable instructions, and the executable instructions are used to be obtained by the processor of the computer.

[0124] The preferred embodiments of the present application are described in detail above in conjunction with the accompanying drawings. However, the present application is not limited to the specific details in the above embodiments. Within the technical concept of the present application, various simple modifications can be made to the technical solution of the present application, and these simple modifications all fall within the scope of protection of the present application.

[0125] It should also be noted that the various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction. To avoid unnecessary repetition, this application will not further describe various possible combinations.

[0126] In addition, the various implementation methods of the present application may be arbitrarily combined, and as long as they do not violate the concept of the present application, they should also be regarded as the contents disclosed in the present application.

Claims

1. An image processing method, characterized in that: The method includes: Obtain the original image to be displayed; Dividing the original image into grids to generate a target grid; Stretching the edge points of the target grid as control points to form the outer contour of the target image, and obtaining pixel coordinates corresponding to the edge points after stretching; generating a standard grid corresponding to the target grid, parallelizing opposite sides of the standard grid to form a first standard side and a second standard side, and distributing grid points on the first standard side and the second standard side at equal distances; Determining the coordinates of the grid points on another pair of sides of the standard grid based on the coordinates of the points on the first standard side and the second standard side and the matrix mapping of the edge points corresponding to the stretched target grid; Traversing the internal filling points of the stretched target grid, obtaining coordinates of a grid point in the standard grid corresponding to the internal filling point, and obtaining pixel coordinates of the internal filling point based on a matrix mapping of the first standard edge and the second standard edge corresponding to the grid point; The texture of the original image is deformed based on the pixel coordinates of each grid point of the stretched target grid to obtain the texture of the target image.

2. The image processing method according to claim 1, wherein: The gridding of the original image to generate the target grid includes: The original image is divided into M*N equidistant grids to generate a target grid.

3. The image processing method according to claim 2, wherein: The generating of a standard grid corresponding to the target grid, parallelizing opposite sides of the standard grid to form a first standard side and a second standard side, and distributing grid points on the first standard side and the second standard side at equal distances includes: generating a standard grid corresponding to the target grid; The edges of the first column and the Nth column of the standard grid are used as the first standard edge and the second standard edge respectively, and the first standard edge and the second standard edge are processed in parallel; The grid points on the first standard side and the second standard side are distributed at equal distances.

4. The image processing method according to claim 3, wherein: The step of deriving the coordinates of the grid points on another pair of sides of the standard grid based on the coordinates of the points on the first standard side and the second standard side and the matrix mapping of the edge points corresponding to the stretched target grid comprises: Calculate the point a of the target grid after stretching 11 、Point a 21 、Point a 1N and point a 2N To point b of the standard grid 11 、Point b 21 、Point b 1N and point b 2N The first perspective transformation matrix Matrix_1 of the mapping is such that: b 11 =Matrix_1*a 11 b 21 =Matrix_1*a 21 b 1N =Matrix_1*a 1N b 2N =Matrix_1*a 2N ; Based on the first perspective transformation matrix Matrix_1, the coordinates of the remaining grid points in the first row of the standard grid are calculated as follows: b 1k =Matrix_1*a 1k Among them, b 1k is the coordinate of the grid point in the first row and the kth column of the standard grid, a 1k is the coordinate of the grid point in the first row and the kth column of the target grid after stretching, where k is any integer from 2 to (N-1); Calculate the point a of the target grid after stretching M1 、Point a (M-1)1 、Point a MN and point a (M-1)N To point b of the standard grid M1 、Point b (M-1)1 、Point b MN and point b (M-1)N The second perspective transformation matrix Matrix_2 is mapped so that: b M1 =Matrix_2*a M1 b (M-1)1 =Matrix_2*a (M-1)1 b MN =Matrix_2*a MN b (M-1)N =Matrix_2*a (M-1)N ; Based on the second perspective transformation matrix Matrix_2, the coordinates of the remaining grid points in the Mth row of the standard grid are calculated as follows: b Mk =Matrix_2*a Mk Among them, b Mk is the coordinate of the grid point in the Mth row and the kth column of the standard grid, a Mk is the coordinate of the grid point in the Mth row and the kth column of the target grid after stretching, where k is any integer from 2 to (N-1).

5. The image processing method according to claim 4, characterized in that The traversing the internal filling points of the target grid after stretching, obtaining the coordinates of the grid point corresponding to the internal filling point in the standard grid, and deriving the pixel coordinates of the internal filling point based on the matrix mapping of the first standard side and the second standard side corresponding to the grid point includes: Traversing the internal filling points of the target mesh after stretching; Calculate the internal filling point a ij The pixel coordinates include: Get point b of the standard grid 1j and point b Mj The coordinates of point b 1j and point b Mj Divide the lines evenly to obtain M equal points and get point b ij The coordinates of are as follows: b ij =(b Mj -b 1j )*(j-1) / (M-1)+b 1j ; Calculate the point b of the standard grid (i-1)1 、Point b i1 、Point b (i-1)N and point b iN Point a of the target mesh after stretching (i-1)1 、Point a i1 、Point a (i-1)N and point a iN The third perspective transformation matrix Matrix_3 of the mapping is such that: a (i-1)1 =Matrix_3*b (i-1)1 a i1 =Matrix_3*b i1 a (i-1)N =Matrix_3*b (i-1)N a iN =Matrix_3*b iN ; Calculate the point b of the standard grid (i+1)1 、Point b i1 、Point b (i+1)N and point b iN Point a of the target mesh after stretching (i+1)1 、Point a i1 、Point a (i+1)N and point a iN The fourth perspective transformation matrix Matrix_4 of the projection is such that: a (i+1)1 =Matrix_4*b (i+1)1 a i1 =Matrix_4*b i1 a (i+1)N =Matrix_4*b (i+1)N a iN =Matrix_4*b iN ; Based on point b ij The coordinates of the third perspective transformation matrix Matrix_3 and the fourth perspective transformation matrix Matrix_4 are used to obtain the internal filling point a ij The pixel coordinates are as follows: a ij =(Matrix_3*b ij +Matrix_4*b ij ) / 2。 6. The image processing method according to any one of claims 1 to 5, characterized in that: The deforming the texture of the original image based on the pixel coordinates of each grid point of the stretched target grid to obtain the texture of the target image includes: triangulate the unstretched target mesh to triangulate the texture of the original image; Based on the pixel coordinates of each grid point of the stretched target grid, the target image is grid-filled and triangulated; According to the triangulation of the target image, the texture of the original image after the triangulation is deformed to obtain the texture of the target image.

7. The image processing method according to any one of claims 1 to 5, characterized in that: The image processing method is applied to a HUD, and the method further includes: Acquire environmental parameters of a road surface in front of the vehicle and / or eye point position parameters of the driver, and adjust the deformed shape of the target image based on the environmental parameters and / or eye point position parameters.

8. An image processing system, characterized in that: include: An acquisition module, used for acquiring the original image to be displayed; The image processing module includes a grid division unit, an edge stretching unit, a coordinate calculation unit and a texture filling unit, wherein: The grid division unit is used to divide the original image into grids to generate a target grid; The edge stretching unit is configured to stretch the edge points of the target grid as control points to form an outer contour of the target image, and obtain pixel coordinates corresponding to the edge points after stretching; The coordinate calculation unit is configured to generate a standard grid corresponding to the target grid, perform parallel processing on opposite sides of the standard grid to form a first standard side and a second standard side, and distribute grid points on the first standard side and the second standard side at equal distances; obtain coordinates of grid points on another pair of sides of the standard grid based on the coordinates of points on the first standard side and the second standard side and a matrix mapping of edge points corresponding to the stretched target grid; traverse internal filling points of the stretched target grid to obtain coordinates of grid points within the standard grid corresponding to the internal filling points, and obtain pixel coordinates of the internal filling points based on the matrix mapping of the first standard side and the second standard side corresponding to the grid points; The texture filling unit is configured to deform the texture of the original image based on the pixel coordinates of each grid point of the stretched target grid to obtain the texture of the target image.

9. A vehicle, characterized in that: The image processing system comprises the image processing system according to claim 8 and executes the image processing method according to any one of claims 1 to 7, wherein the image processing method and the image processing system are applied to a HUD.

10. A readable storage medium, characterized in that: The readable storage medium stores executable instructions, which are used to be read by a processor of a computer to execute the image processing method according to any one of claims 1 to 7.

Images