Image processing method and device, electronic equipment and storage medium

Abstract

The invention discloses an image processing method and device, electronic equipment and a readable storage medium, and the method comprises the steps: determining a map matrix according to the position information of a plurality of target road points on a road surface in a target shot image shot by a camera of a vehicle; determining a camera matrix according to the position information of the camera; determining an obstacle matrix according to the position information and the size of an obstacle in the target shot image; determining a color editing matrix according to color editing information for the target shot image; performing diffusion processing based on the map matrix, the camera matrix, the obstacle matrix and the color editing matrix to obtain target features; decoding the target feature to obtain a target editing image; the target edited image presents an effect after editing the target shot image according to the editing instruction. According to the method provided by the invention, the acquisition difficulty of the target edited image is relatively low, and the acquisition efficiency of the target edited image is relatively high.

Description

Technical Field

[0001] The present application relates to the field of digital information technology, and more specifically, to an image processing method, device, electronic device and computer-readable storage medium. Background Art

[0002] With the continuous development of automobile autonomous driving technology, various types of vehicles equipped with autonomous driving functions have flooded into the market. At present, the iteration process of autonomous driving functions relies on road data collected during vehicle driving, but the acquisition of road data is limited by time, weather, etc., which makes it difficult to obtain road data and the efficiency of obtaining road data is low, making the iteration of autonomous driving functions difficult and inefficient. Summary of the invention

[0003] The present application proposes an image processing method, device, electronic device and computer-readable storage medium to improve the above-mentioned defects.

[0004] In a first aspect, an embodiment of the present application provides an image processing method, the method comprising:

[0005] Determine a map matrix according to the position information of a plurality of target road points on the road surface in the target shooting image shot by the camera of the vehicle; the map matrix is ​​used to indicate the positions of the plurality of target road points;

[0006] Determine the camera matrix according to the position information of the camera; the camera matrix is ​​used to indicate the position of the camera;

[0007] Determine an obstacle matrix according to the position information and size of the obstacle in the target captured image; the obstacle matrix is ​​used to indicate the position and size of the obstacle;

[0008] Determine a color editing matrix according to color editing information for a target captured image; the color editing information is obtained by parsing an editing instruction for the target captured image; the color editing matrix is ​​used to indicate a color value indicated by the editing instruction;

[0009] Diffusion processing is performed based on the map matrix, camera matrix, obstacle matrix and color editing matrix to obtain target features;

[0010] The target features are decoded to obtain a target edited image; the target edited image presents the effect of editing the target captured image according to the editing instructions.

[0011] In a second aspect, an embodiment of the present application further provides an image processing device, the device comprising:

[0012] A first determination module is used to determine a map matrix according to position information of a plurality of target road points on a road surface in a target shooting image shot by a camera of the vehicle; the map matrix is ​​used to indicate the positions of the plurality of target road points;

[0013] A second determination module is used to determine a camera matrix according to the position information of the camera; the camera matrix is ​​used to indicate the position of the camera;

[0014] A third determination module is used to determine an obstacle matrix according to the position information and size of the obstacle in the target captured image; the obstacle matrix is ​​used to indicate the position and size of the obstacle;

[0015] A fourth determination module is used to determine a color editing matrix according to color editing information for the target captured image; the color editing information is obtained by parsing the editing instruction for the target captured image; the color editing matrix is ​​used to indicate the color value indicated by the editing instruction;

[0016] Diffusion module, used to perform diffusion processing based on map matrix, camera matrix, obstacle matrix and color editing matrix to obtain target features;

[0017] The decoding module is used to decode the target features to obtain a target edited image; the target edited image presents the effect of editing the target captured image according to the editing instructions.

[0018] In a third aspect, an embodiment of the present application further provides an electronic device, characterized in that the electronic device comprises: one or more processors; a memory; one or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to execute the above method.

[0019] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, which stores a program code executable by a processor, and when the program code is executed by the processor, the processor executes the above method.

[0020] The present application provides an image processing method, device, electronic device and computer-readable storage medium. In the present application, based on the target captured image captured by the vehicle's camera, the map matrix, camera matrix and obstacle matrix are determined, and color editing information is obtained after parsing the editing instructions for the target captured image to determine the color editing matrix. Then, diffusion processing is performed based on the map matrix, camera matrix, obstacle matrix and color editing matrix to obtain target features, and the target features are decoded to obtain a target edited image, thereby achieving the purpose of automatically editing the target captured image according to the color editing information corresponding to the editing instructions. Moreover, a large number of target edited images can be generated according to different target captured images and different editing instructions, thereby improving the efficiency of obtaining the target edited images, and there is no need to obtain images through road sampling, which greatly reduces the difficulty of obtaining the target edited images, thereby improving the iteration efficiency of the autonomous driving function and reducing the iteration difficulty of the autonomous driving function.

[0021] Other features and advantages of the embodiments of the present application will be described in the subsequent description, and partly become apparent from the description, or can be understood by practicing the embodiments of the present application. The purposes and other advantages of the embodiments of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0023] Figure 1 A flow chart of an image processing method proposed according to an embodiment of the present application is shown.

[0024] Figure 2 A schematic diagram showing a process of acquiring color editing information in an embodiment of the present application is shown.

[0025] Figure 3 A schematic diagram of a diffusion treatment process in an embodiment of the present application is shown.

[0026] Figure 4 A schematic diagram of an image processing process in an embodiment of the present application is shown.

[0027] Figure 5 A structural block diagram of an image processing device proposed in an embodiment of the present application is shown.

[0028] Figure 6A structural block diagram of an electronic device provided according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0029] In order to make those skilled in the art better understand the present application scheme, the technical scheme in the present application embodiment will be clearly and completely described below in conjunction with the drawings in the present application embodiment. Obviously, the described embodiment is only a part of the present application embodiment, rather than all the embodiments. The components of the present application embodiment usually described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the present application for protection, but merely represents the selected embodiment of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0030] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0031] See also Figure 1 , Figure 1 A flow chart of an image processing method proposed according to an embodiment of the present application is shown, which is used in an electronic device. The method includes:

[0032] S110 determines a map matrix according to position information of a plurality of target road points on a road surface in a target captured image captured by a camera of the vehicle.

[0033] The vehicle in this embodiment may be an electric vehicle or a fuel vehicle, or a car, an SUV, a bus, a truck, etc. The vehicle camera may be a camera installed on the vehicle for collecting environmental information around the vehicle. For example, the vehicle camera may refer to a front camera installed on the head of the vehicle, which is used to collect environmental information in front of the vehicle during the driving process. For another example, the vehicle camera may refer to a rear camera installed on the rear of the vehicle, which is used to collect environmental information behind the vehicle during the driving process.

[0034] The image captured by the vehicle's camera is used as the target captured image. The vehicle's camera performs one capture to obtain a captured image, which is used as the target captured image; the vehicle's camera performs multiple captures to obtain multiple captured images, any one of which can be used as the target captured image, or each captured image can be used as a target captured image to execute the image processing method of the present application, so as to generate corresponding target editing images for multiple target captured images.

[0035] The road surface in the target captured image may refer to the road surface of the road where the vehicle is located in the target image. If the vehicle is in a driving state, the road surface in the target captured image may refer to the road surface of the lane where the vehicle is driving; if the vehicle is in a parked state, the road surface in the target captured image may refer to the road surface of the lane where the vehicle is parked.

[0036] The target road point refers to a point selected from the road surface in the target shot image, which may be a point randomly selected from the road surface in the target shot image, or a point selected from the road surface in the target shot image as required, and this application does not make specific restrictions. The number of target road points can be set based on demand, for example, the number of target road points is 50 or 100, etc.

[0037] After selecting the target road point, the coordinates of the target road point in the SL curve coordinate system can be obtained as the location information of the target road point. The SL curve coordinate system uses the road centerline as a reference, S represents the direction of the road centerline (the value in the S direction is used as the x-coordinate), and L (the value in the L direction is used as the y-coordinate) represents the direction perpendicular to the road centerline. When driving on structured roads, the SL curve coordinate system is more in line with actual needs than the XY coordinate system.

[0038] After obtaining the location information of the target road point, an equation can be constructed according to the location information of the target road point by polynomial fitting, and then a map matrix is ​​determined according to the constructed equation. The map matrix is ​​used to indicate the locations of multiple target road points; in other words, the map matrix is ​​a matrix composed of multiple elements, and the values ​​of the multiple elements included in the map matrix indicate the locations of multiple target road points. For example, the value of a row of elements or a column of elements in the map matrix indicates the location of one target road point.

[0039] In some embodiments, S140 may include: constructing a position linear equation indicating the position information of each target road point according to the position information of each target road point; aggregating the position linear equations corresponding to multiple target road points to obtain a road point equation group; rewriting the road point equation group into the form of matrix product to obtain a road point matrix relationship; obtaining the coefficient matrix in the road point matrix relationship as a map matrix.

[0040] A fitting process may be performed according to the position information of each target road point by using a quadratic polynomial to obtain a position linear equation indicating the position information of each target road point.

[0041] For example, for the position information (x1, y1), (x2, y2), ..., (x n ,y n ), the n target road points are fitted with quadratic polynomials to obtain n position linear equations as follows:

[0042]

[0043] Among them, a1, a2, …, a n , b1, b2, …, b n , c1, c2, …, c n All are coefficients.

[0044] After obtaining the position linear equation of each target road point, the position linear equations of multiple target road points are aggregated into an equation group to achieve the aggregation of the position linear equations corresponding to multiple target road points and obtain the road point equation group E map .

[0045] Then the road point equation group E map The matrix equation is converted into the form of matrix product, and the obtained matrix equation is used as the road point matrix equation. map The form of matrix product is Y1=X1E1, where Y1 is y1, y2, ..., y n A matrix consisting of one column of data X1 is x1, x2, ..., x n A matrix consisting of one column of data E1 is a1, a2, …, a n , b1, b2, …, b n , c1, c2, …, c n The matrix consisting of n rows of data Among them, E1 is the road point equation group E map The matrix composed of the coefficients in , therefore, E1 is also called the coefficient matrix.

[0046] After the road point matrix relational expression is obtained, the coefficient matrix E1 in the road point matrix relational expression is obtained as the map matrix. At this time, the value of a row of elements in the map matrix indicates the position of one target road point.

[0047] It is worth mentioning that the road point equation group E can be obtained map Afterwards, the road point equation group E mapSolve and get the coefficients a1, a2, ..., a n , b1, b2, …, b n , c1, c2, …, c n Respective specific values, so that each element in the obtained map matrix is ​​a specific value.

[0048] S120. Determine a camera matrix according to the position information of the camera.

[0049] The position information of the camera may refer to the position information of the camera in the world coordinate system, and the position information of the camera may also include the pitch angle, yaw angle, roll angle and other information of the camera. After determining the position information of the camera, an equation may be constructed according to the position information of the camera by polynomial fitting, and then the camera matrix may be determined according to the constructed equation. Among them, the camera matrix is ​​used to indicate the position of the camera; in other words, the camera matrix is ​​a matrix composed of multiple elements, and the camera matrix indicates the position of the camera through the values ​​of the multiple elements included in it. For example, the value of a row of elements or a column of elements in the camera matrix indicates the position of the camera in a position dimension, wherein the position dimensions involved may include pitch angle, yaw angle, roll angle and coordinates in the world coordinate system, etc.

[0050] In some embodiments, the camera position information includes the pitch angle, yaw angle, roll angle and camera coordinates in the world coordinate system; S120 may include: constructing a pitch angle equation based on the camera coordinates and the pitch angle; constructing a yaw angle equation based on the camera coordinates and the yaw angle; constructing a roll angle equation based on the camera coordinates and the roll angle; aggregating the pitch angle equation, the yaw angle equation and the roll angle equation to obtain a camera position equation group; rewriting the camera position equation group into the form of matrix product to obtain a camera matrix relationship; obtaining the coefficient matrix in the camera matrix relationship as the camera matrix.

[0051] In the present application, the pitch angle equation, the roll angle equation, and the yaw angle equation may be determined by quadratic polynomial fitting. In the process of constructing the pitch angle equation, the roll angle equation, and the yaw angle equation, the number of coefficients of each equation may be the number of the aforementioned target road points. For example, when the number of the aforementioned target road points is n, the constructed pitch angle equation, the roll angle equation, and the yaw angle equation are as follows:

[0052] Pitch=e0+e1·x c +e2·y c +e3·z c +e4·x c 2 +e5·y c 2 +e6·zc 2 +…+e 3m ·z c m

[0053] Yaw=d0+d1·x c +d2·y c +d3·z c +d4·x c 2 +d5·y 2 +d6·z c 2 +…+d 3m ·z c m

[0054] Roll=f0+f1·x c +f2·y c +f3·z c +f4·x c 2 +f5·y c 2 +f6·z c 2 +…+c 3m ·z c m

[0055] Among them, Pitch represents the pitch angle equation, Yaw represents the yaw angle equation, Roll represents the pitch angle equation, e0, e1, ..., e n―1 , d0, d1, …, d n―1 ,f0,f1,…,f n―1 are coefficients, (x c ,y c , z c ) is the coordinate of the camera in the world coordinate system. Where n = 3m + 1.

[0056] After determining the pitch angle equation, roll angle equation and yaw angle equation, the pitch angle equation, roll angle equation and yaw angle equation are summarized into an equation group to achieve the aggregation of the pitch angle equation, roll angle equation and yaw angle equation, and obtain the camera position equation group E camera Then the camera position equation group E camera Convert it into the form of matrix product, and the obtained matrix relationship is used as the camera matrix relationship. For example, the camera position equation group E camera The form of matrix product is Y2=X2E2, where Y2 is a matrix consisting of Pitch, Yaw and Roll, including a column of data. X3 is (xc ,y c , z c ) consists of a matrix with one column of data E2 is e0, e1, …, e 3m , d0, d1, …, d 3m ,f0,f1,…,f 3m The matrix consisting of n columns of data Among them, E2 is the camera position equation group E camera The matrix composed of the coefficients in , so E2 is also called the coefficient matrix.

[0057] After obtaining the camera matrix relation, the coefficient matrix E2 in the camera matrix relation is obtained as the camera matrix. At this time, the value of a row of elements in the camera matrix indicates the position of the camera in a position dimension, and the position dimensions here are the pitch angle, yaw angle and roll angle.

[0058] It is worth mentioning that the camera position equation group E can be obtained camera After that, the camera position equation group E camera Solve and get the coefficients e0, e1, ..., e 3m , d0, d1, …, d 3m ,f0,f1,…,f 3m Each element in the obtained camera matrix has a specific value.

[0059] S130: Determine an obstacle matrix according to the position information and size of the obstacle in the target captured image.

[0060] Obstacles can be detected from the target captured image, and the position information and size of the detected obstacles can be determined, wherein the position information of the obstacle can be the coordinates of the center of mass of the obstacle in the world coordinate system; the size of the obstacle can refer to the value describing the outline of the obstacle, for example, the size of the obstacle can be 4m×2m×3m, indicating that the obstacle is equivalent to a cuboid with a length of 4m, a width of 2m, and a height of 3m.

[0061] In the present application, the obstacle can also be imagined as a sphere, and the center of mass of the obstacle is used as the center of the sphere, and the size of the obstacle is indicated by the radius of the sphere. In this case, the sphere is used as the equivalent sphere of the obstacle. Then, based on the center and radius of the equivalent sphere corresponding to the obstacle, an equation is constructed by polynomial fitting, and then the obstacle matrix is ​​determined according to the constructed equation. Among them, the obstacle matrix is ​​used to indicate the position and size of the obstacle; in other words, the obstacle matrix is ​​a matrix composed of multiple elements, and the obstacle matrix indicates the position and size of the obstacle through the values ​​of the multiple elements included. For example, the value of a row of elements or a column of elements in the obstacle matrix indicates the position and size of an obstacle.

[0062] In some embodiments, S130 may include: determining an equivalent sphere corresponding to the obstacle according to the position information and size of the obstacle; constructing a sphere equation corresponding to each obstacle according to the center and radius of the equivalent sphere corresponding to each obstacle; aggregating the sphere equations corresponding to each obstacle to obtain an obstacle equation group; rewriting the obstacle equation group into a matrix product form to obtain an obstacle matrix relationship; and obtaining a coefficient matrix in the obstacle matrix relationship as an obstacle matrix.

[0063] The process of determining the equivalent sphere is as described above and will not be repeated here. The sphere equation corresponding to each obstacle can be determined by quadratic polynomial fitting, one obstacle corresponds to one equation, wherein in the process of determining the sphere equation corresponding to each obstacle, the number of equations can be the number of the aforementioned target road points.

[0064] Generally speaking, the number of obstacles in the target captured image is limited and may be less than the number of the aforementioned target road points. Some obstacles can be randomly inserted into the target captured image as required to make the number of obstacles the same as the number of the aforementioned target road points, and the equivalent sphere of the inserted obstacle is determined to construct the spherical equation through the equivalent sphere of the obstacle.

[0065] For example, when the number of target road points is n, the constructed sphere equation is as follows:

[0066]

[0067] Among them, (x ci ,y ci , z ci ) is the coordinate of the center of the equivalent sphere corresponding to the ith obstacle in world coordinates (that is, the position information of the ith obstacle), r i is the radius of the equivalent sphere corresponding to the ith obstacle, o1, o2, ..., o n , p1, p2, …, p n ,q1,q2,…,q nAll are coefficients.

[0068] After determining the spherical equations of each obstacle, the spherical equations of multiple obstacles are summarized into an equation group to achieve the aggregation of the spherical equations of multiple obstacles and obtain the obstacle equation group E obj Then the obstacle equations E obj Convert it into the form of matrix product, and the obtained matrix relation is used as the obstacle matrix relation. For example, the obstacle equation group E obj The form of matrix product is Y3=X3E3, where Y3 is r1, r2, ..., r n A matrix consisting of one column of data X3 is (x c ,y c , z c ) consists of a matrix with n rows of data E3 is o1, o2, …, o n , p1, p2, …, p n ,q1,q2,…,q n The matrix consisting of n rows of data Among them, E3 is the obstacle equation group E obj The matrix composed of the coefficients in , so E3 is also called the coefficient matrix.

[0069] After obtaining the obstacle matrix relational expression, the coefficient matrix E3 in the obstacle matrix relational expression is obtained as the obstacle matrix. At this time, for example, the value of a row of elements in the obstacle matrix refers to the position and size of an obstacle.

[0070] It is worth mentioning that the obstacle equations E can be obtained obj Then, the obstacle equations E obj Solve and get the coefficients o1, o2, ..., o n , p1, p2, …, p n ,q1,q2,…,q n Each element in the obstacle matrix is ​​a specific value.

[0071] S140: Determine a color editing matrix according to color editing information for the target captured image.

[0072] The color editing information is obtained by parsing the editing instructions for the target captured image.

[0073] The editing instructions for the target captured image can be voice instructions or text instructions. For text instructions, content analysis can be performed through natural language processing (NLP) technology to obtain content analysis results; for voice instructions, voice instructions can be converted into text instructions through voice recognition technology, and then content analysis is performed on the text instructions through NLP to obtain content analysis results. The content analysis results after content analysis may include editing requirements for the target captured image, for example, the content analysis results may include adjusting the background, adding traffic elements, etc.

[0074] After obtaining the content analysis result, the content analysis result can be further analyzed by a large language model (LLM) to obtain the basic element RGB information of the image as color editing information. The color editing information may include the red pixel value R, the green pixel value G, and the blue pixel value B of each pixel in the target captured image.

[0075] like Figure 2 As shown, firstly, the editing instruction is obtained, content analysis is performed on the editing instruction to obtain content analysis results, and then the content analysis results are parsed to obtain color editing information.

[0076] After obtaining the color editing information of the target captured image, an equation is constructed according to the color editing information of the target captured image by polynomial fitting, and then a color editing matrix is ​​determined according to the constructed equation. The color editing matrix is ​​used to indicate the color value indicated by the editing instruction; the color value indicated by the editing instruction refers to the color value of each pixel in the target captured image after the target captured image is edited according to the editing instruction (the color value of the pixel is also the pixel value of the pixel); the color value indicated by the editing instruction may include the color value of each pixel in the target captured image; for example, if the target captured image includes 256×256 pixels, the color value indicated by the editing instruction may include the color value of each 256×256 pixel, that is, the color value indicated by the editing instruction includes the color value of each 256×256 pixel after the target captured image is edited according to the editing instruction.

[0077] In other words, the color editing matrix is ​​a matrix composed of multiple elements, and the color editing matrix indicates the color value indicated by the editing instruction through the values ​​of each element included. For example, the value of a row of elements or a column of elements in the color editing matrix indicates the color value under a color dimension, where the color dimension may include red, green, and blue.

[0078] In some embodiments, the color editing information includes first color information, second color information, and third color information; S140 may include: obtaining the number of multiple target points as the target number; constructing a first color equation based on the first color information and the target number; constructing a second color equation based on the second color information and the target number; constructing a third color equation based on the third color information and the target number; aggregating the first color equation, the second color equation, and the third color equation to obtain a color equation group; rewriting the color equation group into the form of matrix product to obtain a color matrix relationship; obtaining a coefficient matrix in the color matrix relationship as a color editing matrix. In the present application, the first color information may be the red pixel value of each pixel in the target captured image, the second color information may be the green pixel value of each pixel in the target captured image, and the third color information may be the blue pixel value of each pixel in the target captured image.

[0079] First, the number of target points is determined as a target number, and then the target number is used as the number of coefficients when constructing each color equation to construct the color equation.

[0080] Specifically, the first color equation, the second color equation, and the third color equation may be determined by quadratic polynomial fitting. For example, when the number of target road points is n, the first color equation, the second color equation, and the third color equation are constructed as follows:

[0081] R′=r0+r1R+r2R 2 +r3R 3 +…+r n―1 R n―1

[0082] G′=s0+s1G+s2G 2 +s3G 3 +…+s n―1 G n―1

[0083] B′=t0+t1B+t2B 2 +t3B 3 +…+t n―1 B n―1

[0084] Among them, R' represents the first color equation, G' represents the second color equation, B' represents the third color equation, R is the first color information, G is the second color information, B is the third color information, r0, r1, ..., r n―1 ,s0,s1,…,s n―1 , t0, t1, …, t n―1 All are coefficients.

[0085] After determining the first color equation, the second color equation and the third color equation, the first color equation, the second color equation and the third color equation are summarized into an equation group, and the aggregation of the first color equation, the second color equation and the third color equation is realized to obtain the color equation group E RGB Then the color equation E RGB The matrix is ​​converted into a matrix product form, and the obtained matrix relationship is used as a color matrix relationship.

[0086] For example, the color equation e RGB The form of matrix product is Y4=X4E4, where Y4 is a matrix consisting of R', G', and B' and includes a column of data. X4 is a matrix consisting of R, G and B, including a column of data E4 is r0, r1, ..., r n―1 ,s0,s1,…,s n―1 , t0, t1, …, t n―1 The matrix consisting of n columns of data Among them, E4 is the color equation group E RGB The matrix composed of the coefficients in , therefore, E4 is also called the coefficient matrix.

[0087] After the color matrix relational expression is obtained, the coefficient matrix E4 in the color matrix relational expression is obtained as the color editing matrix. At this time, the value of a row of elements in the color editing matrix indicates the color value under a color dimension, and the color dimensions involved are red, green and blue respectively.

[0088] It is worth mentioning that the color equation group E can be obtained RGB Afterwards, the color equations E RGB Solve and get the coefficients r0, r1, ..., r n―1 ,s0,s1,…,s n―1 , t0, t1, …, t n―1 Each element in the color editing matrix is ​​a specific value.

[0089] S150, performing diffusion processing based on the map matrix, the camera matrix, the obstacle matrix and the color editing matrix to obtain target features.

[0090] After obtaining the map matrix, camera matrix, obstacle matrix and color editing matrix, diffusion processing can be performed through a diffusion model based on the map matrix, camera matrix, obstacle matrix and color editing matrix, and the result after diffusion processing can be obtained as the target feature.

[0091] The diffusion model can be a steady-state diffusion model (DF model) and a diffusion-large language model (Diffusion Large Language Model l, DLLM for short). The diffusion rounds of the diffusion model can be set based on demand, for example, the diffusion rounds are 50.

[0092] In order to achieve the diffusion of the map matrix, camera matrix, obstacle matrix and color editing matrix, when constructing the map matrix, camera matrix, obstacle matrix and color editing matrix, the number of selected target road points, the number of parameters selected for constructing the pitch angle equation, the number of parameters selected for constructing the yaw angle equation, the number of parameters selected for constructing the roll angle equation, the number of obstacles, the number of parameters selected for constructing the first color equation, the number of parameters selected for constructing the second color equation and the number of parameters selected for constructing the third color equation should be consistent, so that the number of rows or columns of the obtained map matrix, camera matrix, obstacle matrix and color editing matrix are consistent.

[0093] In some embodiments, before S150, the method may further include: aligning the columns of the camera matrix and the columns of the color editing matrix with the rows of the map matrix, respectively, to obtain an aligned camera matrix corresponding to the camera matrix and an aligned color editing matrix corresponding to the color editing matrix; accordingly, S150 may include: performing diffusion processing based on the map matrix, the aligned camera matrix, the obstacle matrix and the aligned color editing matrix to obtain target features.

[0094] As mentioned above, the camera matrix and the color editing matrix are both n-column matrices, while the map matrix and the obstacle matrix are n-row matrices. It is necessary to align the columns of the camera matrix and the color editing matrix with the rows of the map matrix (at this time, the camera matrix, the color editing matrix and the obstacle matrix can also be aligned), and the aligned camera matrix and the aligned color editing matrix are obtained. Each element in the aligned camera matrix and the aligned color editing matrix is ​​aligned with the map matrix, and the aligned camera matrix and the aligned color editing matrix are also n-row matrices. At this time, the aligned color editing matrix, the aligned camera matrix, the map matrix and the obstacle matrix are all n×3 matrices.

[0095] In this embodiment, the matrix can be transposed to achieve alignment processing, that is: determine the transposed matrix of the camera matrix as the alignment camera matrix, and determine the transposed matrix of the color editing matrix as the alignment color editing matrix.

[0096] In some embodiments, S150 may include: respectively embedding processing the camera matrix, the obstacle matrix and the color editing matrix to obtain a preprocessed camera matrix corresponding to the camera matrix, a preprocessed obstacle matrix corresponding to the obstacle matrix and a preprocessed color editing matrix corresponding to the color editing matrix; performing attention operation on the preprocessed camera matrix, the preprocessed obstacle matrix, the preprocessed color editing matrix and the map matrix to obtain the attention operation result; performing multiple rounds of diffusion processing on the attention operation result, the preprocessed camera matrix, the preprocessed obstacle matrix and the preprocessed color editing matrix to obtain the target features.

[0097] After obtaining the attention operation results, the attention operation results, the preprocessed camera matrix, the preprocessed obstacle matrix and the preprocessed color editing matrix are all used as inputs for the first round of diffusion processing. They are input into the diffusion model for diffusion processing to obtain the attention operation results of the first round of diffusion processing, the camera matrix of the first round of diffusion processing, the obstacle matrix of the first round of diffusion processing and the color editing matrix of the first round of diffusion processing.

[0098] Afterwards, the attention operation result of the i-th round of diffusion processing, the camera matrix of the i-th round of diffusion processing, the obstacle matrix of the i-th round of diffusion processing, and the color editing matrix of the i-th round of diffusion processing can be used as the input of the i+1-th round of diffusion processing, and they are input into the diffusion model for diffusion processing to obtain the attention operation result of the i+1-th round of diffusion processing, the camera matrix of the i+1-th round of diffusion processing, the obstacle matrix of the i+1-th round of diffusion processing, and the color editing matrix of the i+1-th round of diffusion processing, until the last round of diffusion processing is completed, the attention operation result of the last round of diffusion processing, the camera matrix of the last round of diffusion processing, the obstacle matrix of the last round of diffusion processing, and the color editing matrix of the last round of diffusion processing are fused (the fusion can be achieved by splicing, summing, and finding vector products) to obtain the target feature. Wherein, i here is an integer greater than 0.

[0099] Exemplarily, the process of obtaining the target feature by diffusion processing is as follows: Figure 3As shown, the position information of the road surface, obstacles and the camera are extracted from the target captured image, and the color editing information after the editing instruction is parsed is obtained. The map matrix is ​​determined through the position information of the target road point in the road surface, the obstacle matrix is ​​determined through the position information of the obstacle, the camera matrix is ​​determined through the position information of the camera, and the color editing matrix is ​​determined through the color editing information. Then, the camera matrix, the obstacle matrix and the color editing matrix are embedded respectively to obtain the preprocessed camera matrix corresponding to the camera matrix, the preprocessed obstacle matrix corresponding to the obstacle matrix and the preprocessed color editing matrix corresponding to the color editing matrix. The attention operation result, the preprocessed camera matrix, the preprocessed obstacle matrix and the preprocessed color editing matrix are used as the input of the first round of diffusion processing to perform multiple rounds of diffusion processing to obtain the target features.

[0100] It is worth mentioning that in Figure 3 In , before embedding the camera matrix, obstacle matrix and color editing matrix respectively, the columns of the camera matrix and the columns of the color editing matrix can be aligned with the rows of the map matrix respectively to obtain an aligned camera matrix corresponding to the camera matrix and an aligned color editing matrix corresponding to the color editing matrix; then the aligned camera matrix, obstacle matrix and aligned color editing matrix are embedded respectively to obtain the result of the embedded processing of the aligned camera matrix as the preprocessed camera matrix corresponding to the camera matrix, the result of the embedded processing of the obstacle matrix as the preprocessed obstacle matrix, and the result of the embedded processing of the aligned color editing matrix as the preprocessed color editing matrix corresponding to the color editing matrix.

[0101] S160: Decode the target features to obtain a target edited image.

[0102] The target edited image presents the effect of editing the target captured image according to the editing instructions.

[0103] The target feature can be decoded by a decoder to obtain a decoded image as the target edited image. The decoder can be a decoder in a variational autoencoder (VAE).

[0104] After obtaining the target edited image, the target edited image may be saved in an image file format, such as saving the target edited image in a format such as JPEG, PNG, etc., or the target edited image may be stored in other data formats, such as storing the target edited image in a numpy array, etc.

[0105] In this embodiment, the image processing process is as follows: Figure 4As shown, first input data is determined based on a captured image of a target, the first input data includes a map matrix, a camera matrix and an obstacle matrix; second input data is determined based on an editing instruction, the second input data is a color editing matrix; based on the first input data and the second input data, diffusion processing is performed through a diffusion model to obtain target features, and the target features are decoded to obtain a target edited image, wherein the stages of diffusion processing performed by the diffusion model may include a diffusion stage, a sampling stage and a de-diffusion stage.

[0106] In this embodiment, based on the target captured image taken by the vehicle's camera, the map matrix, camera matrix and obstacle matrix are determined, and based on the color editing information obtained after parsing the editing instructions for the target captured image, the color editing matrix is ​​determined, and then diffusion processing is performed based on the map matrix, camera matrix, obstacle matrix and color editing matrix to obtain target features, and the target features are decoded to obtain the target edited image, thereby achieving the purpose of automatically editing the target captured image according to the color editing information corresponding to the editing instructions, and a large number of target edited images can be generated according to different target captured images and different editing instructions, thereby improving the efficiency of obtaining the target edited images, and there is no need to obtain images through road sampling, which greatly reduces the difficulty of obtaining the target edited images, thereby improving the iteration efficiency of the autonomous driving function and reducing the iteration difficulty of the autonomous driving function.

[0107] Secondly, the existing generative adversarial network to generate images involves image segmentation, obstacle segmentation, background segmentation, encoding process, obstacle latent coding, background latent coding, camera coding, coding optimization, obstacle optimization coding, background optimization coding, camera optimization coding and image reconstruction and other processing processes. The processing processes are numerous, prone to errors, and inefficient. Compared with the existing generation of images through generative adversarial networks, in this application, it is only necessary to determine the map matrix, camera matrix, obstacle matrix and color editing matrix, and perform diffusion processing based on the map matrix, camera matrix, obstacle matrix and color editing matrix. There are fewer processing processes involved, it is not easy to make errors, and the efficiency of generating images is higher.

[0108] Furthermore, for some dangerous driving scenarios, it is difficult to obtain images through road shooting. Through the method of the present application, based on setting danger editing instructions under safe driving scenarios, images of dangerous scenarios can be generated, and there is no need to conduct road shooting to obtain images in dangerous scenarios, thereby avoiding the occurrence of safety accidents caused by collecting images while driving in dangerous scenarios.

[0109] See attached Figure 5 , Figure 5 The structure block diagram of an image processing device proposed in one embodiment of the present application is shown. For electronic equipment, the device 800 includes:

[0110] A first determining module 810 is used to determine a map matrix according to the position information of a plurality of target road points on the road surface in the target captured image captured by the camera of the vehicle; the map matrix is ​​used to indicate the positions of the plurality of target road points;

[0111] A second determination module 820 is used to determine a camera matrix according to the position information of the camera; the camera matrix is ​​used to indicate the position of the camera;

[0112] The third determination module 830 is used to determine an obstacle matrix according to the position information and size of the obstacle in the target captured image; the obstacle matrix is ​​used to indicate the position and size of the obstacle;

[0113] The fourth determining module 840 is used to determine a color editing matrix according to color editing information for the target captured image; the color editing information is obtained by parsing the editing instruction for the target captured image; the color editing matrix is ​​used to indicate the color value indicated by the editing instruction;

[0114] Diffusion module 850, used for performing diffusion processing based on the map matrix, the camera matrix, the obstacle matrix and the color editing matrix to obtain target features;

[0115] The decoding module 860 is used to decode the target features to obtain a target edited image; the target edited image presents the effect of editing the target captured image according to the editing instructions.

[0116] Optionally, the first determination module 810 is further used to construct a position linear equation indicating the position information of each target road point according to the position information of each target road point; aggregate the position linear equations corresponding to multiple target road points to obtain a road point equation group; rewrite the road point equation group into the form of matrix product to obtain a road point matrix relationship; obtain the coefficient matrix in the road point matrix relationship as a map matrix.

[0117] Optionally, the camera position information includes the pitch angle, yaw angle, roll angle and camera coordinates in the world coordinate system; the second determination module 820 is further used to construct a pitch angle equation based on the camera coordinates and the pitch angle; construct a yaw angle equation based on the camera coordinates and the yaw angle; construct a roll angle equation based on the camera coordinates and the roll angle; aggregate the pitch angle equation, the yaw angle equation and the roll angle equation to obtain a camera position equation group; rewrite the camera position equation group into the form of matrix product to obtain a camera matrix relationship; obtain the coefficient matrix in the camera matrix relationship as the camera matrix.

[0118] Optionally, the third determination module 830 is further used to determine the equivalent sphere corresponding to the obstacle according to the position information and size of the obstacle; construct the sphere equation corresponding to each obstacle according to the center and radius of the equivalent sphere corresponding to each obstacle; aggregate the sphere equations corresponding to each obstacle to obtain an obstacle equation group; rewrite the obstacle equation group into the form of matrix product to obtain an obstacle matrix relationship; obtain the coefficient matrix in the obstacle matrix relationship as the obstacle matrix.

[0119] Optionally, the color editing information includes first color information, second color information and third color information; the fourth determination module 840 is also used to obtain the number of multiple target points as the target number; construct a first color equation based on the first color information and the target number; construct a second color equation based on the second color information and the target number; construct a third color equation based on the third color information and the target number; aggregate the first color equation, the second color equation and the third color equation to obtain a color equation group; rewrite the color equation group into the form of matrix product to obtain a color matrix relationship; obtain the coefficient matrix in the color matrix relationship as the color editing matrix.

[0120] Optionally, the diffusion module 850 is also used to align the columns of the camera matrix and the columns of the color editing matrix with the rows of the map matrix, respectively, to obtain an aligned camera matrix corresponding to the camera matrix and an aligned color editing matrix corresponding to the color editing matrix; and to perform diffusion processing based on the map matrix, the aligned camera matrix, the obstacle matrix and the aligned color editing matrix to obtain target features.

[0121] Optionally, the diffusion module 850 is further used to perform embedding processing on the camera matrix, the obstacle matrix and the color editing matrix respectively to obtain a preprocessed camera matrix corresponding to the camera matrix, a preprocessed obstacle matrix corresponding to the obstacle matrix and a preprocessed color editing matrix corresponding to the color editing matrix; perform attention operation on the preprocessed camera matrix, the preprocessed obstacle matrix, the preprocessed color editing matrix and the map matrix to obtain the attention operation result; perform multiple rounds of diffusion processing on the attention operation result, the preprocessed camera matrix, the preprocessed obstacle matrix and the preprocessed color editing matrix to obtain the target features.

[0122] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and modules can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here.

[0123] In addition, each function in each embodiment of the present application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above integrated module can be implemented in the form of hardware or in the form of software function module.

[0124] In addition, each function in each embodiment of the present application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above integrated module can be implemented in the form of hardware or in the form of software function module.

[0125] Please refer to Figure 6 , which shows a structural block diagram of an electronic device provided according to an embodiment of the present application. The electronic device 500 can be an electronic device capable of running applications, such as a smart phone, a tablet computer, an e-book, and a vehicle. The electronic device 500 in the present application may include one or more of the following components: a processor 510, a memory 520, and one or more applications. One or more applications may be stored in the memory 520 and configured to be executed by one or more processors 510, and one or more programs are configured to execute the method described in the aforementioned method embodiment.

[0126] The processor 510 may include one or more processing cores. The processor 510 uses various interfaces and lines to connect various parts of the entire electronic device 500, and executes various functions and processes data of the electronic device 500 by running or executing instructions, programs, code sets or instruction sets stored in the memory 520, and calling data stored in the memory 520. Optionally, the processor 510 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The processor 510 can integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics processing unit (Graphics Processing Unit, GPU), and a modem. Among them, the CPU mainly processes the operating system, user interface, and application programs; the GPU is responsible for rendering and drawing display content; and the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 510, but may be implemented separately through a communication chip.

[0127] The memory 520 may include a random access memory (RAM) or a read-only memory (ROM). The memory 520 may be used to store instructions, programs, codes, code sets or instruction sets. The memory 520 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the following various method embodiments, etc. The data storage area may also store data (such as a phone book, audio and video data, chat record data) created by the electronic device 500 during use.

[0128] In addition, each function in each embodiment of the present application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above integrated module can be implemented in the form of hardware or in the form of software function module.

[0129] On the other hand, the present application also provides a computer-readable storage medium, in which program code is stored. The program code can be called by a processor to execute the method described in the above method embodiment.

[0130] The computer readable storage medium may be an electronic memory such as a flash memory, an EEPROM (electrically erasable programmable read-only memory), an EPROM, a hard disk, or a cluster of ROMs. Optionally, the computer readable storage medium comprises a non-transitory computer-readable storage medium. The computer readable storage medium has storage space for program codes that execute any of the method steps in the above method. These program codes can be read from or written to one or more computer program products. The program code can be compressed, for example, in an appropriate form.

[0131] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. An image processing method, characterized in that: The method comprises: Determine a map matrix according to position information of a plurality of target road points on a road surface in a target shot image shot by a camera of the vehicle; the map matrix is ​​used to indicate the positions of the plurality of target road points; Determine a camera matrix according to the position information of the camera; the camera matrix is ​​used to indicate the position of the camera; Determining an obstacle matrix according to the position information and size of the obstacle in the target captured image; the obstacle matrix is ​​used to indicate the position and size of the obstacle; Determining a color editing matrix according to color editing information for the target captured image; the color editing information is obtained by parsing an editing instruction for the target captured image; the color editing matrix is ​​used to indicate a color value indicated by the editing instruction; Performing diffusion processing based on the map matrix, the camera matrix, the obstacle matrix and the color editing matrix to obtain target features; The target feature is decoded to obtain the target edited image; the target edited image presents the effect of editing the target captured image according to the editing instruction.

2. The method according to claim 1, characterized in that The determining of the map matrix according to the position information of a plurality of target road points on the road surface in the target shooting image shot by the camera of the vehicle comprises: constructing a position linear equation indicating the position information of each target road point according to the position information of each target road point; Aggregating the position linear equations corresponding to the plurality of target road points to obtain a road point equation group; Rewrite the road point equation group into a matrix product form to obtain a road point matrix relationship; The coefficient matrix in the road point matrix relationship is obtained as the map matrix.

3. The method according to claim 1, characterized in that The camera position information includes the pitch angle, yaw angle, roll angle of the camera and the camera coordinates in the world coordinate system; The step of determining a camera matrix according to the position information of the camera includes: Constructing a pitch angle equation based on the camera coordinates and the pitch angle; Constructing a yaw angle equation based on the camera coordinates and the yaw angle; Constructing a rolling angle equation based on the camera coordinates and the rolling angle; Aggregating the pitch angle equation, the yaw angle equation and the roll angle equation to obtain a camera position equation group; Rewrite the camera position equations into a matrix product form to obtain a camera matrix relationship; Obtain a coefficient matrix in the camera matrix relationship as the camera matrix.

4. The method according to claim 1, characterized in that: The step of determining an obstacle matrix according to the position information and size of the obstacle in the target captured image includes: Determine an equivalent sphere corresponding to the obstacle according to the position information and size of the obstacle; Constructing a sphere equation corresponding to each obstacle according to the center and radius of the equivalent sphere corresponding to each obstacle; Aggregating the sphere equations corresponding to the obstacles to obtain an obstacle equation group; Rewrite the obstacle equations into a matrix product form to obtain an obstacle matrix relationship; Obtain a coefficient matrix in the obstacle matrix relationship as the obstacle matrix.

5. The method according to claim 1, characterized in that The color editing information includes first color information, second color information and third color information; The step of determining a color editing matrix according to the color editing information of the target captured image comprises: Acquire the number of the plurality of target points as the target number; Constructing a first color equation according to the first color information and the target quantity; constructing a second color equation according to the second color information and the target quantity; constructing a third color equation according to the third color information and the target quantity; Aggregating the first color equation, the second color equation, and the third color equation to obtain a color equation group; Rewrite the color equation group into a matrix product form to obtain a color matrix relationship; The coefficient matrix in the color matrix relationship is obtained as the color editing matrix.

6. The method according to claim 1, characterized in that Before the diffusion processing is performed based on the map matrix, the camera matrix, the obstacle matrix and the color editing matrix to obtain the target features, the method further includes: Aligning the columns of the camera matrix and the columns of the color editing matrix with the rows of the map matrix respectively, to obtain an aligned camera matrix corresponding to the camera matrix and an aligned color editing matrix corresponding to the color editing matrix; The diffusion processing is performed based on the map matrix, the camera matrix, the obstacle matrix and the color editing matrix to obtain target features, including: Diffusion processing is performed based on the map matrix, the alignment camera matrix, the obstacle matrix and the alignment color editing matrix to obtain target features.

7. The method according to claim 1, characterized in that The diffusion processing is performed based on the map matrix, the camera matrix, the obstacle matrix and the color editing matrix to obtain target features, including: The camera matrix, the obstacle matrix, and the color editing matrix are respectively embedded to obtain a preprocessed camera matrix corresponding to the camera matrix, a preprocessed obstacle matrix corresponding to the obstacle matrix, and a preprocessed color editing matrix corresponding to the color editing matrix; Performing an attention operation on the preprocessed camera matrix, the preprocessed obstacle matrix, the preprocessed color editing matrix, and the map matrix to obtain an attention operation result; Based on the attention operation result, the preprocessed camera matrix, the preprocessed obstacle matrix and the preprocessed color editing matrix, multiple rounds of diffusion processing are performed to obtain the target feature.

8. An image processing device, characterized in that: The device comprises: A first determination module is used to determine a map matrix according to position information of a plurality of target road points on a road surface in a target shot image shot by a camera of the vehicle; the map matrix is ​​used to indicate the positions of the plurality of target road points; A second determination module is used to determine a camera matrix according to the position information of the camera; the camera matrix is ​​used to indicate the position of the camera; A third determination module, used to determine an obstacle matrix according to the position information and size of the obstacle in the target captured image; the obstacle matrix is ​​used to indicate the position and size of the obstacle; A fourth determination module is used to determine a color editing matrix according to color editing information for the target captured image; the color editing information is obtained by parsing an editing instruction for the target captured image; the color editing matrix is ​​used to indicate a color value indicated by the editing instruction; A diffusion module, used for performing diffusion processing based on the map matrix, the camera matrix, the obstacle matrix and the color editing matrix to obtain target features; A decoding module is used to decode the target features to obtain the target edited image; the target edited image presents the effect of editing the target captured image according to the editing instruction.

9. An electronic device, characterized in that: include: one or more processors; Memory; One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, and the one or more applications are configured to execute the method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a program code executable by a processor, and when the program code is executed by the processor, the processor executes the method according to any one of claims 1 to 7.

Images