Image processing method, device, apparatus, medium, vehicle and program product

Abstract

The present invention relates to the field of image processing, and discloses an image processing method, device, equipment, medium, vehicle and program product, wherein the method comprises: when a foreground object is detected in an intersection area of ​​a vehicle-mounted image, the intersection area is divided into a background area image and a foreground area image; the background area image is projected onto a reference plane to obtain a background projection image; the foreground area image is projected onto a torus to obtain a foreground projection image; and a three-dimensional surround view image is generated based on the background projection image and the foreground projection image. The three-dimensional surround view image generated by the present invention has a smaller distortion and is suitable for a vehicle-mounted mobile platform.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to an image processing method, device, equipment, medium, vehicle and program product. Background Art

[0002] In existing vehicle-mounted panoramic imaging systems, defects such as ghosting or misalignment exist when stitching images, which limits their effectiveness and accuracy in practical applications. Summary of the invention

[0003] Embodiments of the present invention provide an image processing method, apparatus, device, medium, vehicle and program product to solve the defects such as ghosting or misalignment mentioned in the background technology.

[0004] An image processing method, comprising:

[0005] When a foreground object is detected in the intersection area of ​​the vehicle-mounted image, the intersection area is segmented into a background area image and a foreground area image;

[0006] Projecting the background area image onto the reference plane to obtain a background projection image;

[0007] Projecting the foreground area image onto the torus surface to obtain a foreground projection image;

[0008] A three-dimensional surround image is generated based on the background projection image and the foreground projection image.

[0009] Before detecting that there is a foreground object in the intersection area of ​​the vehicle image, it also includes:

[0010] Creating a transformation mapping table according to the first transformation matrix;

[0011] The original image is transformed based on the transformation mapping table to obtain the vehicle-mounted image.

[0012] Optionally, before creating the transformation mapping table according to the first transformation matrix, the method further includes:

[0013] Obtain calibration image data of multiple channels;

[0014] A first transformation matrix is ​​determined based on the calibration image data.

[0015] Optionally, the calibration image data includes a plurality of images including a calibration plate; the calibration plate includes a circular ring pattern;

[0016] Determining a first transformation matrix according to the calibration image data includes:

[0017] Extracting a plurality of circle center coordinates and a plurality of corner point coordinates from the calibration image data;

[0018] Determine the focal length of the camera and the pixel coordinates of the camera according to the coordinates of the multiple center points, and determine the intrinsic parameter matrix according to the focal length of the camera and the pixel coordinates;

[0019] A first transformation matrix is ​​determined according to the coordinates of the multiple corner points and the intrinsic parameter matrix.

[0020] Optionally, segmenting the intersection area into a background area image and a foreground area image includes:

[0021] Calculate the motion vectors between adjacent frames in the intersection area;

[0022] The intersection area is segmented according to the motion vectors between adjacent frames to generate a background area image and a foreground area image.

[0023] Optionally, calculating the adjacent inter-frame motion vectors of the intersection region includes:

[0024] The motion vectors between adjacent frames in the intersection area are calculated using the optical flow method.

[0025] Optionally, segmenting the intersection area according to the motion vectors between adjacent frames to generate a background area image and a foreground area image includes:

[0026] Mark pixels whose norm of motion vectors between adjacent frames is greater than a preset motion threshold as foreground area images;

[0027] Pixels whose norm of motion vectors between adjacent frames is less than or equal to a preset motion threshold are marked as background area images.

[0028] Optionally, before projecting the background area image onto the reference plane to obtain the background projection image, the method further includes:

[0029] In the double-layer bowl mold, set the ground plane as the reference plane;

[0030] Set the loop surface based on the bottom and top parameters.

[0031] Optionally, projecting the foreground area image onto the torus surface to obtain a foreground projection image includes:

[0032] The foreground area image is projected onto the annular surface according to the second transformation matrix to obtain a foreground projection image.

[0033] Optionally, before projecting the foreground area image onto the toroidal surface according to the second transformation matrix to obtain the foreground projection image, the method includes:

[0034] Acquire two frames of foreground area images; source channels of the two frames of foreground area images are adjacent;

[0035] Extracting a set of feature point pairs from the foreground area image;

[0036] Filtering out the surface feature point pairs on the toroidal surface from the feature point pair set;

[0037] A second transformation matrix for projecting the foreground object onto the annular surface is determined according to the surface feature point pairs.

[0038] Optionally, surface feature point pairs on the toroidal surface are selected from the feature point pair set, including:

[0039] Calculate the distance between each feature point pair in the feature point pair set and the torus surface;

[0040] The feature point pairs whose distance is less than the preset difference value are determined as the surface feature point pairs.

[0041] Optionally, before filtering out the surface feature point pairs on the toroidal surface from the feature point pair set, the method further includes:

[0042] Get the distance between the foreground object and the current vehicle;

[0043] Adjusts the radius parameter of the loop surface based on the distance.

[0044] Optionally, generating a three-dimensional surround image based on the background projection image and the foreground projection image includes:

[0045] Binding the background projection image and the foreground projection image to the grid table to generate binding data;

[0046] Render the bound data to generate a three-dimensional surround image.

[0047] Optionally, the background projection image and the foreground projection image are bound to the grid table to generate binding data, including:

[0048] Bind the background projection image as the first texture map to the grid table of the ground plane layer of the double-layer bowl mold;

[0049] Bind the foreground projection image as the second texture map to the grid table of the double-layer bowl model moving object layer.

[0050] An image processing device, comprising:

[0051] An image segmentation module, for segmenting the intersection area into a background area image and a foreground area image when a foreground object is detected in the intersection area of ​​the vehicle-mounted image;

[0052] A background projection module, used to project the background area image onto a reference plane to obtain a background projection image;

[0053] A foreground projection module is used to project the foreground area image onto the torus surface to obtain a foreground projection image;

[0054] The rendering module is used to generate a three-dimensional surround image based on the background projection image and the foreground projection image.

[0055] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the above-mentioned image processing method is implemented.

[0056] A computer-readable storage medium stores a computer program, and the computer program implements the above-mentioned image processing method when executed by a processor.

[0057] A vehicle includes a vehicle controller, and the vehicle controller is used to execute the above-mentioned image processing method.

[0058] A computer program product comprises a computer program, wherein the computer program implements the above-mentioned image processing method when executed by a processor.

[0059] The above-mentioned image processing method, device, equipment, medium, vehicle and program product, when detecting the presence of a foreground object in the intersection area of ​​the vehicle-mounted image, divides the intersection area into a background area image and a foreground area image; projects the background area image onto a reference plane to obtain a background projection image; projects the foreground area image onto a toroidal surface to obtain a foreground projection image; generates a three-dimensional surround view image based on the background projection image and the foreground projection image, corrects the perspective distortion of the foreground moving pedestrians / obstacles crossing adjacent cameras in the panoramic photos / videos of the scene taken by the vehicle-mounted panoramic lens, and restores the accurate and true geometric appearance of the background ground plane. Moreover, the method achieves a sufficiently fast processing speed on the vehicle-mounted mobile platform, and the user interface experience is smooth, which is of great significance for improving the viewing experience of the vehicle-mounted panoramic image. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0061] Figure 1 is a flow chart of an image processing method according to an embodiment of the present invention;

[0062] Figure 2 is a schematic diagram of the layout of the calibration plate pattern in one embodiment of the present invention;

[0063] Figure 3 is a three-dimensional stereogram of a double-layer bowl mold in one embodiment of the present invention;

[0064] Figure 4 is a perspective view of a double-layer bowl mold in one embodiment of the present invention;

[0065] Figure 5 is a schematic diagram of the relative positions of a moving vehicle and a torus in one embodiment of the present invention;

[0066] Figure 6 It is a schematic diagram of the structure of an image processing device in one embodiment of the present invention. DETAILED DESCRIPTION

[0067] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0068] In one embodiment, if Figure 1 As shown, an image processing method is provided, comprising the following steps S10-S40.

[0069] S10: When it is detected that there is a foreground object in the intersection area of ​​the vehicle-mounted image, the intersection area is divided into a background area image and a foreground area image.

[0070] It is understandable that the original images of multiple channels for synthesizing the panoramic image can be obtained from the video stream of the CAN bus. In some examples, the original images of the multiple channels include images of the front, back, left, and right channels, which are respectively . After preprocessing the original image, the vehicle image can be obtained. There are intersection areas between the vehicle images of adjacent channels. For example, the vehicle image on the right and the vehicle image below have an intersection area at the lower right. For images in non-intersection areas, they can be directly used to synthesize three-dimensional surround view images. For intersection areas, it can be determined whether there are foreground objects (such as pedestrians or other obstacles) in the intersection area. If there are foreground objects in the vehicle image, this part of the image needs to be segmented to form a background area image and a foreground area image. If there are no foreground objects in the intersection area, it can be directly used to synthesize a three-dimensional surround view image.

[0071] Optionally, before step S10, that is, before detecting that a foreground object exists in the intersection area of ​​the vehicle-mounted image, the method further includes:

[0072] S101, creating a transformation mapping table according to a first transformation matrix;

[0073] S102: Perform a distortion correction transformation on the original image based on the transformation mapping table to obtain a vehicle-mounted image.

[0074] Understandably, the first transformation matrix H may be a transformation matrix for generating a bird's-eye view projection image. A transformation mapping table may be created based on the preset first transformation matrix H. Here, the transformation mapping table is a one-dimensional array, the size of which is the same as the target size of the vehicle-mounted image. In some examples, if the original image is generated based on a fisheye lens, it is necessary to first remove the fisheye distortion and then project it through the transformation mapping table to obtain the vehicle-mounted image. For example, After distortion correction, the vehicle image is generated .

[0075] Optionally, step S101, that is, before creating a transformation mapping table according to the first transformation matrix, includes steps S11-S12.

[0076] S11. Obtain calibration image data of multiple channels.

[0077] Understandably, one can refer to Figure 2 Calibration plates are placed around the vehicle, and cameras in each channel take photos of the plates to form calibration images. Calibration image data can be obtained by further processing the calibration images, such as the coordinates of specific patterns on the calibration plates. Figure 2 In the example, black squares are set at the upper left corner, upper right corner, lower left corner and lower right corner of the vehicle, and white rectangular blocks are set at the left, right, top and bottom of the vehicle, with multiple neatly arranged circles on the white rectangular blocks. The area other than the black squares and white rectangular blocks is white squares with a side length of 2 cm.

[0078] In one example, four front, rear, left and right channel cameras are used to take pictures of the circular patterns on the top, bottom, left and right of the calibration plate respectively, and 10 to 30 calibration images are obtained.

[0079] S12. Determine a first transformation matrix according to the calibration image data.

[0080] Specifically, the calibration image data includes multiple images containing a calibration plate; the calibration plate includes a circular ring pattern. Step S12 can be implemented through steps S121-S123.

[0081] S121. Extract multiple circle center coordinates and multiple corner point coordinates from the calibration image data.

[0082] In one example, the calibration image is preprocessed in grayscale to make the edges sharp and the contrast obvious. Then the Canny edge detection algorithm is used to detect the boundary of the ring. Finally, the Hough transform is used to fit the ellipse and extract several center coordinates (u c ,v c ). Perform corner point detection on the large square grid of the calibration cloth ( Figure 2The calibration image is first processed in grayscale to enhance the edge and contrast, and then the sub-pixel corner detection algorithm is used to analyze the grayscale changes of the image. The calibration image of each channel can detect up to 8 diagonal point coordinates (u a ,v a ).

[0083] S122, determining a camera focal length and a camera pixel coordinate according to a plurality of circle center coordinates, and determining an intrinsic parameter matrix according to the camera focal length and the pixel coordinates;

[0084] S123. Determine a first transformation matrix according to the coordinates of multiple corner points and an intrinsic parameter matrix.

[0085] In one example, the coordinates of the center of the circle (u c ,v c ) is substituted into the left side of equation 1, and the side length of the square is 2cm, which is substituted into the right side of equation 1. Multiple equations for several corner points are listed, and the horizontal focal length f of the camera can be solved through multiple circle center coordinates. x , the vertical focal length of the camera f y , the coordinates of the origin of the image coordinate system in the pixel coordinate system (u0, v0). The unknown quantities can be substituted into 8 pairs of circle center coordinates to solve the unknown quantities and ensure the accuracy of the solution.

[0086] Formula 1: ;

[0087] Among them, u and v are used to substitute the coordinates of the center of the circle (u c ,v c ) value;

[0088] f x is the horizontal focal length of the camera;

[0089] f y is the vertical focal length of the camera;

[0090] R is the rotation matrix;

[0091] T is the translation matrix;

[0092] The coordinates of the origin of the image coordinate system in the pixel coordinate system are (u0, v0);

[0093] K is the internal parameter matrix;

[0094] Can be the ground plane.

[0095] In some examples, the calibration image data includes the coordinates of the center of the circle, the coordinates of the corner points, and the horizontal focal length f of the camera. x , the vertical focal length of the camera f y, the coordinates of the origin of the image coordinate system in the pixel coordinate system (u0, v0) and other parameters.

[0096] The diagonal point coordinates and the internal parameter matrix K obtained in step S201 can be substituted into Formula 2 to solve R and T (Formula 4) and the first transformation matrix H (Formula 3). In some examples, the coordinates of 8 diagonal points can be used for the solution.

[0097] Formula 2: ;

[0098] Formula 3: ;

[0099] Formula 4: ;

[0100] in, is the ground plane, z w is 0.

[0101] In order to obtain the optimal solution, the RANSAC algorithm can be used for parameter estimation, which is sent as the initial estimate to the LM algorithm for iterative adjustment; when the objective function is adjusted to the preset minimum error value, the first transformation matrix H of the optimal estimate can be output.

[0102] In one example, step S10, that is, dividing the intersection area into a background area image and a foreground area image, includes S103-S104.

[0103] In one example, step S10, that is, dividing the intersection area into a background area image and a foreground area image, includes S103-S104.

[0104] S103, calculating the motion vector between adjacent frames in the intersection area.

[0105] In one example, the motion vector between adjacent frames of the intersection region can be calculated by the optical flow method. The optical flow method is a method used in computer vision to estimate the motion of objects in an image sequence. It can calculate the motion vector of each pixel from consecutive image frames, that is, the displacement of the point between adjacent frames. The optical flow method is used to calculate the movement of pixels between consecutive frames to obtain the motion vector between adjacent frames.

[0106] S104, segmenting the intersection area according to the motion vectors between adjacent frames to generate a background area image and a foreground area image.

[0107] According to the motion vectors between these adjacent frames, the image in the intersection area can be segmented into the background area image. , and the foreground image .

[0108] Optionally, step S104, i.e. segmenting the intersection area according to the motion vectors between adjacent frames to generate a background area image and a foreground area image, includes:

[0109] S1041, marking pixels whose norm of motion vectors between adjacent frames is greater than a preset motion threshold as foreground area images;

[0110] S1042: Mark pixels whose norm of motion vectors between adjacent frames is less than or equal to a preset motion threshold as background area images.

[0111] Understandably, pixels whose norm of motion vectors between adjacent frames is greater than a preset motion threshold can be marked as foreground area images, and pixels whose norm of motion vectors between adjacent frames is less than or equal to the preset motion threshold can be marked as background area images. Here, the preset motion threshold can be set according to an empirical value.

[0112] S20, projecting the background area image onto a reference plane to obtain a background projection image.

[0113] It is understandable that a three-dimensional model may be created in advance, and the three-dimensional model includes a reference plane. The background area image may be projected onto the reference plane to obtain a background projection image. In one example, the background area image may be projected onto a reference plane, such as a ground plane, using a first transformation matrix H to obtain a background projection image. The process can be expressed by equation 5.

[0114] Formula 5: .

[0115] Optionally, before step S20, that is, before projecting the background area image onto the reference plane and obtaining the background projection image, the method further includes:

[0116] S21. In the double-layer bowl mold, the ground plane is set as a reference plane;

[0117] S22. Setting a toroidal surface according to bottom surface parameters and top surface parameters.

[0118] Understandably, one can create a Figure 3 and Figure 4 The double-layer bowl mold shown, Figure 3 This is a three-dimensional image of a double-layer bowl mold. Figure 4This is a perspective view of a double-layer bowl mold. The double-layer bowl mold consists of two layers: a reference plane and a torus. The reference plane can be a ground plane, which is used to project the background area image, and the torus is used to project the foreground area image. When creating a double-layer bowl mold, a mesh table is generated for backup. A mesh table is a data structure that contains information about all vertices, edges, and faces in a 3D model. When creating a double-layer bowl mold, the Mesh Table is a key part used to store and organize the model's geometric data.

[0119] In the double-layer bowl mold, the ground plane can be set as the reference plane. The loop surface can be set according to the bottom surface parameters and the top surface parameters.

[0120] The top surface parameters of the toroidal surface are shown in Equation 6.

[0121] Formula 6: ;

[0122] ( ) is the three-dimensional coordinate of the characteristic point on the torus surface, Ta is the major radius of the vertex ellipse of the torus surface (see Figure 5 ), Tb is the minor radius of the vertex ellipse of the torus, and θ is the angle in polar coordinates.

[0123] The bottom surface parameters of the toroidal surface are shown in Equation 7.

[0124] Formula 7: ;

[0125] Ba is the major radius of the base ellipse of the torus (see Figure 5 ), Bb is the minor radius of the base ellipse of the toroidal surface.

[0126] S30, projecting the foreground area image onto the torus surface to obtain a foreground projection image.

[0127] Optionally, step S30, projecting the foreground area image onto the torus surface to obtain a foreground projection image;

[0128] S301. Project a foreground area image onto a torus according to a second transformation matrix to obtain a foreground projection image.

[0129] It can be understood that the second transformation matrix H can be used m Project the foreground area image onto the torus surface to obtain the foreground projection image The process can be expressed by equation 6.

[0130] Formula 6: .

[0131] Optionally, before step S301, that is, before obtaining the second transformation matrix, the process further includes:

[0132] S31, acquiring two frames of foreground area images; source channels of the two frames of foreground area images are adjacent;

[0133] S32, extracting a set of feature point pairs from the foreground area image;

[0134] S33, selecting a surface feature point pair on the toroidal surface from the feature point pair set;

[0135] S34, determining a second transformation matrix for projecting the foreground object onto the toroidal surface according to the surface feature point pairs.

[0136] Understandably, two frames of foreground area images can be obtained, and the source channels of the two frames of foreground area images are adjacent, such as , . Use the ORB algorithm to extract feature point pairs from the two foreground area images, including the first foreground area image Multiple feature points extracted , and from the second foreground image Multiple feature points extracted ,in .

[0137] The feature point pair set can be further filtered to select the surface feature point pairs on the toroidal surface. and (in ).

[0138] The second transformation matrix projected from the moving object layer to the torus surface can be solved by referring to Formula 2: In formula 2, the ground plane is the origin of the center point of the moving vehicle, so let Then, a point in the world coordinates Simplified to , we have formula 8, and we can get the initial .

[0139] Formula 8: ;

[0140] Formula 9: ;

[0141] Among them, Ta is the long radius of the top ellipse of the torus, Tb is the short radius of the top ellipse of the torus; Ba is the long radius of the bottom ellipse of the torus, Bb is the short radius of the bottom ellipse of the torus, is the angle in polar coordinates, and the vertical height from the top ellipse to the bottom ellipse is h. It should be noted that these parameters are all empirical values. After evaluating the blind area of ​​the field of view with reference to the body parameters of the specified vehicle model and the installation position of the camera, the initial default values ​​are set based on experience.

[0142] Optionally, step S33, i.e., selecting surface feature point pairs on the toroidal surface from the feature point pair set, includes:

[0143] S331, calculating the distance between each feature point pair in the feature point pair set and the torus surface;

[0144] S332: Determine a pair of feature points whose distance is less than a preset difference as a pair of surface feature points.

[0145] Understandably, the distances from each feature point pair in the feature point pair set to the toroidal surface can be calculated respectively, and the feature point pairs whose distances are less than the preset difference value are determined as surface feature point pairs. Here, the surface feature point pairs are on the toroidal surface. The preset difference value can be set according to an empirical value.

[0146] Optionally, step S34, i.e., before selecting the surface feature point pairs on the toroidal surface from the feature point pair set, includes:

[0147] S341, obtaining the distance between the foreground object and the current vehicle;

[0148] S342. Adjust the radius parameter of the toroidal surface according to the distance.

[0149] Understandably, the distance between the foreground object and the current vehicle can be obtained, such as the obstacle distance d detected by the radar sent by the CAN line; the radius parameter of the ring surface is adjusted according to the distance. The adjustment rule of the radius parameter can be pre-set. In one example, the adjustment rule can be set as follows: set a certain model of radar to have 4 front, rear, left and right radars, and set one The change value of the obstacle detected in the front or rear direction is , then the adjusted radius of the ring surface is calculated as shown in formula 9; if the distance of the obstacle detected in the left or right direction is , then the adjusted radius of the torus is calculated as shown in Equation 10. If the distances of obstacles detected at the front or rear and the left or right are and , then the adjusted radius of the annular surface is calculated as shown in formula 11.

[0150] Formula 9: ;

[0151] Formula 10: ;

[0152] Formula 11: .

[0153] After the torus is determined, the feature points can be screened based on the torus, and then the second transformation matrix can be optimized by iterating the feature point data subset. , the optimal estimate is obtained which is less than the set error value and the optimal .

[0154] S40: Generate a three-dimensional surround view image based on the background projection image and the foreground projection image.

[0155] In one example, step S40 includes:

[0156] S401, binding the background projection image and the foreground projection image to the grid table to generate binding data;

[0157] S402: Render the bound data to generate a three-dimensional surround view image.

[0158] Understandably, the background projection image and the foreground projection image can be bound to the grid table of the double-layer bowl mold to generate binding data. A specific application can be used to process the binding data to generate a three-dimensional surround image. For example, the texture map and grid table in the binding data are passed to the OpenGLES part of the application, and the GPU is used to render a three-dimensional surround image on the screen. OpenGL ES (OpenGL for Embedded Systems) is a three-dimensional graphics application programming interface (API) optimized for embedded devices.

[0159] The image processing method provided in this embodiment corrects the perspective distortion of moving pedestrians / obstacles in the foreground crossing adjacent cameras in the panoramic photos / videos of the scene taken by the vehicle-mounted panoramic lens, while restoring the accurate and realistic geometric appearance of the background ground plane. Moreover, the method achieves a sufficiently fast processing speed on the mobile platform, and the user interface experience is smooth, which is of great significance for improving the viewing experience of vehicle-mounted panoramic images.

[0160] Optionally, step S401, that is, binding the background projection image and the foreground projection image with the grid table to generate binding data, includes:

[0161] S4011, binding the background projection image as a first texture map to the grid table of the ground plane layer of the double-layer bowl mold;

[0162] S4011, binding the foreground projection image as the second texture map to the grid table of the moving object layer of the double-layer bowl mold.

[0163] Understandably, a background projection image may be used As the first texture map, bind it to the grid table of the double-layer bowl model ground plane layer. As the second texture map, it is bound to the mesh table of the moving object layer (i.e., the toroidal surface) of the double-layer bowl mold. The binding data includes the first texture map, the mesh table of the ground plane layer of the double-layer bowl mold, the second texture map, and the mesh table of the moving object layer of the double-layer bowl mold.

[0164] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0165] In one embodiment, an image processing device is provided, which corresponds one-to-one to the image processing method in the above embodiment. Figure 6 As shown, the image processing device comprises:

[0166] The image segmentation module 10 is used to segment the intersection area into a background area image and a foreground area image when a foreground object is detected in the intersection area of ​​the vehicle-mounted image;

[0167] A background projection module 20 is used to project the background area image onto a reference plane to obtain a background projection image;

[0168] A foreground projection module 30 is used to project the foreground area image onto the annular surface to obtain a foreground projection image;

[0169] The rendering module 40 is used to generate a three-dimensional surround image based on the background projection image and the foreground projection image.

[0170] Optionally, the image processing device further includes a module for acquiring vehicle-mounted images, and the module for acquiring images includes:

[0171] A mapping table creation unit, used to create a transformation mapping table according to the first transformation matrix;

[0172] The transformation unit is used to perform a distortion correction transformation on the original image based on the transformation mapping table to obtain a vehicle-mounted image.

[0173] Optionally, the module for acquiring the vehicle-mounted image further includes:

[0174] A calibration image acquisition unit is used to acquire calibration image data of multiple channels;

[0175] A first matrix determination unit is used to determine a first transformation matrix according to the calibration image data.

[0176] Optionally, the calibration image data includes a plurality of images including a calibration plate; the calibration plate includes a circular ring pattern;

[0177] Determining a first matrix unit includes:

[0178] A coordinate extraction unit, used for extracting a plurality of circle center coordinates and a plurality of corner point coordinates from the calibration image data;

[0179] An internal parameter matrix determination unit is used to determine the focal length of a camera and the pixel coordinates of the camera according to a plurality of circle center coordinates, and to determine the internal parameter matrix according to the focal length of the camera and the pixel coordinates;

[0180] A first matrix unit is generated, which is used to determine a first transformation matrix according to the coordinates of the multiple corner points and the intrinsic parameter matrix.

[0181] Optionally, the image segmentation module 10 includes:

[0182] An inter-frame motion vector unit, used for calculating the inter-frame motion vectors of adjacent frames in the intersection area;

[0183] The image segmentation unit is used to segment the intersection area according to the motion vectors between adjacent frames to generate a background area image and a foreground area image.

[0184] Optionally, the inter-frame motion vector unit is further configured to:

[0185] The motion vectors between adjacent frames in the intersection area are calculated using the optical flow method.

[0186] Optionally, the image segmentation unit is further used for:

[0187] Mark pixels whose norm of motion vectors between adjacent frames is greater than a preset motion threshold as foreground area images;

[0188] Pixels whose norm of motion vectors between adjacent frames is less than or equal to a preset motion threshold are marked as background area images.

[0189] Optionally, the image processing device further includes a model setting module, and the model setting module includes:

[0190] A reference plane setting unit is used to set the ground plane as a reference plane in a double-layer bowl mold;

[0191] The toroidal surface unit is used to set the toroidal surface according to the bottom surface parameters and the top surface parameters.

[0192] Optionally, the foreground projection module is further used for:

[0193] The foreground area image is projected onto the annular surface according to the second transformation matrix to obtain a foreground projection image.

[0194] Optionally, the foreground projection module further includes determining a second matrix unit, and the determining second matrix unit is used for:

[0195] Acquire two frames of foreground area images; source channels of the two frames of foreground area images are adjacent;

[0196] Extracting a set of feature point pairs from the foreground area image;

[0197] Filtering out the surface feature point pairs on the toroidal surface from the feature point pair set;

[0198] A second transformation matrix for projecting the foreground object onto the annular surface is determined according to the surface feature point pairs.

[0199] Optionally, surface feature point pairs on the toroidal surface are selected from the feature point pair set, including:

[0200] Calculate the distance between each feature point pair in the feature point pair set and the torus surface;

[0201] The feature point pairs whose distance is less than the preset difference value are determined as the surface feature point pairs.

[0202] Optionally, before filtering out the surface feature point pairs on the toroidal surface from the feature point pair set, the method further includes:

[0203] Get the distance between the foreground object and the current vehicle;

[0204] Adjusts the radius parameter of the loop surface based on the distance.

[0205] Optionally, the rendering module 40 includes:

[0206] A binding data unit, used for binding the background projection image and the foreground projection image with the grid table to generate binding data;

[0207] The rendering unit is used to render the bound data to generate a three-dimensional surround image.

[0208] Optionally, the binding data unit includes:

[0209] A first binding unit is used to bind the background projection image as a first texture map to a grid table of a ground plane layer of the double-layer bowl mold;

[0210] The second binding unit is used to bind the foreground projection image as the second texture map to the grid table of the moving object layer of the double-layer bowl mold.

[0211] For the specific definition of the image processing device, please refer to the definition of the image processing method above, which will not be repeated here. Each module in the above image processing device can be implemented in whole or in part by software, hardware and a combination thereof. The above modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0212] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the image processing method in the above embodiment is implemented, for example Figure 1 To avoid repetition, S10-S40 are not described here. Alternatively, when the processor executes the computer program, the functions of each module / unit in this embodiment of the image processing device are realized, for example Figure 6The functions of the modules shown are not described here in detail to avoid repetition.

[0213] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the image processing method in the above embodiment is implemented, for example Figure 1 S10-S40 are shown, and will not be described here in detail to avoid repetition. Alternatively, when the computer program is executed by the processor, the functions of each module / unit in the above-mentioned image processing device in this embodiment are realized, for example Figure 6 The functions of the modules shown are not described here in detail to avoid repetition.

[0214] A vehicle includes a vehicle controller, wherein the vehicle controller is used to execute any one of the above-mentioned image processing methods.

[0215] A computer program product comprises a computer program, wherein when the computer program is executed by a processor, any one of the above-mentioned image processing methods is implemented.

[0216] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0217] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0218] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention.

Claims

1. An image processing method, characterized in that: include: When it is detected that there is a foreground object in the intersection area of ​​the vehicle-mounted image, the intersection area is segmented into a background area image and a foreground area image; The intersection area refers to the intersection area of ​​the vehicle-borne images of adjacent channels; Projecting the background area image onto a reference plane to obtain a background projection image; Projecting the foreground area image onto a torus surface to obtain a foreground projection image; Generate a three-dimensional surround image based on the background projection image and the foreground projection image; Before projecting the background area image onto the reference plane to obtain the background projection image, the method further includes: In the double-layer bowl mold, the ground plane is set as the reference plane; The toroidal surface is set according to bottom surface parameters and top surface parameters.

2. The image processing method according to claim 1, characterized in that: Before detecting that a foreground object exists in the intersection area of ​​the vehicle-mounted image, the method further includes: Creating a transformation mapping table according to the first transformation matrix; The original image is subjected to a distortion correction transformation based on the transformation mapping table to obtain the vehicle-mounted image.

3. The image processing method according to claim 2, characterized in that: Before creating the transformation mapping table according to the first transformation matrix, the method further includes: Obtain calibration image data of multiple channels; The first transformation matrix is ​​determined according to the calibration image data.

4. The image processing method according to claim 3, characterized in that: The calibration image data includes a plurality of images containing a calibration plate; the calibration plate includes a circular ring pattern; The determining the first transformation matrix according to the calibration image data comprises: Extracting a plurality of circle center coordinates and a plurality of corner point coordinates from the calibration image data; Determine a camera focal length and a camera pixel coordinate according to the multiple circle center coordinates, and determine an intrinsic parameter matrix according to the camera focal length and the pixel coordinates; The first transformation matrix is ​​determined according to the multiple corner point coordinates and the intrinsic parameter matrix.

5. The image processing method according to claim 1, characterized in that: The step of dividing the intersection area into a background area image and a foreground area image comprises: Calculating the adjacent inter-frame motion vectors of the intersection area; The intersection area is segmented according to the motion vectors between adjacent frames to generate the background area image and the foreground area image.

6. The image processing method according to claim 5, characterized in that: The calculating the adjacent inter-frame motion vectors of the intersection area comprises: The motion vectors between adjacent frames of the intersection area are calculated by the optical flow method.

7. The image processing method according to claim 5, characterized in that: The step of segmenting the intersection area according to the motion vectors between adjacent frames to generate the background area image and the foreground area image comprises: Marking pixels whose norm of motion vectors between adjacent frames is greater than a preset motion threshold as the foreground area image; Pixels whose norms of motion vectors between adjacent frames are less than or equal to the preset motion threshold are marked as the background area image.

8. The image processing method according to claim 1, characterized in that: The step of projecting the foreground area image onto a torus surface to obtain a foreground projection image comprises: The foreground area image is projected onto a torus according to a second transformation matrix to obtain the foreground projection image.

9. The image processing method according to claim 8, characterized in that: Before projecting the foreground area image onto the toroidal surface according to the second transformation matrix to obtain the foreground projection image, the method includes: Acquire two frames of foreground area images; source channels of the two frames of foreground area images are adjacent; Extracting a set of feature point pairs from the foreground area image; Filtering out the surface feature point pairs on the annular surface from the feature point pair set; The second transformation matrix for projecting the foreground object onto the annular surface is determined according to the surface feature point pair.

10. The image processing method according to claim 9, characterized in that: The step of selecting the surface feature point pairs on the annular surface from the feature point pair set includes: Calculating the distance between each feature point pair in the feature point pair set and the annular surface; The feature point pair whose distance is less than the preset difference value is determined as the surface feature point pair.

11. The image processing method according to claim 9, characterized in that: Before selecting the surface feature point pairs on the annular surface from the feature point pair set, the method further includes: Obtaining the distance between the foreground object and the current vehicle; The radius parameter of the annular surface is adjusted according to the distance.

12. The image processing method according to claim 1, characterized in that: The step of generating a three-dimensional surround view image based on the background projection image and the foreground projection image comprises: Binding the background projection image and the foreground projection image to a grid table to generate binding data; The binding data is rendered to generate a three-dimensional surround image.

13. The image processing method according to claim 12, characterized in that: The step of binding the background projection image and the foreground projection image to a grid table to generate binding data includes: Binding the background projection image as a first texture map to a grid table of a ground plane layer of a double-layer bowl mold; The foreground projection image is bound as a second texture map to the grid table of the moving object layer of the double-layer bowl mold.

14. An image processing device, characterized in that: include: An image segmentation module, for segmenting the intersection area into a background area image and a foreground area image when a foreground object is detected in the intersection area of ​​the vehicle-borne image; the intersection area refers to the intersection area of ​​the vehicle-borne images of adjacent channels; A background projection module, used for projecting the background area image onto a reference plane to obtain a background projection image; A foreground projection module, used for projecting the foreground area image onto a torus surface to obtain a foreground projection image; A rendering module for generating a three-dimensional surround image based on the background projection image and the foreground projection image; A model setting module, wherein the model setting module comprises: A reference plane setting unit is used to set the ground plane as a reference plane in a double-layer bowl mold; A toroidal surface unit is provided, which is used to set the toroidal surface according to bottom surface parameters and top surface parameters.

15. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the image processing method according to any one of claims 1 to 13 is implemented.

16. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the image processing method according to any one of claims 1 to 13 is implemented.

17. A vehicle, characterized in that: It comprises a vehicle controller, wherein the vehicle controller is used to execute the image processing method according to any one of claims 1 to 13.

18. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the image processing method according to any one of claims 1 to 13 is implemented.

Images