Method and system for generating in-vehicle surround-view panoramic image

Abstract

Provided in the present application is a method for generating an in-vehicle surround-view panoramic image. The method comprises: acquiring initial in-vehicle fisheye images corresponding to an in-vehicle observation angle; inputting the in-vehicle observation angle into an in-vehicle surround-view model for processing, so as to generate a plurality of fusion region weights corresponding to the initial in-vehicle fisheye images; on the basis of the plurality of fusion region weights, performing mapping processing on the corresponding initial in-vehicle fisheye images, so as to generate a target in-vehicle fisheye image; and performing real-time rendering processing on the target in-vehicle fisheye image, so as to generate an in-vehicle surround-view panoramic image.

Description

A method and system for generating a vehicle-mounted surround view panoramic image Cross-reference to Related Applications

[0001] This application claims priority to the Chinese patent application No. 2024107392125, filed on June 7, 2024, with the Chinese Patent Office, the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0002] Embodiments of the present application relate to, but are not limited to, the image processing technical field, and in particular, relate to a method and system for generating a vehicle-mounted surround view panoramic image. BACKGROUND

[0003] In recent years, with the rapid development of the automobile industry, more and more private cars have become an essential means of transportation for people. With the rapid development of automobile safety technology, the vehicle-mounted panoramic image system has become an essential configuration for cars. SUMMARY

[0004] The purpose of the present application is to provide a method and system for generating a vehicle-mounted surround view panoramic image. The following is a summary of the subject matter described in detail in the present application. This summary is not intended to limit the scope of protection of the claims.

[0005] The present application provides a method for generating a vehicle-mounted surround view panoramic image, comprising: obtaining an initial vehicle-mounted fisheye image corresponding to a vehicle-mounted observation angle; inputting the vehicle-mounted observation angle into a vehicle-mounted surround view model for processing to generate a plurality of fusion region weights corresponding to the initial vehicle-mounted fisheye image; mapping and processing the corresponding initial vehicle-mounted fisheye image according to the plurality of fusion region weights to generate a target vehicle-mounted fisheye image; performing real-time rendering processing on the target vehicle-mounted fisheye image to generate a vehicle-mounted surround view panoramic image.

[0006] In an embodiment of the present application, the obtaining of the initial vehicle-mounted fisheye image corresponding to the vehicle-mounted observation angle comprises: determining a boundary ray vertex coordinate according to the vehicle-mounted observation angle; obtaining an initial vehicle-mounted fisheye image corresponding to the vehicle-mounted observation angle, and the initial vehicle-mounted fisheye image comprises initial vehicle-mounted fisheye images collected by fisheye cameras in at least two directions around the vehicle body.

[0007] In an embodiment of the present application, the plurality of fusion region weights includes a first fusion weight and a second fusion weight, the first fusion weight and the second fusion weight are 1, and the processing of the vehicle-mounted observation view in the vehicle-mounted surround view model to generate the plurality of fusion region weights corresponding to the initial vehicle-mounted fisheye image includes: processing the vehicle-mounted observation view and boundary ray vertex coordinates according to the vehicle-mounted surround view model to generate a first boundary ray and a second boundary ray, a boundary ray included angle and a region to be fused being formed between the first boundary ray and the second boundary ray; obtaining fixed point coordinates according to the vehicle-mounted observation view, and processing the fixed point coordinates according to the first boundary ray and the boundary ray vertex coordinates to generate a fixed point included angle; processing the fixed point included angle and the boundary ray included angle based on the positional relationship between the fixed point coordinates and the boundary ray included angle to generate the first fusion weight and the second fusion weight.

[0008] In an embodiment of the present application, the processing of the vehicle-mounted observation view and boundary ray vertex coordinates according to the vehicle-mounted surround view model to generate a first boundary ray and a second boundary ray, a boundary ray included angle and a region to be fused being formed between the first boundary ray and the second boundary ray includes: performing function mapping processing on the vehicle-mounted observation view to generate a target boundary ray influence coefficient; performing ray slope function processing on the target boundary ray influence coefficient based on the length of the vehicle body and the width of the vehicle body to generate a ray slope data set, the ray slope data set including a first ray slope and a second ray slope; performing ray generation processing on the boundary ray vertex coordinates according to the first ray slope and the second ray slope, respectively, to correspondingly generate a first boundary ray and a second boundary ray; wherein a boundary ray included angle is formed between the first boundary ray and the second boundary ray, and the region corresponding to the boundary ray included angle is a region to be fused.

[0009] In an embodiment of the present application, the function mapping processing on the vehicle-mounted observation view to generate a target boundary ray influence coefficient includes: performing initial function mapping processing on the vehicle-mounted observation view to generate a first influence coefficient; performing intermediate function mapping processing on the first influence coefficient to generate a second influence coefficient; and performing target function mapping processing on the second influence coefficient to generate a target boundary influence coefficient.

[0010] In an embodiment of the present application, the first fusion weight and the second fusion weight are generated by processing the fixed point angle and the boundary ray angle based on the positional relationship between the fixed point coordinate and the boundary ray angle, including: determining whether the fixed point coordinate is located within the boundary ray angle; if the fixed point coordinate is located within the boundary ray angle, determining the first fusion weight according to the proportion of the fixed point angle in the boundary ray angle, and determining the second fusion weight according to the first fusion weight; if the fixed point coordinate is located outside the boundary ray angle, determining whether the fixed point coordinate is close to the first boundary ray; if the fixed point coordinate is close to the first boundary ray, determining the first fusion weight as 1 and the second fusion weight as 0; if the fixed point coordinate is close to the second boundary ray, determining the first fusion weight as 0 and the second fusion weight as 1.

[0011] In an embodiment of the present application, the target vehicle-mounted fisheye image is generated by performing mapping processing on the initial vehicle-mounted fisheye image according to the plurality of fusion region weights, including: performing clipping processing on the initial vehicle-mounted fisheye image in a corresponding direction according to the first fusion weight and the second fusion weight, to generate a plurality of to-be-fused vehicle-mounted fisheye images; performing extrinsic parameter transformation processing on the plurality of to-be-fused vehicle-mounted fisheye images, to correspondingly generate a plurality of extrinsic parameter transformation vehicle-mounted fisheye images; performing intrinsic parameter mapping processing on the plurality of extrinsic parameter transformation vehicle-mounted fisheye images, to correspondingly generate a plurality of intermediate vehicle-mounted fisheye images; and performing weighted fusion processing on the plurality of intermediate vehicle-mounted fisheye images according to the first fusion weight and the second fusion weight, to generate the target vehicle-mounted fisheye image.

[0012] In an embodiment of the present application, the extrinsic parameter transformation processing on the plurality of to-be-fused vehicle-mounted fisheye images to correspondingly generate the plurality of extrinsic parameter transformation vehicle-mounted fisheye images includes: performing extrinsic parameter rotation matrix processing on the plurality of to-be-fused vehicle-mounted fisheye images in a corresponding direction, to generate a plurality of extrinsic parameter rotation fisheye images; and performing extrinsic parameter translation matrix processing on the plurality of extrinsic parameter rotation fisheye images in a corresponding direction, to generate the plurality of extrinsic parameter transformation vehicle-mounted fisheye images.

[0013] In an embodiment of the present application, the intrinsic parameter mapping processing on the plurality of extrinsic parameter transformation vehicle-mounted fisheye images to correspondingly generate the plurality of intermediate vehicle-mounted fisheye images includes: performing intrinsic parameter transformation processing on the plurality of extrinsic parameter transformation vehicle-mounted fisheye images in a corresponding direction, to generate a plurality of corresponding intrinsic fisheye images; and performing de-distortion mapping processing on the plurality of intrinsic fisheye images in a corresponding direction, to generate the plurality of intermediate vehicle-mounted fisheye images.

[0014] The application further provides a vehicle-mounted surround view panoramic image generation system, comprising: an information acquisition module configured to acquire an initial vehicle-mounted fisheye image corresponding to a vehicle-mounted observation angle; a model generation module configured to input the vehicle-mounted observation angle into a vehicle-mounted surround view model for processing to generate a plurality of fusion region weights corresponding to the initial vehicle-mounted fisheye image; a fusion mapping module configured to map and process the initial vehicle-mounted fisheye image corresponding to the plurality of fusion region weights to generate a target vehicle-mounted fisheye image; and a rendering processing module configured to perform real-time rendering processing on the target vehicle-mounted fisheye image to generate a vehicle-mounted surround view panoramic image.

[0015] As described above, the application provides a vehicle-mounted surround view panoramic image generation method and system, which can avoid problems such as misplacement or ghosting in the fusion region of the vehicle-mounted surround view panoramic image, improve the clarity of the vehicle-mounted surround view panoramic image, and further improve the user experience.

[0016] Other aspects can become apparent from a review of the drawings and detailed description. BRIEF DESCRIPTION OF DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the application, the following will briefly introduce the drawings needed to be used in the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the application, and for those skilled in the art, other drawings can also be obtained from these drawings without creative labor.

[0018] FIG. 1 is a flowchart of a vehicle-mounted surround view panoramic image generation method according to an example embodiment of the application.

[0019] FIG. 2 is a flowchart of step S1010 in the embodiment shown in FIG. 1 according to an example embodiment.

[0020] FIG. 3 is a schematic diagram of the vehicle body coordinate system in FIG. 2.

[0021] FIG. 4 is a flowchart of step S200 in the embodiment shown in FIG. 1 according to an example embodiment.

[0022] FIG. 5 is a flowchart of step S220 in the embodiment shown in FIG. 4 according to an example embodiment.

[0023] FIG. 6 is a flowchart of step S221 in the embodiment shown in FIG. 5 according to an example embodiment.

[0024] FIG. 7 is a flowchart of step S240 in the embodiment shown in FIG. 4 according to an example embodiment.

[0025] FIG. 8 is a schematic diagram of the to-be-fused region generated in FIG. 5.

[0026] Fig. 9 shows a schematic diagram of the fixed point coordinates in Fig. 7 being located in the region to be fused.

[0027] Fig. 10 is a flow chart of step S300 in the embodiment shown in Fig. 1 in an exemplary embodiment.

[0028] Fig. 11 is a flow chart of step S320 in the embodiment shown in Fig. 10 in an exemplary embodiment.

[0029] Fig. 12 is a flow chart of step S330 in the embodiment shown in Fig. 10 in an exemplary embodiment.

[0030] Fig. 13 is a schematic diagram of a system for generating a vehicle-mounted surround view panoramic image in the embodiment shown in Fig. 1. DETAILED DESCRIPTION

[0031] The present application is described herein with reference to particular specific embodiments, and the description of these embodiments is not intended to limit the scope of the application. Persons skilled in the art will readily recognize that the application can be practiced with other embodiments, and that the scope of the application is not limited to the embodiments described and shown herein. The description of the embodiments is merely intended to illustrate the principles of the application, and the scope of the application is not limited to the embodiments described and shown herein. The features of the embodiments described and shown herein can be combined with each other, provided that there is no conflict. It should be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of the application. The processing methods described in the following embodiments are generally performed under conventional conditions or under conditions recommended by the manufacturers, unless otherwise specified.

[0032] Please refer to Figs. 1 to 13. It should be understood that the structures, proportions, sizes, etc. shown in the drawings of the present specification are merely intended to illustrate the content disclosed in the present specification, to be understood and read by those skilled in the art, and do not define the limiting conditions for the implementation of the present application, and therefore do not have technical significance. Any modification of the structure, change of the proportional relationship, or adjustment of the size, without affecting the effects and purposes that can be achieved by the present application, should still fall within the scope of the technical content disclosed by the present application. Meanwhile, the terms such as "upper", "lower", "left", "right", "middle", and "one" used in the present specification are merely for the purpose of clear understanding and description, and are not intended to limit the scope of the implementation of the present application. The change or adjustment of the relative relationship, without substantially changing the technical content, is also considered as the scope of the implementation of the present application.

[0033] Due to the lack of camera depth information and the inherent error of calibration, the generated vehicle surround view panoramic image may have obvious misplacement or ghosting, affecting the clarity of the panoramic image and reducing user experience. Therefore, the present application provides a vehicle surround view panoramic image generation method and system, which relates to the field of image processing technology, and can be specifically applied to real-time generation of vehicle surround view panoramic images of a car, avoiding obvious misplacement and ghosting in the vehicle surround view panoramic image, improving the clarity of the vehicle surround view panoramic image, and thus improving user experience.

[0034] Please refer to FIG. 1, which shows a flowchart of a vehicle surround view panoramic image generation method provided by the present application. In an embodiment of the present application, the vehicle surround view panoramic image generation method can include the following steps.

[0035] Step S100, obtaining an initial vehicle fisheye image corresponding to a vehicle observation viewing angle.

[0036] Step S200, inputting the vehicle observation viewing angle into a vehicle surround view model for processing to generate a plurality of fusion region weights corresponding to the initial vehicle fisheye image.

[0037] Step S300, mapping and processing the corresponding initial vehicle fisheye image according to the plurality of fusion region weights to generate a target vehicle fisheye image.

[0038] Step S400, performing real-time rendering processing on the target vehicle fisheye image to generate a vehicle surround view panoramic image.

[0039] Please refer to FIG. 2 and FIG. 3, in an embodiment of the present application, when step S100 is performed, i.e. obtaining an initial vehicle fisheye image corresponding to a vehicle observation viewing angle. Specifically, step S100 can include steps S110 to S120, which are described in detail as follows.

[0040] Step S110, determining the boundary ray vertex coordinates according to the vehicle observation viewing angle.

[0041] Step S120, obtaining an initial vehicle fisheye image corresponding to the vehicle observation viewing angle, and the initial vehicle fisheye image includes initial vehicle fisheye images collected by fisheye cameras in at least two directions around the car body.

[0042] In an embodiment of the present application, when step S110 is performed, the vehicle-mounted observation view angle can be obtained according to manual selection by a user, and the range of the vehicle-mounted observation view angle can be greater than or equal to 0° and less than or equal to 360°. That is, the vehicle-mounted observation view angle can be any angle around the vehicle body. For example, referring to FIG. 3, the vehicle-mounted observation view angle of the front of the vehicle body can be defined as 0°, and the vehicle-mounted observation view angle increases counterclockwise with the direction of the top view of the vehicle body as the standard. That is, with the direction of the top view of the vehicle body as the standard, the vehicle-mounted observation view angle of the left view angle directly opposite the vehicle body is 90°, the vehicle-mounted observation view angle of the rear view angle directly opposite the vehicle body is 180°, and the vehicle-mounted observation view angle of the right view angle directly opposite the vehicle body is 270°.

[0043] In addition, a vehicle body coordinate system can be established with the center of the vehicle body as the origin, the direction of the front of the vehicle body as the positive X-axis, and the direction of the right of the vehicle body as the positive Y-axis. The boundary ray vertex coordinate can be the coordinate of one of the four vertices of the vehicle body in the vehicle body coordinate system, and the boundary ray vertex coordinate can be determined according to the change of the vehicle-mounted observation view angle. For example, referring to FIG. 3, when the vehicle-mounted observation view angle is 90°, the boundary ray vertex coordinate can be the coordinate of the front left vertex of the vehicle body in the vehicle body coordinate system. Of course, this is not limited, and in addition to the vehicle-mounted observation view angle, the length and width of the vehicle body can also be obtained, and the length and width of the vehicle body can be preset values, and the length and width of the vehicle body can not be specifically limited. For example, the length of the vehicle body can be 4770 mm, and the width of the vehicle body can be 1895 mm.

[0044] In an embodiment of the present application, when step S120 is performed, the initial vehicle-mounted fisheye image of different directions of the vehicle body can be determined according to the vehicle-mounted observation view angle, and the initial vehicle-mounted fisheye image can include at least two initial vehicle-mounted fisheye images collected by fisheye cameras in different directions around the vehicle body, and the initial vehicle-mounted fisheye image can be collected and obtained by the fisheye cameras in different directions around the vehicle body. For example, the initial vehicle-mounted fisheye images in different directions around the vehicle body can be collected and obtained by fisheye cameras in front, rear, left, and right directions of the vehicle body. When the vehicle-mounted observation view angle is between 0° and 90°, the initial vehicle-mounted fisheye image can include an initial vehicle-mounted fisheye image collected and obtained by a front fisheye camera and an initial vehicle-mounted fisheye image collected and obtained by a left fisheye camera. In addition, the camera angle of the fisheye camera can be between 150° and 180°.

[0045] Referring to FIG. 4, in an embodiment of the present application, when step S200 is performed, the vehicle-mounted observation view angle is input into the vehicle-mounted surround view model for processing to generate a plurality of fusion region weights corresponding to the initial vehicle-mounted fisheye image. Specifically, step S200 can include steps S210 to S240, which are described in detail as follows.

[0046] Step S210, acquire the vehicle-mounted surround view model.

[0047] Step S220, process the vehicle-mounted observation view angle and the boundary ray vertex coordinates according to the vehicle-mounted surround view model, generate the first boundary ray and the second boundary ray, and form the boundary ray included angle and the to-be-fused region between the first boundary ray and the second boundary ray.

[0048] Step S230, acquire the fixed point coordinates according to the vehicle-mounted observation view angle, and process the fixed point coordinates according to the first boundary ray and the boundary ray vertex coordinates, to generate the fixed point included angle.

[0049] Step S240, process the fixed point included angle and the boundary ray included angle based on the positional relationship between the fixed point coordinates and the boundary ray included angle, to generate the first fusion weight and the second fusion weight.

[0050] In an embodiment of the present application, when step S210 is performed, the vehicle-mounted surround view model can be a bowl-shaped model, the bottom of which is circular, and the bowl wall is a quadratic curve. For example, refer to FIG. 3, the top view of the vehicle-mounted surround view model is circular, and a vehicle body coordinate system can be established with the center of the vehicle body as the origin, the direction of the front of the vehicle body as the X-axis, and the direction of the right of the vehicle body as the Y-axis. The quadratic curve has the feature that the central part of the view is wide, and gradually narrows when extending upwards. This design can better reflect the distance sense and depth of field of the environment around the vehicle.

[0051] Refer to FIG. 5, in an embodiment of the present application, when step S220 is performed, that is, the vehicle-mounted observation view angle and the boundary ray vertex coordinates are processed according to the vehicle-mounted surround view model, to generate the first boundary ray and the second boundary ray, and form the boundary ray included angle and the to-be-fused region between the first boundary ray and the second boundary ray. Specifically, step S220 can include steps S221 to S223, which are described in detail as follows.

[0052] Step S221, perform function mapping processing on the vehicle-mounted observation view angle, to generate a target boundary ray influence coefficient.

[0053] Step S222, perform ray slope function processing on the target boundary ray influence coefficient based on the length of the vehicle body and the width of the vehicle body, to generate a ray slope data set, which includes a first ray slope and a second ray slope.

[0054] Step S223, perform ray generation processing on the boundary ray vertex coordinates according to the first ray slope and the second ray slope, respectively, to correspondingly generate the first boundary ray and the second boundary ray, wherein the boundary ray included angle is formed between the first boundary ray and the second boundary ray, and the region corresponding to the boundary ray included angle is the to-be-fused region.

[0055] Please refer to FIG. 5 and FIG. 6, in one embodiment of the present application, when step S221 is performed, the function mapping processing is performed on the vehicle-mounted observation visual angle to generate the target boundary ray influence coefficient. Specifically, step S221 can include steps S2211 to S2213, which are described in detail as follows.

[0056] Step S2211, the initial function mapping processing is performed on the vehicle-mounted observation visual angle to generate the first influence coefficient.

[0057] Step S2212, the intermediate function mapping processing is performed on the first influence coefficient to generate the second influence coefficient.

[0058] Step S2213, the target function mapping processing is performed on the second influence coefficient to generate the target boundary influence coefficient.

[0059] In one embodiment of the present application, when step S2211 is performed, the initial function mapping processing is performed on the vehicle-mounted observation visual angle to generate the first influence coefficient, so as to map the vehicle-mounted observation visual angle from 0°-360° to a specific numerical value. The initial function mapping processing performed on the vehicle-mounted observation visual angle can satisfy the following formula (1), factor1 = ||yaw-180.0|-90.0| / 90.0-0.5 (1).

[0060] Wherein, yaw can represent the vehicle-mounted observation visual angle, yaw is greater than or equal to 0° and less than or equal to 360°, and factor1 can represent the first influence coefficient. Of course, it is not limited to this, the vehicle-mounted observation visual angle can also satisfy other formulas.

[0061] In one embodiment of the present application, when step S2212 is performed, the intermediate function mapping processing is performed on the first influence coefficient to generate the second influence coefficient, that is, the second influence coefficient can be determined by the first influence. The intermediate function mapping processing performed on the first influence coefficient can satisfy the following formula (2),

[0062] Wherein, factor1 can represent the first influence coefficient, factor2 can represent the second influence coefficient, * can represent the multiplication symbol, and 0.5 can be an empirical value. That is, the intermediate function mapping processing performed on the first influence coefficient can also satisfy other formulas, as long as it can satisfy that when yaw is in the interval of 0°-90° or 180°-270°, factor2 can change from 1 to 0, and when yaw is in the interval of 90°-180° or 270°-360°, factor2 can change from 0 to 1.

[0063] In one embodiment of the present application, when step S2213 is performed, the target mapping function processing on the second influence coefficient can satisfy the following formula (3), kScale = 0.5 - factor2 * 0.375 (3).

[0064] Wherein, factor2 can represent the second influence coefficient, * can represent the multiplication symbol, kScale can represent the target boundary influence coefficient, and 0.5 and 0.375 can represent empirical values for adjusting the target boundary influence coefficient, that is, 0.5 and 0.375 can also be replaced by other numbers.

[0065] Referring to FIGS. 5, 6, 8 and 9, in one embodiment of the present application, when step S222 is performed, the vehicle body length and the vehicle body width are preset values, which can vary according to different vehicle models. Wherein, the ray slope function processing on the target boundary ray influence coefficient based on the vehicle body length and the vehicle body width can include but is not limited to first ray slope processing on the vehicle body length, the vehicle body width and the target boundary ray influence coefficient to generate a first ray slope. Then, coefficient multiplication processing is performed on the first ray slope to generate a second ray slope.

[0066] Further, the first ray slope can satisfy the following formula (4), K1 = (L / W) * kScale (4).

[0067] Wherein, L can represent the vehicle body length, W can represent the vehicle body width, * can represent the multiplication symbol, kScale can represent the target boundary influence coefficient, and K1 can represent the first ray slope.

[0068] Further, the coefficient multiplication processing on the first ray slope to generate the second ray slope can satisfy the following formula (5), K2 = t * K1 (5).

[0069] Wherein, K1 can represent the first ray slope, t can represent a constant coefficient, and t can be 1.5 or other numerical values, and K2 can represent the second ray slope.

[0070] Referring to FIGS. 5, 6, 8 and 9, in one embodiment of the present application, when step S223 is performed, the ray generation processing on the boundary ray vertex coordinates according to the first ray slope and the second ray slope respectively means that the boundary ray vertex coordinates are taken as the starting point, the Y-axis of the vehicle body coordinate system is taken as the basis line, and the boundary ray is drawn according to the first ray slope and the second ray slope to correspondingly generate the first boundary ray and the second boundary ray.

[0071] The first boundary ray and the second boundary ray can form a boundary ray included angle on the same plane, and a region corresponding to the boundary ray included angle can be defined as the to-be-fused region. For example, referring to FIG. 8, when the vehicle-mounted observation visual angle is 180°, the to-be-fused region can be A', and when the vehicle-mounted observation visual angle is 270°, the to-be-fused region can be B'. In addition, when the observation visual angle is converted from the vehicle body rear visual angle 180° to the vehicle body right visual angle 270°, the initial vehicle-mounted fisheye image on the left side of the vehicle body and the initial vehicle-mounted fisheye image on the front side of the vehicle body can be selected, and the to-be-fused region corresponding to the generated to-be-fused region can be adjusted from A' to B'. Referring to FIG. 9, the first boundary ray can be K1, the second boundary ray can be K2, the region formed by K1 and K2 can be the to-be-fused region, the boundary ray vertex coordinate O can be the vertex coordinate of the left front of the vehicle body, and the to-be-fused region formed by the second boundary ray K2 can fall on the left front of the vehicle body.

[0072] Referring to FIGS. 3 and 9, in an embodiment of the present application, when step S230 is performed, the corresponding fixed point coordinate can be obtained according to the vehicle-mounted observation visual angle, and the fixed point coordinate can be a preset arbitrary point coordinate in the vehicle-mounted surround view model, which can fall in the plane formed by the first boundary ray and the second boundary ray. For example, referring to FIG. 9, the fixed point can be defined as point C. Processing the fixed point coordinate according to the first boundary ray and the boundary ray vertex coordinate means that a ray can be emitted from the boundary ray vertex coordinate to the fixed point coordinate to form a ray between the fixed point coordinate and the boundary ray vertex coordinate, and then the included angle between the ray formed between the fixed point coordinate and the boundary ray vertex coordinate and the first boundary ray can be defined as the fixed point included angle theta.

[0073] Referring to FIGS. 3 and 7, in an embodiment of the present application, when step S240 is performed, that is, the fixed point included angle and the boundary ray included angle are processed based on the positional relationship between the fixed point included angle and the boundary ray included angle, the first fusion weight and the second fusion weight are generated. Specifically, step S240 can include steps S241 to S245, which are described in detail as follows.

[0074] Step S241, determining whether the fixed point coordinate is located in the boundary ray included angle.

[0075] Step S242, if the fixed point coordinate is located in the boundary ray included angle, determining the first fusion weight according to the proportion of the fixed point included angle in the boundary ray included angle, and determining the second fusion weight according to the first fusion weight.

[0076] Step S243, if the fixed point coordinate is located outside the boundary ray included angle, determining whether the fixed point coordinate is close to the first boundary ray.

[0077] Step S244, if the fixed point coordinate is close to the first boundary ray, the first fusion weight is determined as 1 and the second fusion weight is determined as 0.

[0078] Step S245, if the fixed point coordinate is close to the second boundary ray, the first fusion weight is determined as 0 and the second fusion weight is determined as 1.

[0079] Referring to FIG. 3, in an embodiment of the present application, when step S240 is performed, the plurality of fusion region weights can include a first fusion weight and a second fusion weight, and the first fusion weight and the second fusion weight are equal to 1. The first fusion weight can represent a proportion of the initial vehicle-mounted fisheye image on the left side of the vehicle-mounted observation visual angle in the to-be-fused region, and the second fusion weight can represent a proportion of the initial vehicle-mounted fisheye image on the right side of the vehicle-mounted observation visual angle in the to-be-fused region. Of course, the first fusion weight can also represent a proportion of the initial vehicle-mounted fisheye image on the right side of the vehicle-mounted observation visual angle in the to-be-fused region, and the second fusion weight can correspondingly represent a proportion of the initial vehicle-mounted fisheye image on the left side of the vehicle-mounted observation visual angle in the to-be-fused region. For example, referring to FIG. 8, when the boundary ray vertex coordinate is the vertex coordinate of the left front side of the vehicle body and the to-be-fused region formed by the first boundary ray and the second boundary ray falls on the left front side of the vehicle body, the first fusion weight can represent a proportion of the initial vehicle-mounted fisheye image on the left side of the vehicle body in the to-be-fused region. The second fusion weight can represent a proportion of the initial vehicle-mounted fisheye image on the front side of the vehicle body in the to-be-fused region, and the first fusion weight and the second fusion weight are equal to 1.

[0080] Further, the first fusion weight and the second fusion weight can be determined by the following method. First, it can be judged whether the fixed point coordinate is located in the boundary ray included angle. If the fixed point coordinate is located in the boundary ray included angle, the first fusion weight can be determined according to a proportion of the fixed point included angle in the boundary ray included angle, and the second fusion weight can be determined according to the first fusion weight. When the fixed point coordinate is located in the boundary ray included angle, the first fusion weight can be a ratio between the fixed point included angle and the boundary ray included angle, and the second fusion weight can be determined by subtracting the first fusion weight from 1.

[0081] Further, if the fixed point coordinate is located outside the boundary ray angle, it is needed to determine whether the fixed point coordinate is close to the first boundary ray to further determine the first fusion weight and the second fusion weight. Wherein, if the fixed point coordinate is close to the first boundary ray, the first fusion weight is determined as 1 and the second fusion weight is determined as 0. That is, it is explained that the initial vehicle-mounted fisheye images on both sides of the vehicle-mounted observation angle do not need to be fused, and the initial vehicle-mounted fisheye image corresponding to the first fusion weight can be directly selected as the to-be-fused vehicle-mounted fisheye image. Of course, it is not limited to this, if the fixed point coordinate is close to the second boundary ray, the first fusion weight is determined as 0 and the second fusion weight is determined as 1. That is, the initial vehicle-mounted fisheye image corresponding to the second fusion weight can be directly selected as the to-be-fused vehicle-mounted fisheye image. In some embodiments, the fixed point coordinate close to the first boundary ray means that the vertical distance between the fixed point coordinate and the first boundary ray is less than the vertical distance between the fixed point coordinate and the second boundary ray. In some embodiments, the fixed point coordinate close to the second boundary ray means that the vertical distance between the fixed point coordinate and the first boundary ray is greater than the vertical distance between the fixed point coordinate and the second boundary ray.

[0082] Please refer to FIG. 1 and FIG. 10, in an embodiment of the present application, when step S300 is performed, that is, the corresponding initial vehicle-mounted fisheye images are mapped according to the plurality of fusion region weights to generate the target vehicle-mounted fisheye image. Specifically, step S300 can include steps S310 to S340, which are described in detail as follows.

[0083] Step S310, the initial vehicle-mounted fisheye images in the corresponding direction are respectively intercepted according to the first fusion weight and the second fusion weight to generate a plurality of to-be-fused vehicle-mounted fisheye images.

[0084] Step S320, the plurality of to-be-fused vehicle-mounted fisheye images are respectively subjected to extrinsic parameter transformation processing to correspondingly generate a plurality of extrinsic parameter transformation vehicle-mounted fisheye images.

[0085] Step S330, the plurality of extrinsic parameter transformation vehicle-mounted fisheye images are respectively subjected to intrinsic parameter mapping processing to correspondingly generate a plurality of intermediate vehicle-mounted fisheye images.

[0086] Step S340, the plurality of intermediate vehicle-mounted fisheye images are subjected to weighted fusion processing according to the first fusion weight and the second fusion weight to generate the target vehicle-mounted fisheye image.

[0087] In an embodiment of the present application, when step S310 is performed, the initial vehicle-mounted fisheye images of the corresponding directions can be intercepted according to the first fusion weight and the second fusion weight respectively to generate a plurality of vehicle-mounted fisheye images to be fused. For example, referring to FIG. 8, when the vehicle-mounted observation viewing angle is between 0° and 90°, the initial vehicle-mounted fisheye images can be the initial vehicle-mounted fisheye image acquired by the left-view fisheye camera and the initial vehicle-mounted fisheye image acquired by the front-view fisheye camera respectively. When the first fusion weight is 0 and the second fusion weight is 1, it means that the initial vehicle-mounted fisheye image acquired by the front-view fisheye camera is directly used in the fusion region. When the first fusion weight is 1 and the second fusion weight is 0, it means that the initial vehicle-mounted fisheye image acquired by the left-view fisheye camera is directly used in the fusion region. When the first fusion weight and the second fusion weight are both between 0 and 1, the initial vehicle-mounted fisheye images of the left-view and the front-view need to be intercepted according to the first fusion weight and the second fusion weight respectively to correspondingly generate a plurality of vehicle-mounted fisheye images to be fused.

[0088] Referring to FIGS. 10 and 11, in an embodiment of the present application, when step S320 is performed, the plurality of vehicle-mounted fisheye images to be fused are subjected to extrinsic parameter transformation processing respectively to correspondingly generate a plurality of vehicle-mounted fisheye images subjected to extrinsic parameter transformation. Specifically, step S320 can include steps S321 to S322, which are described in detail as follows.

[0089] Step S321, the plurality of vehicle-mounted fisheye images to be fused are subjected to extrinsic parameter rotation matrix processing of the corresponding directions respectively to generate a plurality of extrinsic parameter rotation fisheye images.

[0090] Step S322, the plurality of extrinsic parameter rotation fisheye images are subjected to extrinsic parameter translation matrix processing of the corresponding directions respectively to generate a plurality of vehicle-mounted fisheye images subjected to extrinsic parameter transformation.

[0091] In an embodiment of the present application, when step S321 is performed, the extrinsic parameter rotation matrix processing of the corresponding directions on the plurality of vehicle-mounted fisheye images to be fused means that the coordinates in the plurality of vehicle-mounted fisheye images to be fused are multiplied by the extrinsic parameter rotation matrices of the fisheye cameras on the two sides of the vehicle-mounted observation viewing angle respectively to generate a plurality of extrinsic parameter rotation fisheye images.

[0092] Further, for example, the coordinates in the vehicle-mounted fisheye image to be fused can be multiplied by the extrinsic parameter rotation matrix of the left fisheye camera to obtain the extrinsic parameter rotation fisheye image of the left fisheye camera. Then, the coordinates in the vehicle-mounted fisheye image to be fused can be multiplied by the extrinsic parameter rotation matrix of the front fisheye camera to obtain the extrinsic parameter rotation fisheye image of the front fisheye camera.

[0093] In an embodiment of the present application, when step S322 is performed, the performing inner parameter mapping on the plurality of outer parameter transformed vehicle-mounted fisheye images respectively to generate a plurality of intermediate vehicle-mounted fisheye images is specifically: adding the coordinates of the outer parameter rotated fisheye images of the plurality of fisheye cameras to the inner parameter matrices of the plurality of fisheye cameras respectively to generate the plurality of intermediate vehicle-mounted fisheye images.

[0094] Further, for example, the coordinates of the outer parameter rotated fisheye images of the left fisheye camera can be added to the outer parameter translation matrix of the left fisheye camera to generate the outer parameter transformed vehicle-mounted fisheye image of the left fisheye camera. The coordinates of the outer parameter rotated fisheye images of the front fisheye camera can be added to the outer parameter translation matrix of the front fisheye camera to generate the outer parameter transformed vehicle-mounted fisheye image of the front fisheye camera, so as to transfer the coordinate points on the surround view model to the fisheye camera coordinate system.

[0095] Referring to FIGS. 10 and 12, in an embodiment of the present application, when step S330 is performed, the performing inner parameter mapping on the plurality of outer parameter transformed vehicle-mounted fisheye images respectively to generate a plurality of intermediate vehicle-mounted fisheye images. Specifically, step S330 can include steps S331 to S332, which are described in detail as follows.

[0096] Step S331, performing inner parameter mapping on the plurality of outer parameter transformed vehicle-mounted fisheye images respectively to generate a plurality of corresponding inner parameter fisheye images.

[0097] Step S332, performing de-distortion mapping on the plurality of inner parameter fisheye images respectively to generate a plurality of intermediate vehicle-mounted fisheye images.

[0098] In an embodiment of the present application, when step S331 is performed, the performing inner parameter mapping on the plurality of outer parameter transformed vehicle-mounted fisheye images respectively to generate a plurality of corresponding inner parameter fisheye images is specifically: performing inner parameter matrix mapping on the coordinates of the plurality of outer parameter transformed vehicle-mounted fisheye images respectively to generate a plurality of inner parameter fisheye images in the coordinate systems of the corresponding fisheye cameras.

[0099] In an embodiment of the present application, when step S332 is performed, the de-distortion processing of the plurality of reference fisheye images in corresponding directions respectively refers to the de-distortion parameter mapping processing of the coordinates in the reference fisheye images under the corresponding fisheye camera. The de-distortion parameter mapping processing is a technology in the field of image processing and computer vision, which is used to correct the image distortion caused by the lens distortion of the fisheye camera. Among them, the lens distortion of the fisheye camera includes two main forms of radial distortion and tangential distortion. The radial distortion is a distortion caused by the shape of the camera lens, mainly manifested as the compression or stretching of the pixels at the center and edge of the image, making the straight lines in the image appear curved. The tangential distortion is a distortion caused by the non-parallelism of the camera lens and the image sensor, manifested as the skew of the objects in the image in the horizontal or vertical direction. The de-distortion parameter mapping processing can restore the reference fisheye image to a perspective-corrected image by performing inverse distortion transformation on the reference fisheye image, thereby improving the accuracy and quality of the generated image to achieve one-to-one mapping of the pixel points in the initial vehicle fisheye image in the vehicle surround view model.

[0100] Referring to FIG. 10, in an embodiment of the present application, when step S340 is performed, the weighted fusion processing of the plurality of intermediate vehicle fisheye images according to the first fusion weight and the second fusion weight can satisfy the following formula (6), rgb = weight1 * rgb1 + weight2 * rgb2 (6).

[0101] Among them, weight1 can represent the first fusion weight, rgb1 can represent the pixel value of the pixel point in the intermediate vehicle fisheye image under the fisheye camera corresponding to the first fusion weight, weight2 can represent the second fusion weight, rgb2 can represent the pixel value of the pixel point in the intermediate vehicle fisheye image under the fisheye camera corresponding to the second fusion weight, and rgb can represent the pixel value of the pixel point in the corresponding generated target vehicle fisheye image.

[0102] Referring to FIG. 1, in an embodiment of the present application, when step S400 is performed, the real-time rendering processing of the target vehicle fisheye image refers to the process of converting the generated target vehicle fisheye image into a two-dimensional image. In computer graphics and computer vision, rendering is the process of converting a virtual scene or three-dimensional model into a final two-dimensional image or video. This process simulates factors such as lighting, shadows, perspective, material, etc. that affect the appearance of objects in the real world to produce realistic image or video effects. Among them, the rendering processing can include but is not limited to ray tracing, rasterization rendering, voxel rendering, etc. These methods combine the principles of mathematics, physics and computer graphics, and finally generate high-quality two-dimensional images or videos through modeling and processing of the scene.

[0103] Please refer to FIG. 13, which shows a structural schematic diagram of a vehicle-mounted surround view panoramic image generation system provided in the present application. In an embodiment of the present application, the vehicle-mounted surround view panoramic image generation system can include, but is not limited to, an information acquisition module 510, a model generation module 520, a fusion mapping module 530, and a rendering processing module 540. The functions of each module are described in detail as follows.

[0104] In an embodiment of the present application, the information acquisition module 510 can be used to acquire an initial vehicle-mounted fisheye image corresponding to a vehicle-mounted observation angle. The vehicle-mounted observation angle can be obtained by manual selection by a user, and the range of the vehicle-mounted observation angle can be greater than or equal to 0° and less than or equal to 360°. That is, the vehicle-mounted observation angle can be any angle around the vehicle body. For example, please refer to FIG. 3, the vehicle-mounted observation angle of the front of the vehicle body can be defined as 0°, and the vehicle-mounted observation angle increases counterclockwise with the direction of the top view of the vehicle body as the standard. That is, with the direction of the top view of the vehicle body as the standard, the vehicle-mounted observation angle of the left view opposite to the vehicle body is 90°, the vehicle-mounted observation angle of the rear view opposite to the vehicle body is 180°, and the vehicle-mounted observation angle of the right view opposite to the vehicle body is 270°.

[0105] In addition, a vehicle body coordinate system can be established with the center of the vehicle body as the origin, the direction of the front of the vehicle body as the positive X-axis, and the direction of the right of the vehicle body as the positive Y-axis. The boundary ray vertex coordinate can be the coordinate of one of the four vertices of the vehicle body in the vehicle body coordinate system, and the boundary ray vertex coordinate can be determined according to the change of the vehicle-mounted observation angle. For example, please refer to FIG. 3, when the vehicle-mounted observation angle is 90°, the boundary ray vertex coordinate can be the coordinate of the front-left vertex of the vehicle body in the vehicle body coordinate system. Of course, this is not limited, and in addition to the vehicle-mounted observation angle, the length and width of the vehicle body can also be acquired, and the length and width of the vehicle body can be preset values, which are not specifically limited. For example, the length of the vehicle body can be 4770 mm, and the width of the vehicle body can be 1895 mm.

[0106] Further, the initial vehicle-mounted fisheye images in different directions of the vehicle body can be determined according to the vehicle-mounted observation angle, and the initial vehicle-mounted fisheye images can include at least two initial vehicle-mounted fisheye images collected by fisheye cameras in the directions around the vehicle body, which can be acquired by fisheye cameras in the directions around the vehicle body, respectively. For example, the initial vehicle-mounted fisheye images in the directions around the vehicle body can be acquired by fisheye cameras in the front, rear, left, and right directions of the vehicle body, respectively. When the vehicle-mounted observation angle is between 0° and 90°, the initial vehicle-mounted fisheye images can include an initial vehicle-mounted fisheye image acquired by a front-view fisheye camera and an initial vehicle-mounted fisheye image acquired by a left-view fisheye camera. In addition, the camera angle of the fisheye camera can be between 150 degrees and 180 degrees.

[0107] The model generation module 520 can be configured to input the vehicle observation view into the vehicle surround view model for processing to generate a plurality of fusion region weights corresponding to the initial vehicle fisheye image. The inputting the vehicle observation view into the vehicle surround view model for processing can include, but is not limited to, processing the vehicle observation view and the boundary ray vertex coordinates according to the vehicle surround view model to generate a first boundary ray and a second boundary ray, the first boundary ray and the second boundary ray forming a boundary ray included angle and a region to be fused therebetween. A fixed point coordinate is obtained according to the vehicle observation view, and the fixed point coordinate is processed according to the first boundary ray and the boundary ray vertex coordinates to generate a fixed point included angle. The fixed point included angle and the boundary ray included angle are processed based on the positional relationship between the fixed point coordinate and the boundary ray included angle to generate a first fusion weight and a second fusion weight.

[0108] Further, the processing of the vehicle observation view and the boundary ray vertex coordinates according to the vehicle surround view model can include, but is not limited to, function mapping processing of the vehicle observation view to generate a target boundary ray influence coefficient. The target boundary ray influence coefficient is processed by a ray slope function based on the length of the vehicle body and the width of the vehicle body to generate a ray slope dataset, the ray slope dataset including a first ray slope and a second ray slope. The boundary ray vertex coordinates are processed by ray generation according to the first ray slope and the second ray slope, respectively, to correspondingly generate the first boundary ray and the second boundary ray, wherein the first boundary ray and the second boundary ray form a boundary ray included angle, and the region corresponding to the boundary ray included angle is the region to be fused.

[0109] Further, the function mapping processing of the vehicle observation view can include initial function mapping processing of the vehicle observation view to generate a first influence coefficient. The first influence coefficient is processed by intermediate function mapping to generate a second influence coefficient. The second influence coefficient is processed by target function mapping to generate a target boundary influence coefficient. Specifically, the initial function mapping processing of the vehicle observation view generates the first influence coefficient to map the vehicle observation view from 0°-360° to a specific numerical value. The initial function mapping processing of the vehicle observation view can satisfy the following formula (1), factor1 = ||yaw-180.0|-90.0| / 90.0-0.5 (1).

[0110] Wherein, yaw can represent the vehicle observation view, yaw is greater than or equal to 0° and less than or equal to 360°, and factor1 can represent the first influence coefficient. Of course, it is not limited to this, the vehicle observation view can also satisfy other formulas.

[0111] The first influence coefficient is subjected to an intermediate function mapping process to generate a second influence coefficient, that is, the second influence coefficient can be determined by the first influence. The intermediate function mapping process on the first influence coefficient can satisfy the following formula (2),

[0112] Wherein, factor1 can represent the first influence coefficient, factor2 can represent the second influence coefficient, * can represent the multiplication symbol, and 0.5 can be an empirical value, that is, the intermediate function mapping process on the first influence coefficient can also satisfy other formulas, as long as it can satisfy that when yaw is in the interval of 0°-90° or 180°-270°, factor2 can change from 1 to 0, and when yaw is in the interval of 90°-180° or 270°-360°, factor2 can change from 0 to 1.

[0113] The target mapping function process on the second influence coefficient can satisfy the following formula (3), kScale=0.5-factor2*0.375 (3).

[0114] Wherein, factor2 can represent the second influence coefficient, * can represent the multiplication symbol, kScale can represent the target boundary influence coefficient, and 0.5 and 0.375 can represent the empirical values for adjusting the target boundary influence coefficient, that is, 0.5 and 0.375 can also be replaced by other numbers.

[0115] It is worth further explaining that the ray slope function process on the target boundary ray influence coefficient based on the vehicle body length and the vehicle body width can include but is not limited to first ray slope processing on the vehicle body length, the vehicle body width and the target boundary ray influence coefficient to generate a first ray slope. Then, the first ray slope is subjected to a coefficient multiplication process to generate a second ray slope.

[0116] The first ray slope can satisfy the following formula (4), K1=(L / W)* kScale (4).

[0117] Wherein, L can represent the vehicle body length, W can represent the vehicle body width, * can represent the multiplication symbol, kScale can represent the target boundary influence coefficient, and K1 can represent the first ray slope.

[0118] The coefficient multiplication process on the first ray slope to generate the second ray slope can satisfy the following formula (5), K2=t* K1 (5).

[0119] Wherein, K1 can represent the first ray slope, t can represent a constant coefficient, and t can be 1.5 or other numerical values, and K2 can represent the second ray slope.

[0120] Specifically, the ray generation processing of the boundary ray vertex coordinates according to the first ray slope and the second ray slope respectively refers to that the boundary ray is drawn with the boundary ray vertex coordinates as the starting point and the Y axis of the vehicle body coordinate system as the base line according to the first ray slope and the second ray slope, so as to correspondingly generate the first boundary ray and the second boundary ray.

[0121] Wherein, the first boundary ray and the second boundary ray can form a boundary ray included angle on the same plane, and the area corresponding to the boundary ray included angle can be defined as the to-be-fused area. For example, please refer to FIG. 8, when the vehicle-mounted observation visual angle is 180°, the to-be-fused area can be A', and when the vehicle-mounted observation visual angle is 270, the to-be-fused area can be B'. In addition, when the observation visual angle is converted from the vehicle body rear visual angle 180° to the vehicle body right visual angle 270°, the initial vehicle-mounted fisheye image on the left side of the vehicle body and the initial vehicle-mounted fisheye image on the front side of the vehicle body can be selected, and the to-be-fused area corresponding to the generated to-be-fused area can be adjusted from A' to B'. Please refer to FIG. 9, the first boundary ray can be K1, the second boundary ray can be K2, the area formed by K1 and K2 can be the to-be-fused area, the boundary ray vertex coordinate O can be the vertex coordinate of the left front of the vehicle body, and the to-be-fused area formed by the second boundary ray K2 can fall on the left front of the vehicle body.

[0122] It is worth further explaining that the corresponding fixed point coordinate can be obtained according to the vehicle-mounted observation visual angle, and the fixed point coordinate can be a preset arbitrary point coordinate in the surround view model, which can fall in the plane formed by the first boundary ray and the second boundary ray. For example, please refer to FIG. 9, the fixed point can be defined as C point. The processing of the fixed point coordinate according to the first boundary ray and the boundary ray vertex coordinate refers to that a ray can be emitted from the boundary ray vertex coordinate to the fixed point coordinate to form a ray between the fixed point coordinate and the boundary ray vertex coordinate, and then the included angle between the ray formed between the fixed point coordinate and the boundary ray vertex coordinate and the first boundary ray can be defined as the fixed point included angle theta.

[0123] Specifically, the plurality of fusion region weights can include a first fusion weight and a second fusion weight, and the first fusion weight and the second fusion weight sum to 1. The first fusion weight can represent a proportion of the initial vehicle-mounted fisheye image on the left side of the vehicle-mounted observation view angle in the to-be-fused region, and the second fusion weight can represent a proportion of the initial vehicle-mounted fisheye image on the right side of the vehicle-mounted observation view angle in the to-be-fused region. Of course, the first fusion weight can also represent a proportion of the initial vehicle-mounted fisheye image on the right side of the vehicle-mounted observation view angle in the to-be-fused region, and the second fusion weight can correspondingly represent a proportion of the initial vehicle-mounted fisheye image on the left side of the vehicle-mounted observation view angle in the to-be-fused region. For example, referring to FIG. 8, when the boundary ray vertex coordinate is the vertex coordinate of the left front of the vehicle body, and the to-be-fused region formed by the first boundary ray and the second boundary ray falls on the left front of the vehicle body, the first fusion weight can represent a proportion of the initial vehicle-mounted fisheye image in the left direction of the vehicle body in the to-be-fused region. The second fusion weight can represent a proportion of the initial vehicle-mounted fisheye image in the front direction of the vehicle body in the to-be-fused region, and the first fusion weight and the second fusion weight sum to 1.

[0124] Further, the first fusion weight and the second fusion weight can be determined by the following method. First, it can be determined whether the fixed point coordinate is located within the boundary ray included angle. If the fixed point coordinate is located within the boundary ray included angle, the first fusion weight can be determined according to a proportion of the fixed point included angle in the boundary ray included angle, and the second fusion weight can be determined according to the first fusion weight. When the fixed point coordinate is located within the boundary ray included angle, the first fusion weight can be a ratio between the fixed point included angle and the boundary ray included angle, and the second fusion weight can be determined by subtracting the first fusion weight from 1.

[0125] Further, if the fixed point coordinate is located outside the boundary ray included angle, it is necessary to determine whether the fixed point coordinate is close to the first boundary ray to further determine the first fusion weight and the second fusion weight. If the fixed point coordinate is close to the first boundary ray, the first fusion weight is determined to be 1 and the second fusion weight is determined to be 0. That is, it is not necessary to perform fusion processing on the initial vehicle-mounted fisheye images on both sides of the vehicle-mounted observation view angle, and the initial vehicle-mounted fisheye image corresponding to the first fusion weight can be directly selected as the to-be-fused vehicle-mounted fisheye image. Of course, if the fixed point coordinate is close to the second boundary ray, the first fusion weight is determined to be 0 and the second fusion weight is determined to be 1. That is, the initial vehicle-mounted fisheye image corresponding to the second fusion weight can be directly selected as the to-be-fused vehicle-mounted fisheye image.

[0126] The fusion mapping module 530 can be configured to map the initial vehicle-mounted fisheye images according to a plurality of fusion region weights to generate target vehicle-mounted fisheye images. The fusion mapping module 530 can be configured to respectively intercept the initial vehicle-mounted fisheye images in corresponding directions according to first and second fusion weights to generate a plurality of vehicle-mounted fisheye images to be fused. The plurality of vehicle-mounted fisheye images to be fused are respectively subjected to extrinsic parameter transformation processing to generate a plurality of extrinsic parameter transformed vehicle-mounted fisheye images. The plurality of extrinsic parameter transformed vehicle-mounted fisheye images are respectively subjected to intrinsic parameter mapping processing to generate a plurality of intermediate vehicle-mounted fisheye images. Finally, the plurality of intermediate vehicle-mounted fisheye images are subjected to weighted fusion processing according to the first and second fusion weights to generate the target vehicle-mounted fisheye images.

[0127] Specifically, the fusion mapping module 530 can respectively intercept the initial vehicle-mounted fisheye images in corresponding directions according to first and second fusion weights to generate a plurality of vehicle-mounted fisheye images to be fused. For example, referring to FIG. 8, when the vehicle-mounted observation viewing angle is between 0° and 90°, the initial vehicle-mounted fisheye images can be the initial vehicle-mounted fisheye images collected by the left-view fisheye camera and the initial vehicle-mounted fisheye images collected by the front-view fisheye camera, respectively. When the first fusion weight is 0 and the second fusion weight is 1, it means that the initial vehicle-mounted fisheye images collected by the front-view fisheye camera are directly used as the vehicle-mounted fisheye images to be fused. When the first fusion weight is 1 and the second fusion weight is 0, it means that the initial vehicle-mounted fisheye images collected by the left-view fisheye camera are directly used as the vehicle-mounted fisheye images to be fused. When the first and second fusion weights are both between 0 and 1, the initial vehicle-mounted fisheye images of the left view and the initial vehicle-mounted fisheye images of the front view are respectively intercepted according to the first and second fusion weights to generate a plurality of vehicle-mounted fisheye images to be fused.

[0128] The extrinsic parameter transformation processing of the plurality of vehicle-mounted fisheye images to be fused can include extrinsic parameter rotation matrix processing of the plurality of vehicle-mounted fisheye images to be fused in corresponding directions to generate a plurality of extrinsic parameter rotation fisheye images. The plurality of extrinsic parameter rotation fisheye images are respectively subjected to extrinsic parameter translation matrix processing in corresponding directions to generate a plurality of extrinsic parameter transformed vehicle-mounted fisheye images. The extrinsic parameter rotation matrix processing of the plurality of vehicle-mounted fisheye images to be fused in corresponding directions means that the coordinates in the plurality of vehicle-mounted fisheye images to be fused are respectively multiplied by the extrinsic parameter rotation matrices of the fisheye cameras on both sides of the vehicle-mounted observation viewing angle to generate a plurality of extrinsic parameter rotation fisheye images.

[0129] For example, the coordinates in the to-be-fused vehicle-mounted fisheye image can be multiplied by the extrinsic rotation matrix of the left fisheye camera to obtain an extrinsic rotation fisheye image of the left fisheye camera. The coordinates in the to-be-fused vehicle-mounted fisheye image can be multiplied by the extrinsic rotation matrix of the front fisheye camera to obtain an extrinsic rotation fisheye image of the front fisheye camera.

[0130] Further, the extrinsic rotation fisheye images are respectively subjected to extrinsic translation matrix processing in corresponding directions to generate a plurality of extrinsic transformation vehicle-mounted fisheye images.

[0131] For example, the coordinates of the extrinsic rotation fisheye image of the left fisheye camera can be added to the extrinsic translation matrix of the left fisheye camera to generate an extrinsic transformation vehicle-mounted fisheye image of the left fisheye camera. The coordinates of the extrinsic rotation fisheye image of the front fisheye camera can be added to the extrinsic translation matrix of the front fisheye camera to generate an extrinsic transformation vehicle-mounted fisheye image of the front fisheye camera, so as to transfer the coordinate points on the surround-view model to the fisheye camera coordinate system.

[0132] Further, the fusion mapping module 530 can respectively perform intrinsic transformation processing in corresponding directions on the plurality of extrinsic transformation vehicle-mounted fisheye images to generate a plurality of corresponding intrinsic fisheye images. The plurality of intrinsic fisheye images are respectively subjected to de-distortion mapping processing in corresponding directions to generate a plurality of intermediate vehicle-mounted fisheye images.

[0133] Specifically, the extrinsic transformation vehicle-mounted fisheye images are respectively subjected to intrinsic transformation processing in corresponding directions, that is, the coordinates of the plurality of extrinsic transformation vehicle-mounted fisheye images are respectively subjected to intrinsic matrix processing in corresponding fisheye cameras to obtain intrinsic fisheye images in the coordinate systems of the corresponding fisheye cameras.

[0134] The de-distortion processing of the plurality of reference fisheye images in corresponding directions respectively refers to the de-distortion parameter mapping processing of the coordinates in the reference fisheye images under the corresponding side fisheye camera. The de-distortion parameter mapping processing is a technology in the field of image processing and computer vision, which is used to correct the image distortion caused by the lens distortion of the fisheye camera. Among them, the lens distortion of the fisheye camera includes two main forms of radial distortion and tangential distortion. The radial distortion is a distortion caused by the shape of the camera lens, mainly showing that the pixels at the center and edge of the image are compressed or stretched, making the straight lines in the image appear curved. The tangential distortion is a distortion caused by the non-parallelism of the camera lens and the image sensor, showing that the objects in the image are sheared in the horizontal or vertical direction. The de-distortion parameter mapping processing can restore the reference fisheye image to a perspective correct image through inverse distortion transformation, thereby improving the accuracy and quality of the generated image, to realize one-to-one mapping of the pixel points in the initial vehicle fisheye image in the vehicle surround view model.

[0135] It is worth further explaining that the weighted fusion processing of the plurality of intermediate vehicle fisheye images according to the first fusion weight and the second fusion weight can satisfy the following formula (6), rgb=weight1*rgb1+weight2*rgb2 (6).

[0136] Among them, weight1 can represent the first fusion weight, rgb1 can represent the pixel value of the pixel point in the intermediate vehicle fisheye image under the fisheye camera corresponding to the first fusion weight, weight2 can represent the second fusion weight, rgb2 can represent the pixel value of the pixel point in the intermediate vehicle fisheye image under the fisheye camera corresponding to the second fusion weight, and rgb can represent the pixel value of the pixel point in the corresponding generated target vehicle fisheye image.

[0137] The rendering processing module 540 can be used to perform real-time rendering processing on the target vehicle fisheye image to generate a vehicle surround view panoramic image. Specifically, the real-time rendering processing of the target vehicle fisheye image refers to the process of converting the generated target vehicle fisheye image into a two-dimensional image. In computer graphics and computer vision, rendering is the process of converting a virtual scene or three-dimensional model into a final two-dimensional image or video. This process simulates factors such as light, shadow, perspective, material, etc. that affect the appearance of objects in the real world to produce realistic image or video effects. Among them, the rendering processing can include but is not limited to ray tracing, rasterization rendering, voxel rendering, etc. These methods combine the principles of mathematics, physics and computer graphics, and finally generate high-quality two-dimensional images or videos through modeling and processing of the scene.

[0138] The specific definition of the generation system of the vehicle-mounted surround view panoramic image can refer to the definition of the generation method of the vehicle-mounted surround view panoramic image, which is not repeated here. Each module in the generation system can be realized by software, hardware, and a combination thereof, in whole or in part. The above modules can be embedded in or independent of the processor in the computer device in hardware form, or stored in the memory in the computer device in software form, so as to be called and executed by the processor to perform the operations corresponding to each module.

[0139] In summary, the generation method and system of the vehicle-mounted surround view panoramic image provided by the present application can determine the initial vehicle-mounted fisheye image according to the vehicle-mounted observation angle, determine the to-be-fused region and the fusion region weight corresponding to the initial vehicle-mounted fisheye image according to the vehicle-mounted surround view model, perform regional mapping processing on the corresponding initial vehicle-mounted fisheye image according to the fusion region weight, and perform real-time rendering processing on the target vehicle-mounted fisheye image after the fusion mapping processing, thereby avoiding the problems such as misplacement or ghosting in the fusion region of the vehicle-mounted surround view panoramic image, improving the clarity of the vehicle-mounted surround view panoramic image, and further improving the user experience.

[0140] In an exemplary embodiment, a vehicle is also provided, including a processor and a memory; the memory stores instructions executable by the processor; the processor is configured to execute the instructions to implement any of the methods provided by the above embodiments.

[0141] In an exemplary embodiment, a readable storage medium is also provided, including software instructions, which, when executed on a processor of a vehicle, cause the processor to execute any of the methods provided by the above embodiments.

[0142] In an exemplary embodiment, a computer program product containing computer execution instructions is also provided, which, when executed on a processor of a vehicle, causes the processor to execute any of the methods provided by the above embodiments.

[0143] In the above embodiments, all or part can be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part can be implemented in the form of a computer program product. The computer program product includes one or more computer-executable instructions. When the computer-executable instructions are loaded and executed on a computer, all or part generates the flow or function according to the embodiments of the present application. The computer-executable instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another computer-readable storage medium, for example, the computer-executable instructions can be transferred from one website site, computer, server or data center to another website site, computer, server or data center through wired (for example, coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (for example, infrared, wireless, microwave, etc.) manner. The computer-readable storage medium can be any available medium that can be accessed by a computer or data storage device such as one or more servers, data centers, etc. integrated with one or more media. The available medium can be a magnetic medium (for example, floppy disk, hard disk, magnetic tape), optical medium (for example, DVD) and the like.

[0144] In the description of the present specification, the description of the terms "the present embodiment", "example", "specific example" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0145] The above disclosed embodiments of the present application are only used to help explain the present application. The embodiments do not describe all the details, nor limit the application to the specific embodiments described. Obviously, many modifications and changes can be made according to the content of the present specification. The present specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present application, so that those skilled in the art can well understand and utilize the present application. The present application is limited by the claims and their entire scope and equivalents.

Claims

1. A method for generating a vehicle-mounted surround-view panoramic image, comprising: obtaining an initial vehicle-mounted fisheye image corresponding to a vehicle-mounted observation angle; inputting the vehicle-mounted observation angle into a vehicle-mounted surround-view model for processing to generate a plurality of fusion region weights corresponding to the initial vehicle-mounted fisheye image; performing mapping processing on the initial vehicle-mounted fisheye image corresponding to the plurality of fusion region weights to generate a target vehicle-mounted fisheye image; and performing real-time rendering processing on the target vehicle-mounted fisheye image to generate a vehicle-mounted surround-view panoramic image. The obtaining of the initial vehicle-mounted fisheye image corresponding to the vehicle-mounted observation angle comprises: 2.The method of claim 1, wherein, determining boundary ray vertex coordinates according to the vehicle-mounted observation angle; obtaining the initial vehicle-mounted fisheye image corresponding to the vehicle-mounted observation angle, wherein the initial vehicle-mounted fisheye image comprises initial vehicle-mounted fisheye images collected by fisheye cameras in at least two directions around a vehicle body. The inputting of the vehicle-mounted observation angle into the vehicle-mounted surround-view model for processing to generate the plurality of fusion region weights corresponding to the initial vehicle-mounted fisheye image comprises: 3.The method of claim 1, wherein, processing the vehicle-mounted observation angle and the boundary ray vertex coordinates according to the vehicle-mounted surround-view model to generate a first boundary ray and a second boundary ray, wherein a boundary ray included angle and a region to be fused are formed between the first boundary ray and the second boundary ray; obtaining fixed point coordinates according to the vehicle-mounted observation angle; processing the fixed point coordinates according to the first boundary ray and the boundary ray vertex coordinates to generate a fixed point included angle; processing the fixed point included angle and the boundary ray included angle based on a positional relationship between the fixed point coordinates and the boundary ray included angle to generate a first fusion weight and a second fusion weight as the plurality of fusion region weights, wherein the sum of the first fusion weight and the second fusion weight is 1. The processing of the vehicle-mounted observation angle and the boundary ray vertex coordinates according to the vehicle-mounted surround-view model to generate the first boundary ray and the second boundary ray comprises: 4.The method of claim 3, wherein, performing function mapping processing on the vehicle-mounted observation angle to generate a target boundary ray influence coefficient; performing ray slope function processing on the target boundary ray influence coefficient based on a vehicle body length and a vehicle body width to generate a ray slope dataset, wherein the ray slope dataset comprises a first ray slope and a second ray slope; performing ray generation processing on the boundary ray vertex coordinates according to the first ray slope and the second ray slope, respectively, to correspondingly generate the first boundary ray and the second boundary ray; wherein the region corresponding to the boundary ray included angle formed between the first boundary ray and the second boundary ray is the region to be fused. The function mapping processing on the vehicle-mounted observation angle to generate the target boundary ray influence coefficient comprises: 5.The method of claim 4, wherein, performing initial function mapping processing on the vehicle-mounted observation angle to generate a first influence coefficient; performing intermediate function mapping processing on the first influence coefficient to generate a second influence coefficient; performing target function mapping processing on the second influence coefficient to generate the target boundary influence coefficient. ​ 6.The method of claim 3, wherein, The processing of the fixed point angle and the boundary ray angle based on the positional relationship between the fixed point coordinate and the boundary ray angle includes: If the fixed point coordinate is located within the boundary ray angle, the first fusion weight is determined according to the proportion of the fixed point angle in the boundary ray angle, and the second fusion weight is determined according to the first fusion weight; If the fixed point coordinate is located outside the boundary ray angle and close to the first boundary ray, the first fusion weight is determined as 1 and the second fusion weight is determined as 0; If the fixed point coordinate is located outside the boundary ray angle and close to the second boundary ray, the first fusion weight is determined as 0 and the second fusion weight is determined as 1. 7.The method of claim 1, wherein, The mapping processing of the initial vehicle-mounted fisheye image according to the plurality of fusion region weights includes: According to the first fusion weight and the second fusion weight, the initial vehicle-mounted fisheye image in the corresponding direction is intercepted to generate a plurality of to-be-fused vehicle-mounted fisheye images; The plurality of to-be-fused vehicle-mounted fisheye images are respectively subjected to external parameter transformation processing to correspondingly generate a plurality of external parameter transformation vehicle-mounted fisheye images; The plurality of external parameter transformation vehicle-mounted fisheye images are respectively subjected to internal parameter mapping processing to correspondingly generate a plurality of intermediate vehicle-mounted fisheye images; The plurality of intermediate vehicle-mounted fisheye images are subjected to weighted fusion processing according to the first fusion weight and the second fusion weight to generate the target vehicle-mounted fisheye image. 8.The method of claim 7, wherein, The external parameter transformation processing of the plurality of to-be-fused vehicle-mounted fisheye images to correspondingly generate a plurality of external parameter transformation vehicle-mounted fisheye images includes: The plurality of to-be-fused vehicle-mounted fisheye images are respectively subjected to external parameter rotation matrix processing in the corresponding direction to generate a plurality of external parameter rotation fisheye images; The plurality of external parameter rotation fisheye images are respectively subjected to external parameter translation matrix processing in the corresponding direction to generate a plurality of external parameter transformation vehicle-mounted fisheye images. 9.The method of claim 7, wherein, The internal parameter mapping processing of the plurality of external parameter transformation vehicle-mounted fisheye images to correspondingly generate a plurality of intermediate vehicle-mounted fisheye images includes: The plurality of external parameter transformation vehicle-mounted fisheye images are respectively subjected to internal parameter transformation processing in the corresponding direction to generate a plurality of internal parameter fisheye images; The plurality of internal parameter fisheye images are respectively subjected to distortion removal mapping processing in the corresponding direction to generate the plurality of intermediate vehicle-mounted fisheye images.

10. A vehicle-mounted surround view panoramic image generation system, comprising: an information acquisition module configured to acquire an initial vehicle-mounted fisheye image corresponding to a vehicle-mounted observation viewing angle; a model generation module configured to input the vehicle-mounted observation viewing angle into a vehicle-mounted surround view model for processing to generate a plurality of fusion region weights corresponding to the initial vehicle-mounted fisheye image; a fusion mapping module configured to map process the initial vehicle-mounted fisheye image according to the plurality of fusion region weights to generate a target vehicle-mounted fisheye image; and a rendering processing module configured to perform real-time rendering processing on the target vehicle-mounted fisheye image to generate a vehicle-mounted surround view panoramic image.

10. A vehicle-mounted surround view panoramic image generation system, comprising: an information acquisition module configured to acquire an initial vehicle-mounted fisheye image corresponding to a vehicle-mounted observation viewing angle; a model generation module configured to input the vehicle-mounted observation viewing angle into a vehicle-mounted surround view model for processing to generate a plurality of fusion region weights corresponding to the initial vehicle-mounted fisheye image; a fusion mapping module configured to map process the initial vehicle-mounted fisheye image according to the plurality of fusion region weights to generate a target vehicle-mounted fisheye image; and a rendering processing module configured to perform real-time rendering processing on the target vehicle-mounted fisheye image to generate a vehicle-mounted surround view panoramic image.

Images