Method for obtaining installation tolerance of fisheye camera on line

Abstract

The invention provides a method for online obtaining the installation tolerance of a fisheye camera, which is applied to an online simulation platform for optimizing external parameters of the fisheye camera for panoramic stitching and is used for obtaining the installation tolerance of the fisheye camera arranged on a vehicle. The method includes: generating a reference external parameter from one of a plurality of simulated fisheye cameras disposed on a simulated vehicle; enabling the simulation fisheye camera to generate a third fisheye image by using the reference external parameter; performing image stitching on the third fisheye image and a third fisheye image generated by an adjacent simulation fisheye camera adjacent to the simulation fisheye camera to generate a third panoramic stitching image; and when the common-view-area error of the third panoramic spliced image is higher than a common-view-area error threshold value, taking an offset value in the current reference external parameter as the installation tolerance.

Description

Technical Field

[0001] The present invention relates to a method for setting initial external parameters, and in particular to a method for online calibration of external parameters of a fisheye camera of a vehicle surround system. Background Art

[0002] At present, the vehicle-mounted surround view system usually uses the offline calibration method to calibrate the camera parameters, and the external parameter calibration of the camera is the technical focus, because the calibration accuracy of each camera directly determines the auxiliary effect of the vehicle-mounted surround view system on the driver. However, the construction process of the camera external parameter calibration of the surround view system is relatively cumbersome, consuming manpower and financial resources, and has requirements on the technical level of the operator. In addition, there may not be a suitable large site or a suitable vehicle model for the construction of the camera external parameter calibration, and the requirements for light, environment, climate, etc. are relatively strict. Summary of the invention

[0003] A method for obtaining the installation tolerance of a fisheye camera online according to an embodiment of the present invention is applied to an online simulation platform for optimizing the external parameters of a fisheye camera for surround stitching, so as to obtain the installation tolerance of a fisheye camera arranged on a vehicle. The method comprises: generating reference external parameters from one of a plurality of simulated fisheye cameras arranged on a simulated vehicle; using the reference external parameters to enable the simulated fisheye camera to generate a third fisheye image; using the third fisheye image and a third fisheye image generated by an adjacent simulated fisheye camera adjacent to the simulated fisheye camera to perform image stitching to generate a third surround stitching image; and when a common view zone error of the third surround stitching image is higher than a common view zone error threshold, using the offset value in the current reference external parameters as the installation tolerance.

[0004] The following is a detailed description of the present invention with reference to the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] Figure 1 The figure is a flow chart of a method for obtaining initial external parameters according to an embodiment of the present invention.

[0006] Figure 2A It is a flowchart of online optimization of a fisheye camera of a vehicle surround view system according to an embodiment of the present invention.

[0007] Figure 2B The invention provides a flow chart of generating an installation tolerance line of a fisheye camera according to an embodiment of the present invention, wherein the reference external parameters are generated by inputting an offset value into an initial external parameter or an optimized external parameter.

[0008] Figure 3 FIG. 4 is a schematic diagram of a fisheye image including a checkerboard calibration plate according to an embodiment of the present invention.

[0009] Figure 4 FIG. 4 is a schematic diagram of a fisheye image without a checkerboard calibration plate according to an embodiment of the present invention.

[0010] Figure 5A Schematic diagram showing the checkerboard calibration plate included for stitching images, but without optimizing the extrinsic parameters of the fisheye camera.

[0011] Figure 5B Schematic diagram showing that the checkerboard calibration pattern is included for stitching images, but the extrinsics of the fisheye camera have been optimized.

[0012] Fig. 6A Schematic diagram of stitching images without the checkerboard calibration plate, but without optimizing the extrinsic parameters of the fisheye camera.

[0013] Figure 6B Schematic diagram showing that the checkerboard calibration plate is not included for stitching images, but the extrinsic parameters of the fisheye camera have been optimized.

[0014] Figure 7 The flowchart of processing an object appearing at a splicing position of a spliced ​​image of a vehicle surrounding system according to an embodiment of the present invention is shown.

[0015] Figure 8 For the embodiment of the present invention Figure 7 Detailed flowchart of step S706.

[0016] Fig. 9 For the embodiment of the present invention Figure 8 Detailed flowchart of step S804.

[0017] Fig. 10A It is a schematic diagram of a stitched image 1000 in the prior art when an object appears at the stitching position of the stitched images of the vehicle surrounding system.

[0018] Fig. 10B 1 is a schematic diagram of a stitched image 1002 in which an object appears at a stitching position of stitched images of a vehicle surround system in an embodiment of the present invention.

[0019] Fig.11 FIG. 1 is a schematic diagram of an electronic device 1100 according to an embodiment of the present invention.

[0020] Among them, the simple description of the symbols in the accompanying drawings is as follows:

[0021] S100, S102, S104, S106: steps; S200, S202, S204, S206, S208,

[0022] S210, S212, S214: steps; S240, S242, S244, S246, S248, S250: steps; 300, 302, 304, 306: fisheye image; 310: checkerboard calibration plate; 400, 402, 404, 406: fisheye image; 500, 502: stitched image; 510, 512, 514, 516, 520, 522, 524, 526: checkerboard calibration plate; 530: simulated vehicle; 600, 602: stitched image; 610, 620, 650, 660: marking line; 630, 670: object; S700, S702, S704, S706, S70 8. S710: step; S800, S802, S804: step; S900, S902, S904, S906, S908, S910, S912: step; 1000, 1002: stitched image; 1010: first level warning box; 1012: second level warning box; 1020: representative point; 1030, 1030': stitching seam; 1040: target object; 1050: simulated vehicle; 1100: electronic device; 1102: processor; 1104: multiple fisheye cameras; 1110: simulation platform; 1112: external parameter extraction algorithm; 1114: checkerboard corner detection algorithm; 1116: surround stitching algorithm. DETAILED DESCRIPTION

[0023] Reference will now be made in detail to exemplary embodiments of the present invention, examples of which are illustrated in the accompanying drawings. Whenever possible, the same reference numerals are used in the drawings and the description to refer to the same or like parts.

[0024] Certain words are used throughout the specification and claims to refer to specific components. It is understood by those skilled in the art that electronic device manufacturers may refer to the same components by different names. It is not intended to distinguish between components that have the same function but different names. In the following specification and claims, the words "including" and "comprising" are open-ended words and should be interpreted as "including but not limited to...".

[0025] The directional terms mentioned herein, such as "up", "down", "front", "back", "left", "right", etc., are only with reference to the directions of the drawings. Therefore, the directional terms used are for illustration and not for limiting the present invention. In the drawings, each figure illustrates the general characteristics of the methods, structures and / or materials used in a particular embodiment. However, these drawings should not be interpreted as defining or limiting the scope or nature covered by these embodiments. For example, for clarity, the relative size, thickness and position of each film layer, region and / or structure may be reduced or enlarged.

[0026] The structure (or layer, component, substrate) described in the present invention is located on / above another structure (or layer, element, substrate), which may refer to the two structures being adjacent and directly connected, or it may refer to the two structures being adjacent but not directly connected. Indirect connection means that there is at least one intermediate structure (or intermediate layer, intermediate component, intermediate substrate, intermediate interval) between the two structures, the lower surface of one structure is adjacent to or directly connected to the upper surface of the intermediate structure, and the upper surface of the other structure is adjacent to or directly connected to the lower surface of the intermediate structure. The intermediate structure can be composed of a single-layer or multi-layer physical structure or a non-physical structure, without limitation. In the present invention, when a certain structure is disposed "on" another structure, it may refer to that the certain structure is "directly" on the other structure, or that the certain structure is "indirectly" on the other structure, that is, at least one structure is sandwiched between the certain structure and the other structure.

[0027] The terms "approximately," "equal," "equal," or "same," "substantially," or "approximately" are generally interpreted as within 20% of a given value or range, or within 10%, 5%, 3%, 2%, 1% or 0.5% of a given value or range.

[0028] The ordinal numbers used in the specification and claims, such as "first", "second", etc., are used to modify the elements. They do not imply or represent any previous ordinal numbers of the element (or elements), nor do they represent the order of one element and another element, or the order of the manufacturing method. The use of these ordinal numbers is only used to make the element with a certain name clearly distinguishable from another element with the same name. The claims and the specification may not use the same words. Accordingly, the first component in the specification may be the second component in the claim.

[0029] The electrical connection or coupling described in the present invention may refer to direct connection or indirect connection. In the case of direct connection, the endpoints of the components on the two circuits are directly connected or connected to each other by a conductor segment, and in the case of indirect connection, there is a switch, diode, capacitor, inductor, resistor, other suitable components, or a combination of the above components between the endpoints of the components on the two circuits, but it is not limited thereto.

[0030] In the present invention, the thickness, length and width can be measured by an optical microscope, and the thickness or width can be measured by a cross-sectional image in an electron microscope, but it is not limited thereto. In addition, any two values ​​or directions used for comparison may have a certain error. In addition, the terms "equal", "equal", "same", "substantially" or "substantially" mentioned in the present invention generally represent a range of 10% within a given value or range. In addition, the terms "a given range is a first value to a second value", "a given range falls within the range of a first value to a second value" indicate that the given range includes the first value, the second value and other values ​​therebetween. If the first direction is perpendicular to the second direction, the angle between the first direction and the second direction may be between 80 and 100 degrees; if the first direction is parallel to the second direction, the angle between the first direction and the second direction may be between 0 and 10 degrees.

[0031] It should be noted that the following embodiments can replace, reorganize, or mix features in several different embodiments to complete other embodiments without departing from the spirit of the present invention. The features between the embodiments can be mixed and matched as long as they do not violate the spirit of the invention or conflict with each other.

[0032] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meanings as commonly understood by those skilled in the art to which the present invention belongs. It is understood that these terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning consistent with the background or context of the relevant technology and the present invention, and should not be interpreted in an idealized or overly formal manner unless specifically defined in the embodiments of the present invention.

[0033] In order to better describe the embodiments of the present invention, the terms used in the present invention are first explained below.

[0034]

Terminology

[0035] Fisheye lens: A fisheye lens is an ultra-wide-angle lens that produces strong visual distortion to produce wide panoramic or hemispherical images. Current vehicles are equipped with some cameras with fisheye lenses (referred to as fisheye cameras), and then the images obtained are spliced ​​together to display the real scene around the vehicle for the driver's reference. The external parameters such as the position and rotation angle of these fisheye cameras installed on the vehicle are key parameters for establishing a surround image. Therefore, it is very important to optimize the external parameters of the fisheye camera before the vehicle goes on the road to obtain the optimal position and rotation angle. The external parameter optimization in the present invention is performed on the fisheye camera installed on the vehicle, but the present invention sometimes uses the fisheye lens as the subject of the external parameter optimization process during the external parameter optimization process, such as when the internal parameters of the fisheye lens are mentioned during the external parameter optimization process.

[0036] Extrinsic parameters of fisheye camera: The extrinsic parameters of a fisheye camera (also called "extrinsic parameters") describe its position and rotation angle, which are derived from its displacement and tilt / rotation. Users can run an extrinsic optimization program to adjust the extrinsic parameters of the fisheye camera mounted on the simulated vehicle until the optimized extrinsic parameters required for the surround effect are found.

[0037] Intrinsic parameters of fisheye lens (also referred to as "intrinsic parameters"): The intrinsic parameters of fisheye lens describe the pixel mapping from the scene in front of the fisheye lens to the final image, including the focal length, optical center, lens distortion (such as radial lens distortion and tangential lens distortion of fisheye lens), etc.

[0038] Adjacent fisheye cameras: Adjacent fisheye cameras installed on a vehicle can show at least one identical object in the fisheye images they capture, and the same object can be used to create a stitched image. For example, in a six-camera vehicle, the six cameras that go around the vehicle counterclockwise are the front side, the left front side, the left rear side, the rear side, the right rear side, and the right front side. The fisheye images captured by the two cameras on the front side and the right front side of the vehicle may show at least one identical object, but the fisheye images generated by the two cameras on the left rear side and the right front side of the vehicle may not show any identical objects.

[0039] Simulation platform: The present invention is an online system for simulating the vehicle surround effect. The simulation platform has built-in a large number of simulated vehicles, simulated fisheye cameras, simulated chessboard calibration plates, simulated venues, etc., which provide users with the ability to obtain the initial external parameters of the fisheye camera online or optimize the external parameters of the fisheye camera of the vehicle surround system.

[0040] Simulated vehicle: A vehicle in a simulation platform has the same vehicle characteristic parameters as the relevant actual vehicle, such as size (eg, length, width, height, shape, etc.), weight, speed, braking capability, etc.

[0041] Simulated fisheye lens: The internal parameters of the fisheye lens in the simulation platform are the same as those of the relevant real fisheye lens, and the simulated fisheye camera contains at least one simulated fisheye lens.

[0042] Simulated chessboard calibration plate: a chessboard calibration plate in a simulation platform, which has the same chessboard features as the actual chessboard features (such as the edges, shapes, grids, patterns, etc. on the chessboard, referred to as "chessboard features").

[0043] Simulation site: The place or position in the simulation platform used to simulate the vehicle's surrounding effects.

[0044] Reference external parameters (referred to as "reference external parameters"): external parameters that add offset values ​​to initial external parameters (or optimized external parameters). The simulated fisheye camera can generate an ambience stitching image based on the reference external parameters, and when the ambience stitching image is judged to be correct (for example, the error of the overlap region of the ambience stitching image does not exceed a predetermined overlap region error threshold), the offset value added to the initial parameters is gradually increased to gradually increase the deviation of the reference external parameters, until the ambience stitching image generated after the initial external parameter is added with a certain offset value is incorrect (for example, the overlap region error of the ambience stitching image at this time has exceeded the predetermined overlap region error threshold). Finally, the reference external parameters of the correct ambience stitching image can be generated, and the offset value added to the initial external parameters is the maximum offset value of the initial external parameters, and the maximum offset value is installed with a tolerance (Tolerance), and is used as a reference when actually installing the fisheye camera.

[0045] When the external parameters of the fisheye camera are calibrated offline through a real vehicle and a checkerboard calibration plate, the vehicle must be driven to a suitable site, and a suitable checkerboard calibration plate must be placed at a suitable distance from the vehicle, and the fisheye camera must be physically installed to obtain a fisheye image. Subsequently, the fisheye image is generated based on the external parameters of the fisheye camera, and the surrounding image spliced ​​by the fisheye image is calibrated, so that the external parameters of the fisheye camera for constructing the vehicle surrounding system can be optimized. Since the offline fisheye camera and surrounding system optimization process requires manual and repeated related operations to be completed, for example, the fisheye camera is manually installed on the vehicle, the actual position / angle of the fisheye camera is obtained, and the vehicle is moved to generate fisheye images at different angles to optimize the splicing image effect of the fisheye camera external parameters and the surrounding system. In addition, if the camera external parameters are to be re-optimized, the vehicle needs to be driven to a suitable site again and all the above steps need to be repeated, which will extend the construction schedule of the vehicle surrounding system. Furthermore, if the vehicle is a special vehicle (such as a mine truck, a garbage truck, etc.), it may be necessary to arrange additional time and find a suitable venue, which may not only waste time but also incur additional vehicle dispatch fees.

[0046] The inventor of this case is aware of the problems of the prior art, so all offline calibration work is moved to the simulation platform, by building a simulation scene, including building the surrounding environment, selecting a simulated vehicle consistent with the actual vehicle parameters, setting a simulated checkerboard calibration board, and setting the simulated fisheye camera according to the internal parameters of the simulated fisheye lens in the preset simulated fisheye camera, and then the simulation platform generates the initial external parameters corresponding to all simulated fisheye cameras. In addition, the simulation platform can also automatically generate simulated fisheye images with the initial external parameters of the simulated fisheye camera according to the requirements of the vehicle's surrounding system, and automatically optimize the spliced ​​images of the simulated fisheye images, so that the external parameters of the simulated fisheye camera can be optimized according to the requirements of the vehicle's surrounding system. Therefore, the user can actually set the fisheye camera on the vehicle according to the initial external parameters generated by the simulation platform, and can also obtain the optimization results of the vehicle's surrounding system on the simulation platform. On the other hand, since the simulation platform of the present invention has many simulated fisheye cameras with different focal lengths or different models built in, the user can select a fisheye camera on the simulation platform for external parameter acquisition and optimization for the selected vehicle until the most suitable fisheye camera is found.

[0047] In addition, the inventors also realize that there are still many vehicles that are not equipped with any cameras when they leave the factory. However, when installing cameras on vehicles in the aftermarket, it is very inconvenient to use an offline method to obtain the initial external parameters of each vehicle. However, if a simulated vehicle can be generated on a simulation platform based on vehicle characteristic parameters, and then a simulated fisheye camera is generated based on the internal parameters of the fisheye camera and set on the simulated vehicle, the initial external parameters of the fisheye camera can be obtained through the simulation platform as a reference for actual installation, and the external parameters of the fisheye camera can be optimized in the aftermarket based on the requirements of the vehicle surround system. Furthermore, since the person who actually installs the fisheye camera on the vehicle in the aftermarket may be the owner, and his / her installation skills may not be proficient, large deviations may occur during installation. If the installation tolerance of the fisheye camera can be generated through the simulation platform, in addition to knowing the error range during installation, if the user wants to add a shell to the fisheye camera to further protect it, the user can also refer to the installation tolerance to find a shell with a suitable thickness to protect the fisheye camera.

[0048] Figure 1 Flow chart of a method for obtaining initial external parameters according to an embodiment of the present invention. Figure 1 The disclosed flowchart is applicable to a simulation platform for simulating a vehicle including multiple fisheye cameras. In some embodiments, the simulation platform may be, for example, Carla, PerScan, CarSim, VIRES VTD, PTV Vissim, and TESS NG, but the present invention is not limited thereto. Figure 1As shown, the initial external parameter acquisition method of the present invention includes: setting a plurality of simulated chessboard calibration plates around a simulated vehicle corresponding to the actual vehicle (step S100); establishing a plurality of simulated fisheye cameras corresponding to the actual fisheye camera in the simulation platform according to an internal parameter of the fisheye camera, and setting the simulated fisheye camera on the simulated vehicle according to the vehicle characteristic parameters of the simulated vehicle (step S102); obtaining a first fisheye image through the simulated fisheye camera (step S104); calculating an initial external parameter of the simulated fisheye camera on the simulated vehicle according to the first characteristic point in the first fisheye image (step S106). Therefore, when the user selects the simulated vehicle on the simulation platform in step S100 and places the simulated chessboard calibration plates around the simulated vehicle, and then sets the simulated fisheye camera on the simulated vehicle in step S102, the initial external parameters can be automatically generated through steps S104 and S106, without the need for cumbersome manual operation to obtain the initial external parameters. In addition, since the simulation platform generates initial external parameters through software, the software that correctly corresponds the camera coordinate system to the world coordinate system can be automatically called after the initial external parameters are generated, so that the coordinate system of the initial external parameters can be correctly matched. Therefore, when the coordinate system of the initial external parameters is correctly matched, the software that optimizes the camera external parameters under specific applications can be used to optimize the initial external parameters of the fisheye camera according to specific needs. This part will be explained in detail later.

[0049] Figure 1 The initial external parameters of each fisheye camera set on the simulated vehicle can be calculated based on the fisheye image generated by itself. For example, since the coordinates of the checkerboard calibration plate are known to the simulation platform (that is, the coordinates of the checkerboard calibration plate are determined after placement), the simulation platform can calculate the distance between the checkerboard calibration plate and the fisheye camera and the tilt / rotation angle of its fisheye camera through the world coordinates. For example, the initial external parameters of the front fisheye camera of a four-camera vehicle can be obtained by taking the internal vertices of the checkerboard calibration plate (such as a 5x5 checkerboard) taken by itself and calculating the distance between the checkerboard calibration plate and the front fisheye camera with reference to the world coordinates, thereby calculating the tilt / rotation angle of the front fisheye camera, and then converting the displacement and tilt / rotation angle of the front fisheye camera into a matrix (vector) to provide a basis for further calculations. An example of a matrix (vector) converted based on the displacement and tilt / rotation angle (such as the rotation matrix and translation matrix described below) will be given later. On the other hand, since the external parameters of the fisheye camera are an important basis for establishing the panoramic stitching image, if the fisheye camera coordinate system and the world coordinate system are not correctly aligned, the stitching image produced by it will have some deviations or discontinuous sections, thus affecting the effect of the panoramic stitching image. Figure 5A / Fig. 6AIt is the panoramic stitching image produced when the coordinate system of the fisheye camera is not correctly aligned. Figure 5B / Figure 6B is the panoramic stitching image produced after the fisheye camera coordinate system has been correctly aligned, and Figure 5B / Figure 6B The panoramic splicing effect is significantly better than Figure 5A / Fig. 6A In addition, when the coordinate system is correctly aligned, the initial external parameters can be used as a reference to calibrate the stitched image of multiple second fisheye images generated by the simulated fisheye camera, so that the external parameters of the simulated fisheye camera for constructing the vehicle surround system can be optimized. Figure 2A In addition, the present invention can also add an offset value to the initial external parameter (or optimized external parameter) to become a reference external parameter, and then import the online optimization process of the spliced ​​image to obtain the tolerance during installation, which can serve as a reference for the user to actually install the fisheye camera. The generation process of the reference external parameter will be described later. Figure 2B Furthermore, because the surround view system used in vehicles needs to take special consideration of the vehicle's driving safety, the present invention specifically processes objects that appear in the stitching region of the surround view stitching image. The steps related to this will be described in detail in Figure 7 A more detailed explanation is given in the relevant narrative.

[0050] In step S100, the method of the present invention first creates a simulated chessboard calibration board corresponding to the real environment in a simulation platform (such as Carla) according to the chessboard features of the chessboard calibration board used in the real environment (including the style and size of the chessboard, etc.). Afterwards, the present invention sets a suitable position for placing the simulated chessboard calibration board in the simulation environment of the simulation platform according to the site environment of the real environment and the size and shape of the vehicle, and then places the simulated chessboard calibration board on the ground around the vehicle body. In some embodiments, the present invention sets a shortcut key in the user interface of the simulation platform, such as shortcut key E. When the user wants to execute step S100, he only needs to press shortcut key E, and the present invention will execute it. Figure 1 Step S100 is a step of the user's own creation of a simulated vehicle, and the user presses / starts the shortcut key E to automatically place a preset checkerboard calibration plate around the simulated vehicle. However, the present invention is not limited thereto. In addition, step S100 may be written into a plug-in program, such as a plug-in program or a dynamic link library (DLL). When the user wants to execute step S100, the plug-in program may be started to execute the step. Figure 1 Step S100.

[0051] In some embodiments, since the specific style and size of the chessboard calibration plate in the real environment are not fixed, the present invention builds in a variety of specifications of simulated chessboard calibration plates in the simulation platform so that corresponding selections can be made according to actual needs in specific applications. Figure 1 In the embodiment of step S100, the checkerboard calibration plate is a 5*5 checkerboard, and the side length of each small grid is, for example, 30.5 cm. The present invention places four simulated checkerboard calibration plates around the body of the simulated vehicle. Taking the ground landing point of the simulated vehicle as the coordinate origin, the direction pointing to the front of the vehicle is, for example, the x direction, and the direction pointing to the right of the vehicle is, for example, the y direction. Therefore, the coordinates of the center points of the above four checkerboard calibration plates can be, for example, (+4, +3), (-4, +3), (-4, -3), (+4, -3), where the coordinate unit is meter. In addition, the direction pointing to the top of the vehicle can be defined as the z direction, so the three directions of x / y / z can define a vehicle coordinate system (Vehicle Coordinate System) with the center point of the vehicle as the origin, and this vehicle coordinate system can be used as the world coordinate system (World Coordinate System) when optimizing the external parameters of the fisheye camera of the vehicle surround system, but the present invention is not limited to this.

[0052] In step S102, the present invention obtains an internal parameter of each of a plurality of fisheye cameras that are expected to be installed on a vehicle in a real environment. In some embodiments, the number of fisheye cameras that are expected to be installed on a vehicle may be, for example, four or six, but the present invention is not limited thereto. In some embodiments, if the number of fisheye cameras is four, the fisheye cameras may be installed on the front side, rear side, left side, and right side of the vehicle. If the number of fisheye cameras is six, the fisheye cameras may be installed on the front side, left front side, left rear side, rear side, right rear side, and right front side of the vehicle. In some embodiments, the internal parameters of the fisheye lens may be, for example, focal length, optical center, and lens distortion, but the present invention is not limited thereto. In step S102, the present invention creates a plurality of simulated fisheye cameras corresponding to the fisheye cameras in a real environment according to the above internal parameters in a simulation environment, and sets the simulated fisheye cameras on a simulated vehicle. Therefore, the internal parameters of the simulated fisheye lens in step S102 will be equal to the internal parameters of the aforementioned fisheye lens in this paragraph.

[0053] In some embodiments, the internal parameters of the simulated fisheye lens may be, for example, the parameter K

[0054]

[0055] Among them, f x =312.8667907714844,f y =312.0426940917969, c x=643.1307983398438, c y =366.03363037109377.

[0056] The distortion of the simulated fisheye lens can be expressed as parameter D

[0057] D=[0.19537385,-0.09064766,0.027709538,-0.0042501888]

[0058] The resolution of the fisheye image generated by the simulated fisheye camera according to the above parameters K and D may be, for example, 1280*720, but the present invention is not limited thereto.

[0059] In step S104, the method of surround stitching of the present invention further sets a saving path and a saving frame rate for the first fisheye image, and uses the saving path and the saving frame rate to store the first fisheye image. In some embodiments, the saving path may, for example, be saved to a local disk directory. In some embodiments, the first fisheye image may be stored, for example, in the form of a picture file or an image file. In some embodiments, before obtaining the initial external parameters in step S106, the method of the present invention first obtains an actual parameter of a vehicle in an actual environment (for example, the user inputs the actual parameter to the simulation system). Then, the method of the present invention determines the position and rotation angle of the simulated fisheye camera on the simulated vehicle based on the actual parameters. In some embodiments, the actual vehicle characteristic parameters may, for example, include the length, width, and position of the vehicle in the actual environment, but the present invention is not limited to this. Figure 1 In the embodiment of the present invention, the first fisheye image in step S104 includes a checkerboard calibration plate, and the internal vertex of the checkerboard calibration plate is used as the first feature point. In some embodiments, the first fisheye image must include a checkerboard calibration plate before the subsequent step S106, i.e., the step of obtaining the initial external parameter, can be executed. In these embodiments, the processing can be paused / terminated. Figure 1 The process informs the user that the first fisheye image must contain a checkerboard calibration plate in order to calculate the initial extrinsic parameters.

[0060] In step S102, the present invention sets a connection type of the simulated fisheye camera, and sets the simulated fisheye camera on the simulated vehicle according to the connection type. In some embodiments, the connection type of the fisheye camera of the present invention is a rigid connection, such as being screwed to a fixed position, but the present invention is not limited to this. In steps S102 and S104, the method of the present invention sets a shortcut key in the user interface of the simulation platform, such as shortcut key Y, but the present invention is not limited to this. For example, if steps S102 and S104 are set to shortcut key Y, then when the user wants to execute steps S102 and S104, they only need to press shortcut key Y, and the method of the present invention will execute Figure 1In step S102, the settings of the simulated fisheye camera are set, and the fisheye image captured by the fisheye camera is saved to a specified path. In one embodiment, the simulated fisheye camera can be selected by the user on the simulation platform, or it can be a simulated fisheye camera predetermined by the simulation platform. Alternatively, step S104 can be written into a plug-in program, for example, step S104 can be written into a plug-in program or a dynamic link function library. When the user wants to execute step S104, the plug-in program or the dynamic link function library can be loaded to execute Figure 1 Step S104.

[0061] In step S106, the present invention calculates an initial external parameter of the simulated fisheye camera on the simulated vehicle based on the first feature point of the first fisheye image (for example, using a nonlinear least square method (such as a Levenberg-Marquardt algorithm)). In some embodiments, the initial external parameters of the simulated fisheye camera on the simulated vehicle may include a rotation matrix and a translation matrix (such as the two vectors Transform / Rotation mentioned in the next paragraph) for converting the camera coordinate system of the simulated fisheye camera to the world coordinate system, but the present invention is not limited thereto. In some embodiments, the above rotation matrix and translation matrix are included in the initial external parameters. In some embodiments, when the first feature point is an internal vertex of the checkerboard calibration plate, the method of the present invention uses a checkerboard corner point detection algorithm to detect the internal vertex in the first fisheye image in step S104. Then, the method of the present invention calculates the initial external parameters based on the coordinates of the internal vertices of the checkerboard calibration plate in the coordinate system of the fisheye camera and in the world coordinate system. In some embodiments, when the initial external parameters in step S106 have been obtained, the present invention converts the initial external parameters into the displacement and rotation of the simulated fisheye camera. Afterwards, the present invention sets the position and rotation angle of the simulated fisheye camera on the simulated vehicle in the simulated environment accordingly according to the displacement and rotation of the simulated fisheye camera.

[0062] In some embodiments, the method of the present invention sets the position and rotation angle of the simulated fisheye camera in the world coordinate system according to the initial external parameters calibrated by the fisheye camera in the real environment, which is called method 1. In some embodiments, the position and rotation angle of the simulated fisheye camera in the world coordinate system can be set directly in the simulated environment (for example, the user can directly input the position and rotation angle of the fisheye camera in the Clara system), which is called method 2. Taking method 1 as an example, and taking four simulated fisheye cameras as an example, the initial external parameters of the four simulated fisheye cameras can be, for example, as follows.

[0063] Transform(Location(x=1.700000,y=-0.090000,z=1.860000);

[0064] Rotation(pitch=0.000000,yaw=0.000000,roll=0.000000))External parameter 1

[0065] Transform(Location(x=-0.600000,y=-0.800000,z=1.860000),

[0066] Rotation(pitch=0.000000,yaw=-90.000000,roll=0.000000))External reference 2

[0067] Transform(Location(x=-1.700000,y=0.000000,z=1.370000),

[0068] Rotation(pitch=0.000000,yaw=-180.000000,roll=0.000000)) External reference three

[0069] Transform(Location(x=0.000000,y=0.590000,z=1.200000),

[0070] Rotation(pitch=0.000000,yaw=90.000000,roll=0.000000))External reference 4

[0071] In the above external references 1, 2, 3 and 4, the length unit is meter and the angle unit is degree.

[0072] In one embodiment, after step S106 is executed, the first fisheye images of all adjacent fisheye cameras may be used to generate the first surround stitching image. However, as mentioned above, if the correspondence between the coordinate systems of the initial external parameters is incorrect, the following may occur: Figure 5A / Fig. 6AThe stitched image shown. Of course, if the user believes that the surround effect of the first surround stitched image generated by the initial external parameters is acceptable, he / she can skip the step of adjusting the coordinate correspondence. In addition, as mentioned above, since the simulation platform generates the initial external parameters through software, after the initial external parameters are generated in step S106, the software that correctly corresponds the camera coordinate system to the world coordinate system can be automatically called (for example, calling OpenCV's projectPoints, solvePnP, Rodrigues... and other functions) so that the coordinate system of the initial external parameters can correspond correctly (for example, using software calibration on the simulation platform to fine-tune the translation distance and rotation angle of the translation matrix / rotation matrix of the initial external parameters). Subsequently, the initial external parameters of the fisheye camera can be optimized according to specific needs based on the software that optimizes the camera external parameters under specific applications, for example, using Figure 1 After obtaining the initial external parameters, restart Figure 2A The process is to optimize the initial external parameters of the fisheye camera based on the requirements of the vehicle surround system. This part will be described in detail later. Furthermore, in the aftermarket application, if the fisheye camera has not yet been installed on the vehicle, a simulated vehicle / simulated fisheye camera can be established on the simulation platform with the vehicle characteristic parameters and the internal parameters of the fisheye camera, and a fisheye image can be generated after the simulated fisheye camera is installed on the simulated vehicle (for example, Figure 3 The four fisheye images shown in Figure 1 The process is used to calculate the initial external parameters of each simulated fisheye camera, so that the user can refer to the initial external parameters to actually install the fisheye camera. On the other hand, if the fisheye camera has been installed on the vehicle, the fisheye image taken by the fisheye camera (for example Figure 4 The four fisheye images shown in the figure are input into the simulation platform to re-optimize the fisheye camera external parameters of the vehicle surround view system (for example, the camera external parameters change due to external forces), but the present invention is not limited to this.

[0073] Since it is necessary to optimize the external parameters of each fisheye camera after installation when constructing the vehicle surrounding system, the simulation platform of the present invention provides that after the simulated fisheye camera is installed in the simulated vehicle, the image stitching line optimization of the simulated fisheye camera is automatically performed in the simulation platform, so as to make the construction of the vehicle surrounding system more efficient. Figure 2AThe flowchart of the optimization of the external parameter line of the fisheye camera of the vehicle surround system of the embodiment of the present invention comprises the following steps: each simulated fisheye camera generates at least one second fisheye image based on its own initial external parameter (step S200); detects the coordinates of the second feature point in the second fisheye image (step S202); obtains the third feature point in the surround stitching image according to the external parameter of the simulated fisheye camera currently used for surround stitching and the second feature point (step S204); determines whether it is necessary to continue to optimize the current fisheye camera used for surround stitching according to whether the third feature point is qualified (for example, whether the error at the stitching is too large); The external parameters of the simulated fisheye camera are optimized (step S206); if the result of the judgment in step S206 is unqualified, the external parameters of the current simulated fisheye camera used for the surround stitching are optimized (step S208), and then the process returns to step S204; if the result of the judgment is qualified, the external parameters of the next simulated fisheye camera used for the surround stitching are selected (step S210), and then it is determined whether there are still external parameters of simulated fisheye cameras that need to be optimized (step S212); if there are still external parameters of simulated fisheye cameras that need to be optimized, the process returns to step S208; if the external parameters of all simulated fisheye cameras have participated in the optimization of the surround stitching image, the process is terminated (step S214).

[0074] Before executing step S200, if it is found that a simulated fisheye camera has no initial external parameters, the following steps can be performed first: Figure 1 The process is used to obtain the initial external parameters, and then execute Figure 2AIn addition, the number of second fisheye images captured in step S200 will not be affected by the real environment when the simulation platform generates fisheye images, so only one fisheye image can be generated for each simulated fisheye camera. In one embodiment, if the user wishes to obtain multiple second fisheye images (for example, the fisheye camera is used in dynamic scenes such as self-driving cars, robot navigation, etc.), the simulated checkerboard calibration plate can be moved on the simulation platform so that the simulated fisheye camera can generate images of the simulated checkerboard calibration plate at different positions and angles, or the vehicle can be moved so that the simulated fisheye camera can generate images of different scenes and perspectives. It should be noted that when multiple second fisheye images are generated on the simulation platform, the simulation platform can automatically generate a specified number of second fisheye images after setting the moving distance of the simulated checkerboard calibration plate, or setting the distance or angle that the simulated vehicle is to move, and the present invention does not limit this. As for the second feature point of step S202, it can be the edge, internal vertex, or other recognizable texture (such as lane lines or other recognizable objects) of the simulated checkerboard calibration plate to ensure that the correspondence between the camera coordinate system and the world coordinate system is correct. In some embodiments, when the second feature point is the internal vertex or other corner point of the checkerboard calibration plate, the external parameter optimization method of the present invention can use the checkerboard corner point detection algorithm to detect the coordinates of the internal vertex or other corner point of the checkerboard calibration plate in the coordinate system of the simulated fisheye camera in the second fisheye image, but the present invention does not limit the number of second fisheye images and the second identification point. In addition, in step S202, the external parameters of each simulated fisheye camera can be optimized (for example, optimized via the Gauss-Newton Algorithm) through the second feature points in multiple fisheye images, but the present invention is not limited to this.

[0075] The third feature point in step S204 can be a recognizable texture (such as a lane line) in the stitching area; in addition, the third feature point in step S204 is obtained by converting the correspondence between the camera coordinates and the world coordinates constructed by the second feature point and the external parameters of the simulated fisheye camera currently used for surround stitching. Figure 3 / Figure 4 Take the fisheye image as an example, if the second feature point is Figure 3 / Figure 4 The third feature point is the edge of the checkerboard calibration plate / lane line in the camera coordinate system and the world coordinate system. Figure 5B / Figure 6B The checkerboard calibration plate edge / lane line in the top-down view image of the vehicle shown. Figure 3 / Figure 4The distorted checkerboard calibration plate edge / lane line in the fisheye image can be processed in step S204 as shown in FIG. Figure 5B / Figure 6B Correctly displayed. Before stitching adjacent fisheye images, step S204 may compare and align the adjacent fisheye images in the common view area before stitching (referred to as "stitched image"). Step S206 is used to determine whether the error at the stitching of the common view area is too large, which can be achieved by using whether the matching of the third feature point is correct. The specific implementation method can be achieved by calculating whether the error at the stitching of the stitching image is higher than the preset stitching error threshold. In addition, if it is determined in step S206 that the error of the stitching image is greater than the preset stitching error threshold, the external parameters of the simulated fisheye camera used for stitching can be optimized again in step S208 based on the current external parameters (initial external parameters, or external parameters after the last optimization) (for example, the content of the translation matrix / rotation matrix is ​​adjusted again to fine-tune the translation distance and rotation angle), and then return to step S204 to obtain the third feature point again, but the present invention is not limited to this. In one embodiment, before selecting the next simulated fisheye camera for surround stitching in step S210, the current stitched image may be optimized, including minimizing the reprojection error of the stitched image (e.g., using a bundle adjustment method), adjusting the color of the stitched image (e.g., adjusting the color and brightness of the stitched image, because the colors or brightness generated by adjacent simulated fisheye cameras may be different), performing a smooth transition to reduce / eliminate the boundaries or inconsistencies at the stitching location, etc., but the present invention is not limited thereto. Step S212 selects the next simulated fisheye camera according to the preset sequence. For example, a four-camera vehicle is used. The cameras on the front side / left side / rear side / right side of the vehicle are numbered as 1 / 2 / 3 / 4 in a counterclockwise manner around the vehicle. Then, the simulated fisheye camera 1 / 2 (front side / left side of the vehicle) can be optimized for the surround stitching image first, and then the simulated fisheye camera 2 / 3 (left side / right side of the vehicle), the simulated fisheye camera 3 / 4 (rear side / right side of the vehicle), and the simulated fisheye camera 4 / 1 (right side / front side of the vehicle) are processed in sequence. If there is no problem, the process of image stitching optimization of the surround system is terminated by proceeding to step S214. In addition, in one embodiment, the panoramic stitching image generated by all simulated fisheye cameras is optimized for color, brightness, and smooth transition before the process is terminated in step S214, but the present invention is not limited to this.

[0076] For original vehicles that do not have fisheye cameras installed at the factory, if you need to install additional fisheye cameras in the aftermarket as a vehicle surround view system, you need to find a suitable place on the vehicle to install the fisheye camera. However, if there are many vehicles of the same model (for example, a fleet of the same vehicles) that need to be installed, the offline operation process must be used to obtain the initial external parameters for each vehicle of the same model, which will take a lot of time and require many of the same steps to be repeated manually. In this case, if the initial external parameters can be obtained using the simulation platform of the present invention (for example, via Figure 1 In addition, if the fisheye camera of the vehicle is affected by external forces and deviates (for example, someone accidentally hits the fisheye camera, causing the angle of the fisheye lens to deviate), the stitched image displayed by the vehicle's surround view system may become similar to Figure 5A / Fig. 6A Then, all the fisheye cameras installed on the vehicle take a fisheye image, and then the simulation platform of the present invention executes Figure 2A The process is used to re-optimize the external force of the fisheye camera to stitch the image of the surround view system. When the fisheye camera is offset by external force, the fisheye image of the vehicle surround view system can be stitched correctly again (for example, the image in the Figure 5B / Figure 6B The simulation platform of the present invention can also provide the original manufacturer with an evaluation of the constructed surround system after installing fisheye cameras for each vehicle. For example, since the simulation platform has built-in internal parameters corresponding to many simulated fisheye cameras, when the original manufacturer determines the vehicle characteristic parameters of the new car, the vehicle characteristic parameters can be added to the simulation platform, and different fisheye cameras can be selected from the simulation platform to perform surround image stitching until the fisheye camera that best suits the vehicle model and the recommended optimal installation position and angle are selected. This will be more efficient for optimizing the external parameters of the vehicle surround system of the designed new car. In some embodiments, if the user only knows the approximate size of the vehicle, or is unable to input the vehicle characteristics into the simulation platform for some reason (for example, the vehicle parameters have not yet been determined, or the vehicle parameter characteristics need to be kept confidential, etc.), a vehicle characteristic parameter that is closest to the desired initial external parameter can be selected on the simulation platform (for example, a simulated vehicle with a similar vehicle size is selected), and the initial external parameter is generated based on the simulated vehicle. In one embodiment, the user does not have to select a simulated vehicle with vehicle characteristic parameters closest to the original vehicle to generate the initial external parameters. He / she only needs to select a simulated vehicle in the simulation platform that can generate the required initial external parameters (for example, a simulated vehicle with a larger size) to start the subsequent initial external parameter acquisition and external parameter optimization procedures.

[0077] It should be noted that due to Figure 1 (get the initial external parameter) and Figure 2AThe process of (fisheye camera external parameter optimization of vehicle surround system) can be written into two different modules, so the simulation platform can be executed by Figure 1 The process obtains and stores the initial external parameters. At any time in the future, objects other than the checkerboard calibration plate can be used in the simulation platform and combined with Figure 2A The process optimizes the external parameters of the fisheye camera through the stitching of the surround images. It does not need to re-establish the previous simulation environment (such as re-selecting the same simulation site and placing the same simulation checkerboard calibration plate around the simulated vehicle, etc.), but performs external parameter optimization for all objects that the vehicle may encounter (such as lane lines). Figure 2A The process can be based on different vehicles (such as garbage trucks / mining trucks) and reference different objects (such as pedestrians / zebra crossings / lane lines, or workers / walls / cranes, etc.) to optimize the external parameters of the fisheye camera. Figure 1 and Figure 2A In the embodiments combined into one module, step S200 may be directly executed after step S106 is completed. Figure 2A The object referenced for fisheye camera optimization is the checkerboard calibration plate rather than other objects. In some embodiments, the initial external parameters are used to set the position and rotation angle of the simulated fisheye camera in the world coordinate system. In some embodiments, the world coordinate system is the coordinate system of the panoramic splicing image. Figure 2A Although the simulated fisheye camera is based on the external parameter optimization implementation of the vehicle surrounding system, there are still applications other than the vehicle surrounding system in reality, such as object detection and recognition, 3D reconstruction, augmented reality (AR) applications, machine vision and navigation, motion capture and posture estimation, real-time tracking of the camera's position and posture, and visual real-time positioning and mapping (Simultaneous Localization and Mapping, SLAM)... and other applications require optimization of the external parameters of the fisheye camera. For example, through the simulation platform of the present invention, Figure 1 After obtaining the initial external parameters of the fisheye camera through the process, the algorithm required for each application is called according to the actual needs of each application (for example, Figure 2A The process of optimizing the external parameters of the fisheye camera of the vehicle surround system is changed to the process of optimizing the external parameters of the fisheye camera of the 3D reconstruction to complete the online external parameter optimization of the fisheye camera in various applications, which is not limited by the present invention.

[0078] Since the actual installation of the fisheye camera will produce some deviation from the external parameters used for surround stitching, it would be helpful for the user to provide the tolerance for the installation of the fisheye camera. Based on this demand, the inventor of this case provides a simulation platform that uses the image stitching process of the surround system to obtain the installation tolerance online as a reference for the user when actually installing the fisheye camera. Figure 2B The flow chart of reference external parameter generation of the fisheye camera of the embodiment of the present invention is shown in FIG. 1 , wherein the reference external parameter is generated by adding an offset value to the initial external parameter. Figure 2B As shown, the method of the present invention includes: adding an offset value to the initial external parameter (or optimized external parameter) of one of the adjacent simulated fisheye cameras to become a reference external parameter (step S240); the simulation platform generates a third fisheye image of the simulated fisheye camera according to the reference external parameter (step S242); splicing the third fisheye image with the fisheye image generated by another adjacent simulated fisheye camera that does not add the offset value to the initial external parameter (or optimized external parameter) to generate a third surround stitching image (step S244); judging whether the third surround stitching image is qualified (step S246); if the third surround stitching image is still qualified, increasing the offset value (step S248), and then returning to step S240 to add the incremental offset value to the initial external parameter of one of the adjacent simulated fisheye cameras, so that the simulated fisheye camera whose external parameter is increased by the offset value and its adjacent simulated fisheye camera (no offset value is added to the external parameter) are combined again to generate an updated third surround stitching image. If the updated third surround stitching image is determined to be unqualified in step S246, the offset value at this time will become the installation tolerance for the user's reference (step S250). It should be noted that in the above-mentioned installation tolerance generation process, the user can set the offset value and the simulation platform will automatically generate it.

[0079] In one embodiment, in step S240, an offset value is added to one of the adjacent fisheye cameras. Taking the simulated fisheye cameras 1 / 2 (front side / left side of the vehicle) in the four-camera vehicle described above as an example, an offset value can be added to the initial external parameters of the simulated fisheye camera 1 (front side of the vehicle), while the initial external parameters of the simulated fisheye camera 2 (left side of the vehicle) remain unchanged. In step S244, since the external parameters of the simulated fisheye camera 1 are changed to reference external parameters, the third surround stitched image stitched by the simulated fisheye cameras 1 / 2 at this time will produce an error in the common view area compared to the stitched image generated by the optimized external parameters without adding any offset value. Therefore, when determining whether the error in the common view area exceeds the common view area error threshold in step S246, if the error in the common view area does not exceed the common view area error threshold, the process can go to step S248 to increase the offset value, and return to step S240 to regenerate the stitched image.

[0080] In some embodiments, the installation tolerance in the reference external parameter can also be used as a reference for some special applications, such as in applications where it is necessary to add a shell to the fisheye camera for additional protection (for example, a shell is added to the fisheye camera installed on the mine car to prevent the fisheye camera from being damaged by external forces (such as falling rocks)), and this installation tolerance can be used as a reference value for the shell thickness (for example, the shell thickness is less than the displacement indicated by the tolerance). In some embodiments, the installation tolerance can be automatically generated by the simulation platform after setting the offset value increase method. In the method of increasing the offset value in step S248, the position of the initial external parameter of the fisheye camera can be moved by only a preset position offset value each time (for example, 0.1 cm each time) without changing the angle, or only rotated by a preset rotation offset value each time from the angle of the initial external parameter (for example, rotated by 0.05°), and after obtaining one of the tolerances (for example, the movement value), the tolerance of the other (for example, the rotation value) is obtained. In some embodiments, the offset value can be added to the position and angle of the initial external parameter at the same time (for example, 0.1 cm and rotated by 0.05° each time). In another embodiment, preset rotation offset values ​​may be added to the yaw angle (Yaw), roll angle (Roll), and pitch angle (Pitch) respectively. In some embodiments, the reference external parameters obtained may be represented by a displacement (movement tolerance) of ±1 cm of the initial external parameters, and a rotation angle of ±0.5° (rotation tolerance) in step S250, but the present invention is not limited thereto. In some embodiments, the reference external parameters are generated by adding random noise to the initial external parameters of the fisheye camera in the simulation platform to simulate the situation where the position or angle of the fisheye camera of the real surrounding scene is offset. In actual operation, when the random noise is too small, a larger random noise can be obtained and then added to the initial external parameters to regenerate the fisheye image. In some embodiments, the reference external parameters after the camera position or angle is offset can only be known in the simulation scene (because the value of the external parameters after adding noise is known because the amount of noise is known).

[0081] Figure 3 FIG. 4 is a schematic diagram of a fisheye image including a checkerboard calibration plate according to an embodiment of the present invention. Figure 3 The fisheye image 300, the fisheye image 302, the fisheye image 304 and the fisheye image 306 are included. In some embodiments, the fisheye image 300 may be, for example, from the front fisheye camera of the simulated vehicle, the fisheye image 302 may be, for example, from the right fisheye camera of the simulated vehicle, the fisheye image 304 may be, for example, from the rear fisheye camera of the simulated vehicle, and the fisheye image 306 may be, for example, from the left fisheye camera of the simulated vehicle, but the present invention is not limited thereto. Figure 3As shown, the fisheye image 300, the fisheye image 302, the fisheye image 304 and the fisheye image 306 all include a checkerboard calibration plate 310. Therefore, the method for calibrating the external parameters of the present invention can be based on the internal vertices of the checkerboard calibration plate in the fisheye image 300, the fisheye image 302, the fisheye image 304 and the fisheye image 306 (that is, let the internal vertices become Figure 1 The first feature point in the simulation is used to calculate the initial external parameters of the simulated fisheye camera on the simulated vehicle, that is, Figure 1 Step S106. It is worth noting that Figure 3 Each fisheye image can be viewed on the simulation platform separately. Figure 1 In addition to calculating the initial external parameters of the corresponding fisheye camera through the process, the four fisheye images can also be combined on the simulation platform and Figure 2A The process is used to optimize the initial external parameters according to the needs of the vehicle surrounding system. Therefore, the simulation platform of the present invention can generate the initial external parameters of each fisheye camera at the same time and optimize the fisheye camera external parameters of the vehicle surrounding system when each fisheye camera only generates one fisheye image.

[0082] Figure 4 FIG. 4 is a schematic diagram of a fisheye image without a checkerboard calibration plate according to an embodiment of the present invention. Figure 4 The fisheye image 400, the fisheye image 402, the fisheye image 404 and the fisheye image 406 are included. In some embodiments, the fisheye image 400 may be, for example, from the front fisheye camera of the simulated vehicle, the fisheye image 402 may be, for example, from the right fisheye camera of the simulated vehicle, the fisheye image 404 may be, for example, from the rear fisheye camera of the simulated vehicle, and the fisheye image 406 may be, for example, from the left fisheye camera of the simulated vehicle, but the present invention is not limited thereto. Figure 4 As shown, fisheye image 400, fisheye image 402, fisheye image 404, and fisheye image 406 do not include a checkerboard calibration plate, so the external parameters of the fisheye camera cannot be calculated through the internal vertices of the checkerboard calibration plate, and other algorithms need to be used to obtain the initial external parameters of the fisheye camera (for example, using the Scale-Invariant Feature Transform (SIFT) algorithm and combining multiple fisheye images at different angles). However, the fisheye image 400, fisheye image 402, fisheye image 404, and fisheye image 406 can still be used to obtain the initial external parameters of the fisheye camera. Figure 2A To optimize the fisheye camera extrinsics of the vehicle surround view system.

[0083] Figure 5A The stitched image 500 includes a checkerboard calibration plate, in which the fisheye camera extrinsic parameters generated for the fisheye image are not optimized because the camera coordinate system and the world coordinate system do not correspond correctly. Figure 5AAs shown, the stitched image 500 includes a simulated vehicle 530, a checkerboard calibration plate 510, a checkerboard calibration plate 512, a checkerboard calibration plate 514, and a checkerboard calibration plate 516. In some embodiments, the checkerboard calibration plate 510 is disposed on the left front side of the simulated vehicle 530, the checkerboard calibration plate 512 is disposed on the right front side of the simulated vehicle 530, the checkerboard calibration plate 514 is disposed on the left rear side of the simulated vehicle 530, and the checkerboard calibration plate 516 is disposed on the right rear side of the simulated vehicle 530, but the present invention is not limited thereto. Since the camera coordinate system of the stitched image 500 does not correspond correctly to the world coordinate system, as shown in FIG. Figure 5A As shown, the checkerboard calibration plate 510 , the checkerboard calibration plate 512 , the checkerboard calibration plate 514 and the checkerboard calibration plate 516 all include discontinuous sections, and the splicing effect is not ideal.

[0084] Figure 5B For the stitched image 502 including the checkerboard calibration plate, the fisheye camera extrinsics for generating the corresponding fisheye image have been optimized because its camera coordinate system and the world coordinate system have been correctly aligned. Figure 5B As shown, the stitched image 502 includes a simulated vehicle 530, a checkerboard calibration plate 520, a checkerboard calibration plate 522, a checkerboard calibration plate 524, and a checkerboard calibration plate 526. In some embodiments, the checkerboard calibration plate 520 is disposed on the left front side of the simulated vehicle 530, the checkerboard calibration plate 522 is disposed on the right front side of the simulated vehicle 530, the checkerboard calibration plate 524 is disposed on the left rear side of the simulated vehicle 530, and the checkerboard calibration plate 526 is disposed on the right rear side of the simulated vehicle 530, but the present invention is not limited thereto. Since the stitched image 502 is generated by a fisheye camera whose camera coordinate system and the world coordinate system are correctly aligned, as shown in FIG. Figure 5B As shown, the checkerboard calibration plate 520, the checkerboard calibration plate 522, the checkerboard calibration plate 524 and the checkerboard calibration plate 526 do not have the problem of discontinuous sections, and the stitching effect of the stitched image 502 is obviously better than that of the stitched image 502. Figure 5A The stitched image 500 in FIG.

[0085] Fig. 6A FIG. 6 is a schematic diagram of a stitched image 600 that does not include a checkerboard calibration plate, wherein the fisheye camera extrinsic parameters corresponding to the fisheye image are generated and are not optimized because the camera coordinate system and the world coordinate system do not correspond correctly. Fig. 6A As shown in FIG. 6 , the stitched image 600 includes a simulated vehicle 640, a marking line 610, a marking line 620, and an object 630. In some embodiments, the marking line 610 is located on the left side of the simulated vehicle 640. The marking line 620 and the object 630 are located on the right side of the simulated vehicle 640, but the present invention is not limited thereto. Since the camera coordinate system of the stitched image 600 does not correspond correctly to the world coordinate system, Fig. 6AAs shown, marking line 610 , marking line 620 and object 630 all include discontinuous sections, and the splicing effect is not ideal.

[0086] Figure 6B The stitched image 602 includes a schematic diagram of a checkerboard calibration plate, and the fisheye camera extrinsics for generating the corresponding fisheye image have been optimized because the camera coordinate system and the world coordinate system have been correctly aligned. Figure 6B As shown in FIG. 6 , the stitched image 602 includes a simulated vehicle 640, a marking line 650, a marking line 660, and an object 670. In some embodiments, the marking line 650 is located on the left side of the simulated vehicle 640. The marking line 660 and the object 670 are located on the right side of the simulated vehicle 640, but the present invention is not limited thereto. Since the camera coordinate system of the stitched image 602 does not correspond correctly to the world coordinate system, Figure 6B As shown, the marking line 650, the marking line 660 and the object 670 have no discontinuous section problem, and the stitching effect of the stitched image 602 is obviously better than Fig. 6A The stitched image 600 in FIG.

[0087] Figure 7 The present invention is a flowchart of adjusting the stitching seam of the surround stitching when an object is found at the stitching of the surround stitching image in an embodiment of the present invention. Although the surround stitching image can provide the vehicle driver with a real-time situation around the vehicle for reference, when an object appears at the stitching, because the position of the stitching seam is fixed during the image stitching process, the object on the stitching seam disappears or cannot provide sufficient information for the vehicle driver to refer to, which may cause an accident while the vehicle is driving. Based on this consideration, the present invention is particularly directed to adjusting the stitching seam when an object appears at the stitching of the surround stitching image so that the stitching seam can provide the vehicle driver with more information about the object for reference. Figure 7As shown, the method for adjusting the seam of the present invention includes: receiving a fourth fisheye image from a first fisheye camera and a fifth fisheye image from a second fisheye camera (step S700); detecting a target object in the fourth fisheye image and the fifth fisheye image (step S702); projecting the fourth fisheye image, the fifth fisheye image, and the target object to a stitched image, the stitched image including the seam, and the target object is located on the seam (step S704); calculating an adjustment direction of the seam according to the distance between the target object and the first fisheye camera and the second fisheye camera in the stitched image (step S706); determining whether the seam reaches a limit value (step S708); and adjusting the seam according to the adjustment direction (step S710). In some embodiments, the first fisheye camera in step S700 is adjacent to the second fisheye camera, so the target object will appear in both the fourth fisheye image and the fifth fisheye image generated by it. In some embodiments, the target object is a specific object (e.g., a bus). In some embodiments, the fourth fisheye image in step S700 is projected to the left side of the stitched image in step S704, the fifth fisheye image in step S700 is projected to the right side of the stitched image in step S704, and the stitching seam in step S704 is located between the fourth fisheye image and the fifth fisheye image in the stitched image, but the present invention is not limited thereto. When the fourth / fifth fisheye images are stitched, some information is deleted based on the position of the stitching seam. For example, the information on the right side of the stitching seam of the fourth fisheye image may not be displayed in the surround stitching image after stitching; and the information on the left side of the stitching seam of the fifth fisheye image may not be displayed in the surround stitching image after stitching. Therefore, when the position of the stitching seam is fixed, even if a certain fisheye camera captures more information about the target object, this information may be discarded when the images are stitched, so only part of it is displayed or even not displayed in the surround stitching image. However, if the target object information that was originally excluded can be displayed to the vehicle driver when the images are stitched, the occurrence of traffic accidents can be reduced. In addition, since the stitching of the panoramic stitching image is to stitch the common view area that appears in the fisheye images taken by the adjacent fisheye cameras, there will be a stitching seam at the stitching (which will not be displayed in the actual stitching image), and the images on both sides of the stitching seam are basically from different adjacent fisheye cameras, so adjusting the direction of the stitching seam can actually change the content displayed at the stitching (for example, let the image taken by one of the adjacent fisheye cameras show more information in the stitching image). Therefore, adjusting the direction of the stitching seam in step S706 is to determine which fisheye image has more target object information among the fisheye images taken by the adjacent fisheye cameras, and then adjusting the direction of the stitching seam to display more object information to the vehicle driver.

[0088] In some embodiments, the limit value in step S708 is an external parameter associated with the first fisheye camera and the second fisheye camera, such as the image range limit that the fisheye camera can capture based on the current position, rotation angle, and field of view (FOV) of the vehicle. In some embodiments, the present invention Figure 7 The seam adjustment method can be Figure 1 The method for adjusting the seam of the present invention is verified in a simulation platform (for example, Carla), but the present invention is not limited thereto. In step S708, when the method for adjusting the seam of the present invention determines that the seam has reached the limit value, it returns to step S700, that is, it continues to receive a sixth fisheye image from the first fisheye camera and a seventh fisheye image from the second fisheye camera. In some embodiments, the surround stitching is drawn by the drawing software OPENGL (Open Graphics Library). When the seam reaches the limit value (for example, the slope of the seam is too large), jagged edges or other errors may appear on the seam because all fisheye images do not have pixels with target object information. In step S708, when the method for adjusting the seam of the present invention determines that the seam has not reached the limit value, the method for adjusting the seam of the present invention executes step S710, that is, according to the adjustment direction obtained in step S706, the seam is adjusted. In some embodiments, when no target object is detected at the joint in step S702, the method for adjusting the seam of the present invention does not execute subsequent steps. It should be noted that when the limit value is reached, the stitching seam will remain unchanged to generate a surround stitching image for display to the vehicle driver. In addition, when the target object disappears from the stitching seam, the slope of the stitching seam can be regressed to optimize the stitching seam slope used by the external parameter. The advantage of this is that when an object appears at the stitching, the content of the surround stitching image can be adjusted in real time to provide the vehicle driver with reference to prevent the vehicle from colliding with the object, and after the object disappears from the stitching, the stitching seam slope of the external parameter is immediately regressed to optimize to provide the driver with an optimized surround stitching image.

[0089] Figure 8 For the embodiment of the present invention Figure 7 The detailed flow chart of step S706 is shown in FIG. Figure 8As shown, the stitching seam adjustment method of the present invention includes: capturing a representative point of the target object according to the type of the target object in step S702 (step S800); projecting the representative point to the stitching image in step S704 (step S802); and calculating the adjustment direction of the stitching seam according to the distance between the representative point and the first fisheye camera and the second fisheye camera in the stitching image (step S804). In detail, the type of the target object in step S800 may include, for example, a bus, a car, a pedestrian, a truck, and a rider, but the present invention is not limited thereto. For example, when the target object is a bus, the stitching seam adjustment method of the present invention sets the type of the target object to 0. When the target object is a car, the stitching seam adjustment method of the present invention sets the type of the target object to 1. When the target object is a pedestrian, the stitching seam adjustment method of the present invention sets the type of the target object to 2. When the target object is a truck, the stitching seam adjustment method of the present invention sets the type of the target object to 3. When the target object is a knight, the method for adjusting the seam of the present invention sets the type of the target object to 4. However, the present invention is not limited thereto.

[0090] In some embodiments, when the seam adjustment method of the present invention determines that the target object is a bus, a car, or a truck, the seam adjustment method of the present invention generates a target frame around the target object. Then, the seam adjustment method of the present invention captures the geometric center point of the target frame as the representative point of the target object in step S800. When the seam adjustment method of the present invention determines that the target object is a pedestrian or a rider, the seam adjustment method of the present invention generates a target frame around the target object. Then, the seam adjustment method of the present invention captures the lower edge corner point of the target frame as the representative point of the target object in step S800.

[0091] Fig. 9 For the embodiment of the present invention Figure 8 The detailed flow chart of step S804 is shown in FIG. Fig. 9 In the embodiment of FIG. 5 , the fourth fisheye image is projected to the left side of the stitched image, and the fifth fisheye image is projected to the right side of the stitched image. Fig. 9As shown, the stitching seam adjustment method of the present invention includes: calculating a first distance between the representative point and the first fisheye camera (step S900); calculating a second distance between the representative point and the second fisheye camera (step S902); comparing the first distance and the second distance (step S904); when it is determined that the first distance is less than the second distance (step S906), reducing the slope of the stitching seam by a preset value so that the weight of the first fisheye image in the stitching image is increased (step S908); when it is determined that the second distance is less than the first distance (step S910), increasing the slope of the stitching seam by a preset value so that the weight of the second fisheye image in the stitching image is increased (step S912). The reason for determining the first distance and the second distance in step S904 is to find out the fisheye camera that is closer to the target object between the first fisheye camera and the second fisheye camera, and the fisheye camera that is closer to the target object can generally capture more target object areas, and thus can provide more target object information. In some embodiments, if the slope variation of step S908 is too large, the spliced ​​image may flicker during continuous frame playback. If the slope variation of step S908 is too small, the spliced ​​image may not be easily perceived during continuous frame playback. The slope variation of each frame of the spliced ​​image of the present invention is 0.005, so that the spliced ​​image is smoothly presented during continuous frame playback. That is, the above-mentioned preset value is 0.005, but the present invention is not limited thereto.

[0092] It should be noted that there may be more than one target object at the joint (e.g., multiple pedestrians), and the types of the multiple target objects at the joint may also be different (e.g., there may be riders and buses). Therefore, on the simulation platform of the present invention, the target objects that may appear at the joint seams of the stitching images of the surround view systems of different types of vehicles can be simulated before they are actually on the road, in order to reduce possible traffic accidents.

[0093] Fig. 10A FIG. 1 is a schematic diagram of a stitched image 1000 including a stitching seam in the prior art. The stitched image 1000 does not adopt the stitching seam adjustment method of the present invention. Fig. 10A As shown, the stitched image 1000 includes a simulated vehicle 1050, a first-level warning box 1010, a second-level warning box 1012, a stitching seam 1030, a target object 1040, and a representative point 1020. In some embodiments, the first-level warning box 1010 and the second-level warning box 1012 surround a center point of the simulated vehicle 1050 in the stitched image 1000. The distance between the point on the first-level warning box 1010 and the center point of the simulated vehicle 1050 is less than the distance between the point on the second-level warning box 1012 and the center point of the simulated vehicle 1050. In some embodiments, in the stitched image 1000, the stitching seam 1030 is set between the image generated by the fisheye camera on the front side of the simulated vehicle 1050 and the image generated by the fisheye camera on the right side of the simulated vehicle 1050. Fig. 10A Other seams are also included, but the present invention takes the seam 1030 as an example.

[0094] like Fig. 10A As shown, since the spliced ​​image 1000 does not adopt the splicing seam adjustment method of the present invention, although there is a target object 1040 at the splicing position and a representative point 1020 is generated according to the position of the target object 1040, the image of the target object 1040 only appears in the first-level warning frame 1010, and the image of the target object 1040 disappears between the first-level warning frame 1010 and the second-level warning frame 1012. In other words, it is difficult for the driver to observe the target object 1040 in the spliced ​​image 1000, resulting in a visual blind spot, which is easy to cause danger during driving.

[0095] Fig. 10B FIG. 1 is a schematic diagram of a stitched image 1002 including a stitching seam using an embodiment of the present invention. The stitched image 1002 uses the stitching seam adjustment method of the present invention. Fig. 10B As shown, the stitched image 1002 includes a simulated vehicle 1050, a first-level warning box 1010, a second-level warning box 1012, a stitching seam 1030′, a target object 1040, and a representative point 1020. In some embodiments, the first-level warning box 1010 and the second-level warning box 1012 surround the center point of the simulated vehicle 1050 in the stitched image 1002. The distance between the point on the first-level warning box 1010 and the center point of the simulated vehicle 1050 is smaller than the distance between the point on the second-level warning box 1012 and the center point of the simulated vehicle 1050. In some embodiments, in the stitched image 1002, the stitching seam 1030′ is set between the image generated by the fisheye camera on the right side of the simulated vehicle 1050 and the image generated by the fisheye camera on the front side of the simulated vehicle 1050. Fig. 10A Other seams are also included, but the present invention takes the seam 1030 as an example.

[0096] like Fig. 10B As shown, since the stitching image 1002 adopts the stitching seam adjustment method of the present invention, the absolute value of the slope of the adjusted stitching seam 1030' is less than Fig. 10AThe absolute value of the slope of the stitching seam 1030 allows the stitching image 1002 to display a complete image of the target object 1040 (because it contains more image information captured by the right fisheye camera which is closer to the target object 1040, and also contains more image texture information of the target object 1040). In some embodiments, when the stitching seam adjustment method of the present invention determines that the target object 1040 is within the first-level warning box 1010 or the second-level warning box 1012, the stitching seam adjustment method of the present invention adjusts the stitching seam 1030' according to the adjustment direction calculated in step S706. In some embodiments, when the stitching seam adjustment method of the present invention determines that the target object 1040 is not within the first-level warning box 1010 or the second-level warning box 1012, the stitching seam adjustment method of the present invention does not adjust the stitching seam 1030. In some embodiments, Fig. 10A / Fig. 10B The stitching seams 1030 / 1030 ′ appearing in the figure are only used for a clearer illustration. There is no visible stitching seam in the real stitched image, so the stitching seams 1030 / 1030 ′ are only marked later for better highlighting.

[0097] Fig.11 FIG. 1 is a schematic diagram of an electronic device 1100 according to an embodiment of the present invention. Fig.11 As shown, the electronic device 1100 includes a processor 1102 and a plurality of fisheye cameras 1104. The processor 1102 may execute a simulation platform 1110, an external parameter extraction algorithm 1112, a checkerboard corner detection algorithm 1114, and a surround stitching algorithm 1116. In some embodiments, the processor 1102 may obtain the fisheye image and its internal parameters from the fisheye camera 1104. In some embodiments, the simulation platform 1110 may be, for example, Carla, PerScan, CarSim, VIRES VTD, PTV Vissim, and TESS NG, but the present invention is not limited thereto. The processor 1102 executes the simulation platform 1110 to execute the external parameter extraction algorithm 1112 to obtain the process of initial external parameters, execute the checkerboard corner detection algorithm 1114 to obtain the feature points of the checkerboard calibration plate, or execute the surround stitching algorithm 1116 to complete the fisheye camera external parameter optimization process of the vehicle surround system.

[0098] In some embodiments, the processor 1102 executes the simulation platform 1110 to complete Figure 2B The reference external parameter processing flow and the execution of the external parameter extraction algorithm 1112 are used to Figure 2B The process obtains feature points and executes the ambient stitching algorithm 1116 to determine whether the offset value in the reference external parameter is too large.

[0099] In some embodiments, the processor 1102 executes the ambient stitching algorithm 1116 to complete Figure 7 , Figure 8 and Fig. 9 process.

[0100] The surround stitching method of the present invention can effectively improve the efficiency of developers in verifying and improving algorithms, and can accelerate the development process of surround view systems by automatically obtaining the initial external parameters of the fisheye camera and automatically calibrating the external parameters of the fisheye camera according to the requirements of the vehicle surround view system. The stitching seam adjustment method of the present invention can dynamically adjust the stitching seam position when a target object is detected at the stitching position of the vehicle surround view system stitching image, so that the stitching image re-displays the disappeared target object (such as a pedestrian), thereby improving the integrity of the surround view system and enhancing the driving safety of the vehicle.

[0101] Although embodiments of the present invention are described above, it should be understood that what is presented above is merely an example and not a limitation. Many changes to the above exemplary embodiments can be implemented without violating the spirit and scope of the disclosure according to this embodiment. Therefore, the breadth and scope of the present invention should not be limited by the embodiments described above. Rather, the scope of the present invention should be defined by the following claims and their equivalents. Although the above disclosure has been illustrated and described by one or more related executions, equivalent changes and modifications will be thought of by others who are familiar with this field based on the above specifications and drawings. In addition, although a particular feature of the present invention has been demonstrated by one of the related multiple executions, the above feature may be combined with one or more other features so that there may be a need and help for any known or special application.

[0102] The professional terms used in this specification are only for the purpose of describing specific embodiments and are not intended to be used as limitations of the present invention. Unless the context clearly indicates otherwise, the singular form, one, this and the above-mentioned meanings as used herein also include plural forms. Furthermore, the words "include", "comprise", "have", "have", or their variations are either used as detailed descriptions or as claims. The above-mentioned words mean to include, and to some extent are equivalent to the word "include". Unless there are different definitions, all the words used in this article (including technical or scientific terms) can be generally understood by people with general skills in the above-disclosed technology. We should be more aware that the above-mentioned words, such as those defined in commonly used dictionaries, should be interpreted as the same meaning in the context of the relevant technology. Unless explicitly defined in this article, the above-mentioned words will not be interpreted as idealized or overly formal.

Claims

1. A method for obtaining the installation tolerance of a fisheye camera online, which is applied to an online simulation platform for optimizing the external parameters of a fisheye camera for surround stitching, and is used to obtain the installation tolerance of a fisheye camera installed on a vehicle, the method comprising: generating reference external parameters from one of a plurality of simulated fisheye cameras disposed on the simulated vehicle; Using the reference external parameters to enable the simulated fisheye camera to generate a third fisheye image; Using the third fisheye image and a third fisheye image generated by an adjacent simulated fisheye camera adjacent to the simulated fisheye camera to perform image stitching to generate a third surround stitching image; and When the common view area error of the third surround stitching image is higher than the common view area error threshold, the offset value in the current reference external parameter becomes the installation tolerance.

2. The method according to claim 1, further comprising: When the common view area error of the third surround stitching image is lower than the common view area error threshold, the offset value is increased by an increment to generate an updated reference external parameter.

3. The method according to claim 2, wherein: The increment is a preset position offset value or a preset rotation offset value.

4. The method according to claim 2, wherein: The increment is either a random position offset value or a random rotation offset value.

5. The method according to claim 2, further comprising: The simulation platform generates an updated third fisheye image according to the updated reference external parameters; The updated third fisheye image is stitched with the third fisheye image generated by the adjacent simulated fisheye camera adjacent to the simulated fisheye camera to generate an updated third surround stitching image; and It is determined whether the common view area error of the updated third surround stitching image is higher than the common view area error threshold.

6. The method according to claim 1, further comprising: When a fisheye camera corresponding to the simulated fisheye camera is additionally equipped with a housing, the thickness of the housing needs to be smaller than the displacement indicated by the installation tolerance.

7. The method according to claim 1, wherein: The reference external parameter is generated by adding the initial external parameter or the optimized external parameter of the simulated fisheye camera into the offset value.

Images