Panoramic image processing method and device, electronic equipment and storage medium

Abstract

The invention relates to a panoramic image processing method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining target images collected by a plurality of cameras, the distortion parameters corresponding to the plurality of cameras being different; determining a distortion parameter corresponding to each camera from a pre-calibrated incidence relation between the camera and the distortion parameter; and performing distortion correction on the target image correspondingly acquired by the camera according to the distortion parameter corresponding to the camera, and performing fusion based on all corrected images to obtain a panoramic image. According to the panoramic image processing method, distortion correction and splicing processing can be carried out on the 360-degree panoramic module with different cameras installed in a mixed mode, and compatibility is good.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a panoramic image processing method, device, electronic equipment and storage medium. Background Art

[0002] Panoramic imaging refers to capturing a wide angle of view, usually a 360-degree field of view, through a camera. This type of imaging allows viewers to see the entire surrounding environment in one image or video. Panoramic imaging is usually based on a camera device with a combination of 4 cameras, which simultaneously captures images in different directions and then stitches these images into a complete panorama. It can be used in the fields of automobiles, virtual reality (VR), security monitoring, and video conferencing. It is widely used in the automotive field. Panoramic imaging (360-degree surround view camera) can provide a more comprehensive field of view, helping drivers better understand the surrounding environment, avoid collisions, and enhance driving safety.

[0003] At present, the cameras of 360-degree surround-view cameras mainly include lenses, sensors, ISPs (Image Signal Processors), power chips, storage chips, SERDES (video transmission chips), etc. In related technologies, image processing methods (such as distortion correction algorithms and stitching algorithms) are not compatible with different types of cameras, which means that the types of the four cameras must be consistent (for example, the parameters of the cameras must be consistent). Therefore, in related technologies, surround-view cameras are combined with cameras from the same supplier, and cannot be mixed based on different cameras, and the image processing methods have poor compatibility. Summary of the invention

[0004] One of the purposes of the present invention is to provide a panoramic image processing method to solve the problem of poor compatibility of image processing methods in the prior art, so that the surround-view camera equipment can be mixed with cameras from different suppliers; the second purpose is to provide a panoramic image processing device; the third purpose is to provide an electronic device.

[0005] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0006] A panoramic image processing method, the method comprising:

[0007] Obtain target images captured by multiple cameras respectively, where the distortion parameters corresponding to the multiple cameras are different;

[0008] Determine the distortion parameters corresponding to each camera from the association relationship between the pre-calibrated cameras and the distortion parameters;

[0009] The target image acquired by the camera is subjected to distortion correction according to the distortion parameters corresponding to the camera, and all the corrected images are fused to obtain a panoramic image.

[0010] According to the above technical means, the product data corresponding to the cameras provided by various suppliers are stored in the storage unit in advance. When the installed cameras are calibrated, the product data corresponding to different cameras can be retrieved to calibrate the cameras. After the cameras are calibrated, when the images collected by different cameras are processed during power-on use, the distortion parameters, camera internal parameters, camera external parameters and other information corresponding to each camera can be retrieved based on the pre-calibrated distortion parameters and pre-written product data, and the images collected by each camera can be converted, distorted and spliced ​​to obtain panoramic images. In this way, cameras provided by different suppliers can be mixed to form a 360-degree surround view module.

[0011] Further, the step of acquiring target images captured by multiple cameras respectively includes:

[0012] Query the product data of the camera from a preset storage unit, and determine the transformation matrix of the original image captured by the camera from the camera coordinate system to the world coordinate system according to the product data of the camera;

[0013] In the world coordinate system, a bowl-shaped model is established with the target point as the origin;

[0014] Based on the transformation matrix, the original image is mapped to the bowl-shaped model under the world coordinates to obtain a target image, and the original images of multiple cameras are mapped to the bowl-shaped model to obtain multiple target images.

[0015] According to the above technical means, the original images are converted into the same bowl-shaped model. On the one hand, images from different suppliers can be converted into the same coordinate system. On the other hand, the target image is displayed in a bowl-shaped model, so that the target image is closer to the image captured by the surround-view camera (or fisheye camera), the image distortion correction is more accurate, and the display effect of the stitched image is better.

[0016] Further, the step of acquiring target images captured by multiple cameras respectively includes:

[0017] When it is detected that the camera is powered on, a pre-calibrated camera is queried, wherein the pre-calibrated camera is correspondingly associated with a distortion parameter and is stored in a storage unit;

[0018] When it is determined that the pre-calibrated cameras include a powered-on camera, a target image captured by the powered-on camera is obtained, and when multiple cameras are powered on, target images captured by the multiple cameras are correspondingly obtained.

[0019] According to the above technical means, each time the camera is powered on, it is checked whether the powered-on camera is calibrated. In the mixed installation solution of multiple types of cameras, it can be guaranteed that the subsequent image acquisition and image processing processes can proceed normally.

[0020] Furthermore, the step of acquiring target images captured by multiple cameras respectively further includes:

[0021] When it is determined that the pre-calibrated cameras do not include the powered-on camera, querying product data corresponding to the powered-on camera from a preset storage unit, wherein the product data includes at least one of basic information of the camera, lens information, camera internal parameters, and camera external parameters;

[0022] According to the preset distortion correction algorithm and the product data of the powered-on camera, distortion calibration is performed to obtain distortion parameters;

[0023] Based on the powered-on camera and the distortion parameter, after updating the association relationship between the pre-calibrated camera and the distortion parameter, image information respectively captured by multiple cameras is obtained.

[0024] According to the above technical means, each time the camera is powered on, it is checked whether the powered-on camera is calibrated, and if it is calibrated, the camera is calibrated. In some applications, the above means can increase the flexibility of camera combination replacement.

[0025] Further, querying the product data of the camera from the preset storage unit includes:

[0026] According to the mapping relationship between the image sensor of the camera and the storage unit, access the corresponding storage unit;

[0027] Accessing the camera information base address in the storage unit, and obtaining basic information, lens information, camera internal parameters and camera external parameters of the corresponding camera based on the base address;

[0028] Access the lens information offset address in the storage unit, and obtain the image sensor model and firmware version of the corresponding camera, the camera's vendor information, software and hardware numbers and version numbers, and configurable parameters based on the offset address, wherein the configurable parameters include video output pixels, video frame rate, and data format.

[0029] According to the above technical means, when a 360-degree surround view module is composed of a mix of cameras provided by multiple suppliers, the product data corresponding to the cameras can be retrieved, and then the initial images captured by different cameras can be processed to obtain the target image that meets the stitching requirements.

[0030] Further, a method of obtaining configurable parameters based on the offset address:

[0031] The configurable parameters corresponding to the preset specifications are obtained based on the offset address, and the preset specifications corresponding to all cameras are the same.

[0032] According to the above technical means, the configurable parameters of each camera can be configured consistently to ensure the display effect.

[0033] A panoramic image processing device, the device comprising:

[0034] An acquisition module is used to acquire target images respectively acquired by multiple cameras, where the distortion parameters corresponding to the multiple cameras are different;

[0035] A determination module, used to determine the distortion parameters corresponding to each camera from the association relationship between the pre-calibrated cameras and the distortion parameters;

[0036] The distortion correction fusion module is used to perform distortion correction on the target image acquired by the camera according to the distortion parameters corresponding to the camera, and to fuse all the corrected images to obtain a panoramic image.

[0037] An electronic device, comprising: a memory, a processor;

[0038] The memory is used to store computer programs / instructions; the processor is used to implement the above-mentioned method according to the computer programs / instructions stored in the memory.

[0039] Optionally, the electronic device includes a car.

[0040] A computer-readable storage medium stores a computer program / instruction, wherein the computer program / instruction is used to implement the method described above when executed by a processor.

[0041] A computer program product, comprising a computer program / instruction, wherein the computer program / instruction is used to implement the method as described above when executed by a processor.

[0042] Beneficial effects of the present invention:

[0043] (1) Pre-store the product data corresponding to cameras provided by various suppliers in a storage unit. When calibrating the installed cameras, the product data corresponding to different cameras can be retrieved to calibrate the cameras. After the cameras are calibrated, when the images collected by different cameras are processed during power-on use, the distortion parameters, camera internal parameters, camera external parameters and other information corresponding to each camera can be retrieved based on the pre-calibrated distortion parameters and pre-written product data. The images collected by each camera can be converted, distorted and spliced ​​to obtain a panoramic image. In this way, cameras provided by different suppliers can be mixed to form a 360-degree surround view module.

[0044] (2) Before distortion correction and stitching, the present invention converts the image into the same bowl-shaped model based on the product data of the camera. Then, during distortion correction, an SDK algorithm is used to call the distortion parameters corresponding to the camera to perform distortion correction on the target image captured by the camera. This can achieve distortion correction and stitching of images captured by cameras of different schemes. There is no need to use an SDK algorithm package for distortion correction for each camera, thereby reducing the deployment of the SDK algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 A schematic diagram of an application scenario of the panoramic image processing method provided in an embodiment of the present application;

[0046] Figure 2 A schematic diagram of a flow chart of a panoramic image processing method provided in one embodiment of the present application;

[0047] Figure 3 for Figure 2 A further detailed flowchart of step S201;

[0048] Figure 4 A schematic diagram of a flow chart of a panoramic image processing method provided by another embodiment of the present application;

[0049] Figure 5 A schematic diagram of a method for storing product data of a camera provided in one embodiment of the present application. DETAILED DESCRIPTION

[0050] The following will describe the embodiments of the present invention with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be understood that the preferred embodiments are only for illustrating the present invention, not for limiting the scope of protection of the present invention.

[0051] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and thus the drawings only show components related to the present invention rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0052] Panoramic imaging refers to capturing a wide angle of view, usually a 360-degree field of view, through a camera. This type of imaging allows viewers to see the entire surrounding environment in one image or video. Panoramic imaging is usually based on a camera device with a combination of 4 cameras, which simultaneously captures images in different directions and then stitches these images into a complete panorama. It can be used in the fields of automobiles, virtual reality (VR), security monitoring, and video conferencing. It is widely used in the automotive field. Panoramic imaging (360-degree surround view camera) can provide a more comprehensive field of view, helping drivers better understand the surrounding environment, avoid collisions, and enhance driving safety.

[0053] At present, the camera of 360-degree surround view camera equipment mainly includes lens, sensor, ISP (Image Signal Processor), power chip, storage chip, SERDES (video transmission chip) and other parts. In the related technology, generally, multiple cameras on the same product are tendered together and installed with cameras from the same supplier. Therefore, the current image processing methods for 360-degree surround view camera equipment (such as distortion correction algorithms and stitching algorithms) are all adapted to the cameras of the same supplier, and are not compatible with different types of cameras (cameras from different suppliers), that is, the types of the four cameras are required to be consistent (for example, the parameters of the cameras are consistent). Therefore, in the related technology, surround view camera equipment is combined with cameras from the same supplier, and cannot be mixed based on different cameras, and the compatibility of image processing methods is poor.

[0054] In some special products, cameras from different suppliers are installed on the same product, but generally different SDK algorithm packages are called to process images collected by cameras from different suppliers. This method requires high computing power.

[0055] Based on this, an embodiment of the present application provides a panoramic image processing method. In a scenario where cameras from different suppliers are combined and mixed, based on an SDK algorithm package, distortion correction and stitching processing can be performed on images captured by the mixed cameras.

[0056] The following takes the 360-degree surround view module applied in a vehicle as an example to explain various implementation examples of the panoramic image processing method in detail. It should be noted that the application of the panoramic image processing method proposed in the embodiment of the present application includes but is not limited to the 360-degree surround view module of a vehicle.

[0057] Figure 1 A schematic diagram of a scenario provided in an embodiment of the present application, such as Figure 1 As shown, the 360-degree surround view module of the vehicle includes at least four cameras 10, and the four cameras 10 are respectively distributed in the front, rear, left and right sides of the vehicle body. The viewing angles of the four cameras 10 synthesize the 360-degree surround view of the vehicle.

[0058] In this embodiment, the camera 10 includes a lens, an image sensor (sensor), an image signal processor (ISP), a power chip, a storage chip, and a video transmission chip (SERDES). The ISP can be integrated into the sensor or exist as a separate chip in the camera assembly.

[0059] Among them, the lens is used to capture light and focus it onto the image sensor. The image sensor (Sensor) is used to convert light signals into electrical signals to generate raw digital image data. Each lens corresponds to a sensor. The image signal processor (ISP) is used to perform preliminary processing on the raw image data generated by the sensor, including denoising, white balance adjustment, color correction, gamma correction, and sharpening. Ensure that the image quality remains consistent during subsequent processing and stitching. The video transmission chip is used to transmit the processed image data to the central processing unit (CPU) or image processing unit (GPU). The transmission can be completed through different interfaces, such as MIPI, USB, Ethernet, etc.

[0060] In this embodiment, the four cameras are allowed to be provided by different suppliers, that is, the distortion parameters of the four cameras are allowed to be different. However, in vehicle applications, since the video transmission chip and the video decoding chip on the multimedia host need to be used in pairs, and the general panoramic image video decoding chip is an integrated four-in-one solution, that is, one decoding chip completes the decoding of the image signals of the four panoramic cameras, so the video transmission chips of the four cameras are required to be consistent, and the video transmission chip solution selection can only affect the underlying driver configuration software, and cannot affect the panoramic image SDK algorithm package, so that one SDK algorithm package can be used to process images collected by cameras from different suppliers. Therefore, different suppliers can use the same model or the same manufacturer's video transmission chip to achieve mixed installation of cameras.

[0061] The power chip provides power for the entire camera module and automatically matches the corresponding voltage of each chip. It only affects the power-on timing of the camera and does not affect the SDK algorithm package software. Therefore, the type of power chip in cameras from different suppliers has no effect on the image processing process.

[0062] The lens's horizontal field of view angle HFOV, vertical field of view angle VFOV, distortion coefficient, internal and external parameters, and the sensor chip's pixels, pixel size, and other parameters have a significant impact on image distortion correction and 360-degree surround stitching algorithms, and will also directly affect the panoramic imaging SDK algorithm package. Therefore, the panoramic audio SDK algorithm package is compatible with cameras from different suppliers, which is actually a compatible adaptation of different combinations of the two key components of the lens and sensor.

[0063] In the related art, the pixel specifications of surround view camera sensors include 1 million and 3 million. If four cameras of similar size are installed on the same vehicle, the clarity of the four angles after the surround view is stitched will be too different due to the large pixel difference, affecting the user experience. Therefore, when selecting cameras from different suppliers, it is necessary to select sensors with the same pixel specifications. On this basis, cameras from different suppliers can be mixed on the same vehicle.

[0064] The image processing process is described in detail below.

[0065] Figure 2 A flowchart of a panoramic image processing method provided by an embodiment of the present application is shown in FIG. Figure 2 As shown, the method comprises the following steps:

[0066] S201, obtaining target images respectively captured by multiple cameras, where the distortion parameters corresponding to the multiple cameras are different;

[0067] First, the product data corresponding to the cameras of multiple suppliers are written in advance in FLASH. The product data includes at least one of the basic information of the camera, lens information, camera internal parameters and camera external parameters. And establish the association relationship between the camera and the product data storage unit to facilitate the retrieval of the product data corresponding to the camera, distortion calibration and / or image conversion processing.

[0068] It should be noted that the basic information of the camera includes the image sensor model and firmware version, the camera supplier information, software and hardware number and version number, video output pixels, video frame rate, and data format. The camera lens information includes the lens model, lens horizontal angle / lens vertical angle, lens distortion parameters, etc.

[0069] Then, after the camera is installed on the vehicle for the first time, the camera is calibrated for distortion. The distortion coefficient corresponding to the installed camera is retrieved through the pre-written product data corresponding to the cameras of each supplier, and the installed camera is calibrated through a preset calibration method (such as the chessboard method) to obtain the distortion parameters corresponding to the installed camera. Since the product data of each camera on the vehicle is different, during the calibration process, calibration is performed based on the product data of each camera respectively to obtain the distortion parameters corresponding to each camera, and the distortion parameters and product data are associated to form an association relationship between the camera and the distortion parameters. Optionally, the association relationship between the camera (product data) and the distortion parameters can be reflected in the form of a table.

[0070] After the camera is calibrated, power can be supplied to the camera. In this embodiment, the POC power supply method is adopted. After the vehicle-side multimedia host powers on and starts the four-way surround view camera, the communication between the vehicle-side multimedia host and the camera is established through the i2c signal. The bottom-level driver identifies the sensor model through the sensor ID in a polling manner, and loads the corresponding firmware to light up the camera (power on).

[0071] After the camera is powered on, it starts to collect image information. The video transmission chip transmits the image to the image processor, which obtains the initial images collected by multiple cameras. After the image processor obtains the initial images collected by each camera, it performs coordinate conversion processing on the initial images to obtain the target images collected by multiple cameras, and then performs distortion correction and fusion processing on the target images.

[0072] As an example, since the initial image output by each camera is output in its own camera coordinate system, the coordinate systems of the initial images output by different cameras are different, and it is impossible to perform distortion correction and stitching fusion on multiple images. Therefore, it is necessary to perform coordinate transformation processing on the initial image captured by the camera to obtain the target image in the same coordinate system.

[0073] Alternatively, if Figure 3 As shown, taking a camera as an example, the initial image captured by the camera is processed to obtain the target image in the following specific ways:

[0074] S301, querying product data of a camera from a preset storage unit, and determining a transformation matrix for transforming an original image captured by the camera from a camera coordinate system to a world coordinate system according to the product data of the camera.

[0075] For example, based on the camera's intrinsic parameters, extrinsic parameters and other information, determine the corresponding transformation matrix for converting the original image captured by the camera from the camera coordinates to the world coordinates. Different cameras have different intrinsic and extrinsic parameters and different transformation matrices, so it is necessary to retrieve the corresponding intrinsic and extrinsic parameters according to the camera to calculate the transformation matrix.

[0076] As an example, Figure 5 As shown in the figure, the product data corresponding to the cameras of multiple suppliers are written in advance in FLASH. In order to improve the compatibility of the application layer software, the product data corresponding to the cameras of each supplier are written in the form of base address + offset address. That is, the required information is fixed in a certain address segment and the order remains consistent. The base address is provided by the underlying driver or bound to the sensor model, so that the same place specified by the offset address of different schemes can be accessed, and the information obtained is consistent.

[0077] Therefore, the electronic device in the embodiment of the present application can obtain the product data of the camera in the following manner:

[0078] According to the mapping relationship between the image sensor of the camera and the storage unit, access the corresponding storage unit;

[0079] Accessing the camera information base address in the storage unit, and obtaining basic information, lens information, camera internal parameters and camera external parameters of the corresponding camera based on the base address;

[0080] Access the lens information offset address in the storage unit, and obtain the image sensor model and firmware version of the corresponding camera, the camera's vendor information, software and hardware numbers and version numbers, and configurable parameters based on the offset address, wherein the configurable parameters include video output pixels, video frame rate, and data format.

[0081] When you need to obtain the camera's intrinsic and extrinsic parameters, you can retrieve the corresponding stored camera's intrinsic and extrinsic parameters through the base address of the camera information. Then, you can calculate the transformation matrix corresponding to the camera's transformation to the world coordinates through the camera's intrinsic and extrinsic parameters.

[0082] When you need to obtain the basic information of the camera, you can obtain the corresponding basic information through the camera information offset address, including the image sensor model and firmware version, the camera's vendor information, software and hardware numbers and version numbers, and configurable parameters.

[0083] In this embodiment, configurable parameters refer to parameters with multiple groups of selectable configurations, such as multiple groups of video output pixels, multiple groups of video frame rates, and multiple data formats. In this embodiment, when obtaining configurable parameters, configurable parameters are selected based on preset specifications, and all cameras obtain configurable parameters based on preset specifications. In this way, the configurable parameters of all cameras can be consistent, and the display effect can be guaranteed in applications where cameras are mixed.

[0084] In this embodiment, the product data of each camera corresponds to a storage unit, and the image sensor model of the camera is associated with the storage unit. The uniqueness of the image sensor model enables the electronic device to find accurate product data. In addition, in this embodiment, for the same type of parameters, the same offset address is used, and the product data corresponding to each camera can be accurately found through the offset address.

[0085] Through the above settings, when the cameras provided by multiple suppliers are mixed to form a 360-degree surround view module, the product data corresponding to the cameras can be retrieved, and then the initial images collected by different cameras can be processed to obtain the target image that meets the stitching requirements.

[0086] S302: In the world coordinate system, a bowl-shaped model is established with the target point as the origin.

[0087] In the world coordinate system, the initial image captured by each camera can be converted to the world coordinate system through its corresponding transformation matrix. In combination with the bowl-shaped model, the initial image of each camera can be converted to the bowl-shaped model to present the target image corresponding to each camera in the bowl-shaped model.

[0088] It should be noted that, taking vehicle application as an example, the world coordinates refer to the world coordinates of the vehicle (body coordinate system).

[0089] The bowl-shaped model simulates a three-dimensional surface in the real world, which is a 3D bowl-shaped surface. The image captured by the camera (especially the fisheye camera) is projected onto the 3D bowl-shaped surface. The 3D bowl-shaped surface is composed of multiple small patches. By calculating the correspondence between the small patches and the actual image, the pixels on the image are mapped to the 3D bowl-shaped surface.

[0090] The creation of the bowl-shaped model is suitable for the generation of surround-view images of autonomous driving systems, especially in automatic parking assistance systems (AVM). By presenting the scene around the vehicle to the driver in three-dimensional form, the visual realism and three-dimensionality are enhanced.

[0091] S303: Based on the transformation matrix, map the original image to the bowl-shaped model in the world coordinates to obtain a target image.

[0092] It can be understood that each pixel coordinate of the original image is mapped to the bowl-shaped model based on the coordinate transformation matrix, thereby realizing the transformation of the initial image into the bowl-shaped model.

[0093] As an example, the target point can be the center of the vehicle body, and four cameras are arranged along the four directions of the center of the vehicle body. Then, a bowl-shaped model is established with the center of the vehicle body as the origin, and the original image of the camera is mapped to the bowl-shaped model. The converted image can evenly present images in four directions, so that the image stitching effect is better.

[0094] Multiple cameras obtain corresponding transformation matrices respectively in the above manner, and then map the original images to the bowl-shaped model based on the corresponding transformation matrices. In this way, a bowl-shaped image composed of multiple target images is formed in the bowl-shaped model.

[0095] That is, in the above process, as long as different cameras call the correct lens distortion table and internal and external parameters, it can be guaranteed that the stitching is normal when projected to the world coordinates, and thus the stitching effect will not be affected.

[0096] In this embodiment, the original images are converted into the same bowl-shaped model. On the one hand, images from different suppliers can be converted into the same coordinate system. On the other hand, the target image is displayed in a bowl-shaped model, so that the target image is closer to the image captured by the surround-view camera (or fisheye camera), the image distortion correction is more accurate, and the display effect of the stitched image is better.

[0097] S202, determining the distortion parameter corresponding to each camera from the association relationship between the pre-calibrated cameras and the distortion parameters;

[0098] Since the distortion parameters corresponding to the multiple cameras are different, when performing distortion correction and fusion processing on the target images respectively acquired by the multiple cameras, it is necessary to call different distortion parameters respectively to perform distortion correction on the target images.

[0099] In this embodiment, when calibrating the camera, the camera is calibrated based on the product data of each camera to form a distortion table (the association between the camera and the distortion parameter) of each camera. When determining the camera on the vehicle, the distortion parameters corresponding to the camera can be retrieved from the association between the pre-calibrated camera and the distortion parameter based on the product data of the camera.

[0100] In this embodiment, step S201 and step S202 are not limited by the described action sequence, and step S201 and step S202 can be performed in other sequences or simultaneously.

[0101] S203, performing distortion correction on the target image acquired by the camera according to the distortion parameters corresponding to the camera, and fusing all the corrected images to obtain a panoramic image.

[0102] When the SDK algorithm package performs distortion correction on a target image of a camera, it retrieves the distortion parameters corresponding to the camera, and then uses the distortion parameters to perform distortion correction on the target image of the camera. In addition, after the distortion correction is performed on the target image, multiple target images are spliced ​​and fused to form a panoramic image.

[0103] Based on the above process, a specific example is used to illustrate:

[0104] Take the example of a vehicle with 4 cameras installed, all of which are mixed cameras from different suppliers, namely camera n 11 、Camera 12 、Camera 21 、Camera 22 Among them, camera n 11 and camera 12 A camera provided by a supplier, camera n 11 and camera 12 The product data is the same as that of camera n 21 and camera 22 Camera provided by another supplier, camera n 21 and camera 22 The product data is the same as that of the camera n 11 、Camera 12 With camera 21 、Camera 22 The product data is different. The vehicle memory has multiple storage units, each storage unit stores at least one set of product data of the supplier's camera, and the image sensor model in the product data is associated with the storage unit, and the image sensor model is used as a search index. Assume that there are 6 sets of product data, namely product data groups M1 to M6.

[0105] Calibrate the camera installed on the vehicle to 11 Calibration as an example: through camera n 11 The image sensor model is searched for the associated storage unit, specifically by comparing the image sensor model with the search index, and finding the camera n in the storage unit associated with the same image sensor model. 11 Then according to the preset calibration template and camera n 11 Product data M1, for camera n 11 Perform distortion calibration to obtain camera n 11 Distortion parameter X1, establish camera n 11-Correlation of distortion parameter X1. It is worth noting that the product data of the camera includes lens information, and the lens information includes the horizontal angle of the lens and the vertical angle of the lens. When the horizontal angle of the lens or the vertical angle of the lens changes, the distortion of the collected image is different. Based on this, it is necessary to calibrate the lens at different horizontal angles or vertical angles of the lens. Therefore, the correlation between the camera and the distortion parameters includes the correlation between the lens information and the distortion parameters. One camera can correspond to multiple sets of distortion parameters, and each set of distortion parameters corresponds to different lens information.

[0106] Through the above method, camera n is 12 、Camera 21 、Camera 22 Calibrate. Among them, camera n 11 and camera 12 The cameras are provided by the same supplier, so their corresponding product data are all M1. 11 and camera 12 The installation position is different, so the lens information is different, and the corresponding distortion parameters are different. For example, camera n 11 -Distortion parameter X1 establishes an association relationship, camera n 12 -Distortion parameter X1 establishes an association relationship. Camera n 21 and camera 22 Cameras from the same vendor, so camera n 21 and camera 22 The product data is the same, assuming it is M2. Correspondingly, if the installation position is different and the lens information is different, the distortion parameters obtained by calibration will also be different.

[0107] After the four cameras are powered on, they start collecting images. 11 The image processing process is explained by taking the collected image as an example: the processor obtains the camera n 11 The initial image is transmitted and transmitted through the camera n 11 Image sensor model, call camera n 11 The product data M1. Calculate the camera n through the camera internal parameters and camera external parameters in M1. 11 Convert the transformation matrix from camera coordinates to world coordinates and create a bowl-shaped model in world coordinates, then place the camera n 11 The transmitted initial image is mapped to the bowl-shaped model based on the transformation matrix to obtain target image 1. The other three cameras are similar, and the images collected by the cameras are mapped to the bowl-shaped model in turn to obtain target images 2, 3, and 4. Among them, since camera n 11 、Camera 12 With camera21 、Camera 22 The product data is different, so camera n 11 、Camera 12 With camera 21 、Camera 22 The corresponding transformation matrices are different, so when processing the initial image, it is necessary to call the corresponding product data to calculate the transformation matrix.

[0108] After generating four target images in the bowl-shaped model, the four target images are distorted and corrected respectively, and then the distorted target images are fused and spliced. 11 Take the acquired target image 1 for example for distortion correction: 11 Image sensor signal, find camera n 11 Corresponding to the distortion parameter X1, the SDK algorithm package uses the distortion parameter X1 to perform distortion correction processing on the target image 1. Similarly, the distortion correction processing of the target images acquired by the other three cameras is the same as the above method.

[0109] In this embodiment, the target image is spliced ​​in such a way that feature fusion is performed based on overlapping positions.

[0110] There may be cameras provided by multiple suppliers at the assembly site of the vehicle's 360-degree surround view module. In the related art, it is required that the four cameras of the 360-degree surround view module installed on the same vehicle are from the same supplier. In this case, the cameras need to be distinguished first, and then installed on the vehicle. In this scenario, it is also easy to install incorrectly, resulting in the 360-degree surround view module not being able to work properly after installation. In the embodiment of the present application, cameras provided by different suppliers can be installed on the same vehicle to form a 360-degree surround view module. In this scenario, the installer can use different cameras for assembly without distinguishing the cameras, thereby improving installation efficiency and avoiding the impact of mixed cameras on the normal operation of the 360-degree surround view module.

[0111] In this embodiment, the product data corresponding to the cameras provided by various suppliers are stored in the storage unit in advance. When the installed cameras are calibrated, the product data corresponding to different cameras can be retrieved to calibrate the cameras. After the cameras are calibrated, when the images collected by different cameras are processed during power-on use, the distortion parameters, camera internal parameters, camera external parameters and other information corresponding to each camera can be retrieved based on the pre-calibrated distortion parameters and pre-written product data, and the images collected by each camera can be converted, distorted and spliced ​​to obtain panoramic images. In this way, cameras provided by different suppliers can be mixed to form a 360-degree surround view module.

[0112] Figure 4 This is a flowchart of a panoramic image processing method provided by another embodiment of the present application. This embodiment is based on the above embodiment. Since this embodiment is applicable to mixed installation of cameras provided by different suppliers, the camera may be replaced when the camera fails or based on user needs. In order to avoid the inability to process the image captured by the new camera in the scenario where the camera is replaced due to a failure or a change in user needs, the embodiment of the present application solves this problem in the following manner, such as the method includes:

[0113] S401, when it is detected that the camera is powered on, querying a pre-calibrated camera, wherein the pre-calibrated camera is associated with a distortion parameter and stored in a storage unit;

[0114] It should be noted that when the vehicle is started, the camera is powered on, and at this time it is detected that the camera is powered on.

[0115] In one example, when a camera is powered on, it is queried whether the powered-on camera has been pre-calibrated. In this example, whether the powered-on camera needs to be calibrated is determined by determining whether the pre-calibrated camera in the system is the same as the powered-on camera.

[0116] The image sensor model of the pre-calibrated camera is associated with the distortion parameter. The image sensor model of each camera can be queried from the camera-distortion parameter table to obtain the pre-calibrated camera.

[0117] S402, determining whether the pre-calibrated cameras include a powered-on camera;

[0118] By comparing the image sensor signal of the powered-on camera with the image sensor models of the pre-calibrated cameras, it can be determined whether the pre-calibrated cameras include the powered-on camera.

[0119] If it is determined that the pre-calibrated camera includes the powered-on camera, that is, the powered-on camera is calibrated, then S403 is executed to obtain the target image captured by the powered-on camera, and multiple cameras are powered on, and corresponding target images captured by multiple cameras are obtained. Then, the following steps S407 and S408 are executed to process the target images captured by each powered-on camera.

[0120] That is, if the powered-on camera has been calibrated, the product parameters corresponding to the powered-on camera can be directly called to process the image captured by the powered-on camera.

[0121] If it is determined that the pre-calibrated cameras do not include the powered-on camera, that is, the powered-on camera has not been calibrated, execute S404 to query the product data corresponding to the powered-on camera from a preset storage unit, where the product data includes at least one of the basic information of the camera, lens information, camera internal parameters and camera external parameters.

[0122] If the powered-on camera is not one of the four cameras calibrated previously, the product data corresponding to the powered-on camera, such as distortion coefficient, internal parameters, external parameters and other information, is queried from a preset storage unit to calibrate the powered-on camera.

[0123] S405, performing distortion calibration according to a preset distortion correction algorithm and product data of the powered-on camera to obtain distortion parameters;

[0124] In this embodiment, one calibration method is used for all four cameras. When calibrating each camera, each camera is calibrated using its own distortion coefficient to obtain corresponding distortion parameters.

[0125] This embodiment takes the calibration of a fisheye camera as an example: the fisheye camera has barrel distortion, so the fisheye model is selected as the imaging model, and the camera is calibrated using the chessboard calibration method. The specific method is as follows:

[0126] Make a black and white chessboard as a template for image information calibration; observe the calibration template at different angles to obtain the image information of the calibration template at different viewing angles; and extract the feature information in the black and white chessboard template. The camera and the black and white chessboard calibration plate can be moved at will during the process of obtaining the template image information. Finally, based on the fisheye model and the extracted feature point information, the distortion model parameters are calculated to achieve the distortion correction and calibration of the camera.

[0127] S406, based on the product data and distortion parameters of the powered-on camera, after updating the association relationship between the pre-calibrated camera and the distortion parameters, image information respectively captured by the multiple cameras is obtained.

[0128] After obtaining the distortion parameters of the powered-on camera, the camera and the distortion parameters are used to update the association relationship between the pre-calibrated camera and the distortion parameters. In this way, the distortion parameters corresponding to the camera are stored in the storage unit, which is convenient for calling the distortion parameters corresponding to the camera during the image processing process.

[0129] After the cameras are calibrated, the following steps S407 and S408 are executed to process the target images acquired by each powered-on camera:

[0130] S407, determining the distortion parameter corresponding to each camera from the association relationship between the pre-calibrated cameras and the distortion parameters;

[0131] S408, performing distortion correction on the target image acquired by the camera according to the distortion parameters corresponding to the camera, and fusing all the corrected images to obtain a panoramic image.

[0132] In this embodiment, steps S407 and S408 are the same as those described above. Figure 2 S202 and S203 in the illustrated embodiment are similar, and reference may be made to the above embodiment for details.

[0133] In this embodiment, through the above method, the camera can be replaced according to needs without affecting the normal use of the 360-degree surround view module, and the compatibility is strong.

[0134] Optionally, in some examples, if the camera that the user wants to replace does not belong to the pre-written camera corresponding to the supplier, the user can also initiate a replacement request through the after-sales service end. The after-sales service end receives the replacement request and determines the product data of the camera to be replaced based on the replacement request, and writes the product data of the camera to be replaced into the storage unit of the vehicle end. After the storage unit of the vehicle end is updated, the user can replace the camera, and the compatibility with the camera to be updated is achieved in the above manner. Therefore, in this example, the flexibility of the combination method of the 360-degree surround view module is improved.

[0135] The present application also provides a panoramic image processing device, the device comprising:

[0136] An acquisition module is used to acquire target images respectively acquired by multiple cameras, where the distortion parameters corresponding to the multiple cameras are different;

[0137] A determination module, used to determine the distortion parameters corresponding to each camera from the association relationship between the pre-calibrated cameras and the distortion parameters;

[0138] The distortion correction fusion module is used to perform distortion correction on the target image acquired by the camera according to the distortion parameters corresponding to the camera, and to fuse all the corrected images to obtain a panoramic image.

[0139] The panoramic image processing device provided in the embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and this embodiment will not be described in detail here.

[0140] The embodiment of the present application also provides an electronic device, the electronic device comprising: a memory, a processor;

[0141] The memory is used to store computer programs / instructions; the processor is used to execute the computer programs / instructions stored in the memory to implement the methods involved in the above embodiments.

[0142] The electronic device also includes a communication interface and a CAN bus, wherein the processor is used to provide computing power and control capabilities, and can be a GPU, CPU, NPU, MCU, FPGA, etc. The storage device includes an internal memory and a non-volatile memory. The non-volatile memory stores a computer program that implements the above method. The internal memory provides an environment for program startup and operation. The communication interface is used to communicate with an external terminal by wire or wireless.

[0143] In this embodiment, the electronic device may be an electronic component in a car, such as a car terminal, or may be a car.

[0144] Take a car as an example. It includes a multimedia host, a connection harness, and four surround-view cameras with various solutions. Among them, the surround-view camera is a module solution composed of different combinations of key components such as lenses and sensors.

[0145] The multimedia host and the camera are physically connected via LVDS, and GMSL is used for communication to transmit the camera video signal. At the same time, I2C communication is based on LVDS. It is also a processing module for SDK algorithm packages such as integrated panoramic image distortion correction and stitching.

[0146] Due to the large number of fisheye lens models and the increasing number of optional sensor solutions due to technological advancement, in order to realize the hardware platformization of the surround-view camera, the multimedia host is used to read the solution information of the connected camera and perform corresponding distortion correction and stitching processing, etc., and display the 360° panoramic image on the central control display screen, thereby improving the compatibility of the panoramic image algorithm, realizing the mixed installation of multiple solution modules in one vehicle, and reducing the risk of hardware supply.

[0147] The present invention also provides a computer-readable storage medium / computer program product, in which computer control instructions are stored / the computer program product includes computer control instructions, and when the computer control instructions are executed by a processor, they are used to implement the methods involved in the above-mentioned embodiments.

[0148] The above embodiments are only preferred embodiments for fully illustrating the present invention, and the protection scope of the present invention is not limited thereto. Any equivalent substitution or change made by a person skilled in the art based on the present invention is within the protection scope of the present invention.

Claims

1. A panoramic image processing method, characterized in that: The method comprises: Obtain target images captured by multiple cameras respectively, where the distortion parameters corresponding to the multiple cameras are different; Determine the distortion parameters corresponding to each camera from the association relationship between the pre-calibrated cameras and the distortion parameters; The target image acquired by the camera is subjected to distortion correction according to the distortion parameters corresponding to the camera, and all the corrected images are fused to obtain a panoramic image.

2. The method according to claim 1, characterized in that The step of acquiring target images captured by multiple cameras respectively includes: Query the product data of the camera from a preset storage unit, and determine the transformation matrix of the original image captured by the camera from the camera coordinate system to the world coordinate system according to the product data of the camera; In the world coordinate system, a bowl-shaped model is established with the target point as the origin; Based on the transformation matrix, the original image is mapped to the bowl-shaped model under the world coordinates to obtain a target image, and the original images of multiple cameras are mapped to the bowl-shaped model to obtain multiple target images.

3. The method according to claim 1, characterized in that The step of acquiring target images captured by multiple cameras respectively includes: When it is detected that the camera is powered on, a pre-calibrated camera is queried, wherein the pre-calibrated camera is correspondingly associated with a distortion parameter and is stored in a storage unit; When it is determined that the pre-calibrated cameras include a powered-on camera, a target image captured by the powered-on camera is obtained, and when multiple cameras are powered on, target images captured by the multiple cameras are correspondingly obtained.

4. The method according to claim 3, characterized in that The step of acquiring target images acquired by multiple cameras respectively further includes: When it is determined that the pre-calibrated cameras do not include the powered-on camera, querying product data corresponding to the powered-on camera from a preset storage unit, wherein the product data includes at least one of basic information of the camera, lens information, camera internal parameters, and camera external parameters; According to the preset distortion correction algorithm and the product data of the powered-on camera, distortion calibration is performed to obtain distortion parameters; Based on the powered-on camera and the distortion parameter, after updating the association relationship between the pre-calibrated camera and the distortion parameter, image information respectively captured by multiple cameras is obtained.

5. The method according to any one of claims 2 to 4, characterized in that: Query the camera's product data from the preset storage unit, including: According to the mapping relationship between the image sensor of the camera and the storage unit, access the corresponding storage unit; Accessing the camera information base address in the storage unit, and obtaining basic information, lens information, camera internal parameters and camera external parameters of the corresponding camera based on the base address; Access the lens information offset address in the storage unit, and obtain the image sensor model and firmware version of the corresponding camera, the camera's vendor information, software and hardware numbers and version numbers, and configurable parameters based on the offset address, wherein the configurable parameters include video output pixels, video frame rate, and data format.

6. The panoramic image processing method according to claim 5, characterized in that: A method for obtaining configurable parameters based on the offset address: The configurable parameters corresponding to the preset specifications are obtained based on the offset address, and the preset specifications corresponding to all cameras are the same.

7. A panoramic image processing device, characterized in that: The device comprises: An acquisition module is used to acquire target images respectively acquired by multiple cameras, where the distortion parameters corresponding to the multiple cameras are different; A determination module, used to determine the distortion parameters corresponding to each camera from the association relationship between the pre-calibrated cameras and the distortion parameters; The distortion correction fusion module is used to perform distortion correction on the target image acquired by the camera according to the distortion parameters corresponding to the camera, and to fuse all the corrected images to obtain a panoramic image.

8. An electronic device, characterized in that: include: Memory, processor; The memory is used to store computer programs / instructions; The processor is configured to implement the method according to any one of claims 1 to 6 according to the computer program / instructions stored in the memory.

9. The electronic device according to claim 8, characterized in that: The electronic device includes an automobile.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program / instruction, and the computer program / instruction is used to implement the method according to any one of claims 1 to 6 when executed by a processor.

11. A computer program product, characterized in that The computer program product comprises a computer program / instructions, which are used to implement the method according to any one of claims 1 to 6 when executed by a processor.

Images