Camera parameter calibration methods, devices, processors, and vehicles

Abstract

A camera parameter calibration method, apparatus, processor, and vehicle are disclosed, relating to the field of image processing. The method includes: the processor acquiring an image sequence of a first scene using a first camera; calibrating the principal point parameters of the initial intrinsic parameters of the first camera based on the center points of at least two frames with different exposure parameters in the image sequence; addressing the issue that low image brightness and interference from vehicle-mounted structural components on the camera images can lead to poor image edge extraction accuracy during optical center calibration, resulting in poor optical center calibration accuracy and affecting the accuracy of principal point calibration; calibrating the principal points based on multiple frames of images with different exposures reduces the impact of low image brightness and interference from vehicle-mounted structural components on the principal point calibration, improving the accuracy of principal point calibration; then, the processor obtains the calibrated extrinsic parameters based on the calibrated intrinsic parameters and calibration board images, eliminating the need for production line intrinsic parameter calibration procedures requiring specific calibration scenarios, improving the site adaptability of camera parameter calibration, and reducing the operational complexity of camera parameter calibration.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image processing, and in particular to a camera parameter calibration method and device, a processor and a vehicle. BACKGROUND

[0002] A vehicle-mounted panoramic surround view system is used to collect images around a vehicle, and a panoramic image around the vehicle is formed after image processing. The panoramic image is displayed on a vehicle display screen to intuitively present the position and surrounding situation of the vehicle to a user, thereby assisting the user in controlling the vehicle. Before collecting the image, camera parameters (such as intrinsic parameters and extrinsic parameters) of a vehicle-mounted camera are calibrated to improve the accuracy of the panoramic image. At present, the intrinsic parameters and the extrinsic parameters of the camera can be calibrated based on images of multiple calibration scenes, and the calibration operation steps are complex. Therefore, how to reduce the calibration operation complexity of the camera parameters and improve the calibration accuracy of the camera parameters has become a problem to be solved. SUMMARY

[0003] The present application provides a camera parameter calibration method, device, processor and vehicle, which solves the problem of how to reduce the calibration operation complexity of the camera parameters and improve the calibration accuracy of the camera parameters.

[0004] In a first aspect, a camera parameter calibration method is provided, which can be executed by a processor. Specifically, the method includes the following steps: the processor obtains an image sequence of a first scene photographed by a first camera, calibrates a principal point parameter included in initial intrinsic parameters of the first camera according to a center of a circle in at least two images with different exposure parameters in the image sequence, obtains calibrated intrinsic parameters including the calibrated principal point parameter, and then, the processor obtains calibrated extrinsic parameters according to the calibrated intrinsic parameters and a calibration board image, which is an image obtained by the first camera photographing the calibration board. The image sequence includes multiple images collected under different exposure parameters.

[0005] The camera intrinsic parameters are calibrated based on multiple images collected under different exposure parameters obtained by the camera, the influence of low image brightness on the principal point calibration is reduced, and the accuracy of the principal point calibration is improved, wherein the reduction of the image brightness is usually caused by lens flare or external obstruction. Moreover, the principal point calibration can be performed based on images photographed by the camera in any scene, and is not limited by specific calibration objects such as black and white grids and chessboard grids, and only needs to set an extrinsic parameter calibration scene and a calibration object to complete the intrinsic parameter calibration and the extrinsic parameter calibration of the camera, thereby simplifying the operation complexity of the camera parameter calibration.

[0006] In a possible implementation, the processor calibrates the principal point parameter included in the initial intrinsic parameters of the first camera according to the center of the circle in the at least two images with different exposure parameters in the image sequence, including: the processor calibrates the principal point parameter included in the initial intrinsic parameters of the first camera according to the center coordinates of the circle in the at least two images with different exposure parameters in the image sequence.

[0007] When the camera is blocked or there is a halo, the image captured by the camera can have a problem of low brightness, resulting in poor accuracy of image effective area recognition and errors in positioning the center of the effective area, thereby reducing the accuracy of the principal point parameter calibration. Since the image sequence includes multiple images captured under different exposure parameters, the principal point calibration of the camera is performed by integrating the center coordinates in multiple images with different exposure parameters, thereby reducing the influence of the low-brightness image with a halo or blockage on the positioning of the center of the effective area and improving the accuracy of the principal point parameter calibration.

[0008] In a possible implementation, before the processor calibrates the principal point parameter included in the initial intrinsic parameter of the first camera according to the center of the image in at least two images with different exposure parameters in the image sequence, the method further includes: determining, by the processor, the effective area of each image in the image sequence; and determining, by the processor, the center coordinates of each image according to the effective area of each image.

[0009] The center extraction needs to identify the edge of the image, and then calculate the center coordinates. Since the exposure parameters of each image are different, for example, the edge fitting error is large in the local area of the edge of the image with low exposure, and the effective area of each image is determined first, which can reduce the edge fitting error and further reduce the error of the center coordinate calculation, thereby improving the positioning accuracy of the center coordinates.

[0010] In a possible implementation, the processor calibrates the principal point parameter included in the initial intrinsic parameter of the first camera according to the center coordinates of each image in the image sequence, including: the processor calibrates the principal point parameter included in the initial intrinsic parameter of the first camera according to the centralized position of the center coordinates of at least two images with different exposure parameters in the image sequence.

[0011] The effective areas of images with different exposure parameters do not completely coincide, so the centers determined based on multiple images with different exposure parameters in the image sequence are not at the same position, and the more centralized the position of the center coordinates is, the less the influence of the halo or low brightness is. Therefore, the principal point parameter is calibrated according to the centralized position of the center coordinates of the images with different exposure parameters, thereby improving the accuracy of the principal point parameter calibration.

[0012] According to the different hardware configurations of the processor and the first camera, various possible ways can be used to obtain images with different exposure parameters, thereby improving the applicability of the camera calibration method.

[0013] For example, the multiple images captured under different exposure parameters are obtained by the first camera under different exposure time settings.

[0014] Under the condition that the processor can configure the first camera, the exposure of the first camera is adjusted to obtain images with different brightness, thereby improving the efficiency of obtaining the image sequence.

[0015] For example, multiple frames of images captured under different exposure parameters are obtained by the first camera under different light intensities.

[0016] Under conditions where the processor does not support camera configuration adjustment or the processor load is high, adjusting the external light intensity of the first camera to obtain images of different brightness reduces the hardware requirements for image sequence acquisition and improves applicability.

[0017] In one possible implementation, the processor obtains the extrinsic parameters after calibration based on the intrinsic parameters and the calibration board image, including: performing parameter fitting based on the intrinsic parameters, the corner coordinates of the calibration board in the calibration board image, and the world coordinates corresponding to the corner coordinates to obtain the extrinsic parameters after calibration.

[0018] In one possible implementation, after the processor obtains the calibrated extrinsic parameters, the process further includes: the processor acquiring a first image of the second scene through a first camera; the processor acquiring a second image of the second scene through a parameter-calibrated second camera; and the processor adjusting the first and second images according to the calibrated intrinsic and calibrated extrinsic parameters of the first and second cameras to obtain a third image of the second scene to be displayed.

[0019] Secondly, this application provides a camera parameter calibration apparatus, which includes various units for performing the camera parameter calibration method in the first aspect or any possible design of the first aspect, and may include: an image acquisition unit for acquiring an image sequence of a first scene, the image sequence including multiple frames of images acquired under different exposure parameters; a calibration unit for calibrating the principal point parameters included in the initial intrinsic parameters of the first camera based on the center of at least two frames of images with different exposure parameters in the image sequence, to obtain calibrated intrinsic parameters, the calibrated intrinsic parameters including calibrated principal point parameters; the calibration unit is further used to obtain calibrated extrinsic parameters based on the calibrated intrinsic parameters and a calibration board image, the calibration board image being an image acquired by the first camera from the calibration board.

[0020] In one possible implementation, the calibration unit is specifically used to: calibrate the principal point parameters contained in the initial intrinsic parameters of the first camera based on the center coordinates of at least two frames with different exposure parameters in the image sequence.

[0021] In one possible implementation, the calibration unit is specifically used to: calibrate the principal point parameters contained in the initial intrinsic parameters of the first camera based on the concentrated position of the center coordinates of at least two frames with different exposure parameters in the image sequence.

[0022] In one possible implementation, the calibration unit is specifically used to: determine the effective region of each frame in the image sequence; and determine the center coordinates of each frame based on the effective region of each frame.

[0023] In one possible implementation, depending on the different hardware configurations of the processor and the first camera, images with different exposure parameters can be acquired in various ways, thereby improving the applicability of the camera calibration method.

[0024] For example, multiple frames of images captured under different exposure parameters are obtained by the first camera under different exposure duration settings.

[0025] For example, multiple frames of images captured under different exposure parameters are obtained by the first camera under different light intensities.

[0026] In one possible implementation, the calibration unit is specifically used to: perform parameter fitting based on the calibrated intrinsic parameters, the corner coordinates of the calibration board in the calibration board image, and the world coordinates corresponding to the corner coordinates, to obtain the calibrated extrinsic parameters.

[0027] In one possible implementation, the image acquisition unit is further configured to: acquire a first image of the second scene using a first camera; and acquire a second image of the second scene using a parameter-calibrated second camera. The image processing unit is configured to: adjust the first image and the second image according to the calibrated intrinsic and extrinsic parameters of the first and second cameras to obtain a third image of the second scene to be displayed.

[0028] Thirdly, this application provides a processor for executing the operational steps of the camera parameter calibration method in the first aspect or any possible design of the first aspect.

[0029] Fourthly, this application provides a computing device including at least one processor and a memory, the memory being used to store a set of computer instructions; when the processor executes the set of computer instructions as an execution device in the first aspect or any possible implementation of the first aspect, it executes the operation steps of the camera parameter calibration method in the first aspect or any possible implementation of the first aspect.

[0030] Fifthly, this application provides a camera that includes the processor of the third aspect or the computing device of the fourth aspect, wherein when computer software instructions are executed in the computing device or processor, the computing device or processor performs the operation steps of the camera parameter calibration method as described in the first aspect or any possible implementation thereof.

[0031] Sixthly, this application provides a computer-readable storage medium, comprising: computer software instructions; when the computer software instructions are executed in a computing device, causing the computing device to perform the operation steps of the camera parameter calibration method as described in the first aspect or any possible implementation of the first aspect.

[0032] In a seventh aspect, this application provides a computer program product that, when run on a computer, causes a computing device to perform the operation steps of the camera parameter calibration method as described in the first aspect or any possible implementation of the first aspect.

[0033] Eighthly, this application provides a chip system including a processor for implementing the functions of the processor in the camera parameter calibration method of the first aspect described above. In one possible design, the chip system further includes a memory for storing program instructions and / or data. This chip system may be composed of chips or may include chips and other discrete devices.

[0034] Ninthly, this application provides a vehicle surround view system, including one or more cameras and the computing device described in the fourth aspect above, wherein a first camera among the one or more cameras is used to acquire an image sequence of a first scene, and the computing device is used to assist the one or more cameras in performing the camera parameter calibration method described in the first aspect above.

[0035] In a tenth aspect, this application provides a vehicle including the vehicle surround view system described in the ninth aspect above, wherein a first camera among the one or more cameras is used to acquire an image sequence of a first scene, and the computing device is used to assist the one or more cameras in performing the camera parameter calibration method described in the first aspect above.

[0036] Based on the implementation methods provided in the above aspects, this application can be further combined to provide more implementation methods. Attached Figure Description

[0037] Figure 1 A schematic diagram of a calibration scenario provided in an embodiment of this application;

[0038] Figure 2 This application provides a schematic diagram of the structure of an in-vehicle surround view system.

[0039] Figure 3 A schematic diagram of a cloud-based command vehicle provided in an embodiment of this application;

[0040] Figure 4 A method flow for camera parameter calibration provided in this application embodiment Figure 1 ;

[0041] Figure 5 A schematic diagram of an image sequence with images of varying exposure from low to high, provided as an embodiment of this application;

[0042] Figure 6 A method flow for calibrating machine parameters provided in this application embodiment Figure 2 ;

[0043] Figure 7 A flowchart illustrating a method for training an effective region extraction model, as provided in this application embodiment;

[0044] Figure 8 A flowchart illustrating a method for calibrating external parameters provided in this application embodiment;

[0045] Figure 9 This is a schematic diagram illustrating the operation of a camera parameter calibration display interface provided in an embodiment of this application.

[0046] Figure 10 This is a schematic diagram illustrating the composition of a camera parameter calibration device provided in an embodiment of this application;

[0047] Figure 11 This is a schematic diagram of the structure of a computing device provided in an embodiment of this application. Detailed Implementation

[0048] The terms "first," "second," and "third," etc., used in this application specification, claims, and the aforementioned drawings are used to distinguish different objects, not to limit a specific order.

[0049] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0050] The technical solutions of the embodiments of this application will now be described with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0051] To ensure clarity and conciseness in the description of the following embodiments, a brief introduction to the related technologies is given first:

[0052] A vehicle-mounted panoramic surround view system synthesizes a panoramic image of the vehicle's surroundings based on images captured by multiple cameras installed on the vehicle. During this synthesis process, the images need to be projected from the pixel coordinate system to the world coordinate system of the panoramic image based on the camera parameters of each camera. Therefore, to ensure the accuracy of the panoramic image, camera parameter calibration is required before the system is used for the first time, thereby improving the accuracy of processing multiple frames into a single panoramic image of the vehicle's perimeter.

[0053] Camera parameters include intrinsic and extrinsic parameters. Intrinsic parameters describe the camera's internal characteristics, including principal point parameters, focal length parameters, and distortion coefficients. The principal point parameter is the coordinates of the principal point's location (Cx, Cy), the focal length parameter is the focal length (fx, fy), and the distortion parameters (k1, k2, k3, and k4) include radial and / or tangential distortion coefficients. The principal point is the intersection of the camera lens's principal line of sight and the perspective plane, or the intersection of a pair of conjugate planes (i.e., principal planes) with a transverse magnification of 1 and the camera's optical axis. Distortion parameters indicate the degree of lens distortion. Distortion, also known as image distortion, is caused by the spherical aberration of the aperture. The height of the intersection point between the principal ray from different fields of view and the Gaussian image plane after passing through the optical system is not equal to the ideal image height; the difference between the two is the distortion. Spherical aberration refers to the imaging error of an optical system caused by a monochromatic conical beam of light emitted from a point on the principal axis. If, after refraction by the optical system, rays from different aperture angles of the original beam do not converge at the same position on the principal axis, resulting in a blurred spot (commonly known as a blur circle) at the ideal image plane on the principal axis, then spherical aberration is the error in the system's imaging. Extrinsic parameters are the parameters of the camera in the world coordinate system, determining the camera's position and orientation in a three-dimensional space. These include camera position (h, w, z), mounting angles (pitch, yaw, roll), etc. It should be understood that the coordinates (Cx, Cy), focal length (fx, fy), distortion coefficients k1, k2, k3, and k4, and camera position h, w, z are predetermined parameters of the computer vision (CV) model. Optionally, when the camera is a panoramic fisheye camera with a fisheye lens, the camera model is an isometric model, and the distortion parameters, distortion coefficients k1, k2, k3 and k4, are radial distortion coefficients and do not include tangential distortion coefficients.

[0054] Currently, in the camera parameter calibration process, intrinsic parameter calibration requires photographing specific calibration objects. These objects, such as black and white grids, need to be moved during intrinsic parameter calibration photography to obtain multiple images of the calibration object at different relative positions to the camera. Intrinsic parameter calibration is then performed based on these multiple images. Therefore, traditional intrinsic parameter calibration methods require photographing images for calibration in a specific location. Since the cameras in vehicle-mounted panoramic surround view systems are mounted on the vehicle body, they are easily obstructed by vehicle components or surrounding objects. Under obstructed conditions, the images acquired by the camera have lower brightness, and the camera itself may exhibit halos due to lens interference, which also affects image brightness. Reduced image brightness leads to poor accuracy in extracting the effective area of ​​the image during intrinsic parameter calibration. Furthermore, extrinsic parameter calibration needs to be completed in a calibration scene with a preset calibration board and its world coordinates. The calibration scene for extrinsic parameter calibration typically cannot meet the requirements of the aforementioned camera intrinsic parameter calibration scene. Therefore, the intrinsic and extrinsic parameter calibration of vehicle-mounted cameras in current surround view panoramic systems requires multiple calibration scenes and different calibration objects for auxiliary operation, resulting in high calibration complexity.

[0055] For example, see reference Figure 1 The external parameter calibration scenario shown has its center 102 as the origin of the world coordinate system. A positioning device is installed at center 102 according to the vehicle's dimensions. When the vehicle is restrained by the positioning device, its center coincides with center 102. Multiple calibration plates are placed at various locations on center 102; for example, calibration plate 101a is placed at the front left, calibration plate 101b at the front right, calibration plate 101c at the rear left, and calibration plate 101d at the rear right. The operator records the true coordinates of the corner points of each calibration plate in the world coordinate system and inputs these coordinates into the vehicle system for use during parameter calibration calculations. When calibrating camera parameters, the vehicle must travel to the calibration scenario and stop upon encountering the positioning device, ensuring its center coincides with center 102.

[0056] This application provides a camera parameter calibration method, particularly a method for calibrating the intrinsic and extrinsic parameters of a vehicle-mounted camera. The processor acquires an image sequence of a first scene using a first camera. Based on the center points of at least two frames with different exposure parameters in the image sequence, the processor calibrates the initial intrinsic parameters of the first camera, including principal point parameters. Exposure parameters include one or more of exposure duration, exposure gain, and aperture size. The exposure gain includes analog gain and / or digital gain. Furthermore, the exposure parameter can also be exposure intensity, which refers to the intensity and duration of light received by the camera, such as adjusting the exposure intensity by changing the exposure duration and / or ambient light intensity. Since the image sequence includes multiple frames acquired under different exposure parameters, the impact of low image brightness on principal point calibration is reduced, improving the accuracy of principal point calibration. Then, the processor obtains the calibrated extrinsic parameters based on the calibrated intrinsic parameters and the calibration board image. Since the success rate and accuracy of extrinsic parameter calibration are mainly related to the camera's principal point accuracy, this embodiment improves the accuracy of extrinsic parameter calibration by calibrating the principal point parameters using the image sequence. Since changing the camera's shooting configuration or the scene brightness can acquire multiple frames of images with different exposure parameters, there are no special requirements for the calibration object and calibration site for the camera's intrinsic parameter calibration. The extrinsic parameter calibration site can meet its image shooting requirements. Therefore, only the extrinsic parameter calibration scene is needed to complete the calibration of the camera's intrinsic and extrinsic parameters, reducing the operational complexity of camera parameter calibration.

[0057] It should be understood that, in addition to vehicles, the camera parameter calibration method provided in this application embodiment can be applied to devices with panoramic surround view systems, such as automobiles, airplanes, ships, drones, and mobile robots. This embodiment takes the in-vehicle surround view system applied to automobiles as an example to illustrate the camera parameter calibration method.

[0058] The embodiments of this application will now be described in conjunction with the accompanying drawings.

[0059] Figure 2 This is a functional block diagram of an in-vehicle surround view system 200 provided in an embodiment of this application. The in-vehicle surround view system 200 is installed in a vehicle and may include various subsystems, such as a sensing system 202, a computer system 204, and a display system 206. Optionally, the in-vehicle surround view system 200 may include more or fewer subsystems, and each subsystem may include multiple components. Furthermore, each subsystem and component of the in-vehicle surround view system 200 can be interconnected via wired or wireless means.

[0060] The sensing system 202 may include several sensors for sensing information about the environment surrounding the vehicle surround view system 200. For example, the sensing system 202 may include a positioning system 212 (which may be a Global Positioning System (GPS), a BeiDou system, or another positioning system) and a camera 214. The sensing system 202 may also include sensors monitoring the internal systems of the vehicle surround view system 200 (e.g., an in-vehicle air quality monitor, fuel gauge, oil temperature gauge, etc.). The vehicle surround view system 200 may include one or more cameras, such as... Figure 1 As shown, camera 214 includes camera 214a mounted at the front of the vehicle, camera 214b mounted at the rear of the vehicle, camera 214c mounted on the left side of the vehicle, and camera 214d mounted at the rear of the vehicle. The fields of view of any two adjacent cameras overlap, thus covering the entire vehicle's field of view of the vehicle surround view system 200. The cameras can employ lens types such as standard lenses, wide-angle lenses, or ultra-wide-angle lenses. For example, in this embodiment, camera 214 can be a fisheye camera using a fisheye lens within an ultra-wide-angle lens.

[0061] In this embodiment, the positioning system 212 is used to obtain the vehicle's positioning information, and the vehicle travels to a location based on the positioning information. Figure 1 In the calibration scenario shown, the vehicle stops at the center of the calibration field 102 upon encountering the positioning device, and the center of the vehicle coincides with the center of the calibration field 102. Camera 214 is used to capture images required for calibrating intrinsic and extrinsic parameters, such as an image of the calibration plate taken when the vehicle is at the center of the calibration field 102. At this time, each camera of the vehicle surround view system 200 can capture two calibration plates, and one of the calibration plates can also be seen in the adjacent view. For example, the image captured by camera 214a contains calibration plates 101a and 101b, and the image captured by camera 214c contains calibration plates 101a and 101c. The views of cameras 214a and 214c are adjacent views, and calibration plate 101a is present in both adjacent views.

[0062] Computer system 204 may include components that provide data storage, computation, and output control commands for the vehicle. Some or all functions of the vehicle surround view system 200 are controlled by computer system 204. Computer system 204 may include at least one processor 216 that executes instructions 217 stored in a non-transitory computer-readable medium such as data storage device 218. Computer system 204 may also be multiple computing devices that control individual components or subsystems of the vehicle in a distributed manner. Data storage device 218 may be a memory. Computer system 204 can process and analyze images captured by camera 214 to identify objects and / or features in the vehicle's surrounding environment. Computer vision system 204 may use object recognition algorithms and fitting algorithms, and may configure and adjust camera 214 through image signal processing (ISP). ISP is mainly used as a unit for processing output signals from front-end image sensors to match different image sensors.

[0063] Processor 216 can be any conventional processor, such as a commercially available central processing unit (CPU). Alternatively, processor 216 can be a dedicated device such as an application-specific integrated circuit (ASIC), a graphics processing unit (GPU), an embedded neural network processing unit, or other hardware-based processor. Although Figure 2 The processor, memory, and other components are functionally illustrated; however, those skilled in the art will understand that the processor, computer, or memory may actually include or may not be stored in the same physical housing. For example, memory may be a hard disk drive or other storage medium located in a housing different from that of computer system 204. Therefore, references to processors or computers will be understood to include a collection of processors or computers or memories that may or may not operate in parallel. Unlike using a single processor to perform the steps described herein, some components, such as the steering assembly and deceleration assembly, may each have their own processor that performs calculations only related to the component's specific function.

[0064] In all aspects described herein, the processor may be located remotely from the vehicle surround view system 200 and may communicate wirelessly with the vehicle surround view system 200. In other aspects, some of the processes described herein are executed on a processor located within the vehicle surround view system 200, while others are executed by a remote processor, including taking the necessary steps to perform a single operation.

[0065] In this embodiment, when calibrating the camera parameters of the cameras in the vehicle surround view system 200, the processor 216 can retrieve an image sequence from the camera. The image sequence consists of multiple frames of images taken by the camera's fisheye lens under different exposure parameters. The camera can be any one of the cameras 214 installed in the vehicle surround view system 200. The processor 216 can also calibrate the principal point parameters of the initial intrinsic parameters of the first camera based on the centers of at least two frames with different exposure parameters in the image sequence, obtaining calibrated intrinsic parameters, and obtain calibrated extrinsic parameters based on the calibrated intrinsic parameters and the calibration board image.

[0066] In some embodiments, the data storage device 218 may include instructions (e.g., program logic) that can be executed by the processor 216 to perform various functions of the vehicle surround view system 200, including those functions described above. The data storage device 218 may also include additional instructions, including instructions for sending data, receiving data from, interacting with, and / or controlling one or more of the sensing system 202, display system 206, etc.

[0067] In addition to instructions, the data storage device 218 can also store data such as road maps, route information, surrounding image information, vehicle position, direction, speed, and other vehicle data. For example, in this embodiment, the data storage device 218 may store positioning information acquired by the positioning system 212 and images acquired by the camera 214.

[0068] Display system 206 provides a means of interaction between the vehicle and the user. For example, display system 206 can receive user input through user operation of the touch screen and can provide visual information to the user, such as obtaining camera model information based on the operator's camera model setting on the touch screen, sending an optical center calibration command to processor 216 based on the operator's optical center calibration trigger operation on the touch screen, sending a calibration command to processor 216 based on the operator's start calibration operation on the touch screen, sending an end command to processor 216 based on the operator's end calibration operation on the touch screen, and displaying a panoramic view or a real-time image obtained by any camera on the touch screen.

[0069] In some embodiments, an Artificial Intelligence (AI) chip is installed within the computer system 204. The AI ​​chip periodically trains the received sample data to generate an effective region extraction model. The AI ​​chip can also be called a neural network processor (NPU), for example, it can be processor 216. The processor 216 stores the effective region extraction model; alternatively, these models can be stored in memory via a system bus or other transmission methods. The AI ​​chip can be a single chip or other physical component, such as a training chip for building a neural network model or an inference chip for using the neural network model for inference. The neural network model can also be called an artificial neural network, neural networks (NNs), or a connection model. An "artificial neural network" is an algorithmic mathematical model that uses a structure similar to the synaptic connections of the brain for distributed parallel information processing. This artificial neural network, depending on the complexity of the system, adjusts the relationships between a large number of interconnected nodes to achieve the purpose of processing information. By periodically training the sample data with the AI ​​chip to generate a target location detection model, it is possible to better adapt to changes in the target location detection model.

[0070] In some embodiments, computer system 204 may also receive information from or transfer information to other computer systems. Alternatively, sensor data collected from the sensing system 202 of the vehicle surround view system 200 may be transferred to another computer for processing. For example, data from computer system 212 may be transmitted via a network to a cloud service center for further processing. Figure 3 This is a schematic diagram illustrating the composition of an in-vehicle surround view system 200 and a cloud service center, as provided in an embodiment of this application. The in-vehicle surround view system 200 can interact with the cloud service center 320 via network 310.

[0071] Network 310 and intermediate nodes may include various configurations and protocols, including the Internet, World Wide Web, Intranet, Virtual Private Network, Wide Area Network, Local Area Network, Private Network using proprietary communication protocols of one or more companies, Ethernet, WiFi, and HTTP (Hypertext Transfer Protocol), as well as various combinations thereof. This communication can be conducted by any device capable of transmitting data to and from other computers, such as modems and wireless interfaces.

[0072] In one example, cloud service center 320 may include servers with multiple computers, such as a load balancing server cluster, which exchange information with different nodes of network 210 for the purpose of receiving, processing, and transmitting data from computer system 204. The cloud service center 320 may be configured similarly to computer system 212, having a processor 330, memory 340, instruction set 350, and data storage 360.

[0073] Based on the received data, the cloud service center 320 runs its stored programs related to controlling the vehicle's autonomous driving to control the in-vehicle surround view system 200. These programs, related to controlling the vehicle's autonomous driving and parameter calibration, may include programs that control the vehicle's route or speed when entering a calibration scene, and programs that control the vehicle to acquire images through cameras and perform image processing, etc.

[0074] In some instances, the cloud service center 320 sends the vehicle suggested solutions for possible scenarios on roads, in closed or open environments (e.g., informing it of a landmark ahead and how to reach the calibration location and how to capture image sequences for calibration). For example, the cloud service center 320 can assist the vehicle in determining how to proceed when facing a landmark in the environment. The cloud service center 320 sends a response to the vehicle surround view system 200 instructing the vehicle and the vehicle surround view system 200 on how they should proceed in a given scenario. For example, based on collected sensor data, the cloud service center 320 can confirm the presence of a landmark ahead on the road and that the lane or scene is marked with a "calibration location". Accordingly, the cloud service center 320 sends a suggested operating mode for the vehicle to reach the center of the calibration scene. As another example, based on collected sensor data, the cloud service center 320 can confirm the presence of a landmark ahead on the road where the vehicle is located. Accordingly, the cloud service center 220 sends a suggested operating mode for the vehicle surround view system 200 to calibrate camera parameters using the landmark. It should be understood that the above-mentioned program for controlling the autonomous driving of vehicles can be implemented based on any existing autonomous driving scheme, and this embodiment does not impose any limitations.

[0075] Next, the camera parameter calibration of the vehicle surround view system 200 will be explained in detail with reference to the attached diagram.

[0076] like Figure 4 The diagram shown is a flowchart of a camera parameter calibration method provided in an embodiment of this application. Here, we will use the example of the processor 216 in the vehicle surround view system 200 controlling the vehicle surround view system 200 to calibrate the camera's intrinsic and extrinsic parameters based on the image acquired by the sensor system 202 as an example for illustration.

[0077] S401, processor 216 acquires an image sequence of the first scene through the first camera.

[0078] It should be understood that the aforementioned first camera can be any one of the cameras 214 in the vehicle surround view system 200. In this embodiment, multiple frames of images acquired under different exposure parameters are used to calibrate the principal point parameters. Therefore, the first scene for intrinsic parameter calibration of the captured image sequence can be the calibration scene or any scene that enables the first camera to image normally and provides shooting conditions.

[0079] An image sequence consists of multiple frames with different exposure parameters. For example, the exposure parameter can be the exposure level, then multiple frames with different exposure parameters would be multiple frames with exposure levels ranging from low to high. Exposure level refers to the intensity of light perceived by the light sensor of an imaging device such as a camera, and the duration for which the light is perceived. For example, such as... Figure 5 As shown, Figure 5 The exposure of (a) in the image is lower than that of (b). The number of images in the sequence can be 3 to 10 frames, and the range of exposure variation can be 30% to 100% of the adaptive exposure set by the first camera based on the current brightness. For example, if the exposure of the first frame image is EV (Exposure Values) = 7, the brightness of the second frame image can be EV = 3, the exposure of the third frame image can be EV = 4, the exposure of the fourth frame image can be 5, the exposure of the fifth frame image can be 6, the exposure of the sixth frame image can be 8, the exposure of the seventh frame image can be 9, the exposure of the eighth frame image can be 10, and the exposure of the ninth frame image can be 11.

[0080] In this embodiment, the processor 216 can obtain the image sequence captured by the first camera from the memory of the first camera. If the first camera has an internal memory for storing images captured by the first camera itself, the processor 216 can obtain the image sequence from the internal memory of the first camera. If the first camera is communicatively connected to the data storage device 218, the first camera can transmit the captured image sequence to the data storage device 218 for storage. The processor 216 can also read the image sequence captured by the first camera and stored in the data storage device 218 from the data storage device 218.

[0081] In this embodiment, assuming the exposure parameter is exposure level, the processor 216 can control the ISP to adjust the exposure duration of the first camera to obtain images with different exposure levels. For example, in a first scene, the parameters and configurations of the first camera, except for the exposure duration, are fixed. The exposure duration of the first camera is sequentially set to 0.1 seconds, 0.2 seconds, 0.3 seconds, 0.4 seconds, and 0.5 seconds. The first camera takes pictures at each exposure duration setting to obtain five images with different exposure levels. For example, in the image sequence of the first camera, image 1 is an image with an exposure duration of 0.1 seconds, image 2 is an image with an exposure duration of 0.2 seconds, image 3 is an image with an exposure duration of 0.3 seconds, image 4 is an image with an exposure duration of 0.4 seconds, and image 5 is an image with an exposure duration of 0.5 seconds, with the exposure levels of images 1 to 5 increasing sequentially.

[0082] In other embodiments, images with different exposures can be obtained by adjusting the external light intensity of the first camera, such as by illuminating the first camera with a light source or controlling the number of lights in the area where the first camera is located. For example, in a first scene, all parameters and configurations of the first camera are fixed, and a lighting lamp with brightness level adjustment is used to illuminate the first camera. The first camera is then used to take pictures at the brightness levels of the lighting lamp, which are respectively the first, second, third, fourth, and fifth levels, to obtain five images with different exposures, wherein the brightness of the first, second, third, fourth, and fifth levels increases or decreases sequentially.

[0083] S402, processor 216 calibrates the principal point parameters contained in the initial intrinsic parameters of the first camera based on the center of at least two frames with different exposure parameters in the image sequence, and obtains the calibrated intrinsic parameters.

[0084] Processor 216 can obtain initial intrinsic parameters before calibrating them. For example, such as... Figure 6 As shown, prior to S401, S601 is executed, where processor 216 obtains initial intrinsic parameters based on the model of the first camera. The initial intrinsic parameters can be obtained from within the first camera or imported from the data storage device 218, corresponding to the pre-stored initial intrinsic parameters for the first camera's model. The camera model can be obtained from within the camera or from receiving configuration instructions from the operator. The communication method by which processor 216 obtains the initial intrinsic parameters or camera model from the first camera can employ the PTZ (Pan / Tilt / Zoom) control protocol.

[0085] In this embodiment, the center coordinates of the image captured by the first camera can be used as the principal point parameter of the first camera.

[0086] The processor 216 calibrates the principal point parameters included in the initial intrinsic parameters of the first camera based on the center coordinates of at least two frames with different exposure parameters in the image sequence.

[0087] In this embodiment, the processor 216 can extract the effective region of each frame of at least two frames with different exposure parameters, and use a circle center localization algorithm to locate the center of the effective region of each frame. The circle center localization algorithm can be any one of the following: centroid method, Hough transform method, Gaussian fitting method, circle fitting method, and ellipse fitting method.

[0088] Optionally, in this embodiment, the processor 216 may employ a semantic segmentation algorithm to extract the effective region of each frame of the image. The semantic segmentation algorithm involves taking some raw data (e.g., planar images) as input and converting them into masks with highlighted regions of interest, used for dense prediction of each pixel, and labeling each pixel with the corresponding object or region category. The method of extracting the effective region of the image based on the semantic segmentation algorithm in this embodiment is similar to the processing method of existing deep learning algorithms. For the specific training process of the effective region extraction model used in this embodiment for extracting effective regions from multiple frames of images with different exposures, please refer to [reference needed]. Figure 7 The details will be explained below, and will not be repeated here.

[0089] After determining the center coordinates of each frame in the image sequence, the processor 216 calibrates the principal point parameter of the first camera based on the convergence of the center coordinates. Optionally, the processor 216 can use the average of all center coordinates of at least two frames with different exposure parameters in the image sequence as the convergence point. The processor 216 can also project the center coordinates of at least two frames with different exposure parameters into the same image and select the point with the most overlapping center coordinates as the convergence point. The projected image can be any image in the image sequence or any image captured by the first camera other than those in the image sequence. In this embodiment, when the camera 214 uses a fisheye lens, the coordinates of the convergence point of all center coordinates in multiple frames of the image sequence are optical center coordinates. The optical center of the fisheye camera and the principal point can be considered to coincide; therefore, the convergence point of all center coordinates of at least two frames with different exposure parameters in the image sequence can be considered as the principal point parameter in this embodiment.

[0090] Calibration based on multiple frames of images with different exposure parameters in an image sequence can improve image quality and facilitate center positioning. On the other hand, it can improve imaging robustness and enhance adaptability to the environment.

[0091] After calibrating the principal point parameters of the first camera, the extrinsic parameters of the first camera can be calibrated based on the calibrated intrinsic parameters containing the calibrated principal point parameters, and step S403 is executed. The intrinsic parameters in the calibrated intrinsic parameters, excluding the principal point parameters, can be the initial intrinsic parameters obtained from camera 214.

[0092] S403 and processor 216 obtain the calibrated extrinsic parameters based on the calibrated intrinsic parameters and the calibration board image.

[0093] In this embodiment, the processor 216 performs parameter fitting based on the calibrated intrinsic parameters, the corner coordinates of the calibration board in the calibration board image, and the world coordinates corresponding to the corner coordinates, to obtain the calibrated extrinsic parameters. The parameter fitting method can be to substitute the calibrated intrinsic parameters, the corner coordinates of the calibration board in the calibration board image, and the world coordinates corresponding to the corner coordinates into the homography matrix for parameter fitting, thereby solving for the extrinsic parameters in the homography matrix to obtain the calibrated extrinsic parameters. Homography describes the positional mapping relationship between the object's position in the world coordinate system and its position in the pixel coordinate system; the corresponding transformation matrix is ​​called the homography matrix. After calibrating the extrinsic parameters of the first camera, the processor 216 can also optimize the intrinsic and extrinsic parameters to improve the calibration accuracy of the camera parameters. The optimization steps for the camera parameters are described below. Figure 8 The example shown.

[0094] For example, the formula for the homography matrix can be expressed as:

[0095]

[0096] Where H represents the homography matrix, s represents the scale factor, (f x f y (u0, v0) represents the camera's focal length parameter, (u0, v0) represents the camera's principal point parameter, γ represents the distortion coefficient, M represents the camera's intrinsic parameter matrix, and [r1 r2 t] represents the camera's extrinsic parameter matrix. The extrinsic parameter matrix includes the camera's extrinsic parameters, such as the camera's position (e.g., h, w, z) and mounting angles (e.g., pitch, yaw, roll).

[0097] By substituting the calibration intrinsic parameters of the first camera, the corner coordinates of the calibration board in the calibration board image, and the corresponding world coordinates of the corner coordinates into the formula for the homography matrix, the computational load is relatively small, enabling a simple and fast calculation of the extrinsic parameters of the first camera. Simultaneously obtaining the calibration extrinsic parameters of the first camera, the complete homography matrix of the first camera can be obtained, and this complete homography matrix can be directly used when converting the image of the first camera into a panoramic surround view image.

[0098] The training process of the effective region extraction model is explained in detail below:

[0099] S701, processor 216 acquires multiple frames of images acquired by the first camera or a camera of the same model as the first camera under different exposure parameters as training set and test set.

[0100] S702 and processor 216 perform data annotation on the effective regions of the training set according to the annotation instructions of the annotators or the automatic annotation instructions.

[0101] The effective regions of each frame in the training set can be labeled by the annotators through manual identification.

[0102] The S703 and processor 216 use the training set to train the selected network for feature extraction.

[0103] The S704 and processor 216 perform weight learning on the selected network.

[0104] S705, processor 216 determines whether the selected network after weight learning has converged. If so, an effective region extraction model is obtained; otherwise, it jumps back to S703.

[0105] The processor 216 can determine whether the selected network has converged based on the error between the effective region output by the selected network after inputting the test set and the true value of the effective region in the test set. After obtaining the network model through training, the processor 216 inputs the image sequence into the network model after acquiring the image sequence to obtain the effective regions of at least two frames with different exposure parameters in the image sequence. The selected network can be FCN (Fully Convolutional Network), UNET (Unity Networking), DEEPLAB, etc. It should be understood that after obtaining the network model through initial training, the trained network model can be directly used for subsequent extraction of effective regions without the need for retraining. The trained network model can be stored in the data storage device 218.

[0106] In some instances, such as Figure 8 As shown, in step S403, this embodiment can also optimize the calibrated intrinsic and calibrated extrinsic parameters obtained from the camera parameter calibration. The steps for calculating extrinsic parameters and optimizing intrinsic and extrinsic parameters are described in detail below.

[0107] S801: Processor 216 acquires calibrated intrinsic parameters.

[0108] Optionally, in this embodiment, the calibrated intrinsic parameters obtained are the intrinsic parameters after calibrating the principal point parameters. The processor 216 can obtain the calibrated intrinsic parameters from the data storage device 218.

[0109] S802: Processor 216 extracts the corner coordinates from the calibration board image.

[0110] S803: The processor 216 obtains the calibrated extrinsic parameters based on the calibrated intrinsic parameters, corner coordinates, and the world coordinates corresponding to the corner coordinates.

[0111] Optionally, the method for calculating the extrinsic parameters after calibration can refer to step S303. As can be seen from the formula for the homography matrix above, the homography matrix of the first camera can be obtained after determining the intrinsic and extrinsic parameters after calibration.

[0112] S804: Processor 216 calculates the reprojection error based on the homography matrix, corner coordinates, and the world coordinates corresponding to the corner coordinates.

[0113] Processor 216 substitutes the pixel coordinates of the corner point into the homography matrix to obtain the pixel coordinates mapped to the world coordinate system. The aforementioned reprojection error is the difference between the true world coordinates of the corner point in the calibration plate image and the world coordinates mapped from the corner point pixel coordinates according to the homography matrix.

[0114] S805: If the reprojection error is greater than or equal to the preset threshold, the processor 216 performs iterative optimization on the calibrated intrinsic parameters and jumps to step S803.

[0115] Alternatively, iterative optimization can employ gradient descent, Newton's iteration method, the Levenberg-Marquardt (LM) algorithm, or a brute-force iterative algorithm.

[0116] S806: When the reprojection error is less than a preset threshold, the processor 216 determines the final calibrated intrinsic and calibrated extrinsic parameters.

[0117] The processor 216 uses the calibrated intrinsic and extrinsic parameters with reprojection errors less than a preset threshold as the final calibrated intrinsic and extrinsic parameters, thereby improving calibration accuracy.

[0118] Optionally, the difference in reprojection error can be 10 cm, 8 cm, or other values ​​that meet the accuracy requirements for optimizing intrinsic and extrinsic parameters.

[0119] In this embodiment, the first camera is any one of the cameras 214 in the vehicle surround view system 200. Therefore, while or after the camera parameters of the first camera are calibrated using the steps in the above-described camera parameter calibration method, all cameras in the panoramic surround view system need to be calibrated according to the camera parameter calibration steps of the first camera in order to output a complete panoramic view. The vehicle surround view system 200 displays a camera parameter calibration operation interface to the operator, allowing the operator to accurately control the camera parameter calibration process. After the camera parameter calibration is completed, a panoramic view is displayed to the user, presenting the operator with the vehicle's location and the surrounding environment in a comprehensive and detailed manner, enabling the operator to clearly understand the usage effect after camera parameter calibration.

[0120] For example, Figure 9This is a schematic diagram of an operation interface for camera parameter calibration provided in an embodiment of this application. After the operator enters the calibration interface through the touch screen of the display system 206, as... Figure 9 As shown, the calibration interface displays options such as "Set Camera Model," "Optical Center Calibration," "Fisheye Map," "Panoramic View," "Start Calibration," and "End Calibration." When the operator clicks the "Set Camera Model" option, system 206 responds to the user's click and sends a "Set Camera Model" command to processor 216. Processor 216 retrieves the camera model based on this command. When the operator clicks the "Optical Center Calibration" option, system 206 responds to the user's click and sends an "Optical Center Calibration" command to processor 216. Processor 216 executes steps S301-S302 to complete the camera's optical center calibration. The optical center and principal point of the cameras in the vehicle surround view system 200 are considered to coincide; optical center calibration is equivalent to calibrating the principal point parameters. When the operator clicks the "Start Calibration" option, system 206 responds to the user's click and sends a calibration command to processor 216, executing step S303 (or steps S303 and S801-S806) to calibrate the camera parameters of all cameras in the vehicle surround view system 200. When the operator clicks the "End Calibration" option, the system 206 responds to the user's click and sends an end command to the processor 216. The processor 216 then ends the camera parameter calibration process according to the end command.

[0121] When the process and results of camera parameter calibration are highly visible to the operator, the operator can more easily control the camera parameter calibration process to obtain better camera parameter calibration results. In this embodiment, the display system 206 can also display the images acquired by the camera during the camera parameter calibration process and the panoramic view output after the camera calibration is completed. When each camera in the vehicle surround view system 200 is calibrated, the display system 206 can display the images captured by the camera being calibrated through the display area of ​​the "fisheye" option. After one or more cameras in the vehicle surround view system 200 have completed camera parameter calibration, the display system 206 can display a panoramic view synthesized from the images of the one or more cameras through the display area of ​​the "panoramic view" option.

[0122] Optionally, the display area of ​​the "Fisheye View" option can display only the images taken by the camera currently being calibrated, or it can be divided into multiple sub-display areas, each displaying images taken by all cameras. The display area of ​​the "Panoramic View" option can display a complete panoramic view after all cameras in the vehicle surround view system 200 have completed camera parameter calibration. Images taken by cameras in the vehicle surround view system 200 that have completed camera parameter calibration are projected one by one into the panoramic view. The area marked "Panoramic View" can display a panoramic view with partially projected images, and the complete panoramic view is displayed after all cameras have completed camera parameter calibration. For example, when projecting partial images from cameras that have completed camera parameter calibration onto the panoramic view one by one, the processor 216 acquires the first image of the second scene through the first camera after camera parameter calibration, and the second image of the second scene through the second camera after camera parameter calibration. Then, the processor 216 synthesizes the first image and the second image in the panoramic view according to the calibration intrinsic and extrinsic parameters of the first and second cameras to obtain the third image of the second scene to be displayed. The processor 216 sends the third image to the display system 206, so that the display system 206 displays the third image in the display area corresponding to the "panoramic view" identifier on the touch screen. The second scene can be the calibration scene of the vehicle surround view system 200, and the first camera and the second camera can be any two of the cameras 214 of the vehicle surround view system 200.

[0123] It should be understood that the above-mentioned options such as "Set Camera Model", "Optical Center Calibration", "Fisheye Map", "Panoramic View", "Start Calibration" and "End Calibration" are not limitations on the identifier name or the corresponding display area. Any changes to the identifier name and the corresponding display area without affecting the relevant functions are within the scope of protection of this application.

[0124] It is understood that, in order to implement the functions in the above embodiments, the vehicle includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should readily recognize that, based on the units and method steps described in conjunction with the embodiments disclosed in this application, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application scenario and design constraints of the technical solution.

[0125] To complement the camera parameter calibration method provided in the above embodiments of this application, this application also provides a possible camera parameter calibration device. Figure 10This is a schematic diagram of a possible camera parameter calibration device. These camera parameter calibration devices can be used to implement the functions of the processor in the above method embodiments, and therefore can also achieve the beneficial effects of the above method embodiments. In the embodiments of this application, the camera parameter calibration device can be as follows: Figure 2 The processor 216 shown.

[0126] like Figure 10 As shown, the camera parameter calibration device 1000 includes an image acquisition unit 1010 and a calibration unit 1020. The camera parameter calibration device 1000 is used to implement the above-mentioned... Figure 4 , Figure 6 , Figure 7 or Figure 8 The processor functionality is shown in the method embodiment illustrated.

[0127] When the camera parameter calibration device 1000 is used to achieve Figure 4 In the method embodiment shown, the processor functions as follows: the image acquisition unit 1010 is used to execute S401, and the calibration unit 1020 is used to execute S402 to S403.

[0128] When the camera parameter calibration device 1000 is used to achieve Figure 6 In the method embodiment shown, the processor functions as follows: the image acquisition unit 1010 is used to acquire initial intrinsic parameters based on the camera model through the processor 216.

[0129] When the camera parameter calibration device 1000 is used to achieve Figure 7 In the method embodiment shown, the processor functions as follows: the calibration unit 1020 is used to execute S701 to S705.

[0130] When the camera parameter calibration device 1000 is used to achieve Figure 8 In the method embodiment shown, the processor functions as follows: the calibration unit 1020 is used to execute S801 to S806.

[0131] For a more detailed description of the image acquisition unit 1010 and calibration unit 1020 mentioned above, please refer to [link / reference]. Figure 4 , Figure 6 , Figure 7 or Figure 8 The relevant descriptions in the method embodiments shown are directly obtained and will not be repeated here.

[0132] To complement the camera parameter calibration method provided in the embodiments of this application, the embodiments of this application also provide a computing device 1100 that is compatible with the camera parameter calibration method. Figure 11This is a schematic diagram of the structure of a computing device 1100 provided in this embodiment. As shown in the figure, the computing device 1100 includes a processor 1110, a bus 1120, a memory 1130, a memory unit 1150 (also referred to as a main memory unit), and a communication interface 1140. The processor 1110, memory 1130, memory unit 1150, and communication interface 1140 are connected via the bus 1120.

[0133] It should be understood that in this embodiment, processor 1110 may be a CPU, and processor 1310 may also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0134] The processor 1110 may also be a GPU, NPU, microprocessor, ASIC, or one or more integrated circuits used to control the execution of the program of the present application.

[0135] The communication interface 1140 is used to enable communication between the computing device 1100 and external devices or components. In this embodiment, the communication interface 1140 is used to interact with the sensing system 202 and the display system 206.

[0136] Bus 1120 may include a pathway for transferring information between the aforementioned components (such as processor 1110, memory unit 1150, and memory 1130). In addition to a data bus, bus 1120 may also include a power bus, control bus, and status signal bus. However, for clarity, all buses are labeled as bus 1120 in the figure. Bus 1120 may be a Peripheral Component Interconnect Express (PCIe) bus, or an Extended Industry Standard Architecture (EISA) bus, a Unified Bus (Ubus or UB), a Compute Express link (CXL), a Cache Coherent Interconnect for Accelerators (CCIX), etc.

[0137] As an example, computing device 1100 may include multiple processors. A processor may be a multi-core (multi-CPU) processor. Here, "processor" can refer to one or more devices, circuits, and / or computing units used to process data (e.g., computer program instructions). Processor 1110 may call computer instructions stored in memory 1130; when processor 1110 executes computer instructions, it performs... Figure 4 , Figure 6 , Figure 7 or Figure 8 The operational steps for implementing camera parameter calibration methods are described in the document.

[0138] It is worth noting that, Figure 11 Taking computing device 1100 as an example, which includes one processor 1110 and one memory 1130, the processor 1110 and the memory 1130 are used to indicate a type of device or equipment. In specific embodiments, the number of each type of device or equipment can be determined according to business needs.

[0139] Memory unit 1150 can correspond to the storage medium used to store information such as image sequences in the above method embodiments. Memory unit 1150 can be volatile memory or non-volatile memory, or it can include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).

[0140] The memory 1130 is used to store data and can be a solid-state drive or a hard disk drive.

[0141] The aforementioned computing device 1100 can be a general-purpose device or a special-purpose device. For example, the computing device 1100 can be an in-vehicle computer or an edge device (e.g., a box carrying a chip with processing capabilities). Optionally, the computing device 1100 can also be a server or other device with computing capabilities.

[0142] It should be understood that the computing device 1100 according to this embodiment can correspond to the effective region extraction model in this embodiment, and can correspond to the execution of the model according to this embodiment. Figure 7 The corresponding main body in the model, and the above and other operations and / or functions of each module in the effective region extraction model are respectively implemented to achieve Figure 7 The corresponding processes are omitted here for the sake of brevity.

[0143] The method steps in this embodiment can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, portable hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and storage medium can reside in an ASIC. Alternatively, the ASIC can reside in a computing device. Of course, the processor and storage medium can also exist as discrete components in a network device or terminal device.

[0144] To complement the camera parameter calibration method provided in the embodiments of this application, the embodiments of this application also provide a processor, camera, and vehicle that are compatible with the camera parameter calibration method.

[0145] The processor in the embodiments of this application has computing power and is used to execute... Figure 4 , Figure 6 , Figure 7 or Figure 8 The operational steps for implementing camera parameter calibration methods are described in the document.

[0146] Cameras include processors or computing devices, which cause the processor to execute instructions when computer software instructions are run within the computing device or processor. Figure 4 , Figure 6 , Figure 7 or Figure 8 The operational steps for implementing camera parameter calibration methods are described in the document.

[0147] The vehicle may include an in-vehicle surround view system 200, which acquires image sequences through one or more cameras, and a computing device in the in-vehicle surround view system 200 is used to assist the one or more cameras in performing operations. Figure 4 , Figure 6 , Figure 7 or Figure 8 Figure 4 Figure 6 Figure 7 Figure 8 The operational steps for implementing camera parameter calibration methods are described in the document.

[0148] The method steps in the embodiments of this application can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, portable hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and storage medium can reside in an ASIC. Additionally, the ASIC can reside in a network device or a terminal device. Alternatively, the processor and storage medium can exist as discrete components in the network device or terminal device.

[0149] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instructions are loaded and executed on a computer, the processes or functions described in the embodiments of this application are performed entirely or partially. The computer can be a general-purpose computer, a special-purpose computer, a computer network, a network device, a user equipment, or other programmable device. The computer program or instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another. For example, the computer program or instructions can be transferred from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium, such as a floppy disk, hard disk, or magnetic tape; it can also be an optical medium, such as a digital video disc (DVD); or it can be a semiconductor medium, such as a solid-state drive (SSD).

[0150] In the various embodiments of this application, unless otherwise specified or in case of logical conflict, the terminology and / or descriptions of different embodiments are consistent and can be referenced by each other. The technical features of different embodiments can be combined to form new embodiments according to their inherent logical relationship.

[0151] In this application, "at least one" means one or more, and "more than one" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. In the textual description of this application, the character " / " generally indicates an "or" relationship between the preceding and following related objects; in the formulas of this application, the character " / " indicates a "division" relationship between the preceding and following related objects.

[0152] It is understood that the various numerical designations used in the embodiments of this application are merely for descriptive convenience and are not intended to limit the scope of the embodiments of this application. The order of the process numbers described above does not imply the order of execution; the execution order of each process should be determined by its function and internal logic.

Claims

1. A camera parameter calibration method, characterized in that, The method comprises: obtaining an image sequence of a first scene by a first camera, the image sequence comprising a plurality of images captured under different exposure parameters; calibrating a principal point parameter contained in an initial intrinsic parameter of the first camera according to a centralized position of a circle center coordinate of at least two images with different exposure parameters in the image sequence to obtain a calibrated intrinsic parameter, the calibrated intrinsic parameter containing a calibrated principal point parameter, the circle center coordinate being obtained based on a circle center positioning algorithm, and the centralized position being a mean value of all circle center coordinates of the at least two images with different exposure parameters in the image sequence; obtaining a calibrated extrinsic parameter according to the calibrated intrinsic parameter and a calibration board image, the calibration board image being an image obtained by the first camera capturing a calibration board.

2. The method of claim 1, wherein, The method of obtaining the image sequence of the first scene by the first camera comprises: controlling an exposure time of the first camera to obtain the plurality of images captured under the different exposure parameters.

3. The method of claim 2, wherein, The plurality of images captured under the different exposure parameters are obtained by the first camera under different light intensities.

4. The method according to any one of claims 1-3, characterized in that, The method of obtaining the calibrated extrinsic parameter according to the calibrated intrinsic parameter and the calibration board image comprises: performing parameter fitting according to the calibrated intrinsic parameter, corner point coordinates of the calibration board in the calibration board image, and world coordinates corresponding to the corner point coordinates to obtain the calibrated extrinsic parameter.

5. The method of claim 4, wherein, After obtaining the calibrated extrinsic parameter according to the calibrated intrinsic parameter and the calibration board image, the method further comprises: obtaining a first image of a second scene by the first camera; obtaining a second image of the second scene by a second camera after camera parameter calibration; synthesizing the first image and the second image in a panoramic surround view according to the calibrated intrinsic parameter and the calibrated extrinsic parameter of the first camera and the second camera to obtain a third image of the second scene to be displayed.

6. A camera parameter calibration apparatus, characterized by, The device comprises: an image acquisition unit configured to obtain an image sequence of a first scene by a first camera, the image sequence comprising a plurality of images captured under different exposure parameters; a calibration unit configured to calibrate a principal point parameter contained in an initial intrinsic parameter of the first camera according to a centralized position of a circle center coordinate of at least two images with different exposure parameters in the image sequence to obtain a calibrated intrinsic parameter, the calibrated intrinsic parameter containing a calibrated principal point parameter, the circle center coordinate being obtained based on a circle center positioning algorithm, and the centralized position being a mean value of all circle center coordinates of the at least two images with different exposure parameters in the image sequence; the calibration unit is further configured to obtain a calibrated extrinsic parameter according to the calibrated intrinsic parameter and a calibration board image, the calibration board image being an image obtained by the first camera capturing a calibration board.

7. The apparatus of claim 6, wherein, The device further comprises a control unit. The control unit is configured to control an exposure time of the first camera to obtain the plurality of images captured under the different exposure parameters.

8. The apparatus of claim 7, wherein, The plurality of images captured under the different exposure parameters are obtained by the first camera under different light intensities.

9. The apparatus of any one of claims 6-8, wherein, The calibration unit is specifically configured to: perform parameter fitting according to the calibrated intrinsic parameter, corner point coordinates of the calibration board in the calibration board image, and world coordinates corresponding to the corner point coordinates to obtain the calibrated extrinsic parameter.

10. The device of claim 9, wherein The image acquisition unit is further configured to acquire a first image of a second scene by the first camera, and acquire a second image of the second scene by the second camera after camera parameter calibration; The device further comprises an image processing unit; The image processing unit is configured to synthesize the first image and the second image in a panoramic surround view according to the calibrated intrinsic parameters and the calibrated extrinsic parameters of the first camera and the second camera, to obtain a third image of the second scene to be displayed.

11. A camera characterized by, The camera comprises a memory and a processor, the camera is configured to acquire an image sequence, the memory is configured to store computer instructions, and when the processor executes the computer instructions, operation steps of the method in any one of claims 1-5 are performed according to the image sequence.

12. A vehicle characterized by comprising: The vehicle comprises a processor, the camera in claim 11 and a display device, the camera is configured to acquire an image sequence, the processor is configured to perform operation steps of the method in any one of claims 1-5, and the display device is configured to display a surround view image synthesized according to the calibrated intrinsic parameters and the calibrated extrinsic parameters.

Images