Camera system calibration method and device, vehicle and storage medium

Abstract

The application discloses a camera system calibration method and device, a vehicle and a storage medium, and belongs to the technical field of vehicles. The method comprises the following steps: in response to a calibration instruction of a camera system of a target vehicle, the target vehicle is controlled to drive to a target position, the target position being a position of the target vehicle in a calibration site; during the driving of the target vehicle to the target position, environment information around the target vehicle is acquired; whether the current scene of the target vehicle is a target calibration scene is determined based on the environment information, the target calibration scene being a scene corresponding to the target position in the calibration site; and in the case that the current scene of the target vehicle is the target calibration scene, parameters of the camera system are calibrated. In this way, automatic identification of the calibration scene can be realized, which can reduce the dependence on manual work, reduce the labor cost, and thus improve the calibration efficiency of the camera system.

Description

Technical Field

[0001] This application relates to the field of vehicle technology, and in particular to a camera system calibration method, apparatus, vehicle, and storage medium. Background Technology

[0002] During vehicle manufacturing, various parameters need to be calibrated, including the camera system installed in the vehicle. Calibration of the camera system mainly involves adjusting and calibrating the camera parameters to ensure that the camera can accurately capture image information.

[0003] In related technologies, when calibrating a vehicle's camera system, workers first drive the vehicle to a designated calibration workshop equipped with pneumatic devices. The pneumatic devices are then used to calibrate the vehicle so that the wheels are positioned correctly. Afterward, workers connect communication equipment to the vehicle's onboard unit and click the calibration button to perform the subsequent calibration of the camera system.

[0004] However, in the above methods, the calibration of the vehicle's camera system requires continuous staff involvement, making the entire process overly reliant on manpower, which increases labor costs and reduces calibration efficiency. Summary of the Invention

[0005] This application provides a camera system calibration method, apparatus, vehicle, and storage medium. When calibrating the parameters of a vehicle's camera system, it can automatically identify whether the vehicle is in a specified calibration scenario, thereby reducing reliance on manual labor and improving the calibration efficiency of the camera system. The technical solution is as follows:

[0006] Firstly, a camera system calibration method is provided, the method comprising:

[0007] In response to a calibration command for the camera system of a target vehicle, the target vehicle is controlled to move toward a target location, wherein the target location is the position of the target vehicle when the camera system is calibrated in the calibration site;

[0008] During the process of the target vehicle traveling towards the target location, environmental information around the target vehicle is acquired;

[0009] Based on the environmental information, it is determined whether the scene where the target vehicle is currently located is the target calibration scene, wherein the target calibration scene is the scene corresponding to the target position in the calibration site;

[0010] When the target vehicle is currently in the same scene as the target calibration scene, the parameters of the camera system are calibrated.

[0011] In this application, when calibrating the camera system of the target vehicle, the target vehicle is first controlled to move towards the target location. That is, the target vehicle is controlled to move towards the designated position for camera system calibration. Subsequently, during the target vehicle's movement towards the target location, environmental information surrounding the target vehicle is acquired, and based on this information, it is determined whether the target vehicle's current scene is the target calibration scene, i.e., whether the target vehicle has reached the target location. Then, if the target vehicle's current scene is the target calibration scene, the parameters of the camera system are calibrated. Thus, by detecting whether the target vehicle is in the target calibration scene based on environmental information during its movement towards the target location, automatic identification of the calibration scene is achieved. This reduces reliance on manual labor, lowers labor costs, and improves the calibration efficiency of the camera system.

[0012] Optionally, determining whether the current scene of the target vehicle is a target calibration scene based on the environmental information includes:

[0013] The environmental information is input into the scene recognition model, and the environmental information is processed by the scene recognition model to obtain the probability that the scene in which the target vehicle is currently located is the target calibration scene;

[0014] If the probability is greater than or equal to a preset probability threshold, the current scene of the target vehicle is determined to be the target calibration scene;

[0015] If the probability is less than the preset probability threshold, it is determined that the current scene of the target vehicle is not the target calibration scene.

[0016] Optionally, the scene recognition model includes multiple convolutional blocks, convolutional layers, pooling layers, and fully connected layers. Each convolutional block consists of at least two convolutional layers. The step of inputting the environmental information into the scene recognition model and processing the environmental information through the scene recognition model to obtain the probability that the current scene of the target vehicle is the target calibration scene includes:

[0017] The environmental information is feature extracted through the convolutional layer to obtain the first feature;

[0018] The first feature is pooled using the pooling layer to obtain the second feature;

[0019] The first feature and the second feature are fused to obtain the third feature;

[0020] The third feature is extracted using the multiple convolutional blocks to obtain the fourth feature;

[0021] The fourth feature is pooled using the pooling layer to obtain the fifth feature;

[0022] The fifth feature is classified by the fully connected layer to obtain the probability that the current scene of the target vehicle is the target calibration scene.

[0023] Optionally, the environmental information includes images of the area surrounding the target vehicle, and the method further includes:

[0024] If the current scene of the target vehicle is not the target calibration scene, obtain the vehicle status of the target vehicle;

[0025] If the target vehicle is in a parked state, based on the image and the standard scene image, determine why the current scene of the target vehicle is not the target calibration scene, where the standard scene image is the image under the target calibration scene;

[0026] If the target vehicle is in a driving state, obtain the current position of the target vehicle; based on the current position, control the target vehicle to drive to the target position; if the target vehicle has driven to the target position, repeat the step of obtaining the environmental information around the target vehicle and the subsequent steps.

[0027] Optionally, when the target vehicle is in a parked state, determining the reason why the current scene of the target vehicle is not the target calibration scene based on the image and the standard scene image includes:

[0028] When the target vehicle is in a parked state, the pixel values ​​of each pixel in the image and the standard scene image are obtained respectively.

[0029] Based on the pixel values ​​of each pixel in the image and the standard scene image, determine the pixel similarity between the image and the standard scene image;

[0030] If the pixel similarity is less than a preset similarity threshold, the image quality of the image is determined to be substandard.

[0031] Optionally, when the target vehicle is in a parked state, determining the reason why the current scene of the target vehicle is not the target calibration scene based on the image and the standard scene image includes:

[0032] When the target vehicle is in a parked state, the RGB values ​​of each pixel in the image and the standard scene image are obtained respectively.

[0033] The RGB values ​​of each pixel in the image and the standard scene image are converted into HSV values ​​respectively;

[0034] Based on the HSV values ​​of each pixel in the image and the standard scene image, a first histogram and a second histogram are determined respectively. The first histogram is the histogram of the image in the HSV space, and the second histogram is the histogram of the standard scene image in the HSV space.

[0035] Determine the histogram similarity between the first and second histograms;

[0036] If the histogram similarity does not meet the preset conditions, the image color of the image is determined to be unqualified.

[0037] Optionally, the camera system includes multiple cameras, and before controlling the target vehicle to drive towards the target location, it further includes:

[0038] Images are captured using the multiple cameras;

[0039] Upon receiving images captured by the plurality of cameras, it is determined that the current state of the camera system meets the target calibration conditions, which are used to indicate whether the camera system is working properly;

[0040] Optionally, controlling the target vehicle to travel to the target location includes:

[0041] If the current state of the camera system meets the target calibration conditions, control the target vehicle to drive to the target location.

[0042] Optionally, the parameters of the camera system include the focal length, rotation direction, and translation position of the camera. When the scene in which the target vehicle is currently located is the target calibration scene, calibrating the parameters of the camera system includes:

[0043] When the target vehicle is currently in the target calibration scene, multiple target images are captured by the multiple cameras;

[0044] The multiple target images are fused to obtain a three-dimensional image;

[0045] Determine the pixel coordinates of the calibrated object from the three-dimensional image;

[0046] Determine the world coordinates of the calibrated object in the world coordinate system;

[0047] Based on the pixel coordinates and world coordinates of the calibrated object, the focal length, the rotation direction, and the translation position are determined.

[0048] Secondly, a camera system calibration device is provided, the device comprising:

[0049] The control module is used to control the target vehicle to drive towards the target position in response to the calibration command of the camera system of the target vehicle. The target position refers to the position of the target vehicle when the camera system is calibrated in the calibration site.

[0050] The first acquisition module is used to acquire environmental information around the target vehicle during the process of the target vehicle driving towards the target location;

[0051] The first determining module is used to determine, based on the environmental information, whether the scene where the target vehicle is currently located is a target calibration scene, wherein the target calibration scene is the scene corresponding to the target position in the calibration site;

[0052] The calibration module is used to calibrate the parameters of the camera system when the current scene of the target vehicle is the target calibration scene.

[0053] Optionally, the first determining module is used to:

[0054] The environmental information is input into the scene recognition model, and the environmental information is processed by the scene recognition model to obtain the probability that the scene in which the target vehicle is currently located is the target calibration scene;

[0055] If the probability is greater than or equal to a preset probability threshold, the current scene of the target vehicle is determined to be the target calibration scene;

[0056] If the probability is less than the preset probability threshold, it is determined that the current scene of the target vehicle is not the target calibration scene.

[0057] Optionally, the scene recognition model includes multiple convolutional blocks, convolutional layers, pooling layers, and fully connected layers, wherein each convolutional block consists of at least two convolutional layers, and the first determining module is used to:

[0058] The environmental information is feature extracted through the convolutional layer to obtain the first feature;

[0059] The first feature is pooled using the pooling layer to obtain the second feature;

[0060] The first feature and the second feature are fused to obtain the third feature;

[0061] The third feature is extracted using the multiple convolutional blocks to obtain the fourth feature;

[0062] The fourth feature is pooled using the pooling layer to obtain the fifth feature;

[0063] The fifth feature is classified by the fully connected layer to obtain the probability that the current scene of the target vehicle is the target calibration scene.

[0064] Optionally, the environmental information includes images of the area surrounding the target vehicle, and the device further includes:

[0065] The second acquisition module is used to acquire the vehicle status of the target vehicle when the current scene of the target vehicle is not the target calibration scene;

[0066] The second determining module is used to determine, based on the image and the standard scene image, why the current scene of the target vehicle is not the target calibration scene when the vehicle state of the target vehicle is a parked state; the standard scene image is an image under the target calibration scene.

[0067] The third acquisition module is used to acquire the current position of the target vehicle when the vehicle status of the target vehicle is in a driving state; control the target vehicle to drive to the target position based on the current position; and, when the target vehicle has driven to the target position, re-execute the step of acquiring the environmental information around the target vehicle and its subsequent steps.

[0068] Optionally, the second determining module is used to:

[0069] When the target vehicle is in a parked state, the pixel values ​​of each pixel in the image and the standard scene image are obtained respectively.

[0070] Based on the pixel values ​​of each pixel in the image and the standard scene image, determine the pixel similarity between the image and the standard scene image;

[0071] If the pixel similarity is less than a preset similarity threshold, the image quality of the image is determined to be substandard.

[0072] Optionally, the second determining module is used to:

[0073] When the target vehicle is in a parked state, the RGB values ​​of each pixel in the image and the standard scene image are obtained respectively.

[0074] The RGB values ​​of each pixel in the image and the standard scene image are converted into HSV values ​​respectively;

[0075] Based on the HSV values ​​of each pixel in the image and the standard scene image, a first histogram and a second histogram are determined respectively. The first histogram is the histogram of the image in the HSV space, and the second histogram is the histogram of the standard scene image in the HSV space.

[0076] Determine the histogram similarity between the first and second histograms;

[0077] If the histogram similarity does not meet the preset conditions, the image color of the image is determined to be unqualified.

[0078] Optionally, the camera system includes multiple cameras, and the device further includes:

[0079] The imaging module is used to capture images through the multiple cameras;

[0080] The third determining module is used to determine, upon receiving images captured by the plurality of cameras, whether the current state of the camera system meets the target calibration conditions, wherein the target calibration conditions are used to indicate whether the camera system is working normally;

[0081] Optionally, the control module is used for:

[0082] If the current state of the camera system meets the target calibration conditions, control the target vehicle to drive to the target location.

[0083] Optionally, the parameters of the camera system include the camera's focal length, rotation direction, and translation position, and the calibration module is used for:

[0084] When the target vehicle is currently in the target calibration scene, multiple target images are captured by the multiple cameras;

[0085] The multiple target images are fused to obtain a three-dimensional image;

[0086] Determine the pixel coordinates of the calibrated object from the three-dimensional image;

[0087] Determine the world coordinates of the calibrated object in the world coordinate system;

[0088] Based on the pixel coordinates and world coordinates of the calibrated object, the focal length, the rotation direction, and the translation position are determined.

[0089] Thirdly, a vehicle is provided, the vehicle comprising:

[0090] Memory, used to store executable program code;

[0091] A processor is configured to call and run the executable program code from the memory, causing the vehicle to perform the aforementioned camera system calibration method.

[0092] Fourthly, a computer-readable storage medium is provided, the computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-described camera system calibration method.

[0093] Fifthly, a computer program product containing instructions is provided, which, when run on a computer, causes the computer to perform the steps of the above-described camera system calibration method.

[0094] It is understood that the beneficial effects of the second, third, fourth, and fifth aspects mentioned above can be found in the relevant descriptions in the first aspect above, and will not be repeated here. Attached Figure Description

[0095] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0096] Figure 1 This is a schematic diagram of the implementation environment of a camera system calibration method provided in an embodiment of this application;

[0097] Figure 2 This is a schematic diagram of a scene for a camera system calibration method provided in an embodiment of this application;

[0098] Figure 3 This is a flowchart of a camera system calibration method provided in an embodiment of this application;

[0099] Figure 4 This is a schematic diagram of the structure of a scene recognition model provided in an embodiment of this application;

[0100] Figure 5 This is a flowchart illustrating the training process of a scene recognition model provided in an embodiment of this application;

[0101] Figure 6 This is a flowchart provided in an embodiment of the present application for determining whether an image quality is qualified;

[0102] Figure 7 This is a flowchart provided in an embodiment of the present application for determining whether an image color is qualified;

[0103] Figure 8 This is a flowchart of another camera system calibration method provided in the embodiments of this application;

[0104] Figure 9 This is a schematic diagram of the structure of a camera system calibration device provided in an embodiment of this application;

[0105] Figure 10 This is a schematic diagram of the structure of a vehicle provided in an embodiment of this application. Detailed Implementation

[0106] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0107] It should be understood that "multiple" as mentioned in this application refers to two or more. In the description of this application, unless otherwise stated, " / " indicates "or," for example, A / B can mean A or B; "and / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist, for example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Furthermore, to facilitate a clear description of the technical solutions of this application, the terms "first," "second," etc., are used to distinguish identical or similar items with essentially the same function and effect. Those skilled in the art will understand that the terms "first," "second," etc., do not limit the quantity or execution order, and that "first," "second," etc., do not necessarily imply differences.

[0108] Before describing the camera system calibration method provided in the embodiments of this application, the implementation environment of the camera system calibration method provided in the embodiments of this application will be described first.

[0109] For example, Figure 1 This is a schematic diagram illustrating the implementation environment of a camera system calibration method provided in this application embodiment. See also... Figure 1 , Figure 1 It includes a surround view camera 101, a cockpit controller 102, an APP (Application) 103, a smart driving controller 104, a chassis controller 105, and an automatic correction device 106.

[0110] The surround view camera 101 is installed on the vehicle and is used to collect environmental information around the vehicle.

[0111] The cockpit controller 102 is used to determine the scene in which the vehicle is located. Specifically, it can identify the current scene of the vehicle based on the environmental information collected by the surround-view camera 101 to determine whether the vehicle is in a specified calibration scene. The cockpit controller 102 can also execute automatic calibration logic for the parameters of the camera system.

[0112] APP103 is a mobile application that can communicate with the vehicle. APP103 is used to receive the judgment or calibration results from the cockpit controller 102. The received judgment or calibration results can then be displayed to the user, allowing the user to confirm the results.

[0113] The intelligent driving controller 104 is a controller in a vehicle used to realize intelligent driving. The intelligent driving controller 104 can be used to automatically control the vehicle, for example, in the embodiments of this application, it can automatically control the vehicle to drive towards a designated parking space.

[0114] The chassis controller 105 is used to adjust the wheel speed, steering wheel angle, etc. of the vehicle to control the vehicle to drive into the designated parking space.

[0115] The automatic calibration device 106 is used to automatically calibrate the vehicle. In this embodiment, after the vehicle enters the calibrated parking space, the automatic calibration device 106 can perform secondary calibration on the vehicle so that the vehicle's wheels, direction, etc., meet the state required for the camera system calibration.

[0116] In this embodiment, when calibrating the vehicle's camera system, the intelligent driving controller 104 and chassis controller 105 can control the vehicle to drive towards the calibration parking space. During the vehicle's movement towards the calibration parking space, the surround-view camera 101 collects environmental information about the vehicle's surroundings and sends this information to the cockpit controller 102. The cockpit controller 102 can identify the vehicle's current location and determine whether the vehicle is within the specified calibration scenario. The cockpit controller 102 can then output the determination result to the APP 103 for user viewing. When the cockpit controller 102 determines that the vehicle is within the specified calibration scenario, the automatic calibration device 106 can perform secondary calibration on the vehicle. Finally, the cockpit controller 102 executes the automatic calibration logic for the camera system parameters to achieve automatic calibration of the vehicle's camera system.

[0117] The application scenarios of the embodiments of this application are described below.

[0118] In the field of computer vision, a camera essentially maps three-dimensional objects in the real world onto a two-dimensional plane; that is, an image is essentially a mapping from three-dimensional spatial coordinates to two-dimensional planar coordinates. This necessitates determining the relationship between the three-dimensional geometric position of a point on the surface of a spatial object and its corresponding point in the two-dimensional image. Furthermore, camera imaging introduces distortion. Therefore, the purpose of calibrating a camera system is to determine this relationship, allowing the establishment of a geometric model of the camera's imaging. The parameters of this geometric model are the camera parameters (including the camera's intrinsic and extrinsic parameters).

[0119] Furthermore, in today's field of intelligent driving, image algorithms are typically used to extract information from images (such as object category and size) and incorporate this information into subsequent logical calculations. Without calibration, the obtained information will be inaccurate, making it impossible to accurately determine objects and distances. Therefore, vehicle camera systems need to be calibrated before leaving the factory to ensure accurate extraction of object information in three-dimensional space.

[0120] The camera system calibration method provided in this application embodiment can be applied to scenarios where the camera system of a vehicle is calibrated.

[0121] For example, Figure 2 A scene diagram illustrating a camera system calibration method provided in this application embodiment is shown below. Figure 2 , Figure 2 This includes the designated site 201, vehicles 202, and designated parking spaces 203.

[0122] The calibration site 201 is used to provide a location for calibrating the camera system of the vehicle 202. That is, the calibration of the vehicle 202 must be carried out in the calibration site 201.

[0123] The calibration parking space 203 refers to the designated location of vehicle 202 when calibrating its camera system. The calibration parking space 203 is located within the calibration area 201. It should be understood that the environment surrounding the calibration parking space 203 includes various elements necessary for calibrating the camera system of vehicle 202 (such as a checkerboard pattern). Therefore, the parameters of the camera system of vehicle 202 need to be calibrated at the calibration parking space 203.

[0124] Specifically, when calibrating the camera system of vehicle 202, vehicle 202 can first be controlled to move towards the designated parking space 203 in the calibration area 201. During the movement of vehicle 202 towards the designated parking space 203, environmental information surrounding vehicle 202 is acquired, and based on this information, it is determined whether the current scene of vehicle 202 corresponds to the scene of designated parking space 203. Then, if the current scene of vehicle 202 corresponds to the scene of designated parking space 203, the parameters of the camera system of vehicle 202 can be calibrated.

[0125] Thus, by detecting whether the vehicle 202 is in the scene corresponding to the calibrated parking space 203 based on environmental information during the process of the vehicle 202 driving towards the calibrated parking space 203, the automatic recognition of the calibrated scene can be achieved. This can reduce the dependence on manual labor, reduce labor costs, and thus improve the calibration efficiency of the camera system.

[0126] The camera system calibration method provided in the embodiments of this application will be explained in detail below.

[0127] Figure 3 This is a flowchart illustrating a camera system calibration method provided in an embodiment of this application. This method can be applied to a vehicle controller. See also... Figure 3 The method includes the following steps.

[0128] Step 301: In response to the calibration command for the camera system of the target vehicle, control the target vehicle to move towards the target position, which refers to the position of the target vehicle when the camera system is calibrated in the calibration site.

[0129] The target vehicle's camera system may include multiple cameras, and calibrating the target vehicle's camera system is equivalent to calibrating the parameters of the multiple cameras installed on the target vehicle. Optionally, if the target vehicle is an autonomous driving vehicle, the target vehicle's camera system may be a panoramic surround view system.

[0130] The calibration command for the target vehicle's camera system is used to instruct the execution of logic for automatically calibrating the target vehicle's camera system. Optionally, the calibration command for the target vehicle's camera system can be triggered by a calibration personnel. For example, the calibration personnel can trigger the calibration command for the target vehicle's camera system through click operations, voice operations, gesture operations, etc. As an example, after the calibration personnel drive the target vehicle into the calibration area, they can say a voice command such as "Start calibration" to trigger the calibration command.

[0131] A calibration site refers to the location where the camera system of a target vehicle is calibrated. The calibration site may include a scenario for calibrating the camera system's parameters. Optionally, the calibration site can be a calibration workshop or an open-air location, etc., and this application embodiment does not limit this. It should be understood that any location where a scenario for calibrating the camera system's parameters exists can be used as a calibration site.

[0132] When calibrating the parameters of the camera system of the target vehicle, the target vehicle needs to be located at the target position.

[0133] Furthermore, various calibration elements, such as a checkerboard and calibration reference objects, are deployed around the target location to calibrate the camera system of the target vehicle. Therefore, the target vehicle can capture images of these calibration elements at the target location, allowing the parameters of each camera in the camera system to be determined only when the target vehicle is in that location. In other words, the target vehicle cannot capture, or cannot fully capture, these calibration elements outside the target location, thus preventing the accurate determination of the parameters of each camera in the camera system from outside that location.

[0134] In this case, by receiving a calibration instruction for the camera system, the target vehicle is first controlled to move towards the target location, so that the subsequent target vehicle can accurately calibrate the camera system at the target location based on the surrounding scene.

[0135] Optionally, step 301 can be performed as follows: in response to the calibration command of the camera system of the target vehicle, determine the target path from the current position to the target position; control the target vehicle to travel along the target path to the target position.

[0136] In this scenario, it's equivalent to first planning a path from the current location to the target location, and then controlling the target vehicle to travel along that path. This makes it much easier to control the target vehicle to reach the target location.

[0137] Optionally, the target path can be determined through global path planning or local path planning, and this application embodiment does not limit this.

[0138] In this embodiment of the application, after receiving the calibration instruction for the camera system of the target vehicle, before controlling the target vehicle to drive to the target location, it can also be determined whether the camera system of the target vehicle is working properly.

[0139] Specifically, images are captured by multiple cameras in the target vehicle's camera system; upon receiving images from multiple cameras, the current state of the camera system is determined to meet the target calibration conditions.

[0140] Target calibration conditions are used to indicate whether the camera system of the target vehicle is working properly.

[0141] Because calibrating the parameters of a target vehicle's camera system requires images to be captured by multiple cameras on the vehicle, if the camera system malfunctions, the cameras will not be able to capture images, which will affect the subsequent calibration. Therefore, before calibrating the camera system, it is possible to determine whether the camera system is functioning properly by capturing images from the cameras.

[0142] In this scenario, receiving images from multiple cameras indicates that these cameras have captured images and sent them to the controller. This means that all cameras are functioning correctly and the communication links between them and the controller are normal. Therefore, it can be determined that the target vehicle's cameras are working properly, and consequently, the vehicle's camera system is functioning correctly. In other words, the current state of the target vehicle's camera system meets the target calibration conditions.

[0143] If images from multiple cameras are not received, it is determined that the current state of the target vehicle's camera system does not meet the target calibration conditions.

[0144] If no images are received from multiple cameras, it indicates that the images were not transmitted to the controller, or that the cameras did not capture any images. This means the cameras are not functioning properly, or there is a problem with the communication link between the cameras and the controller. Therefore, it can be determined that the target vehicle's camera system is not working correctly, meaning the current state of the target vehicle's camera system does not meet the target calibration conditions.

[0145] Furthermore, if the current state of the camera system meets the target calibration conditions, the target vehicle is controlled to drive to the target location.

[0146] If the current state of the camera system meets the target calibration conditions, it means that the multiple cameras on the target vehicle can capture images, and these images can be transmitted to the controller, allowing the controller to subsequently obtain the necessary information from the images. Therefore, in this case, the target vehicle can drive to the target location for camera system calibration.

[0147] Optionally, if the current state of the camera system does not meet the target calibration conditions, the reason why the current state of the camera system does not meet the target calibration conditions is sent to the target APP.

[0148] The target app is a mobile application that can communicate with the vehicle.

[0149] For example, if it is determined that the current state of the camera system does not meet the target calibration conditions, the target APP can be informed that multiple cameras cannot capture images or cannot receive images captured by multiple cameras.

[0150] In this situation, when calibration personnel see the problem feedback received on the target APP, they can investigate the problems of multiple cameras or the links between multiple cameras and the controller, thereby more quickly resolving the issue that the current state of the camera system does not meet the target calibration conditions.

[0151] Step 302: Acquire environmental information around the target vehicle while it is moving toward the target location.

[0152] The environmental information surrounding the target vehicle is used to indicate whether the target vehicle has driven to the target location.

[0153] Since the camera system's parameters can only be calibrated when the target vehicle is at the target location, environmental information around the target vehicle can be obtained while the target vehicle is traveling towards the target location to determine whether the target vehicle has reached the target location.

[0154] As one implementation method, multiple cameras installed on the target vehicle can be used to capture images of the surrounding environment of the target vehicle, thereby obtaining environmental information about the target vehicle's surroundings.

[0155] In this case, the environmental information around the target vehicle can be images taken by multiple cameras or video streams taken by multiple cameras.

[0156] As another implementation method, point cloud data of the surrounding environment of the target vehicle can be collected by a lidar installed on the target vehicle to obtain environmental information about the target vehicle.

[0157] In this case, the environmental information around the target vehicle is the point cloud of the target vehicle's surrounding environment.

[0158] Furthermore, after obtaining the environmental information around the target vehicle, it is possible to determine whether the target vehicle has driven to the target location based on the environmental information, that is, whether the target vehicle is currently in the scene where the target location is located, that is, to continue to execute the following step 303.

[0159] Step 303: Based on the environmental information, determine whether the current scene of the target vehicle is the target calibration scene. The target calibration scene is the scene corresponding to the target position in the calibration site.

[0160] Since the target location is the position of the target vehicle when calibrating the camera system in the calibration site, the scene corresponding to the target location is also the scene where the target vehicle is located when calibrating the camera system. In other words, the target calibration scene is the standard calibration scene when calibrating the camera system of the target vehicle. Thus, the parameter calibration of the camera system of the target vehicle can be achieved under the target calibration scene.

[0161] In this case, by using the environmental information, it can be determined whether the current scene of the target vehicle is the target calibration scene. This allows it to automatically determine whether the current scene of the target vehicle is the standard calibration scene for calibrating the target vehicle's camera system. In other words, it can determine whether the target vehicle has driven to the target location, and thus determine whether the camera system of the target vehicle can be calibrated in the current scene.

[0162] This enables automatic identification of calibration scenes, thereby reducing reliance on manual labor in the calibration process of the camera system, lowering labor costs, and ultimately improving the calibration efficiency of the camera system.

[0163] Optionally, step 303 can be performed as follows: inputting the environmental information into the scene recognition model, processing the environmental information through the scene recognition model to obtain the probability that the scene in which the target vehicle is currently located is the target calibration scene; if the probability is greater than or equal to a preset probability threshold, determining that the scene in which the target vehicle is currently located is the target calibration scene; if the probability is less than the preset probability threshold, determining that the scene in which the target vehicle is currently located is not the target calibration scene.

[0164] In this case, it is equivalent to using a neural network model to predict the probability that the current scene of the target vehicle is the target calibration scene based on the environment around the target vehicle, and then determining whether the current scene of the target vehicle is the target calibration scene.

[0165] This scene recognition model is used to predict the probability that the scene in which the target vehicle is currently located is the target calibration scene, in order to determine whether the scene in which the target vehicle is currently located is the target calibration scene. In the embodiments of this application, the scene recognition model can be a deep convolutional neural network based on deep learning, such as a ResNet neural network.

[0166] The preset probability threshold can be set in advance, and the preset probability threshold can be set relatively high. For example, the preset probability threshold can be set to 0.9.

[0167] If the probability is greater than or equal to the preset probability threshold, it means that the scene recognition model predicts that the scene in which the target vehicle is currently located is the target calibration scene with a high probability. In other words, the scene recognition model recognizes more features of the target calibration scene in the environmental information. In other words, the scene recognition model believes that the features of the scene in which the target vehicle is currently located are similar to the features of the target calibration scene, and thus it can determine that the scene in which the target vehicle is currently located is the target calibration scene.

[0168] If the probability is less than the preset probability threshold, it means that the scene recognition model predicts that the scene in which the target vehicle is currently located is the target calibration scene with a low probability. In other words, the scene recognition model recognizes fewer features of the target calibration scene in the environmental information. This means that the scene recognition model believes that the features of the scene in which the target vehicle is currently located are not similar to the features of the target calibration scene, and thus it can be determined that the scene in which the target vehicle is currently located is not the target calibration scene.

[0169] In one implementation, the scene recognition model may include multiple convolutional blocks, convolutional layers, pooling layers, and fully connected layers. Each of the multiple convolutional blocks may consist of at least two convolutional layers.

[0170] For example, Figure 4 This is a schematic diagram of the structure of a scene recognition model. See also... Figure 4 , Figure 4 It includes a convolutional layer 401, multiple convolutional blocks 402, a pooling layer 403, and a fully connected layer 404.

[0171] In this case, the operation of inputting the environmental information into the scene recognition model and processing the environmental information through the scene recognition model to obtain the probability that the scene in which the target vehicle is currently located is the target calibration scene can be as follows: extracting features from the environmental information through the convolutional layer 401 to obtain the first feature; pooling the first feature through the pooling layer 403 to obtain the second feature; fusing the first feature and the second feature to obtain the third feature; extracting features from the third feature through multiple convolutional blocks 402 to obtain the fourth feature; pooling the fourth feature through the pooling layer 403 to obtain the fifth feature; and classifying the fifth feature through the fully connected layer 403 to obtain the probability that the scene in which the target vehicle is currently located is the target calibration scene.

[0172] This environmental information can be images of the area around the target vehicle captured by multiple cameras, or partial image frames from a video stream captured by multiple cameras.

[0173] Convolutional layer 401 and multiple convolutional blocks 402 are used for feature extraction to extract deep features from the environmental information, thereby improving the receptive field of the scene recognition model and enabling more accurate probability predictions to be made by utilizing the deep features in the environmental information.

[0174] Pooling layer 403 is used to perform pooling operations. Optionally, pooling layer 403 can be max pooling or average pooling; this embodiment of the application does not limit this. When pooling layer 403 is max pooling, texture information in the features can be preserved, and the feature dimension can be reduced, thereby reducing the computational cost of the model. When pooling layer 403 is average pooling, background information in the features can be preserved.

[0175] The fully connected layer 404 is used for classification operations.

[0176] In this embodiment of the application, fusion is also a merging operation. Optionally, the merging operation can be a connection or an accumulation. That is, the first feature is connected to the second feature, or the first feature and the second feature are accumulated to obtain the third feature.

[0177] It is worth noting that in this embodiment, the multiple convolutional blocks 402 are connected using residual blocks. Specifically, the features output by the first convolutional block 402 are first fused with the features output by the second convolutional block 402, and then the fused features are input into the third convolutional block 402 for processing to obtain the features output by the third convolutional block 402. Then, the features output by the second convolutional block 402 are fused with the features output by the third convolutional block 402, and then the fused features are input into the fourth convolutional block 402 for feature extraction. This process continues until all convolutional blocks 402 have undergone feature extraction.

[0178] In this case, when performing feature extraction, the features output from the first two layers can be combined in subsequent feature extractions. This is equivalent to combining features from different levels for feature extraction, which can ensure that a small number of features are lost, thereby enabling accurate feature extraction. In other words, features that can accurately represent environmental information can be obtained.

[0179] It is worth noting that a scene recognition model also needs to be trained.

[0180] First, a dataset can be obtained, which includes images from both standard and non-standard calibration scenarios. The dataset can then be divided into training, validation, and testing datasets according to a predefined ratio.

[0181] The preset ratio can be set in advance, for example, the preset ratio is 3:1:1.

[0182] The training dataset is used to train the scene recognition model. The validation dataset is used to validate the trained scene recognition model to verify its accuracy. The test dataset is used to test the trained and validated scene recognition model.

[0183] Specifically, the server can obtain multiple training samples from the training dataset, use these multiple training samples to train the neural network model, and obtain the scene recognition model.

[0184] A server is a device used to train the scene recognition model. Optionally, the server can be a single server or a server cluster consisting of multiple independent servers.

[0185] The multiple training samples can be pre-set. Each training sample includes a sample image and a sample label. The sample image is an image containing a standard calibration scene or a non-standard calibration scene, and the sample label is the probability that the sample image is a standard calibration scene.

[0186] This neural network model can include multiple network layers, including an input layer, multiple hidden layers, and an output layer. The input layer is responsible for receiving input data; the output layer is responsible for outputting the processed data; the multiple hidden layers are located between the input and output layers and are responsible for processing the data. These hidden layers are not visible to the outside world. For example, this neural network model can be a deep neural network, and it can be a convolutional neural network, etc., within deep neural networks.

[0187] When the server trains the neural network model using multiple training samples, for each training sample, the input data from that training sample is fed into the neural network model to obtain output data. A loss function is used to determine the loss value between the output data and the sample labels in that training sample. The parameters in the neural network model are then adjusted based on this loss value. After adjusting the parameters of the neural network model based on each of the multiple training samples, the neural network model with adjusted parameters becomes the scene recognition model.

[0188] The operation of adjusting the parameters in the neural network model based on the loss value by the server can be referred to in relevant technologies, and will not be described in detail in this embodiment.

[0189] For example, computer equipment can use formulas This allows for the adjustment of any parameter in the neural network model. These are the adjusted parameters. W is the parameter before adjustment. α is the learning rate, which can be preset, such as 0.001, 0.000001, etc., but this application does not limit this to a single value. dw is the derivative of the loss function with respect to W, which can be obtained from the loss value.

[0190] Additionally, the number of iterations can be set when training the scene recognition model. During training, it can also be determined whether the current iteration count has reached the set number of iterations.

[0191] Specifically, if the set number of iterations has been reached, training the scene recognition model can be terminated, and the trained scene recognition model can then be validated. If the set number of iterations has not been reached, a loss function is used to determine the loss value between the output data and the sample labels in this training sample. Then, an optimization function is used to optimize the parameters of the scene recognition model based on this loss value.

[0192] For example, Figure 5 This is a flowchart of the training process for a scene recognition model.

[0193] See Figure 5 When training the scene recognition model, the first step is to obtain the training dataset. Then, the training samples from the training dataset are input into the scene recognition model, which processes these samples to obtain the output. After the scene recognition model obtains the output, it is determined whether the current iteration count has reached the set number of training iterations. If the set number of iterations has been reached, training of the scene recognition model ends. If the set number of iterations has not been reached, the loss value between the output and the sample labels in the training samples is determined. The parameters of the scene recognition model are then adjusted using an optimization function based on this loss value. The adjusted scene recognition model is then used to process the training samples again to obtain the output. This process is repeated until the set number of training iterations is reached.

[0194] It is worth noting that step 303 above can automatically determine whether the current scene of the target vehicle is the target calibration scene. Subsequently, if the current scene of the target vehicle is the target calibration scene, the parameters of the target vehicle's camera system can be calibrated, that is, the following step 304 is executed.

[0195] Step 304: If the target vehicle is currently in the target calibration scene, calibrate the parameters of the camera system.

[0196] In this case, if the scene where the target vehicle is currently located is determined to be the target calibration scene, it means that the target vehicle is currently at the specified position (target position) in the calibration site. At this position, the parameters of the camera system can be accurately calibrated.

[0197] Optionally, the parameters of the camera system may include the camera's focal length, rotation direction, and translation position.

[0198] In this case, step 304 can be performed as follows: when the target vehicle is currently in the target calibration scene, multiple target images are captured by the multiple cameras; the multiple target images are fused to obtain a three-dimensional image; the pixel coordinates of the calibration object are determined from the three-dimensional image; the world coordinates of the calibration object in the world coordinate system are determined; and the focal length, rotation direction, and translation position are determined based on the pixel coordinates and world coordinates of the calibration object.

[0199] The target image is an image of the target calibration scene captured by the camera of the target vehicle. Optionally, the target vehicle may be equipped with at least four cameras.

[0200] Since the images captured by the camera are two-dimensional, parameter calibration for surround-view cameras needs to be performed in three-dimensional space. Therefore, in this embodiment, multiple target images captured by multiple cameras can be fused to obtain a three-dimensional image in three-dimensional space. The three-dimensional image is then processed to calibrate the camera parameters.

[0201] Additionally, in this embodiment, a checkerboard pattern can be arranged in the target calibration scene. The checkerboard pattern will then be imaged in the target image. In this case, the calibration object can be the checkerboard pattern.

[0202] Because the checkerboard pattern is composed of black and white squares, its corner points are clear and easily distinguishable. Determining the pixel coordinates of the calibration object from the 3D image can involve determining the pixel coordinates of each corner point in the checkerboard pattern from the 3D image, and determining the world coordinates of each corner point in the checkerboard pattern from the world coordinate system. In this way, more accurate world and pixel coordinates of the calibration object can be determined, leading to more accurate determination of the camera's focal length, rotation direction, and translation position.

[0203] One possible method for fusing multiple target images to obtain a 3D image is as follows: For any one of the multiple target images, map the pixel coordinates of each pixel in the target image to a 3D space to obtain the corresponding 3D pixel coordinates of the target image; then, stitch together the corresponding 3D pixel coordinates of the multiple target images to obtain the pixel coordinates of each pixel in the 3D image.

[0204] In this scenario, it's equivalent to mapping multiple target images separately into a three-dimensional space, and then stitching these multiple target images together to obtain a three-dimensional image. Since the calibration object captured by the camera is essentially in three-dimensional space, mapping each image separately allows for the determination of a more accurate three-dimensional image, which in turn enables the accurate calibration of the camera system's parameters.

[0205] Optionally, in this embodiment, the pixel coordinates and world coordinates of the calibrated object can be fitted using the least squares method to determine the focal length, rotation direction, and translation position.

[0206] Optionally, if the target vehicle is currently in the target calibration scenario, the target vehicle can be calibrated using a calibration device at the target location to make the parking direction of the target vehicle more consistent with the state of the vehicle during calibration. This will allow the orientation of the multiple cameras installed on the target vehicle to better meet the requirements of parameter calibration, thereby making the parameter calibration of the camera system more accurate.

[0207] Optionally, the environmental information can be images of the area surrounding the target vehicle captured by multiple cameras.

[0208] In this embodiment of the application, if the current scene of the target vehicle is not the target calibration scene, the following steps (1)-(3) may also be included.

[0209] (1) Obtain the vehicle status of the target vehicle when the current scene of the target vehicle is not the target calibration scene.

[0210] The vehicle status of the target vehicle indicates whether the target vehicle is in motion or parked. The vehicle status of the target vehicle includes both in motion and parked states.

[0211] Since the target vehicle's current location is not within the target's designated scene, it indicates that the images captured by multiple cameras suggest the target vehicle is not currently in the scene corresponding to the target position. In this case, the vehicle's status can be obtained, and further determination of whether the target vehicle has reached the target position can be made based on the vehicle's status.

[0212] (2) When the target vehicle is in a parked state, determine the reason why the current scene of the target vehicle is not the target calibration scene based on the image and the standard scene image.

[0213] The standard scene image is an image of the target calibration scene.

[0214] If the target vehicle is in a parked state, it means that the target vehicle has already moved to the target location. However, if the target vehicle is not detected to be in the target calibration scene, it is likely that there is a problem with the images captured by multiple cameras.

[0215] In this case, based on the image and the standard scene image, the reason why the target vehicle is currently in a scene that is not the target calibration scene can be determined. This allows subsequent calibration personnel to investigate the problem more quickly based on the reason, thereby resolving the issue of the target vehicle not being in the target calibration scene more rapidly, and further improving the calibration efficiency of the camera system.

[0216] As one implementation method, step (2) can be performed as follows: when the target vehicle is in a parked state, obtain the pixel values ​​of each pixel in the image and the standard scene image respectively; based on the pixel values ​​of each pixel in the image and the standard scene image, determine the pixel similarity between the image and the standard scene image; if the pixel similarity is less than a preset similarity threshold, determine that the image quality of the image is unqualified.

[0217] If images captured by multiple cameras have issues, it is highly likely that there is a problem with image quality. Generally, image quality is one of the factors affecting overall image quality. Therefore, in this embodiment, it can be determined whether the image quality is acceptable.

[0218] Since the image quality of the standard scene image is not problematic, determining the pixel similarity between this image and the standard scene image is equivalent to determining whether the image quality of this image is close to that of the standard scene image. Thus, using the image quality of the standard scene image as a reference, we can accurately determine whether the image quality of this image is acceptable.

[0219] The preset similarity threshold can be set in advance, and the preset similarity threshold can be set to a large value.

[0220] If the pixel similarity is less than the preset similarity threshold, it means that the pixel similarity between the image and the standard scene image is small. In other words, the image quality of the image is not close to that of the standard scene image, which means that the image quality is poor. Therefore, the image quality of the image is deemed unqualified.

[0221] In this embodiment of the application, if the preset similarity is greater than or equal to the preset similarity threshold, the image quality of the image can be determined to be qualified.

[0222] If the pixel similarity is greater than or equal to the preset similarity threshold, it means that the pixel similarity between the image and the standard scene image is relatively large. In other words, the image quality of the image is close to that of the standard scene image, which means that the image quality of the image is not a problem. Therefore, the image quality of the image can be determined to be qualified.

[0223] Optionally, the operation of determining the pixel similarity between the image and the standard scene image based on the pixel values ​​of each pixel in the image and the standard scene image can be: based on the pixel values ​​of each pixel in the image and the standard scene image, the pixel similarity between the image and the standard scene image can be determined by the following formula (1).

[0224]

[0225] Where S is the pixel similarity between the image and the standard scene image. M is the number of rows in the image, and N is the number of columns in the image. L(i,j) is the pixel value of the i-th row and j-th column in the standard scene image, and K(i,j) is the pixel value of the i-th row and j-th column in the image.

[0226] Optionally, multiple cameras can capture multiple images, resulting in multiple images of the area surrounding the target vehicle. In this case, determining the pixel similarity between the image and the standard scene image can be done as follows: for each of the multiple images, determine the pixel similarity between each image and the standard scene image, obtaining multiple pixel similarities; and determine the average of the multiple pixel similarities as the pixel similarity between the image and the standard scene image.

[0227] The following example illustrates the process of determining whether an image's quality is acceptable. For example, Figure 6 It is a flowchart for determining whether an image quality is up to standard.

[0228] See Figure 6 To determine if the image quality of an image is acceptable, the system acquires the pixel values ​​of each pixel in images of the area surrounding the target vehicle captured by multiple cameras, and then acquires the pixel values ​​of each pixel in a standard scene image. Next, it calculates the pixel similarity between the image and the standard scene image. Then, it determines whether the pixel similarity between the image and the standard scene image meets the standard, that is, whether the pixel similarity is greater than a preset similarity. If the pixel similarity is greater than or equal to the preset similarity, the image quality is considered acceptable. If the pixel similarity is less than the preset similarity, the image quality is considered unacceptable.

[0229] As another implementation, step (2) can be performed as follows: when the target vehicle is in a parked state, obtain the RGB values ​​of each pixel in the image and the standard scene image respectively; convert the RGB values ​​of each pixel in the image and the standard scene image into HSV values ​​respectively; determine the first histogram and the second histogram based on the HSV values ​​of each pixel in the image and the standard scene image respectively; determine the histogram similarity between the first histogram and the second histogram; if the histogram similarity does not meet the preset conditions, determine that the image color of the image is unqualified.

[0230] RGB values ​​refer to the color values ​​of each pixel in a color image in the red, green, and blue channels respectively. RGB values ​​include the R (Red) value of the red channel, the G (Green) value of the green channel, and the B (Blue) value of the blue channel.

[0231] HSV is a color space. It uses hue, saturation, and value to represent color images. Therefore, HSV values ​​can include H (Hue), S (Saturation), and V (Value).

[0232] Among them, RGB space and HSV space can be converted to each other. That is, a color image can be converted from RGB space to HSV space to obtain the representation of each pixel in HSV space, and a color image can be converted from HSV space to RGB space to obtain the representation of each pixel in RGB space.

[0233] The first histogram is the histogram of the image in HSV space, which represents the distribution of contrast, brightness, and intensity of the image. The second histogram is the histogram of a standard scene image in HSV space, which represents the distribution of contrast, brightness, and intensity of the standard scene image.

[0234] Image color can also be one of the factors affecting image quality. Therefore, in this embodiment, it can also be determined whether the image color is acceptable.

[0235] Since the pixel representation in HSV space is closer to the human eye's visual perception of an image, in this case, by converting the image from RGB space to HSV space, the representation of each pixel in HSV space can be obtained, thus allowing for a more accurate determination of whether the image's color is acceptable.

[0236] Since the image colors of the standard scene image conform to the image colors of the target calibration scene in all aspects, by comparing the first and second histograms, it is possible to determine whether there are any problems with the image colors.

[0237] The operation to determine the histogram similarity between the first histogram and the second histogram can be as follows: the histogram similarity between the first histogram and the second histogram is determined by chi-square comparison, as shown in the following formula (2).

[0238]

[0239] Where d(H1, H2) is the histogram similarity between the first histogram and the second histogram. H(1) is the second histogram, and H(2) is the first histogram.

[0240] Of course, in addition to determining the histogram similarity between the first and second histograms through chi-square comparison, other methods can also be used, such as correlation comparison or bus distance. This application does not limit this method.

[0241] It is worth noting that before determining the first and second histograms, normalization can be performed on both histograms.

[0242] Normalization is used to normalize the size of the first and second histograms, so that they can be compared on the same scale in subsequent histogram similarity calculations, thereby improving the efficiency of histogram similarity calculation.

[0243] The following example illustrates the process for determining whether the image colors are acceptable. For example, Figure 7 It is a flowchart for determining whether the color of an image is acceptable.

[0244] See Figure 7 To determine if the image color is acceptable, both the original image and the standard scene image are converted to HSV color space, i.e., their RGB values ​​are obtained and then converted back to HSV values. Next, the first histogram of the original image and the second histogram of the standard scene image are calculated. Both histograms are then normalized to the same scale space. Finally, histogram similarity is calculated between the normalized histograms. If the histogram similarity between the first and second histograms is 0, it indicates that the image color is very similar to the standard scene image, and the image color is considered acceptable. If the histogram similarity is not 0, it indicates that the image color differs significantly from the standard scene image, and the image color is considered unacceptable.

[0245] Optionally, after determining the reason why the target vehicle's current scenario is not the target calibration scenario through the above step (2), the reason can be sent to the target APP.

[0246] In this scenario, by sending the reason why the target vehicle's current environment is not the target calibration environment to the target app, the app can then present this reason to the calibration personnel. The calibration personnel can quickly understand why the target vehicle's current environment is not the target calibration environment, allowing for rapid investigation of multiple cameras and swift resolution of any issues. This, in turn, improves the calibration efficiency of the camera system.

[0247] (3) When the target vehicle is in a driving state, obtain the current position of the target vehicle; based on the current position, control the target vehicle to drive to the target position; when the target vehicle has driven to the target position, re-execute the step of obtaining the environmental information around the target vehicle and its subsequent steps.

[0248] If the target vehicle is in a driving state, it means that the target vehicle has not yet reached the target location. In this case, the images captured by multiple cameras are not images of the target location or the target calibration scene. Therefore, it will be recognized that the scene in which the target vehicle is currently located is not the target calibration scene.

[0249] In this scenario, before the target vehicle reaches the target location, the system controls the target vehicle to reach the target location. After the target vehicle reaches the target location, the system reacquires the environmental information around the target vehicle and re-determines whether the scene in which the target vehicle is located is the target calibration scene.

[0250] To facilitate understanding, we will now combine... Figure 8 The camera system calibration method provided in the embodiments of this application will be described by way of example.

[0251] See Figure 8 The process of automatically calibrating the camera system of the target vehicle includes the following steps.

[0252] (1) After the target vehicle arrives at the calibration site, the status of the target vehicle is checked.

[0253] Specifically, checking the status of the target vehicle means determining whether the target vehicle's camera system meets the target calibration conditions.

[0254] (2) If the camera system of the target vehicle meets the target calibration conditions, automatically control the target vehicle to drive to the target location. If the camera system of the target vehicle does not meet the target calibration conditions, report the problem with the camera system and re-perform the status check.

[0255] (3) Scene recognition is performed while controlling the target vehicle to move to the target location.

[0256] Specifically, during the process of controlling the target vehicle to drive towards the target location, images of the area surrounding the target vehicle are acquired, and the environmental information is used to determine whether the current scene of the target vehicle is the target calibration scene.

[0257] (4) When the target vehicle is currently in the target calibration scene, the parameters of the camera system are calibrated and the calibration results are output to the target APP.

[0258] (5) If the target vehicle is not in the target calibration scenario, determine whether the target vehicle should stop.

[0259] (6) When the target vehicle is in a parked state, analyze the image quality to determine if there are any image quality issues. Then send the analysis results to the target APP.

[0260] (7) When the target vehicle is in a driving state, obtain the current location of the target vehicle and control the target vehicle to continue driving to the target location through the intelligent driving controller and chassis controller.

[0261] (8) After controlling the target vehicle to continue driving to the target location, scene recognition is performed again to determine whether the target vehicle is in the target calibration scene.

[0262] In this embodiment, when calibrating the camera system of the target vehicle, the controller first controls the target vehicle to move towards the target location. That is, it controls the target vehicle to move towards the designated location for camera system calibration. Subsequently, during the target vehicle's movement towards the target location, environmental information surrounding the target vehicle is acquired, and based on this information, it is determined whether the current scene of the target vehicle is the target calibration scene, i.e., whether the current scene corresponds to the target location, or whether the target vehicle has reached the target location. Then, if the current scene of the target vehicle is the target calibration scene, the parameters of the camera system are calibrated. Thus, by detecting whether the target vehicle is in the target calibration scene based on environmental information during its movement towards the target location, automatic identification of the calibration scene is achieved. This reduces reliance on manual labor, lowers labor costs, and improves the calibration efficiency of the camera system.

[0263] Figure 9 This is a schematic diagram of the structure of a camera system calibration device provided in an embodiment of this application. The camera system calibration device can be implemented as part or all of a vehicle by software, hardware, or a combination of both. The vehicle can be described below. Figure 10 The vehicle shown. See also Figure 9 The device includes: a control module 901, a first acquisition module 902, a first determination module 903, and a calibration module 904.

[0264] The control module 901 is used to control the target vehicle to move towards the target position in response to the calibration command of the camera system of the target vehicle. The target position refers to the position of the target vehicle when the camera system is calibrated in the calibration site.

[0265] The first acquisition module 902 is used to acquire environmental information around the target vehicle while the target vehicle is moving towards the target location;

[0266] The first determining module 903 is used to determine, based on the environmental information, whether the scene where the target vehicle is currently located is the target calibration scene, wherein the target calibration scene is the scene corresponding to the target position in the calibration site;

[0267] The calibration module 904 is used to calibrate the parameters of the camera system when the target vehicle is currently in the target calibration scene.

[0268] Optionally, the first determining module 903 is used for:

[0269] The environmental information is input into the scene recognition model, and the scene recognition model processes the environmental information to obtain the probability that the scene in which the target vehicle is currently located is the target calibration scene;

[0270] If the probability is greater than or equal to a preset probability threshold, the current scene of the target vehicle is determined as the target calibration scene;

[0271] If the probability is less than a preset probability threshold, it is determined that the current scene of the target vehicle is not the target calibration scene.

[0272] Optionally, the scene recognition model includes multiple convolutional blocks, convolutional layers, pooling layers, and fully connected layers. Each convolutional block consists of at least two convolutional layers. The first determining module 903 is used for:

[0273] The first feature is obtained by extracting features from the environmental information through a convolutional layer.

[0274] The first feature is pooled using a pooling layer to obtain the second feature;

[0275] The first and second features are fused to obtain the third feature;

[0276] The third feature is extracted by using multiple convolutional blocks to obtain the fourth feature;

[0277] The fourth feature is pooled using a pooling layer to obtain the fifth feature;

[0278] The fifth feature is classified by a fully connected layer to obtain the probability that the current scene of the target vehicle is the target calibration scene.

[0279] Optionally, the environmental information includes images of the area surrounding the target vehicle, and the device further includes:

[0280] The second acquisition module is used to acquire the vehicle status of the target vehicle when the current scene of the target vehicle is not the target calibration scene.

[0281] The second determining module is used to determine, based on the image and the standard scene image, why the current scene of the target vehicle is not the target calibration scene when the vehicle status of the target vehicle is a parked state;

[0282] The third acquisition module is used to acquire the current position of the target vehicle when the vehicle status is in a driving state; based on the current position, control the target vehicle to drive to the target position; and when the target vehicle has driven to the target position, re-execute the step of acquiring the environmental information around the target vehicle and its subsequent steps.

[0283] Optionally, the second determining module is used for:

[0284] When the target vehicle is in a parked state, the pixel values ​​of each pixel in the image and the standard scene image are obtained respectively.

[0285] Based on the pixel values ​​of each pixel in the image and the standard scene image, determine the pixel similarity between the image and the standard scene image;

[0286] If the pixel similarity is less than a preset similarity threshold, the image quality of the image is determined to be substandard.

[0287] Optionally, the second determining module is used for:

[0288] When the target vehicle is in a parked state, obtain the RGB values ​​of each pixel in the image and the standard scene image respectively;

[0289] Convert the RGB values ​​of each pixel in the image and the standard scene image to HSV values ​​respectively;

[0290] Based on the HSV values ​​of each pixel in the image and the standard scene image, a first histogram and a second histogram are determined respectively. The first histogram is the histogram of the image in the HSV space, and the second histogram is the histogram of the standard scene image in the HSV space.

[0291] Determine the histogram similarity between the first and second histograms;

[0292] If the histogram similarity does not meet the preset conditions, the image color of the image is determined to be unqualified.

[0293] Optionally, the camera system includes multiple cameras, and the device also includes:

[0294] A camera module for capturing images using the multiple cameras;

[0295] The third determining module is used to determine, upon receiving images captured by the multiple cameras, whether the current state of the camera system meets the target calibration conditions. The target calibration conditions are used to indicate whether the camera system is working properly.

[0296] Optionally, the control module 901 is used for:

[0297] If the current state of the camera system meets the target calibration conditions, control the target vehicle to drive to the target location.

[0298] Optionally, the parameters of the camera system include the camera's focal length, rotation direction, and translation position. The calibration module 904 is used for:

[0299] When the target vehicle is currently in the target calibration scene, multiple target images are obtained by capturing multiple cameras.

[0300] Multiple target images are fused to obtain a three-dimensional image;

[0301] Determine the pixel coordinates of the calibrated object from the 3D image;

[0302] Determine the world coordinates of the calibrated object in the world coordinate system;

[0303] Based on the pixel coordinates and world coordinates of the calibrated object, the focal length, rotation direction, and translation position are determined.

[0304] In this embodiment, when calibrating the camera system of the target vehicle, the target vehicle is first controlled to move towards the target location. That is, the target vehicle is controlled to move towards the designated location for camera system calibration. Subsequently, during the target vehicle's movement towards the target location, environmental information surrounding the target vehicle is acquired. Based on this environmental information, it is determined whether the target vehicle's current scene is the target calibration scene, i.e., whether the target vehicle has reached the target location. Then, if the target vehicle's current scene is the target calibration scene, the parameters of the camera system are calibrated. Thus, by detecting whether the target vehicle is in the target calibration scene based on environmental information during its movement towards the target location, automatic identification of the calibration scene is achieved. This reduces reliance on manual labor, lowers labor costs, and improves the calibration efficiency of the camera system.

[0305] It should be noted that the camera system calibration device provided in the above embodiments is only an example of the division of the above functional modules when calibrating the parameters of the vehicle's camera system. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0306] The functional units and modules in the above embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of the embodiments of this application.

[0307] The camera system calibration device and camera system calibration method provided in the above embodiments belong to the same concept. The specific working process and technical effects of the units and modules in the above embodiments can be found in the method embodiments section, and will not be repeated here.

[0308] Figure 10 This is a schematic diagram of the structure of a vehicle provided in an embodiment of this application.

[0309] For example, such as Figure 10 As shown, the vehicle includes a memory 101 and a processor 100, wherein the memory 101 stores executable program code 102, and the processor 100 is used to call and execute the executable program code 102 to perform the above-mentioned camera system calibration method.

[0310] This embodiment can divide the vehicle into functional modules according to the above method example. For example, each function can be assigned to a separate module, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware. It should be noted that the module division in this embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods.

[0311] When each functional module is divided according to its corresponding function, the vehicle may include: a control module, a first acquisition module, a first determination module, and a calibration module. It should be noted that all relevant content of each step involved in the above method embodiments can be referenced from the functional descriptions of the corresponding functional modules, and will not be repeated here.

[0312] The vehicle provided in this embodiment is used to perform the above-described camera system calibration method, and therefore can achieve the same effect as the above-described implementation method.

[0313] When using integrated units, the vehicle may include a processing module and a storage module. The processing module is used to control and manage the vehicle's actions. The storage module supports the vehicle in executing program code and data.

[0314] The processing module may be a processor or a controller, which can implement or execute various exemplary logic blocks, modules, and circuits as disclosed in this application. The processor may also be a combination of computing functions, such as a combination of one or more microprocessors, a combination of digital signal processing (DSP) and microprocessors, etc., and the storage module may be a memory.

[0315] This embodiment also provides a computer-readable storage medium storing computer program code. When the computer program code is run on a computer, the computer executes the above-described related method steps to implement the above-described method for calibrating a camera system.

[0316] This embodiment also provides a computer program product that, when run on a computer, causes the computer to perform the aforementioned related steps to implement the above-described method for calibrating a camera system.

[0317] In this embodiment, the vehicle, computer-readable storage medium, computer program product, or chip are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can be referred to the beneficial effects of the corresponding methods provided above, and will not be repeated here.

[0318] Through the above description of the embodiments, those skilled in the art will understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0319] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0320] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A calibration method for a camera system, characterized in that, The method includes: In response to a calibration command for the camera system of a target vehicle, the target vehicle is controlled to move toward a target location, wherein the target location is the position of the target vehicle when the camera system is calibrated in the calibration site; During the process of the target vehicle traveling towards the target location, environmental information around the target vehicle is acquired; Based on the environmental information, it is determined whether the scene where the target vehicle is currently located is the target calibration scene. The target calibration scene is the scene corresponding to the target position in the calibration site. The environmental information includes images around the target vehicle. When the target vehicle is currently in the same scene as the target calibration scene, the parameters of the camera system are calibrated. If the current scene of the target vehicle is not the target calibration scene, obtain the vehicle status of the target vehicle; If the target vehicle is in a parked state, based on the image and the standard scene image, determine the reason why the current scene of the target vehicle is not the target calibration scene. The standard scene image is an image under the target calibration scene. The reason is that there is a problem with the image captured by the camera system. If the target vehicle is in a driving state, obtain the current position of the target vehicle; based on the current position, control the target vehicle to drive to the target position; if the target vehicle has driven to the target position, repeat the step of obtaining the environmental information around the target vehicle and the subsequent steps.

2. The method as described in claim 1, characterized in that, The step of determining whether the current scene of the target vehicle is the target calibration scene based on the environmental information includes: The environmental information is input into the scene recognition model, and the environmental information is processed by the scene recognition model to obtain the probability that the scene in which the target vehicle is currently located is the target calibration scene; If the probability is greater than or equal to a preset probability threshold, the current scene of the target vehicle is determined to be the target calibration scene; If the probability is less than the preset probability threshold, it is determined that the current scene of the target vehicle is not the target calibration scene.

3. The method as described in claim 2, characterized in that, The scene recognition model includes multiple convolutional blocks, convolutional layers, pooling layers, and fully connected layers. Each convolutional block consists of at least two convolutional layers. The process of inputting the environmental information into the scene recognition model and processing the environmental information to obtain the probability that the current scene of the target vehicle is the target calibration scene includes: The environmental information is feature extracted through the convolutional layer to obtain the first feature; The first feature is pooled using the pooling layer to obtain the second feature; The first feature and the second feature are fused to obtain the third feature; The third feature is extracted using the multiple convolutional blocks to obtain the fourth feature; The fourth feature is pooled using the pooling layer to obtain the fifth feature; The fifth feature is classified by the fully connected layer to obtain the probability that the current scene of the target vehicle is the target calibration scene.

4. The method as described in claim 1, characterized in that, When the target vehicle is in a parked state, the reason for determining that the current scene of the target vehicle is not the target calibration scene, based on the image and the standard scene image, includes: When the target vehicle is in a parked state, the pixel values ​​of each pixel in the image and the standard scene image are obtained respectively. Based on the pixel values ​​of each pixel in the image and the standard scene image, determine the pixel similarity between the image and the standard scene image; If the pixel similarity is less than a preset similarity threshold, the image quality of the image is determined to be substandard.

5. The method as described in claim 1, characterized in that, When the target vehicle is in a parked state, the reason for determining that the current scene of the target vehicle is not the target calibration scene, based on the image and the standard scene image, includes: When the target vehicle is in a parked state, the RGB values ​​of each pixel in the image and the standard scene image are obtained respectively. The RGB values ​​of each pixel in the image and the standard scene image are converted into HSV values ​​respectively; Based on the HSV values ​​of each pixel in the image and the standard scene image, a first histogram and a second histogram are determined respectively. The first histogram is the histogram of the image in the HSV space, and the second histogram is the histogram of the standard scene image in the HSV space. Determine the histogram similarity between the first and second histograms; If the histogram similarity does not meet the preset conditions, the image color of the image is determined to be unqualified.

6. The method as described in claim 1, characterized in that, The camera system includes multiple cameras, and before controlling the target vehicle to drive towards the target location, it also includes: Images are captured using the multiple cameras; Upon receiving images captured by the plurality of cameras, it is determined that the current state of the camera system meets the target calibration conditions, which are used to indicate whether the camera system is working properly; Controlling the target vehicle to travel to the target location includes: If the current state of the camera system meets the target calibration conditions, control the target vehicle to drive to the target location.

7. The method as described in claim 1, characterized in that, The camera system includes multiple cameras, and the parameters of the camera system include the focal length, rotation direction, and translation position of the cameras. When the scene in which the target vehicle is currently located is the target calibration scene, the parameters of the camera system are calibrated, including: When the target vehicle is currently in the target calibration scene, multiple target images are captured by the multiple cameras; The multiple target images are fused to obtain a three-dimensional image; Determine the pixel coordinates of the calibrated object from the three-dimensional image; Determine the world coordinates of the calibrated object in the world coordinate system; Based on the pixel coordinates and world coordinates of the calibrated object, the focal length, the rotation direction, and the translation position are determined.

8. A camera system calibration device, characterized in that, The device includes: The control module is used to control the target vehicle to drive towards the target position in response to the calibration command of the camera system of the target vehicle. The target position refers to the position of the target vehicle when the camera system is calibrated in the calibration site. The first acquisition module is used to acquire environmental information around the target vehicle during the process of the target vehicle driving towards the target location; The first determining module is used to determine, based on the environmental information, whether the scene where the target vehicle is currently located is a target calibration scene, wherein the target calibration scene is the scene corresponding to the target position in the calibration site, and the environmental information includes images around the target vehicle; The calibration module is used to calibrate the parameters of the camera system when the target vehicle is currently in the target calibration scene; The second acquisition module is used to acquire the vehicle status of the target vehicle when the current scene of the target vehicle is not the target calibration scene; The second determining module is used to determine, based on the image and the standard scene image, why the current scene of the target vehicle is not the target calibration scene when the vehicle status of the target vehicle is a parked state, and the reason is that there is a problem with the image captured by the camera system. The third acquisition module is used to acquire the current position of the target vehicle when the vehicle status of the target vehicle is in a driving state; control the target vehicle to drive to the target position based on the current position; and, when the target vehicle has driven to the target position, re-execute the step of acquiring the environmental information around the target vehicle and its subsequent steps.

9. A vehicle, characterized in that, The vehicles include: Memory, used to store executable program code; A processor for calling and running the executable program code from the memory, causing the vehicle to perform the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method as described in any one of claims 1 to 7.

Images