Calibration method for vehicle-mounted camera, camera calibration device and computer readable storage medium

Abstract

The invention discloses a calibration method for a vehicle-mounted camera and a camera calibration device. The method comprises the following steps: establishing communication connection between the camera calibration device and a vehicle-mounted control device of engineering machinery; acquiring a real-time video stream of a target camera from the vehicle-mounted control device; displaying the real-time video stream on a display device of a camera calibration device and photographing according to the display content; and calibrating the vehicle-mounted camera based on the calibration picture obtained by photographing. According to the calibration method and the camera calibration device provided by the invention, a user can interact with the vehicle-mounted control device in real time through the camera calibration device to realize online calibration, so that the vehicle-mounted camera of the engineering machinery can be calibrated even not in a production workshop or a professional maintenance place.

Description

Technical Field

[0001] The present invention relates to the technical field of engineering machinery detection, and in particular to a vehicle-mounted camera calibration method, a camera calibration device, and a computer-readable storage medium. Background Art

[0002] Modern construction machinery often uses onboard cameras or cameras for environmental and self-awareness. Calibration of these cameras is typically required after manufacturing and installation to ensure compliance with design specifications and measurement accuracy. Furthermore, once installed on the construction machinery, the cameras may deviate from their ideal operating state or original installation state due to shock, vibration, and other factors during operation. Therefore, regular calibration is also necessary.

[0003] However, current onboard control systems for construction vehicles lack a user interface (UI), preventing users from performing real-time interactive operations related to onboard camera calibration. Consequently, the construction machinery must be returned to the production workshop or a specialized repair facility for onboard camera calibration, which can cause the construction machinery to be offline and significantly impact its availability. Therefore, there is a need for further improvements to existing onboard camera calibration methods and devices. Summary of the Invention

[0004] The present invention proposes a calibration method and a camera calibration device for a vehicle-mounted camera, aiming to overcome one or more of the above-mentioned technical problems and / or other technical problems in the prior art.

[0005] According to one aspect of the present invention, a calibration method for a vehicle-mounted camera is proposed, the method comprising the following steps:

[0006] Establishing a communication connection between the camera calibration device and the onboard control device of the construction machinery;

[0007] Acquire a real-time video stream of a target camera from the vehicle-mounted control device;

[0008] Displaying the real-time video stream on a display device of a camera calibration device and taking photos according to the displayed content;

[0009] Calibrate the vehicle camera based on the calibration images obtained by taking photos.

[0010] According to another aspect of the present invention, a camera calibration device is provided. The camera calibration device comprises an input module, a display module, a communication module, a processor, and a memory, wherein the processor is configured to execute any one embodiment of the aforementioned method.

[0011] According to another aspect of the present invention, a computer-readable storage medium is provided, on which a computer program is stored. The computer program includes executable instructions. When the executable instructions are executed by a processor, any one of the aforementioned embodiments of the calibration method for a vehicle-mounted camera is implemented.

[0012] In the calibration method for vehicle-mounted cameras proposed in the present invention, a communication connection is first established between an external camera calibration device and a vehicle-mounted control device through a data interface to obtain a real-time video stream of the target camera. The real-time video stream can be presented to the user by means of the display device of the camera calibration device. After viewing the video content through the display interface of the camera calibration device, the user can input corresponding photo-taking instructions through the camera calibration device, and the vehicle-mounted camera can take a calibration picture according to the instructions input by the user. Finally, calibration can be performed based on the captured calibration picture. The camera calibration device is designed, for example, as an external removable device. As a result, the vehicle-mounted camera of the construction machinery can be calibrated even outside the production workshop or professional maintenance site, especially online calibration, that is, the construction machinery does not leave the working area during the calibration process. In addition, the user can interact with the vehicle-mounted control device in real time through the camera calibration device to achieve online calibration. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The above and other features and advantages of the present invention will become more apparent by describing in detail exemplary embodiments thereof with reference to the attached drawings.

[0014] Figure 1 is a flow chart of an embodiment of the method according to the present invention;

[0015] Figure 2 A schematic diagram of a technical architecture for implementing the method of the present invention;

[0016] Figure 3 2 is a schematic diagram of the architecture of a preferred embodiment of a camera calibration device according to the present invention. DETAILED DESCRIPTION

[0017] The exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, the exemplary embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that the disclosure will be comprehensive and complete and will fully convey the concepts of the exemplary embodiments to those skilled in the art. In the drawings, the dimensions of some elements may be exaggerated or distorted for clarity. Identical reference numerals in the drawings represent identical or similar structures, and thus their detailed description will be omitted.

[0018] In addition, the described features, structures or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present invention. However, those skilled in the art will appreciate that it is possible to practice the technical solution of the present invention without one or more of the specific details, or to adopt other methods, elements, etc. In other cases, known structures, methods or operations are not shown or described in detail to avoid blurring various aspects of the present invention.

[0019] Figure 1 FIG. 1 is a flow chart of a preferred embodiment of a calibration method for a vehicle-mounted camera according to the present invention. The method comprises the following steps:

[0020] S1: Establishing a communication connection between the camera calibration device and the onboard control device of the construction machinery;

[0021] S2: Obtaining a real-time video stream of the target camera from the vehicle-mounted control device;

[0022] S3: Displaying the real-time video stream on a display device of the camera calibration device and taking photos according to the displayed content;

[0023] S4: Calibrate the vehicle camera based on the calibration image obtained by taking photos.

[0024] Here, the camera calibration device is, for example, a portable mobile device, such as a tablet computer, laptop computer, smartphone, etc. The onboard control device can be a domain control unit (DCU) or an onboard computer installed on the construction machinery. The camera calibration device can be connected to the onboard control device via a data interface. The data interface can be a common network interface, such as an Ethernet interface, a Wi-Fi interface, a Bluetooth interface, etc. Of course, other interfaces suitable for transmitting data between the camera calibration device and the onboard control device can also be considered.

[0025] During the process of establishing a communication connection between the camera calibration device and the on-board control device, the camera calibration device can be authenticated. Authentication refers to the use of a series of security mechanisms to confirm whether the identity of the user or program is legitimate in order to control access rights to resources in the on-board control device. Common authentication technologies are similar to usernames and passwords. Users need to enter the correct username and password to establish a communication connection and access the corresponding resources. Authentication can also be achieved through hardware containing secret keys, such as a USB key. Users need to insert the USB key and enter the password to complete the authentication. Authentication improves system security and enables categorized management of users with different identities. For example, different users have different permission levels, and different users can calibrate different on-board cameras or different parameter ranges.

[0026] To uniquely identify an onboard camera, each onboard camera is assigned a camera number. The camera numbers can be obtained from the onboard control unit and displayed in a list on the display of the camera calibration device. Of course, the camera numbers can also be pre-stored in the memory of the camera calibration device and displayed in a list on the display of the camera calibration device when needed. The user can select the corresponding camera number from the display interface of the camera calibration device. Using the corresponding camera number, the real-time video stream of the target camera can be obtained from the onboard control unit via the aforementioned communication connection. The real-time video captured by the corresponding camera is then displayed to the user on the display of the camera calibration device.

[0027] The calibration plate is, for example, a chessboard pattern consisting of black and white rectangles. During the calibration process, for example, a calibration plate placed at a specific position around the vehicle is photographed by a target camera to obtain a calibration picture. That is to say, the calibration plate is photographed at different positions, angles, and postures, and at least 3 calibration pictures need to be taken, preferably 20 to 30 calibration pictures. The user or calibration personnel views the video stream of the calibration plate photographed by the camera through a display device. If the calibration personnel considers that the image of the calibration plate captured meets the requirements after viewing, a photo-taking instruction can be issued through the calibration device, and the photo obtained by taking the photo is stored as a calibration picture. The calibration picture is stored in the memory of the on-board control device, for example, according to the camera number. It is also possible to consider storing the calibration picture in other locations, such as the memory of the camera calibration device or other external memories, which can be accessed by the calibration program.

[0028] After the calibration picture is taken, the on-board camera can be calibrated based on the calibration picture obtained by taking the picture. Camera calibration can be performed according to existing methods, such as Zhang Zhengyou calibration method (Zhang's calibration method). Zhang's calibration method uses a chessboard as a calibration plate. After obtaining the calibration picture of the calibration plate, the Harris corner detection algorithm is used to obtain the pixel coordinates of each corner point. This method fixes the world coordinate system on the chessboard, that is, the world coordinates of any point on the chessboard. Because the world coordinate system is defined on the chessboard, the size of each grid is also known, so the coordinates of each corner point in the world coordinate system can be calculated. Therefore, the pixel coordinates and world coordinates of the corner points can be used for calibration to obtain the camera's internal parameters, external parameters, and distortion parameters.

[0029] The calibration method and steps are embedded in an onboard control unit, for example, as a software module. By activating the calibration program within the onboard control unit, the onboard camera can be calibrated based on the calibration image captured. The onboard control unit can be a domain control unit (DCU) or an onboard computer.

[0030] After the calibration is completed, it is preferred that the calibration results can also be evaluated. The method for evaluating the calibration results is to reproject the three-dimensional points in space using the obtained internal and external parameters of the camera, obtain the coordinates of the new projection points of the three-dimensional points in space on the image, and calculate the deviation between the projection coordinates and the sub-pixel corner coordinates. The smaller the deviation, the better the calibration result. In other words, if the deviation is within the predetermined range, the camera can continue to be used after calibration. If the deviation is large and exceeds the predetermined range, it means that the camera cannot be restored to normal working state through calibration or calibration. In this case, it is necessary to prompt to replace the camera.

[0031] Figure 2 A schematic diagram of the system architecture for implementing the method according to the present invention is shown. The camera calibration device 400 is designed as a removable device 1 with a display 2. The removable device 1 has multiple functional modules schematically shown in boxes. The display 2 can be integrated into the removable device 1 or can be an independent display device. The removable device 1 can be communicatively connected to an on-board control device in the form of an on-board domain control unit 3 via a data interface. During the process of establishing the communication connection, identity authentication is also performed. Only when the identity authentication is passed can data and instructions be transmitted via the communication connection. If the identity authentication is not passed, the communication connection can be directly interrupted or the transmission of data or instructions can be prohibited. The communication connection can be implemented in a wired or wireless manner based on the data interface. If the identity authentication is passed, the removable device 1 can obtain the video of the specified camera from the on-board domain control unit 3, start the camera to take a picture of the calibration plate 6, and start the calibration program to perform camera calibration based on the picture. The calibration of the on-board camera includes intrinsic parameter calibration and / or distortion calibration.

[0032] The construction machinery may include multiple onboard cameras, but for simplicity, only two onboard cameras 4 and 5 are shown here. The onboard domain control unit 3 includes multiple functional modules, schematically illustrated by blocks, and is signal-connected to the onboard cameras 4 and 5. The onboard domain control unit 3 can capture real-time video from the onboard cameras 4 and 5 and issue a capture command to the corresponding onboard cameras 4 and 5 to control the onboard cameras. The onboard cameras 4 and 5 are installed in different locations and viewing angles on the construction machinery. Each onboard camera 4 and 5 is assigned a unique camera number to distinguish them. The onboard domain control unit 3 can capture video streams from each onboard camera and transmit them to the mobile device 1. The onboard domain control unit 3 also includes a camera calibration module, such as a camera calibration program module, to implement camera calibration. Preferably, after camera calibration is completed, the onboard domain control unit 3 transmits the calibration results to the camera calibration device 400. Calibrators can evaluate the calibration process based on the calibration results released by the onboard domain control unit to determine whether the current camera can be used normally after calibration or requires repair or replacement.

[0033] In an exemplary embodiment of the present application, a computer-readable storage medium is further provided, on which a computer program is stored. The program includes executable instructions that, when executed by, for example, a processor, can implement the steps of the vehicle camera calibration method described in any of the above-described embodiments. In some possible implementations, various aspects of the present application can also be implemented in the form of a program product, which includes program code. When the program product is executed on a terminal device, the program code is used to cause the camera calibration device to perform the steps described in the vehicle camera calibration method according to various exemplary embodiments of the present application.

[0034] The computer-readable storage medium may include a data signal propagated in baseband or as part of a carrier wave, wherein the readable program code is carried. The data signal propagated may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The readable storage medium may also be any readable storage medium other than an optical disc, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination thereof.

[0035] The program code for performing the operations of the present application can be written in any combination of one or more programming languages, including object-oriented programming languages such as Java, C++, etc., and conventional procedural programming languages such as "C" or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0036] Refer to the following Figure 3 The camera calibration device 400 according to this embodiment of the present application is described. Figure 3 The camera calibration device 400 shown is merely an example and should not limit the functions and scope of use of the embodiments of the present application.

[0037] like Figure 3As shown, the camera calibration device 400 is in the form of a portable computing device, such as a PAD, a laptop or a smart phone. The components of the camera calibration device 400 may include but are not limited to: at least one processing unit 410, at least one storage unit 420, a bus 430 connecting different system components (including the storage unit 420 and the processing unit 410), a display unit 440, etc. The storage unit stores a program code, and the program code can be executed by the processing unit 410, so that the processing unit 410 performs the steps of various exemplary embodiments of the present application described in the calibration method for vehicle-mounted cameras in this specification. For example, the processing unit 410 may perform the following steps: Figure 1 Follow the steps shown in .

[0038] The storage unit 420 may include a readable storage medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 4201 and / or a cache memory unit 4202 , and may further include a read-only memory unit (ROM) 4203 .

[0039] The storage unit 420 may also include a program / utility 4204 having a set (at least one) of program modules 4205, such program modules 4205 including but not limited to: an operating system, one or more application programs, other program modules and program data, each of which or some combination may include an implementation of a network environment.

[0040] Bus 430 may represent one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0041] The camera calibration device 400 can also communicate with one or more external devices 500 (e.g., a keyboard, a pointing device, a Bluetooth device, etc.), one or more devices that enable a user to interact with the camera calibration device 400, and / or any device that enables the camera calibration device 400 to communicate with one or more other computing devices (e.g., a router, a modem, etc.). Such communication can be performed via an input / output (I / O) interface 450. Furthermore, the camera calibration device 400 can also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network such as the Internet) via a network adapter 460. The network adapter 460 can communicate with other modules of the camera calibration device 400 via a bus 430. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with the camera calibration device 400, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0042] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described here can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the embodiments of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, or a network device, etc.) to execute the calibration method for a vehicle-mounted camera according to the embodiments of the present application.

[0043] Other embodiments of the present invention will readily occur to those skilled in the art after considering the specification and practicing the disclosure herein. The present invention is intended to cover any variations, uses, or adaptations of the present invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the invention being indicated by the appended claims.

Claims

1. A calibration method for a vehicle-mounted camera, the method comprising the following steps: Establishing a communication connection between the camera calibration device and the onboard control device of the construction machinery; Acquire a real-time video stream of a target camera from the vehicle-mounted control device; Displaying the real-time video stream on a display device of a camera calibration device and taking photos according to the displayed content; Calibrate the vehicle camera based on the calibration images obtained by taking photos.

2. The method according to claim 1, characterized in that Establishing a communication connection between the camera calibration device and the onboard control device of the construction machinery includes authenticating the camera calibration device.

3. The method according to claim 1 or 2, characterized in that The real-time video stream of the target camera is obtained from the vehicle-mounted control device through a camera number, wherein the camera number uniquely identifies the vehicle-mounted camera.

4. The method according to claim 3, characterized in that The operator issues a photo-taking instruction according to the displayed content, and after receiving the corresponding instruction, the target camera takes a photo of the calibration plate.

5. The method according to claim 3 or 4, characterized in that The calibration images obtained by taking photos are stored in the storage area inside the vehicle control device according to the corresponding camera numbers.

6. The method according to claim 1 or 2, characterized in that The calibration program in the vehicle control device is started to calibrate the vehicle camera based on the calibration image obtained by taking the photo.

7. The method according to claim 1 or 2, characterized in that The vehicle-mounted control device is a domain control unit or a vehicle-mounted computer.

8. The method according to claim 1 or 2, characterized in that The calibration of the vehicle-mounted camera includes intrinsic parameter calibration and / or distortion calibration.

9. A camera calibration device, comprising an input module, a display module, a communication module, a processor, and a memory, wherein: The processor is configured to perform the method according to any one of claims 1 to 8. 10 . A computer-readable storage medium having a computer program stored thereon, the computer program comprising executable instructions, and when the executable instructions are executed by a processor, the method according to claim 1 is implemented.

Images