Method and device for determining camera calibration parameters in multi-marshalling articulated vehicle surround view system, train, equipment and medium

Abstract

The invention provides a method and device for determining camera calibration parameters in a multi-marshalling articulated vehicle surround view system, a train, equipment and a medium, and can be applied to the technical field of camera calibration and the technical field of 360-degree surround view. The method comprises the following steps: for any adjacent first target carriage and second target carriage in n carriages, acquiring a first checkerboard image generated by shooting a checkerboard by a first camera and a second checkerboard image generated by shooting the checkerboard by a second camera at each rotation angle, determining a static transformation matrix which corresponds to each rotation angle and is used for representing a transformation relation between the first camera and the second camera; solving a first transformation matrix and a second transformation matrix corresponding to each rotation angle by utilizing a preset equation according to each rotation angle and the static transformation matrix corresponding to each rotation angle, and obtaining a real-time rotation angle between the first target carriage and the second target carriage based on the real-time rotation angle between the first target carriage and the second target carriage; a real-time transformation matrix between the first camera and the second camera is determined.

Description

Technical Field

[0001] The present disclosure relates to the fields of camera calibration technology and 360-degree surround view technology, and in particular to a method, apparatus, train, equipment, and medium for determining camera calibration parameters in a surround view system of a multi-unit articulated vehicle. Background Art

[0002] During multi-carriage train operation, the connections between carriages undergo relative motion due to turning and bending. This causes the relative positions of the cameras mounted on the carriages to constantly change, posing a challenge to the image fusion of the surround view system. Therefore, overcoming the impact of relative motion caused by turning and bending is a technical problem that needs to be solved in related technologies. Summary of the Invention

[0003] In view of this, the present disclosure provides a method, apparatus, train, equipment and medium for determining camera calibration parameters in a surround view system of a multi-unit articulated vehicle.

[0004] One aspect of the present disclosure provides a method for determining camera calibration parameters in a surround view system of a multi-unit articulated train, wherein the multi-unit articulated train includes n carriages, each of which is equipped with a camera. The method includes:

[0005] For any adjacent first target carriage and second target carriage among the n carriages, obtain a first checkerboard image generated by photographing the checkerboard with the first camera and a second checkerboard image generated by photographing the checkerboard with the second camera at each rotation angle.

[0006] The first camera is mounted on a side of the first target carriage, the second camera is mounted on the same side of the second target carriage, the checkerboard is placed within a predetermined area, n is an integer greater than or equal to 2, and the rotation angle represents the relative rotation angle between the first target carriage and the second target carriage;

[0007] Determining, based on the first checkerboard image and the second checkerboard image corresponding to each rotation angle, a static transformation matrix corresponding to each rotation angle and used to represent a transformation relationship between the first camera and the second camera;

[0008] Using a predetermined equation, based on the rotation angles and the static transformation matrices corresponding to the rotation angles, a first transformation matrix corresponding to the rotation angles for representing the transformation relationship between the first camera and the hinge of the vehicle body and a second transformation matrix corresponding to the rotation angles are solved;

[0009] Based on the first transformation matrix and the second transformation matrix corresponding to the above-mentioned rotation angles and the real-time rotation angle between the above-mentioned first target car and the above-mentioned second target car when the above-mentioned multi-unit articulated vehicle is running, the real-time transformation matrix between the above-mentioned first camera and the above-mentioned second camera is determined, wherein the above-mentioned real-time transformation matrix represents the parameters used for camera calibration.

[0010] According to an embodiment of the present disclosure, determining a static transformation matrix corresponding to each rotation angle and used to represent a transformation relationship between the first camera and the second camera based on the first checkerboard image and the second checkerboard image corresponding to each rotation angle includes:

[0011] For each rotation angle, based on a preset number of target points in the checkerboard, a static transformation matrix between the first camera and the second camera is determined according to first coordinates of the target points in the first checkerboard image and second coordinates of the target points in the second checkerboard image.

[0012] According to an embodiment of the present disclosure, a first coordinate system is established with the optical center of the first camera as the origin, a second coordinate system is established with the hinge between the first target carriage and the second target carriage as the origin, and a third coordinate system is established with the optical center of the second camera as the origin. The first transformation matrix represents the transformation relationship from the first coordinate system to the second coordinate system, and the second transformation matrix represents the transformation relationship from the second coordinate system to the third coordinate system.

[0013] The determining of the static transformation matrix between the first camera and the second camera includes:

[0014] According to the transformation matrix, the first coordinate of the target point in the first checkerboard image is transformed into the third coordinate system to obtain the third coordinate of the first coordinate in the third coordinate system, wherein the transformation matrix represents an unknown static transformation matrix; according to the third coordinate and the second coordinate of the target point in the second checkerboard image, an error vector for the target point is constructed; and based on the error vectors of each of the preset number of target points, the static transformation matrix is ​​determined.

[0015] According to an embodiment of the present disclosure, constructing an error vector for the target point based on the third coordinate and the second coordinate for the target point in the second checkerboard image includes:

[0016] An error vector for the target point is obtained by subtracting the second coordinate from the third coordinate, wherein the third coordinate is obtained by multiplying the transformation matrix by the first coordinate.

[0017] According to an embodiment of the present disclosure, determining the static transformation matrix based on the error vectors of the preset number of target points includes:

[0018] Calculating the square of the norm of the error vector of each of the preset number of target points;

[0019] The error function is obtained by summing the squares of the error vectors of the preset number of target points;

[0020] When the error function takes the minimum value, the value of the transfer function in the error function is determined as the static transformation matrix.

[0021] According to an embodiment of the present disclosure, the method for determining camera calibration parameters in the multi-unit articulated vehicle surround view system further includes:

[0022] A first calibration parameter for characterizing a transformation relationship between the first camera and the checkerboard and a second calibration parameter for characterizing a transformation relationship between the second camera and the checkerboard are calculated based on the coordinates of the target point in the checkerboard in world coordinates, the first coordinates, and the second coordinates.

[0023] According to an embodiment of the present disclosure, the method for determining camera calibration parameters in the multi-unit articulated vehicle surround view system further includes:

[0024] The image captured by the first camera is corrected according to the real-time transformation matrix, the first calibration parameter, and the second calibration parameter to obtain a corrected image, so as to fuse the corrected image with the image captured by the second camera.

[0025] Another aspect of the present disclosure provides a device for determining camera calibration parameters in a surround view system for a multi-unit articulated vehicle, comprising:

[0026] An acquisition module is configured to acquire, for any adjacent first target carriage and second target carriage among the n carriages, a first checkerboard image generated by photographing the checkerboard with the first camera and a second checkerboard image generated by photographing the checkerboard with the second camera at each rotation angle, respectively.

[0027] The first camera is mounted on a side of the first target carriage, the second camera is mounted on the same side of the second target carriage, the checkerboard is placed within a predetermined area, n is an integer greater than or equal to 2, and the rotation angle represents the relative rotation angle between the first target carriage and the second target carriage;

[0028] A first determining module is configured to determine, based on the first checkerboard image and the second checkerboard image corresponding to each rotation angle, a static transformation matrix corresponding to each rotation angle and used to represent a transformation relationship between the first camera and the second camera;

[0029] a solving module, configured to solve, using a predetermined equation, a first transformation matrix corresponding to each rotation angle and representing a transformation relationship between the first camera and the hinge of the vehicle body, and a second transformation matrix corresponding to each rotation angle, representing a transformation relationship between the second camera and the hinge of the vehicle body;

[0030] The second determination module is used to determine the real-time transformation matrix between the first camera and the second camera based on the first transformation matrix and the second transformation matrix corresponding to the above-mentioned each rotation angle and the real-time rotation angle between the above-mentioned first target car and the above-mentioned second target car when the above-mentioned multi-unit articulated vehicle is running, wherein the above-mentioned real-time transformation matrix represents the parameters used for camera calibration.

[0031] Another aspect of the present disclosure provides a train comprising: a multi-unit articulated vehicle, wherein the train is equipped with the device for determining camera calibration parameters in the surround view system of the multi-unit articulated vehicle.

[0032] Another aspect of the present disclosure provides an electronic device, comprising: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method described above.

[0033] Another aspect of the present disclosure provides a computer-readable storage medium storing computer-executable instructions, which are used to implement the method described above when executed.

[0034] Another aspect of the present disclosure provides a computer program product comprising computer executable instructions, which are used to implement the method described above when the instructions are executed.

[0035] According to the method, apparatus, train, equipment, and medium for determining camera calibration parameters in a surround-view system for a multi-carriage articulated train provided herein, a first checkerboard image generated by a first camera photographing a checkerboard grid and a second checkerboard image generated by a second camera photographing the checkerboard grid at different rotation angles are obtained. Based on the first and second checkerboard images, a static transformation matrix between the first and second cameras at each rotation angle is determined. Then, based on the relationship between the static transformation matrix, the first transformation matrix, and the second transformation matrix at the same rotation angle, the first transformation matrix and the second transformation matrix can be solved to obtain the first and second transformation matrices. During the operation of a multi-carriage train, changes in the articulation angles at the articulations of adjacent cars are the only factor that changes the transformation relationship between the first and second cameras. Therefore, based on the static transformation matrix, the first transformation matrix, and the second transformation matrix corresponding to each rotation angle, the real-time transformation matrix between the first and second cameras can be obtained when the real-time articulation angle is obtained. This plays a key role in accurately fusion images in the subsequent surround-view system, ensuring the continuity and accuracy of the fused surround-view image. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The above and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0037] Figure 1 A diagram schematically illustrates an application scenario of a method for determining camera calibration parameters in a surround view system for multiple articulated vehicles according to an embodiment of the present disclosure;

[0038] Figure 2 Schematically shows a flow chart of a method for determining camera calibration parameters in a surround view system for multi-unit articulated vehicles according to an embodiment of the present disclosure;

[0039] Figure 3 A schematic diagram of a bicycle surround view system according to an embodiment of the present disclosure is schematically shown;

[0040] Figure 4 A schematic diagram schematically illustrates any adjacent first target carriage and second target carriage according to an embodiment of the present disclosure;

[0041] Figure 5 Schematically illustrates a vehicle cabin analysis diagram for determining camera calibration parameters according to an embodiment of the present disclosure;

[0042] Figure 6 A block diagram schematically illustrates a device for determining camera calibration parameters in a surround view system for multiple articulated vehicles according to an embodiment of the present disclosure; and

[0043] Figure 7The block diagram of an electronic device suitable for implementing a method for determining camera calibration parameters in a surround view system for multi-unit articulated vehicles according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0044] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are merely exemplary and are not intended to limit the scope of the present disclosure. In the detailed description below, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of well-known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0045] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise," "include," etc. used herein indicate the presence of the features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0046] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0047] When expressions such as "at least one of A, B, and C, etc." are used, they should generally be interpreted in accordance with the meaning commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0048] In the embodiments of this disclosure, the collection, updating, analysis, processing, use, transmission, provision, disclosure, and storage of all data involved (including, but not limited to, user personal information) comply with relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. In particular, necessary measures have been taken to prevent unauthorized access to user personal information data and to safeguard the security of user personal information, network security, and national security.

[0049] In the embodiments of the present disclosure, the user's authorization or consent is obtained before obtaining or collecting the user's personal information.

[0050] During the development of this disclosure, it was discovered that a 360-degree surround view system, also known as a panoramic surround view system or a panoramic monitoring system, is a type of driver-assistance technology used in vehicles that provides a 360-degree panoramic view of the vehicle's surroundings. This surround view system typically consists of multiple cameras mounted on the front, rear, left, and right sides of the vehicle. An algorithm is used to synthesize the images captured by each camera into a bird's-eye view.

[0051] The functions of a 360-degree surround view system generally include the following aspects: the 360-degree surround view system can help the driver better understand the environment around the vehicle, reduce blind spots, and thus reduce the risk of traffic accidents, that is, improve safety; by providing comprehensive visual information, the 360-degree surround view system can improve the driver's driving confidence and comfort, that is, enhance the driving experience; for autonomous driving systems, the 360-degree surround view system can provide the necessary environmental perception information to help autonomous driving vehicles better understand the surrounding environment, that is, support autonomous driving; for fleet management or vehicle monitoring applications, the 360-degree surround view system can provide real-time vehicle status monitoring.

[0052] The image synthesis method of the surround view system includes the following steps: image acquisition, namely, multiple cameras in the system simultaneously capture video images around the vehicle; image processing, namely, the captured images need to be pre-processed, including noise removal and contrast enhancement, to improve image quality; feature extraction, namely, based on image processing, extracting key feature points in the image, such as corners and edges; image stitching, namely, using the extracted feature points to stitch the images from multiple cameras into a complete 360-degree panoramic image through an image stitching algorithm. Common stitching algorithms include feature-based stitching and pixel-based stitching; distortion correction, namely, due to differences in the installation positions and angles of the cameras, the stitched panoramic image may be distorted, and the panoramic image needs to be distortion-corrected to restore its true geometric shape; video synthesis, namely, the distortion-corrected panoramic image can be combined with the original video stream to form a complete 360-degree panoramic video; user interface, namely, the synthesized 360-degree panoramic video is presented to the driver through the user interface, allowing the driver to intuitively understand the environment around the vehicle.

[0053] However, during the operation of a multi-carriage train, the connections between cars will experience relative motion due to the train's turning and bending, causing the relative positions of the cameras installed on the cars to constantly change. This poses a challenge to the image fusion of the surround view system. To this end, embodiments of the present disclosure provide a method for determining camera calibration parameters in a surround view system for a multi-carriage articulated train, to overcome the impact of relative motion caused by the train's turning and bending on image fusion.

[0054] Figure 1The application scenario diagram 100 of the method for determining camera calibration parameters in a multi-unit articulated vehicle surround view system according to an embodiment of the present disclosure is schematically shown. It should be noted that, Figure 1 The application scenario diagram 100 shown is only an example to which the embodiments of the present disclosure can be applied, to help those skilled in the art understand the technical content of the present disclosure, but it does not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios.

[0055] like Figure 1 As shown, the application scenario 100 according to this embodiment may include a multi-carriage articulated vehicle surround view system 101 and a server 102. The multi-carriage articulated vehicle surround view system 101 includes a train, which may be a multi-carriage articulated vehicle. The multi-carriage articulated vehicle may include n carriages, and each carriage is equipped with a camera.

[0056] The server 102 can be used to obtain, for any adjacent first target car and second target car in n cars, a first checkerboard image generated by the first camera shooting the checkerboard and a second checkerboard image generated by the second camera shooting the checkerboard at each rotation angle, respectively, at different rotation angles, so as to determine, based on the first checkerboard image and the second checkerboard image corresponding to each rotation angle, a static transformation matrix corresponding to each rotation angle for characterizing the transformation relationship between the first camera and the second camera; and then, using a predetermined equation, based on each rotation angle and the static transformation matrix corresponding to each rotation angle, solve the first transformation matrix corresponding to each rotation angle for characterizing the transformation relationship between the first camera and the car hinge and the second transformation matrix corresponding to the second rotation angle, so as to determine the real-time transformation matrix between the first camera and the second camera based on the first transformation matrix and the second transformation matrix corresponding to each rotation angle and the real-time rotation angle between the first target car and the second target car when the multi-unit articulated vehicle is running.

[0057] Among them, the first camera is installed on the side of the first target carriage, the second camera is installed on the same side of the second target carriage, the checkerboard is placed in a predetermined area, n is an integer greater than or equal to 2, and the rotation angle represents the relative rotation angle between the first target carriage and the second target carriage.

[0058] Among them, the real-time transformation matrix represents the parameters used for camera calibration.

[0059] It should be noted that the method for determining camera calibration parameters in the surround-view system for a multi-unit articulated vehicle provided in the embodiment of the present disclosure can generally be executed by the server 105. Accordingly, the device for determining camera calibration parameters in the surround-view system for a multi-unit articulated vehicle provided in the embodiment of the present disclosure can generally be set in the server 105. The method for determining camera calibration parameters in the surround-view system for a multi-unit articulated vehicle provided in the embodiment of the present disclosure can also be executed by a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Accordingly, the device for determining camera calibration parameters in the surround-view system for a multi-unit articulated vehicle provided in the embodiment of the present disclosure can also be set in a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Alternatively, the method for determining camera calibration parameters in the surround-view system for a multi-unit articulated vehicle provided in the embodiment of the present disclosure may also be executed by the first terminal device 101, the second terminal device 102, or the third terminal device 103, or may also be executed by a terminal device other than the first terminal device 101, the second terminal device 102, or the third terminal device 103. Accordingly, the apparatus for determining camera calibration parameters in the surround-view system for a multi-unit articulated vehicle provided in the embodiment of the present disclosure may also be provided in the first terminal device 101, the second terminal device 102, or the third terminal device 103, or may be provided in a terminal device other than the first terminal device 101, the second terminal device 102, or the third terminal device 103.

[0060] It should be understood that Figure 1 The number of the multi-group articulated vehicle surround view system and the server in the embodiment is only illustrative. According to the implementation requirements, any number of the multi-group articulated vehicle surround view system and the server can be provided.

[0061] The following will be based on Figure 1 The scene described by Figures 2 to 5 A method for determining camera calibration parameters in a surround view system for multi-unit articulated vehicles according to an embodiment of the present disclosure is described in detail.

[0062] Figure 2 The flowchart of the method for determining camera calibration parameters in a surround view system of a multi-unit articulated vehicle according to an embodiment of the present disclosure is schematically shown.

[0063] The multi-carriage articulated train includes n carriages, each of which is equipped with a camera.

[0064] like Figure 2 As shown, the method 200 includes operations S210 to S240.

[0065] In operation S210, for any adjacent first target carriage and second target carriage among the n carriages, a first checkerboard image generated by the first camera photographing the checkerboard and a second checkerboard image generated by the second camera photographing the checkerboard at each rotation angle are respectively obtained at different rotation angles.

[0066] Among them, the first camera is installed on the side of the first target carriage, the second camera is installed on the same side of the second target carriage, the checkerboard is placed in a predetermined area, n is an integer greater than or equal to 2, and the rotation angle represents the relative rotation angle between the first target carriage and the second target carriage.

[0067] According to the embodiments of the present disclosure, the surround view system for multi-carriage articulated trains has a wider field of view than systems for small passenger vehicles. A single carriage can exceed 10 meters in length, and when turning, the carriages of a multi-carriage articulated train are at an angle to each other. Therefore, the surround view system for multi-carriage articulated trains is highly sensitive to camera installation parameters. The system must provide accurate calibration results, and the camera installation parameters must also be accurately compensated. Otherwise, the surround view system's effectiveness will be significantly affected.

[0068] Figure 3 A schematic diagram of a bicycle surround view system according to an embodiment of the present disclosure is schematically shown.

[0069] like Figure 3 As shown, in the bicycle surround view system, cameras 1 to 4 are connected, and their distribution can be as follows: Figure 3 As shown in Figure 2, image stitching is relatively easy for images captured by cameras 1 to 4.

[0070] When the number of carriages in the surround view system of a multi-carriage articulated vehicle is greater than or equal to 2, the images taken by the cameras on the adjacent carriages are difficult to fuse because the adjacent carriages will produce relative movement due to the turning and bending of the carriages.

[0071] Figure 4 A schematic diagram of any adjacent first target car and second target car according to an embodiment of the present disclosure is schematically shown.

[0072] like Figure 4 As shown, there are any two adjacent carriages in the n carriages of the multi-unit articulated vehicle surround view system, namely the first target carriage 410 and the second target carriage 420; the installation position of the camera on the carriage can be Figure 4 shown.

[0073] According to an embodiment of the present disclosure, the first camera may be Figure 4 The camera 5 on the first target carriage 410 is the second camera Figure 4The camera 3 on the second target carriage 420 is positioned at the same side of the first target carriage and the second target carriage.

[0074] like Figure 4 As shown in , an angle sensor is further installed at the hinge of any two adjacent carriages to measure the relative rotation angle between the first target carriage and the second target carriage.

[0075] According to an embodiment of the present disclosure, the predetermined area may be an overlapping area of ​​the field of view of the first camera and the field of view of the second camera, so that both the first camera and the second camera can capture the checkerboard.

[0076] Figure 5 A schematic diagram of a vehicle cabin analysis for determining camera calibration parameters according to an embodiment of the present disclosure is schematically shown.

[0077] like Figure 5 As shown, the field of view of the first camera (camera 5) is as follows Figure 5 As shown in the solid box in , the field of view of the second camera (camera 3) is as follows Figure 5 As shown in the dotted box in FIG, the checkerboard can be placed in the overlapping area of ​​the fields of view of the first camera and the second camera, that is, the predetermined area is the overlapping area of ​​the fields of view.

[0078] According to an embodiment of the present disclosure, a checkerboard is placed in the overlapping area of ​​the fields of view of the first camera and the second camera, so that both the first camera and the second camera can capture the checkerboard, for subsequent calibration of camera parameters based on the captured checkerboard image.

[0079] During the static calibration process, at different rotation angles In this case, the first camera and the second camera may be used to shoot the chessboard to obtain a first chessboard image and a second chessboard image at the same rotation angle.

[0080] In operation S220 , a static transformation matrix corresponding to each rotation angle and used to represent a transformation relationship between the first camera and the second camera is determined based on the first checkerboard image and the second checkerboard image corresponding to each rotation angle.

[0081] According to an embodiment of the present disclosure, since the first checkerboard image and the second checkerboard image are both images obtained by photographing the checkerboard, a static transformation matrix for representing the transformation relationship between the first camera and the second camera can be determined based on the first checkerboard image and the second checkerboard image at the same rotation angle.

[0082] For the same rotation angle, the coordinates of the midpoints in the first checkerboard image can be transformed into the coordinates in the second checkerboard image using a static transformation matrix. Similarly, the coordinates of the midpoints in the second checkerboard image can be transformed into the coordinates in the first checkerboard image using a static transformation matrix.

[0083] In one embodiment, at the same rotation angle, a binocular camera calibration method may be used to determine the static transformation relationship between the first camera and the second camera.

[0084] In operation S230, using predetermined equations, according to each rotation angle and the static transformation matrix corresponding to each rotation angle, the first transformation matrix corresponding to each rotation angle for characterizing the transformation relationship between the first camera and the vehicle body hinge and the second transformation matrix corresponding to each rotation angle are solved.

[0085] Among them, the car hinge is the hinge between the first target car and the second target car; the first coordinate system is established with the optical center of the first camera as the origin, the second coordinate system is established with the hinge between the first target car and the second target car as the origin, and the third coordinate system is established with the optical center of the second camera as the origin. The first transformation matrix represents the transformation relationship from the first coordinate system to the second coordinate system, and the second transformation matrix represents the transformation relationship from the second coordinate system to the third coordinate system.

[0086] The coordinate system can be defined as Figure 5 As shown, the right-hand standard is used.

[0087] Figure 5 In the O1 and O3 coordinate systems, the optical centers of cameras 5 and 3 are taken as their origins respectively. The directions of the X and Z axes in the O1 and O3 coordinate systems are as follows: Figure 5 As shown, the Y axis is perpendicular to the paper surface, that is, the O1 coordinate system can be the first coordinate system, and the O3 coordinate system can be the third coordinate system.

[0088] Since the rotation between the carriages is around the hinge of the carriages, the O2 coordinate system is defined at the hinge and is fixed to the front car. The directions of the X and Y axes in the O2 coordinate system are as follows: Figure 5 As shown, the Z axis is perpendicular to the paper surface, that is, the O2 coordinate system can be the second coordinate system.

[0089] Among them, the relative rotation angle between the first target carriage and the second target carriage is ; A chessboard attached to the ground is used as the calibration target surface. The chessboard is placed in a preset area to ensure that both camera 3 and camera 5 can capture the chessboard.

[0090] Assume that there is a transformation M between the O1 coordinate system and the O2 coordinate system 12 , that is, the first transformation matrix can be M 12; There is a transformation M between the O2 coordinate system and the O3 coordinate system 23 , that is, the second transformation matrix can be M 23 .

[0091] At the same rotation angle, a predetermined equation may be established based on a static transformation matrix between the first camera and the second camera, the rotation angle, and an association relationship between the first transformation matrix and the second transformation matrix.

[0092] In one embodiment, at the same rotation angle, a static transformation matrix between the first camera and the second camera determined based on the first checkerboard image and the second checkerboard image is equal to the product of the first transformation matrix and the second transformation matrix, wherein the rotation angle affects the first transformation matrix and the second transformation matrix.

[0093] For example, Figure 5 As shown in In the case of , the static transformation matrix between the first camera and the second camera can be M, and the first transformation matrix can be M 12 , the second transformation matrix can be M 23 .

[0094] According to the knowledge of camera extrinsic parameter calibration, the three-dimensional coordinates of each point on the chessboard in the camera coordinate system can be obtained. Among them, the transformation matrix from the O1 coordinate system to the O2 coordinate system is It can be expressed as the following formula (1).

[0095] (1)

[0096] in, It can represent the rotation transformation between the O1 coordinate system and the O2 coordinate system. It can represent the translation transformation between the O1 coordinate system and the O2 coordinate system.

[0097] Transformation matrix from O2 coordinate system to O3 coordinate system It can be expressed as the following formula (2).

[0098] (2)

[0099] in, It can represent the rotation transformation between the O2 coordinate system and the O3 coordinate system. It can represent the translation transformation between the O2 coordinate system and the O3 coordinate system.

[0100] According to an embodiment of the present disclosure, for a point on the chessboard, the first coordinate of the point in the first chessboard image can be , that is, the coordinates of the point in the first coordinate system O1 are , using the first transformation matrix , transform the coordinates of the point in the first coordinate system O1 to the second coordinate system O2, then the coordinates of the point in the second coordinate system O2 can be shown as the following formula (3).

[0101] (3)

[0102] According to an embodiment of the present disclosure, the second transformation matrix is ​​used , transform the coordinates of the point in the second coordinate system O2 to the third coordinate system O3, then the coordinates of the point in the third coordinate system O3 can be shown as the following formula (4).

[0103] (4)

[0104] Thus, the static transformation matrix M can be used to transform the coordinates of the points on the first checkerboard image to the third coordinate system. The first transformation matrix M 12 and the second transformation matrix M 23 It can also be used to transform the coordinates under the first coordinate system to the third coordinate system, and the established predetermined equation can be , where M 12 and M 23 and rotation angle related, and The rotation around the Z axis is what the angle sensor can provide.

[0105] Based on the above content, the rotation angle and the static transformation matrix in the predetermined equation are known, and thus the first transformation matrix and the second transformation matrix in the predetermined equation can be solved.

[0106] In operation S240, a real-time transformation matrix between the first camera and the second camera is determined based on the first transformation matrix and the second transformation matrix corresponding to each rotation angle and the real-time rotation angle between the first target car and the second target car when the multi-unit articulated vehicle is running.

[0107] Among them, the real-time transformation matrix represents the parameters used for camera calibration.

[0108] Based on the above operation S230, the first transformation matrix and the second transformation matrix corresponding to each rotation angle can be obtained. On this basis, the rotation angle, the static transformation matrix, the first transformation matrix and the second transformation matrix are corresponding based on the same rotation angle.

[0109] In one embodiment, based on the rotation angle, a static transformation matrix, a first transformation matrix, and a second transformation matrix at the same rotation angle may be determined.

[0110] During the operation of a multi-carriage articulated train, the rotation angle between the first and second target cars changes in real time. Therefore, the transformation relationship between the first and second cameras also changes in real time. An angle sensor at the joint between the first and second target cars can measure the real-time rotation angle between the first and second target cars in real time. Based on this real-time rotation angle, the real-time transformation matrix between the first and second cameras can be determined.

[0111] According to an embodiment of the present disclosure, a first checkerboard image generated by photographing the checkerboard with a first camera and a second checkerboard image generated by photographing the checkerboard with a second camera at different rotation angles are obtained. Based on the first and second checkerboard images, a static transformation matrix between the first and second cameras at each rotation angle is determined. Then, based on the relationship between the static transformation matrix, the first transformation matrix, and the second transformation matrix at the same rotation angle, the first transformation matrix and the second transformation matrix can be solved to obtain the first and second transformation matrices. During multi-carriage train operation, changes in the articulation angles at adjacent car joints are the only factor that changes the transformation relationship between the first and second cameras. Therefore, based on the static transformation matrix, the first transformation matrix, and the second transformation matrix corresponding to each rotation angle, the real-time transformation matrix between the first and second cameras can be obtained when the real-time articulation angle is obtained. This plays a key role in accurately fusion images in the subsequent surround view system, ensuring the continuity and accuracy of the fused surround view images.

[0112] According to an embodiment of the present disclosure, a static transformation matrix corresponding to each rotation angle and used to characterize the transformation relationship between the first camera and the second camera is determined based on the first checkerboard image and the second checkerboard image corresponding to each rotation angle. The method includes: for each rotation angle, based on a preset number of target points in the checkerboard, determining the static transformation matrix between the first camera and the second camera according to first coordinates of the target points in the first checkerboard image and second coordinates of the target points in the second checkerboard image.

[0113] According to an embodiment of the present disclosure, a preset number of target points can be selected in the checkerboard for each rotation angle. The first checkerboard image is obtained by photographing the checkerboard with a first camera, and the second checkerboard image is obtained by photographing the checkerboard with a second camera. For each target point in the checkerboard, points corresponding to the target point can be determined in both the first and second checkerboard images.

[0114] Since the static transformation matrix represents the transformation relationship from the first coordinate system to the third coordinate system, the first coordinate of the target point in the first checkerboard image is in the first coordinate system, and the second coordinate of the target point in the second checkerboard image is in the third coordinate system. Based on the first coordinates and second coordinates corresponding to a preset number of target points, the static transformation matrix between the first camera and the second camera can be determined.

[0115] The preset number of target points in the selected checkerboard grid are not collinear in the checkerboard grid.

[0116] According to an embodiment of the present disclosure, in the process of determining a static transformation matrix, a preset number of target points are selected to reduce the deviation of a single target point, so that the determined static transformation matrix is ​​more accurate.

[0117] According to an embodiment of the present disclosure, determining a static transformation matrix between a first camera and a second camera includes: transforming a first coordinate for a target point in a first checkerboard image into a third coordinate system according to the transformation matrix to obtain a third coordinate for the first coordinate in the third coordinate system, wherein the transformation matrix represents an unknown static transformation matrix; constructing an error vector for the target point according to the third coordinate and the second coordinate for the target point in the second checkerboard image; and determining the static transformation matrix based on the error vectors of each of a preset number of target points.

[0118] According to an embodiment of the present disclosure, for a target point, a first checkerboard image is generated based on a first camera capturing the checkerboard, and the coordinates of the target point in the first checkerboard image are first coordinates. Based on a transformation matrix, the first coordinates of the target point in the first checkerboard image can be transformed to obtain third coordinates of the first coordinates in a third coordinate system. That is, based on the transformation matrix, the first coordinates of the target point in the first checkerboard image are transformed to obtain the coordinates of the target point in the second checkerboard image.

[0119] Theoretically, the third coordinate obtained based on the transformation matrix should be the same as the second coordinate of the target point in the second checkerboard image, but due to the relative motion of the first target carriage and the second target carriage, there will be a certain error between the second coordinate and the third coordinate.

[0120] Therefore, for each target point, an error vector for the target point can be constructed according to the third coordinate corresponding to the target point and the second coordinate for the target point in the second checkerboard image.

[0121] According to an embodiment of the present disclosure, based on the error vectors of the respective preset number of target points, a static transformation matrix between the first camera and the second camera may be determined when the error is minimized.

[0122] According to an embodiment of the present disclosure, for a preset number of target points on a checkerboard, the first coordinates of the target points in the first checkerboard image are transformed based on the transformation matrix to obtain third coordinates of the first coordinates in the second checkerboard image. An error vector is then constructed based on the third coordinates obtained via the transformation matrix and the second coordinates of the target points in the second checkerboard image. Thus, based on the error vectors for the multiple target points, a static transformation matrix between the first and second cameras is determined. The use of the error vectors improves the accuracy of the determined static transformation matrix.

[0123] According to an embodiment of the present disclosure, an error vector for the target point is constructed based on the third coordinate and the second coordinate for the target point in the second checkerboard image, including: subtracting the second coordinate from the third coordinate to obtain the error vector for the target point, wherein the third coordinate is obtained by multiplying the transfer matrix by the first coordinate.

[0124] According to an embodiment of the present disclosure, based on the conversion matrix The third coordinate of the target point obtained in the third coordinate system O3 It can be expressed as the following formula (5).

[0125] (5)

[0126] in, The first coordinate of the target point in the first checkerboard image may be represented.

[0127] Theoretically, the third coordinate of the target point determined by the first transformation matrix and the second transformation matrix in the third coordinate system O3 is The second coordinate of the target point in the second checkerboard image are the same, i.e. .

[0128] Thus, the following formula (6) can be obtained.

[0129] (6)

[0130] According to an embodiment of the present disclosure, each point on the chessboard theoretically satisfies the relationship of the above formula (6).

[0131] In the case of static calibration of the surround view system of multi-unit articulated vehicles, the static transformation matrix is ​​unknown. The static transformation matrix has 6 degrees of freedom, including 3 degrees of freedom each of rotation and translation, that is, the transformation matrix There are 12 unknowns in .

[0132] Therefore, the static transformation matrix can be solved by equations established by more than 12 non-collinear point pairs on the chessboard, that is, the preset number is more than 12.

[0133] For each target point in the chessboard, due to the relative motion between the first target carriage and the second target carriage, there is an error between the second coordinate determined based on the transformation matrix and the third coordinate of the target point in the second chessboard image. Therefore, the error vector for the target point can be expressed as follows (7).

[0134] (7)

[0135] in, is an integer greater than or equal to 1, used to represent the first target points; It can be expressed as the The error vector of the target point; It can be expressed as the second chessboard image for the The second coordinate of the target point; It can be expressed as the first chessboard image for the The first coordinate of the target point.

[0136] According to an embodiment of the present disclosure, based on the error between the third coordinate and the second coordinate of the target point in the second checkerboard image, an error vector for the target point can be obtained, so that the static transformation matrix between the first camera and the second camera can be optimized based on the error vectors of each of a preset number of target points to improve the accuracy of the determined static transformation matrix.

[0137] According to an embodiment of the present disclosure, a static transformation matrix is ​​determined based on the error vectors of each of a preset number of target points, including: calculating the square of the norm of the error vector of each of the preset number of target points; summing the square of the norm of the error vector of each of the preset number of target points to obtain an error function; and when the error function takes a minimum value, determining the value of the conversion matrix in the error function as the static transformation matrix.

[0138] According to an embodiment of the present disclosure, the error function E( ) can be expressed as the following formula (8).

[0139] (8)

[0140] in, It can be expressed as The square of the norm of the error vector of the target point; N is a preset number.

[0141] According to an embodiment of the present disclosure, by adjusting the conversion matrix The value of the error function also changes accordingly, so the value of the transformation matrix corresponding to the minimum value of the error function is determined as the static transformation matrix.

[0142] According to an embodiment of the present disclosure, the static transformation matrix can be solved by the following formula (9): .

[0143] (9)

[0144] According to an embodiment of the present disclosure, an error function is formed by the sum of the squares of the norms of the error vectors of a preset number of target points, and based on the error function, the value of the transformation matrix when the error function is minimized is solved. The value of the transformation matrix at this time can be determined as a static transformation matrix, which improves the accuracy of the static transformation matrix and is also beneficial to improving the accuracy of subsequent image fusion.

[0145] According to an embodiment of the present disclosure, the method for determining camera calibration parameters in the above-mentioned multi-unit articulated vehicle surround view system also includes: calculating, based on the coordinates, first coordinates, and second coordinates of the target point in the chessboard in world coordinates, a first calibration parameter for characterizing the transformation relationship between the first camera and the chessboard and a second calibration parameter for characterizing the transformation relationship between the second camera and the chessboard.

[0146] According to the embodiments of the present disclosure, in actual applications, although the installation of the camera is guaranteed by the tooling, there are still deviations after installation. In particular, when the outer surface of the front rear compartment is a curved surface, the installation height of the camera will cause the orientation to change.

[0147] Assume that the distortion parameters of the image taken by the camera are known and have been distortion corrected.

[0148] For a point on the chessboard, the imaging process in the camera can be expressed as the following formula (10), where the point on the chessboard can be a target point or any other arbitrary point.

[0149] (10)

[0150] Where u and v are the image coordinates of the midpoint of the checkerboard in the image captured by the camera, K is the intrinsic parameter matrix of the camera, R and T are the rotation and translation relationships between the camera and the checkerboard in the world coordinate system, [xyz 1] T is the point in the world coordinate system, that is, the coordinate of the midpoint of the chessboard in the world coordinate system; It can represent the scaling between the image coordinates of the midpoint in the image captured by the camera and the coordinates of the midpoint on the chessboard in the world coordinate system.

[0151] The image coordinates of the points on the image are extrapolated to the X and Y on the chessboard, as shown in the following formula (11).

[0152] (11)

[0153] According to an embodiment of the present disclosure, based on the above formula (10) and formula (11), the monocular camera is calibrated offline. For example, by using the Zhang Zhengyou calibration method, the intrinsic parameter matrix K of each camera and the R and T of the camera relative to the ground checkerboard can be obtained.

[0154] The first calibration parameter may represent the R and T of the first camera relative to the checkerboard, and the second calibration parameter may represent the R and T of the second camera relative to the checkerboard.

[0155] According to an embodiment of the present disclosure, based on the coordinates, first coordinates, and second coordinates of the target point in the checkerboard in the world coordinate system, a first calibration parameter between the checkerboard and the first camera and a second calibration parameter between the checkerboard and the second camera can be calculated, respectively, for use in correcting the image during the image fusion process, which is beneficial to improving the accuracy of image fusion.

[0156] According to an embodiment of the present disclosure, the method for determining camera calibration parameters in the above-mentioned multi-unit articulated vehicle surround view system also includes: correcting the image taken by the first camera according to the real-time transformation matrix, the first calibration parameter and the second calibration parameter to obtain a corrected image, so as to fuse the corrected image with the image taken by the second camera.

[0157] According to the embodiments of the present disclosure, in a surround view system for multi-carriage articulated trains, if the position of cameras on the same side of adjacent cars is more than just translation, their installation angle and height affect the camera's capture range and spatial sampling resolution. Without correction, forcibly fusing the two camera images will inevitably affect the final effect.

[0158] Fix the O2 coordinate system and use and , and They respectively represent the transformation relationship of the two cameras relative to O2. Obviously, and The difference in the Y and Z axis directions is the installation error. Among them, the difference in the X axis direction is caused by the installation distance between the front and rear carriages, which is not an error; and The angle difference of each axis is the error caused by inconsistent installation orientation.

[0159] by Figure 5 Taking the first camera (camera 5) and the second camera (camera 3) in the figure as an example, the parameters of the first camera can be corrected from the angle and position errors based on the real-time transformation matrix, the first calibration parameters, and the second calibration parameters. Based on the corrected parameters, the image taken by the first camera is corrected, so that the corrected image can be fused with the image taken by the second camera.

[0160] The image captured by the first camera can be corrected as shown in the following formula (12).

[0161] (12)

[0162] Among them, R' is the rotation matrix after the first camera correction, and T' is the translation matrix after the first camera correction; and is the coordinate of the midpoint of the corrected image.

[0163] According to the embodiments of the present disclosure, the image captured by the first camera can be corrected based on the real-time transformation matrix, the first calibration parameters, and the second calibration parameters, thereby eliminating image inconsistency problems caused by errors such as installation errors. The corrected image can be fused with the image captured by the second camera, thereby improving the accuracy of image fusion and the continuity and accuracy of the fused surround view image.

[0164] Based on the above content, it can be seen that the method for determining the camera calibration parameters in the surround view system of a multi-unit articulated vehicle disclosed in the present invention can dynamically calculate the image fusion area based on the calibration results of the camera parameters and the real-time angle information of the angle sensor, so that it can respond to the relative movement between the carriages in real time, ensure the continuity and accuracy of the surround view image, thereby providing the driver with more comprehensive and real-time surrounding environment information, and reducing manual participation, improving the level of automation, and the calibration results can improve the quality of the synthesized image.

[0165] Therefore, the method disclosed in the present invention not only improves the performance of the surround view system of multi-unit articulated vehicles, but also reduces the complexity and cost of system maintenance. It also helps to improve the safety of train operation and reduce potential risks caused by blind spots. It is of great significance to improving the overall safety level of rail transit.

[0166] Based on the above-mentioned method for determining camera calibration parameters in a surround view system for a multi-unit articulated vehicle, the present disclosure further provides a device for determining camera calibration parameters in a surround view system for a multi-unit articulated vehicle. Figure 6 The device is described in detail.

[0167] Figure 6 The figure schematically shows a block diagram of a device for determining camera calibration parameters in a surround view system for multiple articulated vehicles according to an embodiment of the present disclosure.

[0168] like Figure 6 As shown, the device 600 for determining camera calibration parameters in a surround view system of a multi-unit articulated vehicle of this embodiment includes an acquisition module 610 , a first determination module 620 , a solution module 630 and a second determination module 640 .

[0169] Acquisition module 610 is configured to acquire, for any adjacent first and second target carriages among n carriages, a first checkerboard image generated by the first camera photographing the checkerboard grid and a second checkerboard image generated by the second camera photographing the checkerboard grid at each rotation angle, respectively. The first camera is mounted on a side of the first target carriage, the second camera is mounted on the same side of the second target carriage, the checkerboard grid is positioned within a predetermined area, n is an integer greater than or equal to 2, and the rotation angle represents the relative rotation angle between the first and second target carriages. In one embodiment, acquisition module 610 may be configured to execute operation S210 described above and will not be further described herein.

[0170] The first determination module 620 is configured to determine, based on the first checkerboard image and the second checkerboard image corresponding to each rotation angle, a static transformation matrix representing the transformation relationship between the first camera and the second camera. In one embodiment, the first determination module 620 may be configured to perform operation S220 described above, and will not be further described herein.

[0171] Solving module 630 is configured to solve the first transformation matrix and the second transformation matrix corresponding to each rotation angle using a predetermined equation based on each rotation angle and the static transformation matrix corresponding to each rotation angle. In one embodiment, solving module 630 may be configured to perform operation S230 described above, which will not be further described herein.

[0172] Among them, a first coordinate system is established with the optical center of the first camera as the origin, a second coordinate system is established with the hinge between the first target carriage and the second target carriage as the origin, and a third coordinate system is established with the optical center of the second camera as the origin. The first transformation matrix represents the transformation relationship from the first coordinate system to the second coordinate system, and the second transformation matrix represents the transformation relationship from the second coordinate system to the third coordinate system.

[0173] A second determination module 640 is configured to determine a real-time transformation matrix between the first camera and the second camera based on the first and second transformation matrices corresponding to the respective rotation angles and the real-time rotation angle between the first and second target cars during operation of the multi-carriage articulated train. The real-time transformation matrix represents parameters used for camera calibration. In one embodiment, the second determination module 640 can be configured to perform operation S240 described above and will not be further described here.

[0174] According to an embodiment of the present disclosure, the first determining module 620 includes a determining submodule.

[0175] A determination submodule is configured to determine, for each rotation angle, a static transformation matrix between the first camera and the second camera based on a preset number of target points in the checkerboard, according to first coordinates of the target points in the first checkerboard image and second coordinates of the target points in the second checkerboard image.

[0176] According to an embodiment of the present disclosure, the determination submodule includes a transforming unit, a constructing unit, and a determining unit.

[0177] The transformation unit is used to transform the first coordinate of the target point in the first checkerboard image into a third coordinate system according to a transformation matrix to obtain a third coordinate of the first coordinate in the third coordinate system, wherein the transformation matrix represents an unknown static transformation matrix.

[0178] The construction unit is used to construct an error vector for the target point according to the third coordinate and the second coordinate for the target point in the second checkerboard image.

[0179] The determining unit is configured to determine a static transformation matrix based on the error vectors of respective preset number of target points.

[0180] According to an embodiment of the present disclosure, the construction unit includes a first obtaining subunit.

[0181] The first obtaining subunit is used to obtain an error vector for the target point by subtracting the second coordinate from the third coordinate, wherein the third coordinate is obtained by multiplying the transfer matrix by the first coordinate.

[0182] According to an embodiment of the present disclosure, the determining unit includes a calculating subunit, a second obtaining subunit, and a determining subunit.

[0183] The calculation subunit is used to calculate the square of the norm of the error vector of a set number of target points.

[0184] The second obtaining subunit is configured to sum the squares of the norms of the error vectors of a preset number of target points to obtain an error function.

[0185] The determination subunit is used to determine the value of the conversion matrix in the error function as the static transformation matrix when the error function takes the minimum value.

[0186] According to an embodiment of the present disclosure, the device 600 for determining camera calibration parameters in a surround view system for multi-unit articulated vehicles further includes a calculation module.

[0187] The calculation module is used to calculate a first calibration parameter for characterizing the transformation relationship between the first camera and the checkerboard and a second calibration parameter for characterizing the transformation relationship between the second camera and the checkerboard based on the coordinates, the first coordinates, and the second coordinates of the target point in the checkerboard in world coordinates.

[0188] According to an embodiment of the present disclosure, the device 600 for determining camera calibration parameters in a surround view system for multi-unit articulated vehicles further includes a correction module.

[0189] The correction module is used to correct the image taken by the first camera according to the real-time transformation matrix, the first calibration parameter and the second calibration parameter to obtain a corrected image, so as to fuse the corrected image with the image taken by the second camera.

[0190] According to the embodiments of the present invention, any number of modules, sub-modules, units, and sub-units, or at least part of the functions of any number of them, can be implemented in one module. According to the embodiments of the present invention, any one or more of the modules, sub-modules, units, and sub-units can be split into multiple modules for implementation. According to the embodiments of the present invention, any one or more of the modules, sub-modules, units, and sub-units can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or can be implemented by hardware or firmware in any other reasonable way of integrating or packaging the circuit, or can be implemented in any one of the three implementation methods of software, hardware, and firmware, or in any appropriate combination of any of them. Alternatively, according to the embodiments of the present invention, one or more of the modules, sub-modules, units, and sub-units can be at least partially implemented as a computer program module, which can perform the corresponding functions when the computer program module is executed.

[0191] For example, any multiple of the acquisition module 610, the first determination module 620, the solution module 630, and the second determination module 640 can be combined into a single module / unit / sub-unit, or any one of these modules / units / sub-units can be split into multiple modules / units / sub-units. Alternatively, at least part of the functionality of one or more of these modules / units / sub-units can be combined with at least part of the functionality of other modules / units / sub-units and implemented in a single module / unit / sub-unit. According to an embodiment of the present disclosure, at least one of the acquisition module 610, the first determination module 620, the solution module 630, and the second determination module 640 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or can be implemented in hardware or firmware by any other reasonable means of circuit integration or packaging, or can be implemented in any one of the three implementation methods of software, hardware, and firmware, or any appropriate combination of any of these. Alternatively, at least one of the acquisition module 610 , the first determination module 620 , the solution module 630 , and the second determination module 640 may be at least partially implemented as a computer program module, which may perform corresponding functions when executed.

[0192] It should be noted that the device for determining camera calibration parameters in the surround-view system of a multi-unit articulated vehicle in the embodiment of the present disclosure corresponds to the method for determining camera calibration parameters in the surround-view system of a multi-unit articulated vehicle in the embodiment of the present disclosure. The description of the device for determining camera calibration parameters in the surround-view system of a multi-unit articulated vehicle refers to the method for determining camera calibration parameters in the surround-view system of a multi-unit articulated vehicle, which will not be repeated here.

[0193] Figure 7 The block diagram of an electronic device suitable for implementing a method for determining camera calibration parameters in a surround view system for multi-unit articulated vehicles according to an embodiment of the present disclosure is schematically shown. Figure 7 The electronic device shown is only an example and should not limit the functions and scope of use of the embodiments of the present disclosure.

[0194] like Figure 7As shown, the electronic device 700 according to an embodiment of the present disclosure includes a processor 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage portion 708 into a random access memory (RAM) 703. The processor 701 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 701 may also include onboard memory for caching purposes. The processor 701 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0195] Various programs and data required for the operation of the electronic device 700 are stored in the RAM 703. The processor 701, ROM 702, and RAM 703 are connected to each other via a bus 704. The processor 701 executes the various operations of the method flow according to the embodiment of the present disclosure by executing the programs in the ROM 702 and / or RAM 703. It should be noted that the programs may also be stored in one or more memories other than the ROM 702 and RAM 703. The processor 701 may also execute the various operations of the method flow according to the embodiment of the present disclosure by executing the programs stored in the one or more memories.

[0196] According to an embodiment of the present disclosure, electronic device 700 may further include an input / output (I / O) interface 705, which is also connected to bus 704. Electronic device 700 may also include one or more of the following components connected to I / O interface 705: an input section 706 including a keyboard, mouse, etc.; an output section 707 including devices such as a cathode ray tube (CRT), liquid crystal display (LCD), and speakers; a storage section 708 including a hard disk; and a communication section 709 including a network interface card such as a LAN card or modem. Communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to I / O interface 705 as needed. Removable media 711, such as a magnetic disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed in drive 710 as needed, so that computer programs read from the removable media can be installed into storage section 708 as needed.

[0197] According to an embodiment of the present disclosure, the method flow according to an embodiment of the present disclosure can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 709, and / or installed from the removable medium 711. When the computer program is executed by the processor 701, the above-mentioned functions defined in the system of the embodiment of the present disclosure are executed. According to an embodiment of the present disclosure, the system, equipment, device, module, unit, etc. described above can be implemented by a computer program module.

[0198] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments, or may exist independently and not be incorporated into the device / apparatus / system. The computer-readable storage medium carries one or more programs, and when executed, implements the method according to the embodiments of the present disclosure.

[0199] According to embodiments of the present disclosure, a computer-readable storage medium may be a non-volatile computer-readable storage medium. Examples include, but are not limited to, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0200] For example, according to an embodiment of the present disclosure, the computer-readable storage medium may include the ROM 702 and / or the RAM 703 described above and / or one or more memories other than the ROM 702 and the RAM 703 .

[0201] An embodiment of the present disclosure also includes a computer program product, which includes a computer program containing program code for executing the method provided by the embodiment of the present disclosure. When the computer program product is run on an electronic device, the program code is used to enable the electronic device to implement the method for determining camera calibration parameters in a multi-group articulated vehicle surround view system provided by the embodiment of the present disclosure.

[0202] When the computer program is executed by the processor 701, the above functions defined in the system / device of the embodiment of the present disclosure are performed. According to the embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0203] In one embodiment, the computer program may be stored on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may be transmitted and distributed in the form of a signal on a network medium, downloaded and installed via the communication portion 709, and / or installed from a removable medium 711. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to wireless, wired, or any suitable combination thereof.

[0204] According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computer programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C" or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect via the Internet).

[0205] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of the systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or part of the code, which contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the boxes can also occur in an order different from that marked in the accompanying drawings. For example, two boxes shown in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, as well as the combination of boxes in the block diagram or flowchart, can be implemented using a dedicated hardware-based system that performs the specified functions or operations, or can be implemented using a combination of dedicated hardware and computer instructions. It will be understood by those skilled in the art that the features described in the various embodiments of the present disclosure can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in the present disclosure. In particular, without departing from the spirit and teachings of the present disclosure, the features described in the various embodiments of the present disclosure may be combined and / or coupled in various ways, and all of these combinations and / or couplings fall within the scope of the present disclosure.

[0206] The above describes the embodiments of the present disclosure. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of the present disclosure. Although each embodiment has been described separately above, this does not mean that the measures in each embodiment cannot be advantageously used in combination. Without departing from the scope of the present disclosure, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present disclosure.

Claims

1. A method for determining camera calibration parameters in a surround view system for multi-unit articulated vehicles, wherein: The multi-unit articulated train includes n carriages, each carriage is equipped with a camera, and the method includes: For any adjacent first target carriage and second target carriage among the n carriages, a first checkerboard image generated by photographing the checkerboard with the first camera and a second checkerboard image generated by photographing the checkerboard with the second camera at each rotation angle are respectively obtained. wherein the first camera is mounted on a side of the first target carriage, the second camera is mounted on the same side of the second target carriage, the checkerboard is placed within a predetermined area, n is an integer greater than or equal to 2, and the rotation angle represents the relative rotation angle between the first target carriage and the second target carriage; Determining, based on the first checkerboard image and the second checkerboard image corresponding to each rotation angle, a static transformation matrix corresponding to each rotation angle and used to represent a transformation relationship between the first camera and the second camera; Using a predetermined equation, based on the respective rotation angles and the static transformation matrices corresponding to the respective rotation angles, a first transformation matrix corresponding to the respective rotation angles for representing the transformation relationship between the first camera and the hinge of the vehicle body and a second transformation matrix corresponding to the respective rotation angles are solved; Based on the first transformation matrix and the second transformation matrix corresponding to the respective rotation angles and the real-time rotation angle between the first target car and the second target car when the multi-unit articulated vehicle is running, the real-time transformation matrix between the first camera and the second camera is determined, wherein the real-time transformation matrix represents parameters used for camera calibration.

2. The method according to claim 1, wherein The determining, based on the first checkerboard image and the second checkerboard image corresponding to each rotation angle, a static transformation matrix corresponding to each rotation angle and used to characterize a transformation relationship between the first camera and the second camera, includes: For each rotation angle, based on a preset number of target points in the checkerboard, a static transformation matrix between the first camera and the second camera is determined according to first coordinates of the target points in the first checkerboard image and second coordinates of the target points in the second checkerboard image.

3. The method according to claim 2, wherein: A first coordinate system is established with the optical center of the first camera as the origin, a second coordinate system is established with the hinge between the first target carriage and the second target carriage as the origin, and a third coordinate system is established with the optical center of the second camera as the origin, the first transformation matrix represents the transformation relationship from the first coordinate system to the second coordinate system, and the second transformation matrix represents the transformation relationship from the second coordinate system to the third coordinate system; The determining of the static transformation matrix between the first camera and the second camera includes: transforming a first coordinate of the target point in the first checkerboard image into the third coordinate system according to a transformation matrix to obtain a third coordinate of the first coordinate in the third coordinate system, wherein the transformation matrix represents an unknown static transformation matrix; constructing an error vector for the target point according to the third coordinate and the second coordinate for the target point in the second checkerboard image; The static transformation matrix is ​​determined based on the error vectors of the preset number of target points.

4. The method according to claim 3, wherein: The step of constructing an error vector for the target point based on the third coordinate and the second coordinate for the target point in the second checkerboard image includes: Subtracting the second coordinate from the third coordinate to obtain an error vector for the target point, wherein the third coordinate is obtained by multiplying the transformation matrix by the first coordinate.

5. The method according to claim 4, wherein The determining the static transformation matrix based on the error vectors of the preset number of target points includes: Calculating the square of the norm of the error vector of each of the preset number of target points; Summing the squares of the norms of the error vectors of the preset number of target points to obtain an error function; When the error function takes a minimum value, the value of the conversion matrix in the error function is determined as the static transformation matrix.

6. The method according to claim 2, further comprising: A first calibration parameter for characterizing a transformation relationship between the first camera and the checkerboard and a second calibration parameter for characterizing a transformation relationship between the second camera and the checkerboard are calculated based on the coordinates of the target point in the checkerboard in world coordinates, the first coordinates, and the second coordinates.

7. The method according to claim 6, further comprising: The image captured by the first camera is corrected according to the real-time transformation matrix, the first calibration parameter, and the second calibration parameter to obtain a corrected image, so as to fuse the corrected image with the image captured by the second camera.

8. A device for determining camera calibration parameters in a surround view system for a multi-unit articulated vehicle, wherein: The multi-unit articulated train includes n carriages, each of which is equipped with a camera, and the device includes: an acquisition module for acquiring, for any adjacent first target carriage and second target carriage among the n carriages, a first checkerboard image generated by photographing the checkerboard with the first camera and a second checkerboard image generated by photographing the checkerboard with the second camera at each rotation angle, respectively. wherein the first camera is mounted on a side of the first target carriage, the second camera is mounted on the same side of the second target carriage, the checkerboard is placed within a predetermined area, n is an integer greater than or equal to 2, and the rotation angle represents the relative rotation angle between the first target carriage and the second target carriage; a first determining module, configured to determine, based on the first checkerboard image and the second checkerboard image corresponding to each rotation angle, a static transformation matrix corresponding to each rotation angle and used to represent a transformation relationship between the first camera and the second camera; a solving module, configured to solve, using a predetermined equation, a first transformation matrix corresponding to each rotation angle and used to represent a transformation relationship between the first camera and the hinge of the vehicle body, and a second transformation matrix corresponding to each rotation angle and used to represent a transformation relationship between the second camera and the hinge of the vehicle body; The second determination module is used to determine the real-time transformation matrix between the first camera and the second camera based on the first transformation matrix and the second transformation matrix corresponding to the respective rotation angles and the real-time rotation angle between the first target car and the second target car when the multi-unit articulated vehicle is running, wherein the real-time transformation matrix represents the parameters used for camera calibration.

9. A train comprising: A multi-unit articulated vehicle is provided with the device for determining camera calibration parameters in the surround view system of a multi-unit articulated vehicle as claimed in claim 8.

10. An electronic device comprising: one or more processors; a memory for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors are enabled to implement the method according to any one of claims 1 to 7.

11. A computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, causes the processor to implement the method according to any one of claims 1 to 7.

Images