Camera equipment calibration method, training method, image processing method and device

By calculating the intersection position and focal length range of reference lines in the image to be processed, the pitch angle and field of view of the camera are determined, which solves the problems of high cost and low efficiency in traffic scenes by traditional manual calibration methods, and realizes efficient automatic calibration of the camera .

CN114937088BActive Publication Date: 2024-11-08ZHEJIANG DAHUA TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210384121.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-04-12
Publication Date
2024-11-08
Estimated Expiration
2042-04-12

AI Technical Summary

Technical Problem

In traffic scenarios, the traditional manual camera calibration method is costly and inefficient, making it difficult to effectively obtain camera parameters for images captured by a monocular camera, especially when the camera installation angles vary.

Method used

By obtaining the pending image containing the reference lines, evacuation point position information is calculated, and shooting parameters, including pitch angle and field of view, are determined based on the focal length range and evacuation point information, to realize automatic calibration of the camera.

Benefits of technology

It realizes efficient calibration of the camera, reduces costs, and is simpler and more efficient than manual calibration, and can automatically obtain the internal and external parameters of the camera.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114937088B_ABST
    Figure CN114937088B_ABST
Patent Text Reader

Abstract

The present application discloses a camera calibration method, a training method, an image processing method and a device, the method comprising: obtaining an image to be processed taken by a camera, the image to be processed contains at least two reference lines, and the straight lines where the at least two reference lines are located have an intersection; based on the intersection, obtaining vanishing point position information; based on the current focal length range and the vanishing point position information, calculating the shooting parameters of the camera taking the image to be processed, the shooting parameters are used to characterize the angle information of the shooting device. Through the above methods, the present application can realize the calibration of the camera with high efficiency.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a camera equipment calibration method, a training method, an image processing method and a device. Background Art

[0002] When realizing target detection or recognition, especially signs and markings on the ground, the distant targets may be too small or unclear due to the camera's viewing angle. The targets such as signs and markings that were originally parallel in the real world no longer have the characteristics of parallelism, while the bird's-eye view of the image can reproduce the parallelism and consistent size of the targets such as ground markings. In order to obtain the bird's-eye view, a geometric mapping relationship between the 3D world coordinates and the camera's 2D coordinates can be established, so the camera needs to be calibrated. Although the traditional manual calibration method can be used for calibration, in some application scenarios, such as traffic scenarios, the installation angles of the cameras are different. If the traditional manual calibration method is used for calibration, the cost is extremely high. Therefore, how to obtain the camera parameters through the images taken by the monocular camera is an urgent problem to be solved. Summary of the invention

[0003] The present application provides a camera calibration method, a training method, an image processing method and a device, which can realize the calibration of the camera equipment with high efficiency.

[0004] In order to solve the above technical problems, the technical solution adopted in the present application is: to provide a camera equipment calibration method, the method comprising: obtaining an image to be processed taken by the camera equipment, the image to be processed contains at least two reference lines, and there is an intersection between the straight lines where at least two reference lines are located; based on the intersection, obtaining the vanishing point position information; based on the current focal length range and the vanishing point position information, calculating the shooting parameters of the camera equipment for shooting the image to be processed, the shooting parameters are used to characterize the angle information of the shooting equipment.

[0005] In order to solve the above-mentioned technical problems, another technical solution adopted in the present application is: to provide a training method for a neural network model, the method comprising: obtaining multiple training samples, the training samples comprising an image to be processed and shooting parameters corresponding to the image to be processed, the shooting parameters comprising a shooting pitch angle and a shooting field of view angle; using multiple training samples to train the neural network model to obtain a trained neural network model; wherein the shooting parameters are obtained using the camera equipment calibration method of the above-mentioned technical solution.

[0006] In order to solve the above-mentioned technical problems, another technical solution adopted in the present application is: to provide an image processing method, the method comprising: obtaining an input image, and processing the input image based on a trained neural network model to obtain shooting parameters of the input image; based on the shooting parameters, generating a homography matrix; based on the homography matrix, processing the input image to obtain a processed image; wherein, the trained neural network model is obtained by adopting the training method of the neural network model of the above-mentioned technical solution.

[0007] In order to solve the above-mentioned technical problems, another technical solution adopted in the present application is: to provide an image processing device, which includes a memory and a processor connected to each other, wherein the memory is used to store a computer program, and when the computer program is executed by the processor, it is used to implement the camera equipment calibration method, neural network model training method or image processing method in the above-mentioned technical solution.

[0008] In order to solve the above-mentioned technical problems, another technical solution adopted in the present application is: to provide a computer-readable storage medium, which is used to store a computer program. When the computer program is executed by a processor, it is used to implement the camera equipment calibration method, neural network model training method or image processing method in the above-mentioned technical solution.

[0009] Through the above scheme, the beneficial effect of the present application is: first, the target monitoring scene is photographed by a camera device to obtain an image to be processed, and the image to be processed includes at least two reference lines, and at least the straight lines where the two reference lines are located have an intersection; then the position information of the intersection is used to generate the vanishing point position information; then, the current focal length range and the vanishing point position information are used to determine the shooting parameters of the camera device when shooting the image to be processed, so as to obtain the internal and external parameters of the camera device, thereby realizing the calibration of the camera device, which is simple to implement, has lower cost and higher efficiency than manual calibration. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them:

[0011] Figure 1 is a schematic diagram of the field of view provided by this application;

[0012] Figure 2 It is a flowchart of an embodiment of a camera device calibration method provided by the present application;

[0013] Figure 3It is a schematic diagram provided by the present application that the intersection point is outside the image to be processed;

[0014] Figure 4 It is a schematic diagram provided by the present application that the intersection point is located within the image to be processed;

[0015] Figure 5 is a schematic diagram of obtaining the vanishing point in the y direction of the image to be processed provided by the present application;

[0016] Figure 6 is a flowchart of another embodiment of the camera device calibration method provided by the present application;

[0017] Figure 7 This is a bird's-eye view provided by this application;

[0018] Figure 8 is another bird’s eye view provided by this application;

[0019] Fig. 9 is a schematic diagram of reference line C and reference line D in the image to be processed provided by the present application;

[0020] Fig.10 It is a flowchart of an embodiment of a training method for a neural network model provided by the present application;

[0021] Fig.11 It is a flowchart of an embodiment of an image processing method provided by the present application;

[0022] Fig.12 is the input image provided by this application;

[0023] Fig.13 is with Fig.12 The corresponding processed image;

[0024] Fig.14 is a structural schematic diagram of an embodiment of an image processing device provided by the present application;

[0025] Fig.15 It is a structural schematic diagram of an embodiment of a computer-readable storage medium provided by the present application. DETAILED DESCRIPTION

[0026] The present application is further described in detail below in conjunction with the accompanying drawings and examples. It is particularly noted that the following examples are only used to illustrate the present application, but are not intended to limit the scope of the present application. Similarly, the following examples are only some embodiments of the present application rather than all embodiments, and all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0027] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0028] It should be noted that the terms "first", "second" and "third" in this application are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Thus, the features defined as "first", "second" and "third" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "multiple" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally also includes steps or units that are not listed, or optionally also includes other steps or units inherent to these processes, methods, products or devices.

[0029] First, the calibration of the imaging device (ie, camera) involved in this application is introduced. The camera calibration is mainly to obtain the intrinsic and extrinsic parameters of the camera. The extrinsic parameter matrix represents the transformation from the world coordinate system to the camera coordinate system, and the intrinsic parameter matrix represents the transformation from the camera coordinate system to the 2D image coordinate system. The intrinsic parameter matrix is ​​as follows:

[0030]

[0031] In formula (1), fx is the horizontal focal length, fy is the vertical focal length, and ideally fx=fy; s is the coordinate axis tilt parameter, which is ideally 0, cx is the horizontal coordinate of the image center, and cy is the vertical coordinate of the image center.

[0032] The rotation matrix of the camera is as follows:

[0033] R=R X ·R Y ·R Z (2)

[0034] In formula (2), R X , R Y , R Z They represent the angles of the camera relative to the real world in different directions. Since the transformation of the bird's-eye view in traffic scenes only requires obtaining the pitch angle of the camera, only R X, the rotation matrix in formula (2) becomes:

[0035]

[0036] Wherein, in formula (3), p is the pitch angle.

[0037] The homography matrix looks like this:

[0038] H=K·R·K -1 (4)

[0039] Transformation in the image coordinate system is performed by relying on the homography matrix. From the above formulas (1) to (4), it can be seen that after simplifying the required parameters, calibration can be achieved by only obtaining the focal length and pitch angle of the camera. Since the focal length changes synchronously with the image size when the image is scaled, the focal length can be transformed into the field of view angle. The transformation formula is as follows:

[0040]

[0041] In formula (5), fov represents the horizontal field of view, such as Figure 1 As shown, w represents the width of the image and f represents the focal length.

[0042] The present application predicts the pitch angle and field of view angle of the camera device, obtains the intrinsic parameter matrix and rotation matrix of the camera device, and finally transforms the input image through the homography matrix to obtain the transformed image (for example, a bird's-eye view), etc. The scheme adopted in the present application is described in detail below.

[0043] See also Figure 1 , Figure 1 : is a flow chart of an embodiment of a camera device calibration method provided by the present application, the method comprising:

[0044] S21: Acquire the image to be processed taken by the camera device.

[0045] First, a camera is used to shoot the target monitoring scene to obtain an image to be processed; specifically, the target monitoring scene can be a traffic scene or a parking lot scene, and the image to be processed contains at least two reference lines, and the straight lines where at least two reference lines are located have an intersection, and the reference lines are parallel lines in the real world. The reference lines can be zebra crossings or lane lines, etc., and the parallel reference lines have a unique intersection in the image to be processed.

[0046] S22: Based on the intersection point, obtain the vanishing point position information.

[0047] After obtaining an image to be processed that meets the requirements, the image to be processed can be processed to calculate the coordinates of the intersection of two reference lines in the image to be processed (recorded as a vanishing point) to obtain the vanishing point position information. The vanishing point may be located inside the image to be processed or outside the image to be processed. For example, Figure 3 As shown, the reference lines L1-L3 in the image to be processed I1 intersect at point H1, and point H1 is outside the image to be processed I1; or Figure 4 As shown, the reference lines L4 - L6 in the image to be processed I2 intersect at a point H2 , and the point H2 is located in the image to be processed I2 .

[0048] In a specific embodiment, taking the target monitoring scene as a traffic scene as an example, there must be lane lines or zebra crossings in the traffic scene, and parallel reference lines in the real world are selected, such as: Figure 5 As shown, reference line A and reference line B are selected, and the intersection of reference line A and reference line B is calculated. The intersection of reference line A and reference line B is the vanishing point in the vertical direction (ie, the y direction) in the image coordinate system, and different parallel lines in the y direction will intersect at this point.

[0049] S23: Calculating shooting parameters of the camera device for shooting the image to be processed based on the current focal length range and the vanishing point position information.

[0050] After obtaining the vanishing point position information, the shooting parameters of the camera device when shooting the image to be processed can be calculated according to the preset current focal length range and the vanishing point position information, and the shooting parameters and the corresponding image to be processed can be stored for subsequent use. The shooting parameters are used to characterize the angle information of the shooting device. For example, the shooting parameters include the shooting pitch angle and the shooting field angle.

[0051] In a specific embodiment, the current focal length range can be set to a larger range, and a focal length is selected from the current focal length range as the current focal length. The current focal length is substituted into formula (1) to obtain the current internal parameter matrix; the current pitch angle is calculated using the relationship between the current internal parameter matrix, the vanishing point position information and the pitch angle; then the current rotation matrix is ​​calculated using the current pitch angle and formula (3); then the current homography matrix is ​​calculated using the current rotation matrix, the current internal parameter matrix and formula (4); then the current homography matrix is ​​used to process the image to be processed to obtain a transformed image; then the transformed image is used to determine whether the current focal length meets the set conditions. If so, the current focal length is used as the focal length in the shooting parameters. If not, the process returns to the step of selecting a focal length in the current focal length range as the current focal length, until all focal lengths in the current focal length range are traversed or the current focal length meets the set conditions.

[0052] In other embodiments, the current focal length range may be set to a slightly smaller range and processed in a similar manner as described above, which will not be described in detail herein. The difference is that: when it is determined that all focal lengths in the current focal length range do not meet the set conditions, the current focal length range is adjusted to become another range, and the above processing is repeated until a current focal length that meets the requirements is found and used as the shooting focal length; then the shooting field of view angle is calculated using the shooting focal length and formula (5), and the shooting pitch angle is calculated using the relationship between the shooting focal length, the vanishing point position information, and the shooting pitch angle.

[0053] This embodiment obtains the vanishing point position information by calculating the intersection coordinates of any two intersecting reference lines in the image to be processed; then determines the shooting focal length by using the relationship between the focal length and the pitch angle and the current focal length range; generates the shooting field angle by using the relationship between the shooting focal length and the shooting field angle; calculates the shooting pitch angle by using the relationship between the shooting field angle, the vanishing point position information and the pitch angle, thereby realizing camera calibration, which is simple to realize, has lower cost and higher efficiency than manual calibration.

[0054] See also Figure 6 , Figure 6 : is a flow chart of another embodiment of the camera device calibration method provided by the present application, the method comprising:

[0055] S61: Acquire an image to be processed taken by a camera device.

[0056] The image to be processed includes a first target object.

[0057] S62: Calculate the coordinates of the intersection of two reference lines that have an intersection in the image to be processed to obtain the vanishing point position information.

[0058] S61-S62 and S21-S22 in the above embodiment are not described in detail here.

[0059] S63: Obtain the real scale ratio of the image to be processed.

[0060] Due to the shooting angle of the camera equipment, not only the scale ratio of the distance is different, but also the scale ratio of the x / y direction at the same position is different. If the scale ratio cannot be obtained, after determining the vanishing point, the homography matrix can be obtained using the vanishing point and any focal length. After transformation, a parallel bird's-eye view will be obtained, but there are differences in the horizontal and vertical scale ratios, such as Figure 7-Figure 8 As shown, both are bird's-eye views looking down and parallel, but the scale ratios in the x / y direction are different.

[0061] In order to obtain the true scale ratio, the length and width of the first target object in the image to be processed can be obtained respectively to obtain the first length and the first width. The first target object can be a zebra crossing, a lane line or other objects; then the ratio of the first width to the first length is calculated to obtain the true scale ratio.

[0062] In a specific embodiment, for a given image to be processed, in order to obtain the scale ratio of a certain pixel point in the x / y direction, the national standard of signs and markings can be used to determine it. The signs and markings on the ground in traffic scenes are all drawn according to the same standard. Therefore, when marking, the zebra crossing and other markings can be used to determine the actual scale ratio in the x / y direction; for example, Fig. 9 As shown, the length and width ratio of zebra crossings is standardized, and the true scale ratio is calculated using the following formula:

[0063]

[0064] In formula (6), is the true scale ratio, Dis real (C) is the actual length of line segment C determined according to the national standard, Dis real (D) is the actual length of line segment D determined according to the national standard.

[0065] S64: Determine whether the current focal length range meets a preset condition.

[0066] The current focal length range includes the current maximum focal length and the current minimum focal length. The deviation value corresponding to the current maximum focal length and the deviation value corresponding to the current minimum focal length are calculated respectively to obtain a first deviation value and a second deviation value; it is determined whether the product of the first deviation value and the second deviation value is less than a third preset value, and the third preset value can be 0; if the product of the first deviation value and the second deviation value is less than the third preset value, it is determined that the current focal length range meets the preset conditions; if the product of the first deviation value and the second deviation value is greater than or equal to the third preset value, it is determined that the current focal length range does not meet the preset conditions.

[0067] Further, if it is detected that the current focal length range meets the preset condition, it indicates that the shooting focal length falls within the current focal length range, and the shooting focal length can be searched within the current focal length range.

[0068] S65: If the current focal length range does not meet the preset condition, the current focal length range is adjusted.

[0069] When it is determined that the current focal length range does not meet the preset conditions, the current focal length range is adjusted, and the focal length range obtained after the adjustment is determined as the current focal length range; then the process returns to the step of determining whether the current focal length range meets the preset conditions, that is, executing S64, until the current focal length range meets the preset conditions, and a current focal length range that meets the requirements is obtained.

[0070] Further, the current maximum focal length can be added to the first adjustment step length to obtain the first adjustment focal length, and the range formed by the current maximum focal length and the first adjustment focal length is used as the current focal length range; or, the current minimum focal length can be subtracted from the second adjustment step length to obtain the second adjustment focal length, and the range formed by the second adjustment focal length and the current minimum focal length is used as the current focal length range; or, the current maximum focal length can be added to the first adjustment step length to obtain the first adjustment focal length, and the current minimum focal length can be subtracted from the second adjustment step length to obtain the second adjustment focal length, and the range formed by the second adjustment focal length and the first adjustment focal length is used as the current focal length range. For example, assuming that the current focal length range is [100, 500] and the second adjustment step length is 30, if the current focal length range does not meet the preset conditions, the current focal length range is [70, 100].

[0071] S66: Calculate the shooting pitch angle and the shooting focal length of the camera device based on the current focal length range and the real scale ratio.

[0072] Select a focal length from the current focal length range to obtain the current search focal length; determine whether the deviation value corresponding to the current search focal length falls within the preset deviation range; if the deviation value corresponding to the current search focal length falls within the preset deviation range, determine the current search focal length as the shooting focal length; if the deviation value corresponding to the current search focal length does not fall within the preset deviation range, return to the step of selecting a focal length from the current focal length range to obtain the current search focal length, until the deviation value corresponding to the current search focal length falls within the preset deviation range.

[0073] In a specific embodiment, the deviation value may be calculated using the following scheme:

[0074] 1) Obtain the current candidate focal length and calculate the current intrinsic parameter matrix based on the current candidate focal length.

[0075] The current candidate focal length is the current search focal length, the current maximum focal length, or the current minimum focal length.

[0076] 2) Based on the current intrinsic parameter matrix and the vanishing point position information, calculate the current pitch angle.

[0077] Calculate the product of the inverse matrix of the current intrinsic parameter matrix and the vanishing point position information to obtain a first vector; divide the first vector by the modulus of the first vector to obtain a second vector; process the second vector to obtain a current pitch angle; specifically, obtain the value of a preset position in the second vector to obtain a second value, where the preset position may be the second position; calculate the arc sine value of the second value to obtain a third value; calculate the negative value of the third value to obtain the current pitch angle, that is, use the following formula to calculate the current pitch angle:

[0078] R=(r1,r2,r3) (7)

[0079]

[0080] p=-arcsin(r3(2)) (9)

[0081] Among them, R is the current rotation matrix, r3 is the third column of the current rotation matrix (i.e., the first vector), K -1 is the inverse matrix of the current internal parameter matrix; v z is the vanishing point position information, v z is a 3×1 vector, the first and second values ​​of which are the horizontal and vertical coordinates respectively, and the third value is 1; “||||” is a modulo operation, p is the current pitch angle, and r3(2) is the second element of the first vector.

[0082] 3) Based on the current pitch angle, calculate the current rotation matrix.

[0083] After obtaining the current pitch angle, the current rotation matrix is ​​calculated using formula (3).

[0084] 4) Based on the current rotation matrix and the current intrinsic parameter matrix, calculate the current homography matrix.

[0085] After obtaining the current rotation matrix and the current intrinsic parameter matrix, substitute them into formula (4) to obtain the current homography matrix.

[0086] 5) Generate a transformed image based on the current homography matrix and the image to be processed.

[0087] Multiply each pixel value of the image to be processed by the current homography matrix to obtain the pixel value of the transformed image.

[0088] 6) Based on the ratio of the transformed image to the true scale, the deviation value is calculated.

[0089] The transformed image includes a second target object corresponding to the first target object, and the length and the width of the second target object are calculated respectively to obtain a second length and a second width; the product of the true scale ratio and the second length is calculated to obtain a first value; and the difference between the second width and the first value is calculated to obtain a deviation value.

[0090] In one embodiment, if Fig. 9 As shown, the first target object is a zebra crossing. The length of line C can be recorded as the length of the zebra crossing, and the length of line segment D can be recorded as the width of the zebra crossing. Through the current homography matrix, the lines C1 and D1 in the transformed image of the lines C and D in the image to be processed are obtained, and the lengths of the lines C1 and D1 in the transformed image are calculated. In theory, the length ratio of the lines C1 and D1 should be the same as the true scale ratio, so the deviation value can be calculated using the following formula:

[0091]

[0092] In formula (10), dis(C1) is the length of line C1, dis(D1) is the length of line D1, and Diff dis is the deviation value, and the following formula is used to determine whether the current focal length range meets the preset conditions:

[0093] G=dis dis (fmin)*dis dis (fmax) (11)

[0094] Among them, G is the deviation product, dis dis (fmax) is the deviation value corresponding to the current maximum focal length (i.e., the first deviation value), dis dis (fmin) is the deviation value corresponding to the current minimum focal length (i.e., the second deviation value). If G<0, it means that the shooting focal length falls within the current focal length range; if G≥0, it means that the shooting focal length does not fall within the current focal length range. At this time, the current focal length range needs to be adjusted until the shooting focal length falls within the current focal length range.

[0095] After determining the current focal length range in which the shooting focal length is located, the shooting focal length is searched in the current focal length range using a method such as a binary search or a nearest neighbor search, so that the shooting focal length satisfies the following formula:

[0096] |Diff dis (f)| <error (12)

[0097] Wherein, in formula (12), f is the determined final focal length (ie, the shooting focal length), error is the preset deviation value, and the preset deviation range is [-error, error].

[0098] S67: Calculate the shooting field of view angle based on the shooting focal length and the size information of the image to be processed.

[0099] The size information is the width of the image to be processed. After obtaining the shooting focal length, it can be substituted into formula (5) to calculate the shooting field of view, that is, the product of the width and the first preset value is calculated to obtain the fourth value; the ratio of the fourth value to the shooting focal length is calculated to obtain the fifth value; the inverse tangent value of the fifth value is calculated to obtain the sixth value; the product of the sixth value and the second preset value is calculated to obtain the shooting field of view; wherein the first preset value can be 0.5 and the second preset value can be 2.

[0100] In view of the characteristics of variable pitch angles and diverse focal lengths of camera equipment in traffic intersection scenes, this embodiment analyzes traffic intersection scenes and finds that not all transformation parameters of the bird's-eye view need to be obtained. Only the pitch angle and field of view angle need to be obtained, which can simplify the parameters to be obtained and simplify the implementation.

[0101] See also Fig.10 , Fig.10 1 is a flow chart of an embodiment of a training method for a neural network model provided by the present application, the method comprising:

[0102] S101: Acquire multiple training samples.

[0103] The training samples include the image to be processed and the shooting parameters corresponding to the image to be processed. The shooting parameters include the shooting pitch angle and the shooting field angle, that is, the pitch angle label and the field angle label. The shooting parameters are obtained by the camera equipment calibration method in the above embodiment.

[0104] S102: Using multiple training samples to train the neural network model to obtain a trained neural network model.

[0105] After obtaining multiple training samples, the pre-established neural network model is trained using the training samples; specifically, the error between the prediction parameters output by the neural network model and the shooting parameters in the training samples can be calculated to obtain the current error; then based on the current error, the model parameters of the neural network model are updated until the neural network model meets the preset stopping conditions, thereby obtaining a trained neural network model.

[0106] Further, the prediction parameters include a predicted value of the pitch angle corresponding to the shooting pitch angle and a predicted value of the field of view angle corresponding to the shooting field of view angle. The error between the shooting pitch angle and the predicted value of the pitch angle can be calculated to obtain a first error, and the error between the shooting field of view angle and the predicted value of the field of view angle can be calculated to obtain a second error. The first error and the second error are weighted and summed to obtain the current error. It can be understood that the model training process in this embodiment is similar to that of a general deep neural network, which will not be repeated here, and this embodiment does not limit the specific structure of the neural network model, as long as the function of predicting the pitch angle and the field of view angle can be achieved.

[0107] Furthermore, the preset stopping conditions include: loss value convergence, that is, the difference between the previous loss value and the current loss value is less than the set value; determining whether the current loss value is less than the preset loss value, the preset loss value is a preset loss threshold, if the current loss value is less than the preset loss value, it is determined that the preset stopping condition is met; the number of training times reaches the set value (for example: training 10,000 times); or the accuracy obtained when testing using the test set reaches the set conditions (for example: exceeding the preset accuracy), etc.

[0108] This embodiment provides a model training method, by determining the vanishing point in one direction, estimating the focal length, obtaining the pitch angle label and the field of view angle label of the camera device, forming a training sample, and using the training sample to train the model, so as to generate a model that can predict the pitch angle and the field of view angle, so as to facilitate the subsequent direct use of the model to predict the pitch angle and the field of view angle corresponding to the input image.

[0109] See also Fig.11 , Fig.11 : is a flow chart of an embodiment of an image processing method provided by the present application, the method comprising:

[0110] S111: Acquire an input image, and process the input image based on the trained neural network model to obtain shooting parameters of the input image.

[0111] The trained neural network model is obtained by using the training method of the neural network model in the above embodiment.

[0112] S112: Generate a homography matrix based on the shooting parameters.

[0113] The shooting field of view angle in the shooting parameters and formula (5) are used to calculate the shooting focal length; the shooting focal length and formula (1) are used to calculate the internal parameter matrix; the shooting pitch angle and formula (4) are used to calculate the rotation matrix; the homography matrix can be obtained by using the internal parameter matrix, the rotation matrix and formula (4).

[0114] S113: Process the input image based on the homography matrix to obtain a processed image.

[0115] The homography matrix is ​​used to perform perspective transformation on the input image, that is, each pixel value of the input image is multiplied by the homography matrix to obtain the pixel value of the processed image, which can be a bird's-eye view or a rectified image. For example, Fig.12 and Fig.13 As shown, Fig.12 is the input image. After the input image is processed by perspective transformation, we can get Fig.13 Processed image shown.

[0116] In one embodiment, the pitch angle of the camera device has an angle range of 0 to 90 degrees; the field of view angle may have an angle range of 0 to 180 degrees due to different camera devices; in order to simplify calculations, trigonometric functions may be used to predict when training a neural network model and using the model for prediction, for example: using a sin function to constrain the predicted value of the pitch angle to be between (0, 1), and using a cos function to constrain the predicted value of the field of view angle to be between (-1, 1).

[0117] This embodiment provides a method for generating a bird's-eye view based on camera self-calibration. Based on a single vanishing point and a current focal length range, an approximation method is used to determine the shooting focal length and the shooting pitch angle to form a label used for training a neural network model. Moreover, by utilizing the fitting ability of the neural network model, the pitch angle and the field of view angle of the camera to which the input image belongs are directly inferred, and then the homography matrix is ​​calculated. The input image can be transformed into a bird's-eye view through the homography matrix. No additional calibration information is required, and no third-party technology is relied on. The implementation is simple and efficient.

[0118] See also Fig.14 , Fig.14 It is a structural diagram of an embodiment of an image processing device provided by the present application. The image processing device 120 includes a memory 121 and a processor 122 connected to each other. The memory 121 is used to store a computer program. When the computer program is executed by the processor 122, it is used to implement the camera equipment calibration method, the neural network model training method or the image processing method in the above-mentioned embodiment.

[0119] See also Fig.15 , Fig.15 It is a structural diagram of an embodiment of a computer-readable storage medium provided in the present application. The computer-readable storage medium 130 is used to store a computer program 131. When the computer program 131 is executed by the processor, it is used to implement the camera calibration method, neural network model training method or image processing method in the above-mentioned embodiments.

[0120] The computer-readable storage medium 130 may be a server, a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, and other media that can store program codes.

[0121] In the several embodiments provided in this application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only illustrative, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0122] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0123] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0124] The above descriptions are merely embodiments of the present application and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A camera equipment calibration method, It is characterized in that include: Acquire an image to be processed taken by a camera device, wherein the image to be processed includes at least two reference lines, and the straight lines where the at least two reference lines are located have an intersection; Based on the intersection point, obtaining vanishing point position information; Calculating shooting parameters of the camera device for shooting the image to be processed based on the current focal length range and the vanishing point position information, wherein the shooting parameters are used to represent angle information of the camera device; The shooting parameters include a shooting field angle and a shooting pitch angle, and the step of calculating the shooting parameters of the image to be processed by the camera device based on the current focal length range and the vanishing point position information includes: Selecting a focal length from the current focal length range to obtain a current search focal length; Determine whether the deviation value corresponding to the current search focal length falls within a preset deviation range; If not, return to the step of selecting a focal length from the current focal length range to obtain the current search focal length, until the deviation value corresponding to the current search focal length falls within the preset deviation range; If yes, the current search focal length is determined as the shooting focal length; Calculate the shooting field of view angle by using the relationship between the shooting focal length and the shooting field of view angle; The shooting pitch angle is calculated by using the relationship between the shooting focal length, the vanishing point position information and the shooting pitch angle.

2. The camera device calibration method according to claim 1, It is characterized in that The step of calculating the shooting parameters of the camera device for shooting the image to be processed based on the current focal length range and the vanishing point position information comprises: Obtaining the real scale ratio of the image to be processed; Based on the current focal length range and the real scale ratio, the shooting pitch angle and the shooting focal length of the camera device are calculated.

3. The camera device calibration method according to claim 2, It is characterized in that The image to be processed includes a first target object, and the step of obtaining the real scale ratio of the image to be processed includes: Respectively acquiring the length and width of the first target object to obtain a first length and a first width; The ratio of the first width to the first length is calculated to obtain the true scale ratio.

4. The camera device calibration method according to claim 2, It is characterized in that Before the step of calculating the shooting pitch angle and the shooting focal length of the camera device based on the current focal length range and the real scale ratio, the step further includes: If it is determined that the current focal length range does not meet the preset condition, the current focal length range is adjusted until the current focal length range meets the preset condition.

5. The camera device calibration method according to claim 4, It is characterized in that The current focal length range includes a current maximum focal length and a current minimum focal length. Before the step of determining that the current focal length range does not meet the preset condition, the step includes: Calculate the deviation value corresponding to the current maximum focal length and the deviation value corresponding to the current minimum focal length respectively to obtain a first deviation value and a second deviation value; Determine whether the product of the first deviation value and the second deviation value is less than a third preset value; If not, it is determined that the current focal length range does not meet the preset condition.

6. The camera device calibration method according to claim 5, It is characterized in that The calculation of the deviation value includes: Obtaining a current candidate focal length, and calculating a current internal parameter matrix based on the current candidate focal length, wherein the current candidate focal length is the current search focal length, the current maximum focal length, or the current minimum focal length; Calculating a current pitch angle based on the current intrinsic parameter matrix and the vanishing point position information; Based on the current pitch angle, calculating a current rotation matrix; Calculate a current homography matrix based on the current rotation matrix and the current intrinsic parameter matrix; Generate a transformed image based on the current homography matrix and the image to be processed; The deviation value is calculated based on the ratio of the transformed image to the true scale.

7. The camera device calibration method according to claim 6, It is characterized in that The transformed image includes a second target object corresponding to the first target object, and the step of calculating the deviation value based on the ratio between the transformed image and the true scale includes: Calculating the length of the second target object and the width of the second target object respectively to obtain a second length and a second width; Calculating the product of the true scale ratio and the second length to obtain a first value; The difference between the second width and the first value is calculated to obtain the deviation value.

8. The camera device calibration method according to claim 6, It is characterized in that The step of calculating the current pitch angle based on the current intrinsic parameter matrix and the vanishing point position information comprises: Calculate the product of the inverse matrix of the current intrinsic parameter matrix and the vanishing point position information to obtain a first vector; Dividing the first vector by the modulus of the first vector to obtain a second vector; The second vector is processed to obtain the current pitch angle.

9. The camera device calibration method according to claim 8, It is characterized in that The step of processing the second vector to obtain the current pitch angle includes: Obtaining a value of a preset position in the second vector to obtain a second value; Calculate the arc sine value of the second value to obtain a third value; The negative value of the third value is calculated to obtain the current pitch angle.

10. The camera device calibration method according to claim 2, It is characterized in that The size information is the width of the image to be processed, and the step of calculating the shooting field of view angle based on the shooting focal length and the size information of the image to be processed includes: Calculating the product of the width of the image to be processed and the first preset value to obtain a fourth value; Calculating a ratio of the fourth value to the shooting focal length to obtain a fifth value; Calculating an arc tangent value of the fifth value to obtain a sixth value; The product of the sixth value and the second preset value is calculated to obtain the shooting field of view angle.

11. A training method for a neural network model, It is characterized in that include: Acquire a plurality of training samples, wherein the training samples include images to be processed and shooting parameters corresponding to the images to be processed, wherein the shooting parameters include a shooting pitch angle and a shooting field angle; Using the multiple training samples to train the neural network model to obtain a trained neural network model; Wherein, the shooting parameters are obtained by using the camera equipment calibration method described in any one of claims 1-10.

12. An image processing method, It is characterized in that include: Acquire an input image, and process the input image based on the trained neural network model to obtain shooting parameters of the input image; Based on the shooting parameters, generating a homography matrix; Based on the homography matrix, the input image is processed to obtain a processed image; Wherein, the trained neural network model is obtained by adopting the training method of the neural network model described in claim 11.

13. An image processing device, It is characterized in that It comprises a memory and a processor connected to each other, wherein the memory is used to store a computer program, and when the computer program is executed by the processor, it is used to implement the camera equipment calibration method described in any one of claims 1 to 10, the neural network model training method described in claim 11, or the image processing method described in claim 12.

14. A computer-readable storage medium for storing a computer program, It is characterized in that When the computer program is executed by a processor, it is used to implement the camera equipment calibration method described in any one of claims 1 to 10, the neural network model training method described in claim 11, or the image processing method described in claim 12.