Camera monitoring method and device, storage medium and chip

Abstract

The present disclosure relates to a camera monitoring method, device, storage medium and chip, the method comprising: acquiring an image collected by a camera in a vehicle; extracting a plurality of ground coordinate points included in a ground in the image; projecting the plurality of ground coordinate points from a pixel coordinate system into a camera coordinate system to obtain a plurality of first coordinate points; determining a plurality of second initial coordinate points based on an unknown depth parameter and the plurality of first coordinate points; projecting the plurality of second initial coordinate points from the camera coordinate system into a vehicle body coordinate system of the vehicle to obtain a plurality of third coordinate points, and outputting the known depth parameter by taking a coordinate value of each third coordinate point in a vertical direction as a constraint; outputting a plurality of second target coordinate points based on the known depth parameter and the plurality of first coordinate points; obtaining a target distance between the camera and the ground based on the plurality of second target coordinate points, and determining a monitoring result based on the target distance and a pre-obtained reference distance.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of automatic driving, and particularly relates to a camera monitoring method and device, a storage medium and a chip. BACKGROUND

[0002] Due to the fact that a camera can sense rich texture features in a scene, it becomes an indispensable part in the automatic driving of a vehicle. In order to utilize the sensing results of the camera in the automatic driving of the vehicle, for example, using lane lines and traffic lights and the like sensing results in driving and parking, it is necessary to project the sensing results to a vehicle body coordinate system, and the installation position of the camera is crucial to the projection of the sensing results to the vehicle body coordinate system.

[0003] However, during the driving of the vehicle, the angle of the camera is often prone to change, and therefore, the angle of the camera needs to be monitored in a timely and accurate manner. SUMMARY

[0004] To overcome the problems in the related art, the present disclosure provides a camera monitoring method and device, a storage medium and a chip.

[0005] According to a first aspect of an embodiment of the present disclosure, a camera monitoring method is provided, comprising:

[0006] obtaining an image captured by a camera in a vehicle;

[0007] extracting a plurality of ground coordinate points included in a ground in the image;

[0008] projecting the plurality of ground coordinate points from a pixel coordinate system constructed based on the image to a camera coordinate system constructed based on the camera, to obtain a plurality of first coordinate points;

[0009] determining a plurality of second initial coordinate points based on an unknown depth parameter and the plurality of first coordinate points, the depth parameter being used to reflect the depth of each second initial coordinate point in the camera coordinate system;

[0010] projecting the plurality of second initial coordinate points from the camera coordinate system to a vehicle body coordinate system of the vehicle, to obtain a plurality of third coordinate points, and taking the coordinate value of each third coordinate point in the vertical direction as zero as a constraint, and outputting a known depth parameter;

[0011] outputting a plurality of second target coordinate points based on the known depth parameter and the plurality of first coordinate points;

[0012] obtaining a target distance between the camera and the ground in the image based on the plurality of second target coordinate points, and determining a monitoring result for characterizing the degree of offset of the angle of the camera based on the target distance and a pre-obtained reference distance.

[0013] In some embodiments, the extracting the ground included in the image comprises:

[0014] Obtaining a region of interest in the image;

[0015] Performing ground segmentation on the region of interest to obtain the ground;

[0016] Obtaining a plurality of ground coordinate points included in the ground.

[0017] In some embodiments, the method further comprises:

[0018] In a case where the number of the plurality of ground coordinate points is greater than or equal to a preset coordinate point number, projecting the plurality of ground coordinate points from the pixel coordinate system to the camera coordinate system to obtain a plurality of the first coordinate points;

[0019] In a case where the number of the plurality of ground coordinate points is less than the preset coordinate point number, repeatedly performing the steps of obtaining the image collected by the camera in the vehicle and extracting the plurality of ground coordinate points included in the ground in the image until the number of the plurality of ground coordinate points extracted is greater than or equal to the preset coordinate point number.

[0020] In some embodiments, the obtaining the target distance between the camera and the ground in the image based on the plurality of second target coordinate points comprises:

[0021] Determining a plurality of distances between the plurality of second target coordinate points and the origin of the camera coordinate system;

[0022] Performing an average operation on the plurality of distances to obtain the target distance between the camera and the ground in the image.

[0023] In some embodiments, the determining a plurality of second initial coordinate points based on the unknown depth parameter and the plurality of first coordinate points comprises:

[0024] Determining the plurality of second initial coordinate points based on the product between the unknown depth parameter and the plurality of first coordinate points.

[0025] In some embodiments, the image comprises a frame, and the method further comprises:

[0026] Repeating the steps of obtaining the image collected by the camera in the vehicle and obtaining the target distance between the camera and the ground in the image based on the plurality of second target coordinate points until a plurality of target distances are obtained and / or the plurality of target distances obtained converge, the plurality of target distances being distances between the camera and the ground in a preset number of frames.

[0027] The monitoring result for representing the degree of deviation of the angle of the camera is determined based on the target distance and a reference distance obtained in advance, and includes:

[0028] The plurality of target distances are averaged to obtain an average target distance.

[0029] The monitoring result for representing the degree of deviation of the angle of the camera is determined based on the average target distance and the reference distance.

[0030] In some embodiments, the monitoring result for representing the degree of deviation of the angle of the camera is determined based on the average target distance and the reference distance, and includes:

[0031] In a case where the average target distance is higher than a first preset threshold or lower than a second preset threshold, the monitoring result for representing the degree of deviation of the angle of the camera is obtained, which is not within an error range, and the first preset threshold and the second preset threshold are obtained based on the reference distance.

[0032] The method further includes:

[0033] Prompting information is output according to the monitoring result.

[0034] According to a second aspect of the embodiments of the present disclosure, a camera monitoring device is provided, which includes:

[0035] An acquisition module is configured to acquire an image captured by a camera in a vehicle.

[0036] An extraction module is configured to extract a plurality of ground coordinate points included in a ground surface in the image.

[0037] A first projection module is configured to project the plurality of ground coordinate points from a pixel coordinate system constructed based on the image into a camera coordinate system constructed based on the camera, to obtain a plurality of first coordinate points.

[0038] A first determination module is configured to determine a plurality of second initial coordinate points based on an unknown depth parameter and the plurality of first coordinate points, the depth parameter being used to reflect a depth of each of the second initial coordinate points in the camera coordinate system.

[0039] A first output module is configured to project the plurality of second initial coordinate points from the camera coordinate system into a vehicle body coordinate system of the vehicle, to obtain a plurality of third coordinate points, and to output a known depth parameter by taking a coordinate value of each of the third coordinate points in a vertical direction as a constraint.

[0040] A second output module is configured to output a plurality of second target coordinate points based on the known depth parameter and the plurality of first coordinate points.

[0041] monitoring, based on the target distance and a reference distance obtained in advance, a monitoring result used to represent a degree of deviation of an angle of the camera.

[0042] According to a third aspect of embodiments of the present disclosure, a camera monitoring apparatus is provided, comprising:

[0043] a processor;

[0044] a memory for storing processor-executable instructions;

[0045] wherein the processor is configured to:

[0046] obtain an image captured by a camera in a vehicle;

[0047] extract a plurality of ground coordinate points included in a ground surface in the image;

[0048] project the plurality of ground coordinate points from a pixel coordinate system constructed based on the image into a camera coordinate system constructed based on the camera, to obtain a plurality of first coordinate points;

[0049] determine a plurality of second initial coordinate points based on an unknown depth parameter and the plurality of first coordinate points, the depth parameter being used to reflect a depth of each of the second initial coordinate points in the camera coordinate system;

[0050] project the plurality of second initial coordinate points from the camera coordinate system into a vehicle body coordinate system of the vehicle, to obtain a plurality of third coordinate points, and set a coordinate value of each of the third coordinate points in a vertical direction to zero as a constraint, and output a known depth parameter;

[0051] output a plurality of second target coordinate points based on the known depth parameter and the plurality of first coordinate points;

[0052] monitor, based on the target distance and a reference distance obtained in advance, a monitoring result used to represent a degree of deviation of an angle of the camera.

[0053] According to a fourth aspect of embodiments of the present disclosure, a computer readable storage medium is provided, having stored thereon computer program instructions, which, when executed by a processor, implement the steps of the camera monitoring method provided in the first aspect of the present disclosure.

[0054] According to a fifth aspect of the embodiments of the present disclosure, a chip is provided, comprising a processor and an interface; the processor is configured to read instructions to execute the method according to any one of the first aspect of the present disclosure.

[0055] The technical solution provided by the embodiments of the present disclosure can have the following beneficial effects: by taking the coordinate value of each third coordinate point in the vertical direction as zero as a constraint, outputting a known depth parameter, and obtaining the depth of the ground coordinate point in the camera coordinate system, the target distance between the camera and the ground in the image can be obtained through the second target coordinate point of the ground coordinate point in the camera coordinate system, and then the angle of the camera can be monitored through the target distance. This monitoring method is not constrained by lane lines and can be more widely applied to natural scenes.

[0056] It should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory, and are not limiting to the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0057] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments consistent with the present disclosure and serve to explain the principles of the present disclosure together with the description.

[0058] Figure 1 is a flowchart of a camera monitoring method according to an example embodiment.

[0059] Figure 2 is an example diagram of a region of interest according to an example embodiment.

[0060] Figure 3 is a block diagram of a camera monitoring device according to an example embodiment.

[0061] Figure 4 is a block diagram of a vehicle according to an example embodiment.

[0062] Figure 5 is a block diagram of a camera monitoring device according to an example embodiment. DETAILED DESCRIPTION

[0063] The example embodiments will be described in detail herein with reference to the attached drawings. When the description below refers to accompanying drawings, the same numbers in different drawings refer to the same or similar elements unless otherwise described. The implementations described in the following example embodiments do not represent all implementations consistent with the present disclosure. Instead, they are merely examples of apparatuses and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0064] It should be noted that all the actions of acquiring signals, information or data in the present disclosure are carried out in accordance with the corresponding data protection regulations and policies of the country where the device is located, and with the authorization given by the owner of the corresponding device.

[0065] As the background art, the angle of the camera often changes during the driving of the vehicle, so it is necessary to monitor the angle of the camera in time and accurately. In the related art, the angle of the camera can be monitored by the following two schemes: Scheme 1: When the vehicle is shipped, the camera takes a picture, detects the key points belonging to the vehicle body; in the actual driving scene of the vehicle, take another picture, and extract the key points belonging to the vehicle body again; compare the change of the positions of the key points twice, if it exceeds the set threshold, it is considered that the external parameter changes greatly, and the angle of the camera changes greatly. Scheme 2: Detect or segment more than two parallel lane lines in the image captured by the camera through a model; and fit the points in the same lane line; calculate the vanishing point of the two lane lines through the equation, and then calculate the external parameter of the camera, and measure the change of the angle of the camera by calculating the deviation between the calculated external parameter and the angle of the line calibration.

[0066] However, scheme 1 needs to extract the key points on the vehicle body, and it is difficult to extract the key points on the pure color part of the vehicle body, so the subsequent matching is difficult, and the personalized accessories on the vehicle body, such as the vehicle body stickers, will affect the matching of the key points; scheme 2 needs more than two parallel lines, which leads to that on the one hand the scene requirement is high, and on the other hand the camera installed on both sides of the vehicle cannot detect more than two parallel lines even if the actual scene has enough lane lines, so the use of scheme 2 is very limited.

[0067] Therefore, the present disclosure proposes a camera monitoring method, device, storage medium and chip, the camera monitoring method of the present disclosure is not restricted by lane lines and can be more widely applied to natural scenes.

[0068] Figure 1 A flowchart of a camera monitoring method according to an example embodiment is shown, which can be performed by a vehicle, as shown in Figure 1 The camera monitoring method can include the following steps.

[0069] Step 110, acquiring an image captured by a camera in a vehicle.

[0070] In some embodiments, the camera can be a camera installed at any position in the vehicle, for example, can be installed on the front side, rear side or B pillar of the vehicle, etc., wherein the B pillar refers to the vertical column between the front and rear doors of the vehicle. The present disclosure does not make any limitation on the position of the camera in the vehicle.

[0071] Step 120, extracting a plurality of ground coordinate points included in the ground in the image.

[0072] In some embodiments, extracting multiple ground coordinate points from the ground in an image may include: performing ground segmentation processing on the image to obtain the ground in the image; and extracting multiple ground coordinate points from the ground.

[0073] In some embodiments, ground segmentation processing of an image can be performed using a pre-trained ground segmentation model. The ground segmentation model can be trained by: acquiring multiple frames of sample images labeled with ground features; iteratively updating the parameters of the initial ground segmentation model based on the multiple frames of sample images to reduce the loss function value corresponding to each sample image, thereby obtaining a trained ground segmentation model; wherein the loss function value corresponding to each sample image is determined by: processing the sample images using the ground segmentation model to obtain the predicted ground; and determining the loss function value based on the difference between the predicted ground and the labeled ground.

[0074] For example, during the training of the ground segmentation model, the parameters of the initial ground segmentation model can be continuously adjusted to reduce the loss function value corresponding to each sample image, so that the loss function value meets a preset condition. For example, the loss function value converges, or the loss function value is less than a preset value. When the loss function meets the preset condition, the model training is complete, and a trained ground segmentation model is obtained. The trained ground segmentation model can perform ground segmentation processing on the image to obtain the ground included in the image.

[0075] In some embodiments, the ground coordinate points can be ground coordinate points included in the ground within a region of interest in the image. Extracting multiple ground coordinate points included in the ground of the image may include: obtaining a region of interest in the image; performing ground segmentation processing on the region of interest to obtain the ground; and obtaining multiple ground coordinate points included in the ground.

[0076] This embodiment of the present disclosure extracts the ground coordinate points included in the region of interest. For different images, it can uniformly measure the target distance between the ground and the camera through the same region of interest in the image. Furthermore, in the process of calculating the target distance, only the ground coordinate points in the region of interest need to be considered, which can reduce the amount of calculation and improve the camera monitoring efficiency.

[0077] In some embodiments, the region of interest (ROI) can be specifically determined according to actual needs; for example, the ROI may be the central region of the image. For instance, the ROI may be as follows: Figure 2 The area within the box shown. By using the central region of the image as the region of interest, it is ensured that there is ground within the region of interest, and that the distance between the ground represented by the central region and the vehicle body is appropriate, thus guaranteeing the accuracy of the calculated target distance.

[0078] In some embodiments, the region of interest can be subjected to ground segmentation processing by a pre-trained ground segmentation model. The ground segmentation model can refer to the foregoing related description, which will not be repeated here.

[0079] At step 130, the plurality of ground coordinate points are projected from the pixel coordinate system constructed based on the image to the camera coordinate system constructed based on the camera, to obtain a plurality of first coordinate points.

[0080] It can be understood that the ground coordinate points are coordinate points in the image, and the pixel coordinate system is constructed based on the image. Therefore, the ground coordinate points can be in the pixel coordinate system. In some embodiments, the plurality of ground coordinate points can be projected from the pixel coordinate system to the camera coordinate system according to the intrinsic matrix, to obtain a plurality of first coordinate points. The intrinsic matrix can be obtained after the camera is calibrated in advance, for example, the camera is calibrated when the vehicle is manufactured, to obtain the intrinsic matrix.

[0081] The intrinsic matrix can be used to reflect the conversion relationship from the camera coordinate system to the pixel coordinate system. According to different imaging models of the camera, the parameters included in the intrinsic matrix of the camera can also be different. The imaging model of the camera can include a pinhole imaging model, a fisheye model, and an omnidirectional model, etc. The specific parameters included in the intrinsic matrix can refer to related technologies, which will not be repeated here.

[0082] The first coordinate point does not include the depth in the camera coordinate system, that is, it does not include the coordinate value in the z-axis direction of the camera coordinate system. For example, taking a single ground coordinate point (u, v) as an example, the first coordinate point (x, y) corresponding to the ground coordinate point can be obtained according to the following formula (1):

[0083]

[0084] Wherein, x represents the coordinate value of the first coordinate point in the x-axis direction of the camera coordinate system, y represents the coordinate value of the first coordinate point in the y-axis direction of the camera coordinate system, u represents the pixel value of the ground coordinate point in the u-axis direction of the pixel coordinate system, v represents the pixel value of the ground coordinate point in the v-axis direction of the pixel coordinate system, K represents the intrinsic matrix, and K -1 represents the inverse matrix of the intrinsic matrix.

[0085] In some embodiments, the camera monitoring method can further include: in the case that the number of the plurality of ground coordinate points is greater than or equal to a preset coordinate point number, projecting the plurality of ground coordinate points from the pixel coordinate system to the camera coordinate system to obtain a plurality of first coordinate points; in the case that the number of the plurality of ground coordinate points is less than the preset coordinate point number, repeatedly performing the steps of acquiring the image collected by the camera in the vehicle and extracting the plurality of ground coordinate points included in the ground in the image until the number of the plurality of ground coordinate points extracted is greater than or equal to the preset coordinate point number.

[0086] The preset coordinate point quantity can be determined according to actual requirements. For example, the preset coordinate point quantity can be determined according to the total number of coordinate points in the image in which the ground is located. For example, the preset coordinate point quantity t1 can be obtained according to the formula t1 = ratio1*N, where ratio1 represents the first weight value, and N represents the total number of coordinate points in the image.

[0087] In some embodiments, the first weight value can be any value in [0, 1]. The first weight value can be determined according to actual requirements. The closer the first weight value is to 1, the higher the requirement for ground coordinate points. In possible implementations, the first weight value can be 0.5, indicating that only half of the ground coordinate points in the image are used to calculate the target distance.

[0088] In the case where the number of ground coordinate points is greater than or equal to the preset coordinate point quantity, the plurality of ground coordinate points are projected from the pixel coordinate system to the camera coordinate system to obtain a plurality of first coordinate points for subsequent calculation of the target distance. In the case where the number of ground coordinate points is less than the preset coordinate point quantity, the steps of obtaining the image captured by the camera in the vehicle and extracting a plurality of ground coordinate points included in the ground in the image are repeatedly performed until the number of extracted ground coordinate points is greater than or equal to the preset coordinate point quantity, that is, in the case where the number of ground coordinate points does not meet the requirement, the next frame of image is obtained and the ground coordinate points of the next frame of image are extracted. Thus, in the case where the number of ground coordinate points is greater than or equal to the preset coordinate point quantity, the target distance is calculated, which can ensure the accuracy of the target distance and improve the accuracy of the calculated target distance.

[0089] In step 140, a plurality of second initial coordinate points are determined based on the unknown depth parameter and the plurality of first coordinate points, and the depth parameter is used to reflect the depth of each second initial coordinate point in the camera coordinate system.

[0090] In some embodiments, determining the plurality of second initial coordinate points based on the unknown depth parameter and the plurality of first coordinate points can include determining the plurality of second initial coordinate points based on the product between the unknown depth parameter and the plurality of first coordinate points.

[0091] For example, taking the unknown depth parameter as λ and the aforementioned single first coordinate point (x, y) as an example, the second initial coordinate point (λx, λy, λ) corresponding to the single first coordinate point (x, y) can be calculated by the following formula (2):

[0092]

[0093] wherein λ, x, and the meaning of each represented are the same as the aforementioned; λx represents the coordinate value of the second initial coordinate point on the x-axis of the camera coordinate system; λy represents the coordinate value of the second initial coordinate point on the y-axis of the camera coordinate system; and λ represents the coordinate value of the second initial coordinate point on the z-axis of the camera coordinate system, i.e., the depth of the second initial coordinate point in the camera coordinate system.

[0094] At step 150, the plurality of second initial coordinate points are projected from the camera coordinate system into the vehicle body coordinate system of the vehicle to obtain a plurality of third coordinate points, and a constraint that the coordinate value of each third coordinate point in the vertical direction is zero is used to output the known depth parameter.

[0095] In some embodiments, the plurality of second initial coordinate points can be projected from the camera coordinate system into the vehicle body coordinate system of the vehicle according to an extrinsic matrix to obtain a plurality of third coordinate points. The extrinsic matrix can be used to reflect the conversion between the camera coordinate system and the vehicle body coordinate system, and the extrinsic matrix can include a rotation matrix R and a translation vector t.

[0096] For example, assuming that the vehicle body coordinate system {x w ,y w ,z w} has an x w -axis, a y w -axis, and a z w -axis, each of which corresponds to a direction of “front”, “left”, and “up”, respectively, the roll angle is obtained by rotating around the x w -axis, the pitch angle is obtained by rotating around the y w -axis, and the yaw angle is obtained by rotating around the z w -axis. Assuming that the conversion between the camera coordinate system and the vehicle body coordinate system requires rotating around the x w -axis, the y w -axis, and the z w -axis of the vehicle body coordinate system by angles of α, β, and γ, respectively, the rotation matrix R can be obtained by rotating around the z w -axis first, then rotating around the y w -axis, and finally rotating around the x w -axis, and the rotation matrix R can be obtained by the following formula (3):

[0097]

[0098] wherein cos represents the cosine function, and sin represents the sine function.

[0099] In order to simplify the expression of the rotation matrix R, it is assumed that the multiplication of the three matrices in the above formula (3) obtains the following formula (4):

[0100]

[0101] It should be noted that the parameters a-i in formula (4) are parameters defined for the purpose of simplifying the rotation matrix R, and the aforementioned formula (3) is not equivalent to formula (4) after expansion, and specific details can be referred to related technologies, which will not be described here.

[0102] After obtaining the aforementioned rotation matrix R, the extrinsic parameter matrix can be obtained in combination with the translation vector t. For example, the extrinsic parameter matrix T can be represented by the following formula (5):

[0103]

[0104] Wherein, T represents the extrinsic parameter matrix, t represents the translation vector, t xw represents the translation component along the x w axis, t yw represents the translation component along the y w axis, t zw represents the translation component along the z w axis.

[0105] In some embodiments, the extrinsic parameter matrix can be obtained after calibrating the camera in advance, for example, calibrating the camera when the vehicle is manufactured, to obtain the extrinsic parameter matrix. It should be noted that the form of the extrinsic parameter matrix can be flexibly determined according to actual needs, and the extrinsic parameter matrix can also reflect the conversion between the vehicle body coordinate system and the camera coordinate system, for example, the inverse matrix of the extrinsic parameter matrix represented by the aforementioned formula (5) can be determined as the final used extrinsic parameter matrix, which can represent the conversion between the vehicle body coordinate system and the camera coordinate system.

[0106] For example, still taking the example of the aforementioned formula (1)-(5), a single second initial coordinate point (λx, λy, λ) can be projected from the camera coordinate system to the vehicle body coordinate system of the vehicle according to the extrinsic parameter matrix by the following formula (6) to obtain a single third coordinate point (x w , y w , z w ):

[0107]

[0108] Wherein, the meanings of the parameters in formula (6) are the same as described above, which will not be described here.

[0109] It can be understood that the aforementioned formula (1), (2) and (6) illustrate the conversion between single coordinate points, and when multiple coordinate points are converted, the multiple coordinate points can be reflected by a coordinate point matrix, so as to convert multiple points at the same time.

[0110] In some embodiments, a constraint that the coordinate value of each third coordinate point in the vertical direction is zero can be used to solve the unknown depth parameter, and output the known depth parameter. The coordinate value of the third coordinate point in the vertical direction can refer to the coordinate value of the third coordinate point in the z-axis direction of the camera coordinate system. w The coordinate value of the third coordinate point in the vertical direction can refer to the coordinate value of the third coordinate point in the z-axis direction of the camera coordinate system.

[0111] For example, taking the aforementioned formula (6) as an example, since the constraint that the coordinate value of the third coordinate point in the vertical direction is zero is used, according to the matrix operation of formula (6), the following can be obtained: g*+h*+i*t zw *1= w =0. Thus, the following can be obtained: wherein g, h, i, and t zm The values of g, h, i, and t can be obtained according to the pre-calibrated extrinsic matrix, and are known quantities. The values of x and y are the coordinate values of the first coordinate point, and are also known quantities. Thus, the unknown depth parameter λ can be solved, and the known depth parameter can be output.

[0112] In step 160, based on the known depth parameter and the plurality of first coordinate points, a plurality of second target coordinate points are output.

[0113] For example, the depth parameter and the plurality of first coordinate points are known quantities. At this time, the plurality of second target coordinate points can be output according to the product between the depth parameter and the plurality of first coordinate points.

[0114] In step 170, based on the plurality of second target coordinate points, a target distance between the camera and the ground in the image is obtained, and based on the target distance and a pre-obtained reference distance, a monitoring result used to represent the degree of offset of the angle of the camera is determined.

[0115] The disclosed embodiments can output the known depth parameter by using the constraint that the coordinate value of each third coordinate point in the vertical direction is zero, and obtain the depth of the ground coordinate point in the camera coordinate system. Thus, the target distance between the camera and the ground in the image can be obtained through the second target coordinate point of the ground coordinate point in the camera coordinate system, and the angle of the camera can be monitored through the target distance. This monitoring method is not constrained by the lane line, and can be more widely applied in natural scenes.

[0116] The target distance between the camera and the ground in the image can be the distance between the origin of the camera coordinate system of the camera and the second target coordinate point of the ground coordinate point included in the ground in the camera coordinate system. In some embodiments, obtaining the target distance between the camera and the ground in the image based on the plurality of second target coordinate points can include: determining a plurality of distances between the plurality of second target coordinate points and the origin of the camera coordinate system; and performing an average operation on the plurality of distances to obtain the target distance between the camera and the ground in the image.

[0117] In some embodiments, the image can include a frame, and by performing a round of steps 110-170, a single target distance between the camera and the ground in the single frame image can be obtained. For the single target distance, based on the target distance and the pre-obtained reference distance, determining the monitoring result for characterizing the degree of deviation of the angle of the camera can include: obtaining the monitoring result for characterizing that the degree of deviation of the angle of the camera is not within the error range when the target distance is higher than a first preset threshold or lower than a second preset threshold, the first preset threshold and the second preset threshold being obtained based on the reference distance.

[0118] The reference distance can be pre-obtained for characterizing a standard distance between the camera and the ground. For example, the reference distance can be a target distance pre-obtained according to the aforementioned steps 110-170 when the vehicle is shipped.

[0119] The first preset threshold and the second preset threshold can be obtained based on the reference distance. For example, the first preset threshold d1 = ratio2 * d, where ratio2 represents a second weight value, and d represents the reference distance, where the second weight value can be a value greater than 1, for example, the second weight value can be 1.1 or 1.2, etc. The second preset threshold d2 = ratio3 * d, where ratio3 represents a third weight value, and d represents the reference distance, where the third weight value can be a value less than 1, for example, the third weight value can be 0.8 or 0.9, etc.

[0120] The angle of the camera can refer to the pitch angle of the camera. In some embodiments, the method further includes: when the monitoring result characterizes that the degree of deviation of the angle of the camera is not within the error range, outputting prompt information according to the monitoring result, the prompt information being used to prompt the driver of the vehicle with at least one of the following information: to use the automatic driving function cautiously, to use the auxiliary driving function cautiously, and to re-calibrate the position of the camera. By outputting the prompt information when it is found that the degree of deviation of the angle of the camera is not within the error range, the present disclosure can prompt the driver that the camera parameters are inaccurate and to use the related automatic driving or auxiliary driving function cautiously. Thus, the driving safety of the vehicle can be improved.

[0121] As described above, the image can include a frame, and in some embodiments, the method further includes: repeatedly performing the steps of obtaining the image captured by the camera in the vehicle, obtaining the target distance between the camera and the ground in the image based on the plurality of second target coordinate points, until a plurality of target distances are obtained, and / or the plurality of obtained target distances converge, the plurality of target distances being distances between the camera and the ground in a preset number of frames of images; and determining the monitoring result for representing the degree of deviation of the angle of the camera based on the target distance and the previously obtained reference distance, including: performing an average operation on the plurality of target distances to obtain an average target distance; and determining the monitoring result for representing the degree of deviation of the angle of the camera according to the average target distance and the reference distance.

[0122] In some embodiments, repeatedly obtaining the image captured by the camera in the vehicle can include: repeatedly obtaining the image captured by the camera in the vehicle during driving of the vehicle. The disclosure obtains the image during driving of the vehicle and obtains the monitoring result of the angle of the camera, which can realize real-time monitoring of the angle of the camera.

[0123] Embodiments of the disclosure repeatedly obtain the image captured by the camera in the vehicle, obtain the target distance between the camera and the ground in the image based on the plurality of second target coordinate points, obtain a plurality of target distances, and determine the monitoring result based on the plurality of target distances, which can realize determination of the monitoring result of the angle of the camera based on data of multiple frames of images, and prevent incorrect monitoring results determined based on a single frame of image due to vehicle shaking and the like.

[0124] In some embodiments, repeatedly obtaining the image captured by the camera in the vehicle, obtaining the target distance between the camera and the ground in the image based on the plurality of second target coordinate points, and obtaining a plurality of target distances can be performed every preset period. For example, the preset period can be 20 seconds, and assuming that 25 frames of images can be obtained per second, 500 frames of images can be processed in the preset period, and 500 target distances can be obtained.

[0125] The stop condition of repeatedly obtaining the image captured by the camera in the vehicle, and obtaining the target distance between the camera and the ground in the image based on the plurality of second target coordinate points can include: obtaining a plurality of target distances, and / or the plurality of obtained target distances converging. In some embodiments, the variance of the plurality of target distances can be calculated, and the plurality of target distances can be determined to converge when the variance is less than a third preset threshold.

[0126] In some embodiments, the method further comprises: determining, according to the average target distance and the reference distance, a monitoring result for representing a degree of deviation of the angle of the camera, wherein: when the average target distance is higher than a first preset threshold or lower than a second preset threshold, the monitoring result is that the degree of deviation of the angle of the camera is out of the error range, and the first preset threshold and the second preset threshold are obtained based on the reference distance; and outputting prompt information according to the monitoring result. For details of the first preset threshold, the second preset threshold and the prompt information, refer to the foregoing description, which will not be repeated here.

[0127] Figure 3 is a block diagram of a camera monitoring apparatus according to an exemplary embodiment. Referring to Figure 3 The camera monitoring apparatus 300 comprises an acquisition module 310, an extraction module 320, a first projection module 330, a first determination module 340, a first output module 350, a second output module 360 and a monitoring module 370.

[0128] The acquisition module 310 is configured to acquire an image captured by a camera in a vehicle;

[0129] The extraction module 320 is configured to extract a plurality of ground coordinate points included in a ground in the image;

[0130] The first projection module 330 is configured to project the plurality of ground coordinate points from a pixel coordinate system constructed based on the image into a camera coordinate system constructed based on the camera, to obtain a plurality of first coordinate points;

[0131] The first determination module 340 is configured to determine a plurality of second initial coordinate points based on an unknown depth parameter and the plurality of first coordinate points, the depth parameter being used to reflect a depth of each of the second initial coordinate points in the camera coordinate system;

[0132] The first output module 350 is configured to project the plurality of second initial coordinate points from the camera coordinate system into a vehicle body coordinate system of the vehicle, to obtain a plurality of third coordinate points, and output a known depth parameter by taking a coordinate value of each of the third coordinate points in a vertical direction as a constraint;

[0133] The second output module 360 is configured to output a plurality of second target coordinate points based on the known depth parameter and the plurality of first coordinate points;

[0134] The monitoring module 370 is configured to obtain a target distance between the camera and the ground in the image based on the plurality of second target coordinate points, and determine a monitoring result for representing a degree of deviation of the angle of the camera based on the target distance and a reference distance obtained in advance.

[0135] In some embodiments, the extraction module 320 is further configured to:

[0136] obtain a region of interest in the image;

[0137] perform ground segmentation on the region of interest to obtain the ground;

[0138] obtain a plurality of ground coordinate points included in the ground.

[0139] In some embodiments, the apparatus further comprises:

[0140] a second projection module configured to, in a case where a number of the plurality of ground coordinate points is greater than or equal to a preset coordinate point number, project the plurality of ground coordinate points from the pixel coordinate system into the camera coordinate system to obtain a plurality of the first coordinate points;

[0141] a first execution module configured to, in a case where the number of the plurality of ground coordinate points is less than the preset coordinate point number, repeatedly execute the steps of obtaining the image collected by the camera in the vehicle and extracting the plurality of ground coordinate points included in the ground in the image until the number of the plurality of ground coordinate points extracted is greater than or equal to the preset coordinate point number.

[0142] In some embodiments, the monitoring module 370 is further configured to:

[0143] determine a plurality of distances between the plurality of second target coordinate points and an origin of the camera coordinate system;

[0144] perform an average operation on the plurality of distances to obtain the target distance between the camera and the ground in the image.

[0145] In some embodiments, the first determination module 340 is further configured to:

[0146] determine a plurality of the second initial coordinate points based on a product between the unknown depth parameter and the plurality of the first coordinate points.

[0147] In some embodiments, the image comprises a frame, and the apparatus further comprises:

[0148] a second execution module configured to repeatedly execute the steps of obtaining the image collected by the camera in the vehicle and obtaining the target distance between the camera and the ground in the image based on the plurality of the second target coordinate points until a plurality of target distances are obtained and / or the plurality of the target distances obtained converge, the plurality of target distances being distances between the camera and the ground in a preset number of frames;

[0149] The monitoring module 370 is further configured to:

[0150] averaging the plurality of target distances to obtain an average target distance;

[0151] determining, according to the average target distance and the reference distance, a monitoring result for representing a degree of deviation of the angle of the camera.

[0152] In some embodiments, the monitoring module 370 is further configured to:

[0153] in a case where the average target distance is higher than a first preset threshold or lower than a second preset threshold, obtaining the monitoring result for representing that the degree of deviation of the angle of the camera is not within an error range, the first preset threshold and the second preset threshold being obtained based on the reference distance;

[0154] The method further comprises:

[0155] outputting prompt information according to the monitoring result.

[0156] As to the device in the above embodiments, the specific manners in which the modules perform operations have been described in detail in the embodiments of the method, and will not be described in detail here.

[0157] The present disclosure also provides a computer-readable storage medium having stored thereon computer program instructions, which, when executed by a processor, implement the steps of the camera monitoring method provided by the present disclosure.

[0158] The apparatus described above can be a part of an independent electronic device, for example, in an embodiment, the apparatus can be an integrated circuit (IC) or a chip, wherein the integrated circuit can be one IC or a collection of multiple ICs; the chip can include but is not limited to the following types: GPU (Graphics Processing Unit), CPU (Central Processing Unit), FPGA (Field Programmable Gate Array), DSP (Digital Signal Processor), ASIC (Application Specific Integrated Circuit), SOC (System on Chip, SoC, System on a Chip or System on Chip), etc. The integrated circuit or chip described above can be used to execute executable instructions (or code) to implement the camera monitoring method described above. The executable instructions can be stored in the integrated circuit or chip, or obtained from other devices or equipment, for example, the integrated circuit or chip includes a processor, a memory, and an interface for communicating with other devices. The executable instructions can be stored in the memory, and when the executable instructions are executed by the processor, the camera monitoring method described above is implemented; or the integrated circuit or chip can receive executable instructions through the interface and transmit them to the processor for execution to implement the camera monitoring method described above.

[0159] In another exemplary embodiment, a computer program product is also provided, which includes a computer program capable of being executed by a programmable device, and the computer program has a code portion for executing the camera monitoring method described above when executed by the programmable device.

[0160] Figure 4 is a block diagram of a vehicle 400 according to an exemplary embodiment. For example, the vehicle 400 can be a hybrid vehicle, an electric vehicle, or other types of vehicles. The vehicle 400 can be an autonomous vehicle or a semi-autonomous vehicle.

[0161] Referring to Figure 4 , the vehicle 400 can include various subsystems, for example, an infotainment system 410, a perception system 420, a decision control system 430, a drive system 440, and a computing platform 450. The vehicle 400 can also include more or fewer subsystems, and each subsystem can include multiple components. In addition, each subsystem of the vehicle 400 and each component can be interconnected by wired or wireless means.

[0162] In some embodiments, infotainment system 410 can include a communication system, an entertainment system, a navigation system, and the like.

[0163] Perception system 420 can include several sensors for sensing information of the environment surrounding vehicle 400. For example, perception system 420 can include a global positioning system (which can be a GPS system, a Beidou system, or other positioning system), an inertial measurement unit (IMU), a lidar, a millimeter wave radar, an ultrasonic radar, and a camera.

[0164] Decision control system 430 can include a computing system, a vehicle controller, a steering system, a throttle, and a braking system.

[0165] Drive system 440 can include components that provide motive power for vehicle 400. In one embodiment, drive system 440 can include an engine, an energy source, a transmission system, and wheels. The engine can be one or a combination of an internal combustion engine, an electric motor, an air compression engine, or the like. The engine can convert energy provided by the energy source into mechanical energy.

[0166] Some or all functions of vehicle 400 are controlled by computing platform 450. Computing platform 450 can include at least one processor 451 and memory 452, and processor 451 can execute instructions 453 stored in memory 452.

[0167] Processor 451 can be any conventional processor, such as commercially available CPUs. The processor can also include a Graphics Processing Unit (GPU), a Field Programmable Gate Array (FPGA), a System on Chip (SOC), an Application Specific Integrated Circuit (ASIC), or a combination thereof.

[0168] Memory 452 can be implemented by any type of volatile or nonvolatile memory or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read only memory (EEPROM), erasable programmable read only memory (EPROM), programmable read only memory (PROM), read only memory (ROM), magnetic memory, flash memory, a magnetic disk, or an optical disk.

[0169] In addition to instructions 453, memory 452 can store data such as road maps, route information, vehicle's position, direction, speed, etc. The data stored by memory 452 can be used by computing platform 450.

[0170] In embodiments of the present disclosure, processor 451 can execute instructions 453 to complete all or part of the steps of the camera monitoring method described above.

[0171] Figure 5 is a block diagram of a camera monitoring apparatus 500 according to an exemplary embodiment. For example, apparatus 500 can be provided as a server. Referring to Figure 5 , apparatus 500 includes a processing component 522, which further includes one or more processors, and a memory resource represented by memory 532, for storing instructions, such as application programs, executable by the processing component 522. The application programs stored in memory 532 can include one or more than one module each corresponding to a set of instructions. In addition, processing component 522 is configured to execute instructions to perform the camera monitoring method described above.

[0172] Apparatus 500 can also include a power supply component 526 configured to perform power management of apparatus 500, a wired or wireless network interface 550 configured to connect apparatus 500 to a network, and an input / output interface 558. Apparatus 500 can operate based on an operating system stored in memory 532, such as Windows Server TM , Mac OS X TM , Unix TM , Linux TM , FreeBSD TM or the like.

[0173] Other embodiments of the present disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the present disclosure. The present disclosure is intended to cover any variations, uses or adaptive changes of the present disclosure following the general principles thereof and including those expressly stated or implied herein. The specification and examples are to be regarded as exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0174] It should be understood that the present disclosure is not limited to the precise structures herein described and illustrated above, and that various modifications and changes can be made without departing from the scope thereof. The scope of the present disclosure is limited only by the claims that follow.

Claims

1. A camera monitoring method characterized by, The method comprises: acquiring an image captured by a camera in a vehicle; extracting a plurality of ground coordinate points included in a ground surface in the image; projecting the plurality of ground coordinate points from a pixel coordinate system constructed based on the image into a camera coordinate system constructed based on the camera, to obtain a plurality of first coordinate points; determining a plurality of second initial coordinate points based on an unknown depth parameter and the plurality of first coordinate points, the depth parameter being used to reflect a depth of each of the second initial coordinate points in the camera coordinate system; projecting the plurality of second initial coordinate points from the camera coordinate system into a vehicle body coordinate system of the vehicle, to obtain a plurality of third coordinate points, and outputting a known depth parameter by taking a coordinate value of each of the third coordinate points in a vertical direction as a constraint; outputting a plurality of second target coordinate points based on the known depth parameter and the plurality of first coordinate points; obtaining a target distance between the camera and the ground surface in the image based on the plurality of second target coordinate points, and determining a monitoring result for representing a degree of offset of an angle of the camera based on the target distance and a reference distance obtained in advance.

2. The camera monitoring method of claim 1, wherein, The extracting a plurality of ground coordinate points included in a ground surface in the image comprises: acquiring a region of interest in the image; performing ground segmentation processing on the region of interest to obtain the ground surface; acquiring a plurality of ground coordinate points included in the ground surface.

3. The camera monitoring method according to claim 1 or 2, characterized in that, The method further comprises: in a case where a number of the plurality of ground coordinate points is greater than or equal to a preset coordinate point number, projecting the plurality of ground coordinate points from the pixel coordinate system into the camera coordinate system to obtain the plurality of first coordinate points; in a case where the number of the plurality of ground coordinate points is less than the preset coordinate point number, repeatedly performing the steps of acquiring the image captured by the camera in the vehicle and extracting the plurality of ground coordinate points included in the ground surface in the image until the number of the plurality of ground coordinate points extracted is greater than or equal to the preset coordinate point number.

4. The camera monitoring method of claim 1, wherein, The obtaining a target distance between the camera and the ground surface in the image based on the plurality of second target coordinate points comprises: determining a plurality of distances between the plurality of second target coordinate points and an origin of the camera coordinate system; performing an average operation on the plurality of distances to obtain the target distance between the camera and the ground surface in the image.

5. The camera monitoring method of claim 1, wherein, The determining a plurality of second initial coordinate points based on an unknown depth parameter and a plurality of first coordinate points comprises: determining the plurality of second initial coordinate points based on a product between the unknown depth parameter and the plurality of first coordinate points.

6. The camera monitoring method of claim 1, wherein, The image comprises one frame, and the method further comprises: repeatedly performing the steps of acquiring the image captured by the camera in the vehicle and obtaining a target distance between the camera and a ground surface in the image based on a plurality of second target coordinate points until a plurality of target distances are obtained and / or the plurality of target distances converge, the plurality of target distances being distances between the camera and ground surfaces in a preset number of frames of images. The monitoring result for representing the deviation degree of the angle of the camera is determined based on the target distance and a reference distance obtained in advance, and the method comprises the following steps: averaging the plurality of target distances to obtain an average target distance; determining the monitoring result for representing the deviation degree of the angle of the camera based on the average target distance and the reference distance.

7. The camera monitoring method of claim 6, wherein, The monitoring result for representing the deviation degree of the angle of the camera is determined based on the average target distance and the reference distance, and the method comprises the following steps: in the case that the average target distance is higher than a first preset threshold value or lower than a second preset threshold value, the monitoring result for representing that the deviation degree of the angle of the camera is not within an error range is obtained, and the first preset threshold value and the second preset threshold value are obtained based on the reference distance; The method further comprises: outputting prompt information according to the monitoring result.

8. A camera monitoring apparatus characterized by comprising: The method comprises the following steps: an acquisition module configured to acquire an image collected by a camera in a vehicle; an extraction module configured to extract a plurality of ground coordinate points included in a ground in the image; a first projection module configured to project the plurality of ground coordinate points from a pixel coordinate system constructed based on the image into a camera coordinate system constructed based on the camera, to obtain a plurality of first coordinate points; a first determination module configured to determine a plurality of second initial coordinate points based on an unknown depth parameter and the plurality of first coordinate points, the depth parameter being used to reflect the depth of each second initial coordinate point in the camera coordinate system; a first output module configured to project the plurality of second initial coordinate points from the camera coordinate system into a vehicle body coordinate system of the vehicle to obtain a plurality of third coordinate points, and output a known depth parameter by taking the coordinate value of each third coordinate point in the vertical direction as zero as a constraint; a second output module configured to output a plurality of second target coordinate points based on the known depth parameter and the plurality of first coordinate points; a monitoring module configured to obtain a target distance between the camera and the ground in the image based on the plurality of second target coordinate points, and determine a monitoring result for representing the deviation degree of the angle of the camera based on the target distance and a reference distance obtained in advance.

9. A camera monitoring apparatus characterized by comprising: The method comprises the following steps: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to: acquire an image collected by a camera in a vehicle; extract a plurality of ground coordinate points included in a ground in the image; project the plurality of ground coordinate points from a pixel coordinate system constructed based on the image into a camera coordinate system constructed based on the camera, to obtain a plurality of first coordinate points; determine a plurality of second initial coordinate points based on an unknown depth parameter and the plurality of first coordinate points, the depth parameter being used to reflect the depth of each second initial coordinate point in the camera coordinate system; project the plurality of second initial coordinate points from the camera coordinate system into a vehicle body coordinate system of the vehicle to obtain a plurality of third coordinate points, and output a known depth parameter by taking the coordinate value of each third coordinate point in the vertical direction as zero as a constraint; Output a plurality of second target coordinate points based on the known depth parameter and the plurality of first coordinate points; Obtain a target distance between the camera and the ground in the image based on the plurality of second target coordinate points, and determine a monitoring result for representing a degree of offset of an angle of the camera based on the target distance and a pre-obtained reference distance.

10. A computer-readable storage medium having stored thereon computer program instructions, wherein, The program instruction is executed by the processor to implement the steps of the method in any one of claims 1-7.

11. A chip, characterized by An interface; the processor is used to read instructions to execute the method in any one of claims 1-7. An interface; the processor is used to read instructions to execute the method in any one of claims 1-7.

Images