Roadside camera displacement detection method and device, electronic equipment and storage medium

Abstract

The invention discloses a roadside camera displacement detection method and device, electronic equipment and a storage medium. The method comprises the following steps: acquiring a first image and a second image in a roadside camera; performing image feature detection and matching on the first image and the second image by adopting a preset feature point detection algorithm and a preset matching algorithm to obtain a homography matrix of feature point pairs; calculating a first re-projection error mean value after the homography matrix change according to the feature points after the homography matrix change and the corresponding feature points of the first image; according to the feature points in the second image and the corresponding feature points of the first image, calculating a second re-projection error mean value which is not subjected to homography matrix change; and determining whether the roadside camera is displaced by comparing the first projection error mean value with the second projection error mean value. According to the invention, detection and automatic correction of the camera pose change are realized.

Description

Technical Field

[0001] The present application relates to the field of camera calibration technology, and in particular to a roadside camera displacement detection method, device, electronic equipment, and storage medium. Background Art

[0002] In the roadside unit, the camera is indispensable as the main perception sensor, and after the camera is installed, the camera's external parameters need to be calibrated. The external parameter calibration plays a decisive role in the positioning of the camera's perception results. After the camera's external parameters are calibrated, the camera's external parameters are theoretically fixed. However, due to the mechanical structure of the camera installation, outdoor wind and gravity, etc., after the camera completes the external parameter calibration, the camera will cause the position change to deviate due to the influence of the above factors for a long time.

[0003] In the related art, if the camera's posture changes, the camera's external parameters will change, causing perceived positioning deviation. In addition, when there are many roadside cameras, manually checking the camera's posture changes and correcting them is a huge task, which consumes a lot of time and labor costs. In addition, it is not convenient to manually check whether a large number of cameras have posture changes every day. Summary of the invention

[0004] The embodiments of the present application provide a roadside camera displacement detection method, device, electronic device, and storage medium to detect and automatically correct camera posture changes.

[0005] The present application embodiment adopts the following technical solutions:

[0006] In a first aspect, an embodiment of the present application provides a roadside camera displacement detection method, wherein the method comprises:

[0007] Acquire a first image and a second image in a roadside camera, wherein the first image at least includes an original image obtained after the roadside camera has been calibrated, and the second image includes a current image in the roadside camera when a change in camera posture needs to be detected;

[0008] Using a preset feature point detection algorithm and a preset matching algorithm, image feature detection and matching are performed on the first image and the second image to obtain a homography matrix of feature point pairs, wherein the homography matrix is ​​used to transform feature points in the second image to obtain feature points after being changed by the homography matrix;

[0009] Calculating a first reprojection error mean after the homography matrix change according to the feature points after the homography matrix change and the corresponding feature points of the first image;

[0010] Calculating a second reprojection error mean after the homography matrix is ​​not changed according to the feature points in the second image and the corresponding feature points in the first image;

[0011] By comparing the first re-projection error mean value with the second re-projection error mean value, it is determined whether the roadside camera is displaced.

[0012] In some embodiments, determining whether the roadside camera is displaced by comparing the first reprojection error mean value with the second reprojection error mean value includes:

[0013] When the first reprojection error mean is greater than the second reprojection error mean, the first reprojection error mean and the second reprojection error mean are subtracted to obtain a first difference. If the first difference is less than a preset judgment threshold, it is determined that the camera has not undergone a posture change.

[0014] If the first difference is greater than a preset threshold, the first reprojection error mean value is significantly different from the second reprojection error mean value but the corrected error is larger, it is determined that the detection correction has failed;

[0015] When the first reprojection error mean is less than or equal to the second reprojection error mean, subtract the first reprojection error mean from the second reprojection error mean to obtain a second difference; if the second difference is less than a preset judgment threshold, the first reprojection error mean is not much different from the second reprojection error mean, and it is judged that the camera has not changed its posture;

[0016] If the second difference is greater than the preset threshold, the first reprojection error mean is significantly different from the second reprojection error mean, the camera detection correction is successful, and the camera corrected reprojection error becomes smaller, and it is judged that the camera posture has changed.

[0017] In some embodiments, after the roadside camera is calibrated, the original external parameter calibration file and the camera internal parameter file of the camera are saved, and the method further includes:

[0018] If it is determined that the roadside camera has changed its posture, the original camera calibration external parameter file is added to the homography matrix so that each pixel point in the image generated by the roadside camera is converted into the pixel coordinate system of the first image in the pixel space through the homography matrix.

[0019] In some embodiments, the first image also includes an image saved after the last correction by the roadside camera, and the method further includes:

[0020] When it is not the first time to detect the change of camera posture, the corrected original image is saved as the image saved after the last correction of the roadside camera and used as the first image;

[0021] When it is not the first time to detect the change in camera posture, the current image in the roadside camera that is reacquired is used as the second image.

[0022] In some embodiments, after performing image feature detection and matching on the first image and the second image using a preset feature point detection algorithm and a preset matching algorithm, the method further includes:

[0023] Dedistort the pixels in the point set according to the camera's internal reference file, update the dedistorted pixel coordinates, and obtain a set of matched feature point pairs;

[0024] Filtering the set of matched feature point pairs, and calculating the mean value D of the relative pixel distances of the feature point pairs in the set of feature point pairs;

[0025] In the process of traversing the set of feature point pairs, calculating the pixel-to-pixel distance of each point pair;

[0026] If the relative pixel distance of each point pair is greater than D+d, it is determined to be a difference feature point pair and is eliminated. The threshold of d can be adjusted.

[0027] In some embodiments, the preset feature point detection algorithm includes:

[0028] Feature point detection is performed on the first image and the second image respectively, and the SuperPoint algorithm is used to detect the feature points.

[0029] In some embodiments, the preset matching algorithm includes:

[0030] The feature points detected by the preset feature point detection algorithm are matched to obtain a set of feature point matching pairs. The SuperGlue algorithm is used to match the feature points.

[0031] In a second aspect, an embodiment of the present application further provides a roadside camera displacement detection device, wherein the device comprises:

[0032] An acquisition module, used to acquire a first image and a second image in a roadside camera, wherein the first image at least includes an original image obtained after the roadside camera has been calibrated, and the second image includes a current image in the roadside camera when a change in camera posture needs to be detected;

[0033] A feature detection and matching module, used to perform image feature detection and matching on the first image and the second image using a preset feature point detection algorithm and a preset matching algorithm, and obtain a homography matrix of feature point pairs, wherein the homography matrix is ​​used to transform feature points in the second image to obtain feature points after being changed by the homography matrix;

[0034] A first calculation module, used for calculating a first reprojection error mean after the homography matrix is ​​changed according to the feature points after the homography matrix is ​​changed and the corresponding feature points of the first image;

[0035] A second calculation module, used for calculating a second reprojection error mean value after the homography matrix is ​​not changed according to the feature points in the second image and the corresponding feature points in the first image;

[0036] A comparison module is used to determine whether the roadside camera is displaced by comparing the first re-projection error mean value with the second re-projection error mean value.

[0037] In a third aspect, an embodiment of the present application further provides an electronic device, comprising: a processor; and a memory arranged to store computer executable instructions, wherein the executable instructions, when executed, cause the processor to perform the above method.

[0038] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, which stores one or more programs. When the one or more programs are executed by an electronic device including multiple application programs, the electronic device executes the above method.

[0039] At least one of the above technical solutions adopted in the embodiment of the present application can achieve the following beneficial effects: by acquiring the first image and the second image in the roadside camera, the first image at least includes the original image obtained after the roadside camera is calibrated, and the second image includes the current image in the roadside camera when the camera posture change needs to be detected. Then, the preset feature point detection algorithm and the preset matching algorithm are used to perform image feature detection and matching on the first image and the second image to obtain the homography matrix of the feature point pair. The homography matrix is ​​used to transform the feature points in the second image to obtain the feature points after the homography matrix change. Further, according to the feature points after the homography matrix change and the corresponding feature points of the first image, the first reprojection error mean after the homography matrix change is calculated; according to the feature points in the second image and the corresponding feature points of the first image, the second reprojection error mean after the homography matrix change is calculated. Finally, by comparing the size between the first reprojection error mean and the second reprojection error mean, it is determined whether the roadside camera has been displaced.

[0040] Through the above method, in the roadside camera perception scenario, by completing the detection and correction of camera posture changes, the problem of inaccurate perception and positioning caused by camera posture changes is solved, and the cost of manually checking the posture changes of a large number of roadside cameras is reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0042] Figure 1 This is a flow chart of a method for detecting displacement of a roadside camera in an embodiment of the present application;

[0043] Figure 2 This is a schematic diagram of the structure of a roadside camera displacement detection device in an embodiment of the present application;

[0044] Figure 3 This is a schematic diagram of the structure of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0046] The technical solutions provided by various embodiments of the present application are described in detail below in conjunction with the accompanying drawings.

[0047] The present application embodiment provides a roadside camera displacement detection method, such as Figure 1 As shown, a schematic flow chart of a roadside camera displacement detection method in an embodiment of the present application is provided, and the method at least includes the following steps S110 to S150:

[0048] Step S110, obtaining a first image and a second image in a roadside camera, wherein the first image at least includes an original image obtained after calibration of the roadside camera, and the second image includes a current image in the roadside camera when a change in camera posture needs to be detected.

[0049] The first image at least includes: the original image obtained by the roadside camera after calibration, that is, the original image captured by the camera after calibration. The second image includes the current image obtained by the roadside camera when the camera posture change needs to be detected.

[0050] In specific implementation, after the roadside camera is installed and external parameter calibration is completed: capture an image and save it as origin.png. At the same time, save the camera's external parameter calibration file and camera internal parameter file. And when the camera posture change needs to be detected for the first time, capture a camera image and save it as new.png.

[0051] Step S120, using a preset feature point detection algorithm and a preset matching algorithm, perform image feature detection and matching on the first image and the second image to obtain a homography matrix of feature point pairs, wherein the homography matrix is ​​used to transform the feature points in the second image to obtain feature points after being changed by the homography matrix.

[0052] For the feature point detection algorithm and feature point matching algorithm preset for the first image and the second image, image feature detection and matching are performed on the feature points in the first image and the second image to obtain a homography matrix of the feature point pairs. By solving the homography matrix H, the feature points after the homography matrix change can be converted. Specifically, through the homography matrix, the feature points in the second image are converted to obtain the feature points after the homography matrix change.

[0053] Step S130, calculating a first reprojection error mean after the homography matrix is ​​changed according to the feature points after the homography matrix is ​​changed and the corresponding feature points of the first image.

[0054] According to the solved homography matrix, the feature points calculated by the image new.png (the second image) are transformed, and then the average distance distance_pixel_1 of the pixels is calculated with the corresponding feature points calculated by the image origin.png (the first image).

[0055] It can be understood that the average distance of pixels distance_pixel_1 is also the mean value of the first reprojection error.

[0056] Step S140, calculating a second reprojection error mean after the homography matrix is ​​not changed according to the feature points in the second image and the corresponding feature points in the first image.

[0057] The feature points calculated for the image new.png (the second image) are not transformed by the homography matrix H, and the average distance distance_pixel_2 of the pixels corresponding to the feature points calculated for the image origin.png (the first image) is calculated.

[0058] It can be understood that the average distance of pixels distance_pixel_2 is also the mean value of the second reprojection error.

[0059] Step S150: Determine whether the roadside camera is displaced by comparing the first re-projection error mean value with the second re-projection error mean value.

[0060] Based on the difference between the first reprojection error mean and the second reprojection error mean, it can be further determined whether the roadside camera has been displaced. Alternatively, it can be determined that the detection and correction have failed considering other situations.

[0061] Specifically, when the posture change of the roadside camera is detected and corrected, it is reflected in the pixel coordinate system of the camera, that is, the change of pixels. The corresponding relationship between the pixel changes of the two images can be detected to determine whether the pixel coordinate system has changed. If so, the posture of the camera has changed. If not, the posture of the camera has not changed.

[0062] Through the above method, feature point detection is performed on the two images in the roadside camera, and then feature point matching is performed. According to the corresponding feature point pairs, the homography change of the two images is obtained. Based on the homography transformation, the corresponding feature point pairs are changed to obtain the reprojection error. The pixel difference between the two images is judged according to the reprojection error. If it reaches the preset threshold, it is judged that the posture of the two images has changed, otherwise the two images have not undergone displacement changes.

[0063] Through the above method, the first image and the second image in the roadside camera are obtained, the first image at least includes the original image obtained after the roadside camera is calibrated, and the second image includes the current image in the roadside camera when the camera posture change needs to be detected. For the two images of the first image and the second image, one of the images is a screenshot when the external parameter calibration is completed after the camera is installed, and the other image is an image captured at the moment of detection when the camera posture change detection is required. In addition, the above method can also meet the needs of continuous detection of roadside camera displacement, that is, if continuous detection in multiple time periods is to be performed, it is also necessary to add an intermediate transition image as the image captured during the last detection.

[0064] Through the above method, the first reprojection error mean after the homography matrix change is calculated based on the feature points after the homography matrix change and the corresponding feature points of the first image; the second reprojection error mean before the homography matrix change is calculated based on the feature points in the second image and the corresponding feature points of the first image; by comparing the first reprojection error mean with the second reprojection error mean, it is determined whether the roadside camera has been displaced. The pixel difference between the two images is judged based on the reprojection error. If it reaches a preset threshold, it is determined that the two images have undergone a posture change, otherwise the two images have not undergone a displacement change.

[0065] In one embodiment of the present application, the method of determining whether the roadside camera has been displaced by comparing the first reprojection error mean and the second reprojection error mean includes: when the first reprojection error mean is greater than the second reprojection error mean, subtracting the first reprojection error mean and the second reprojection error mean to obtain a first difference; if the first difference is less than a preset judgment threshold, it is determined that the camera has not changed its posture; if the first difference is greater than the preset threshold, the first reprojection error mean is significantly different from the second reprojection error mean but the corrected error is larger, then it is determined that Detection and correction failed; when the first reprojection error mean is less than or equal to the second reprojection error mean, the first reprojection error mean and the second reprojection error mean are subtracted to obtain a second difference. If the second difference is less than the preset judgment threshold, the first reprojection error mean and the second reprojection error mean are not much different, and it is judged that the camera has not changed its posture; if the second difference is greater than the preset threshold, the first reprojection error mean and the second reprojection error mean are significantly different, the camera detection and correction is successful, and the camera's corrected reprojection error becomes smaller, and it is judged that the camera has changed its posture.

[0066] To facilitate comparison, we can obtain the following judgment result: if the mean value of the first reprojection error distance_pixel_1 is greater than the mean value of the second reprojection error distance_pixel_2, it is judged that the camera has no posture change.

[0067] In order to accurately obtain the judgment result: if the first reprojection error mean value distance_pixel_1 is less than or equal to the second reprojection error mean value distance_pixel_2, and the reprojection error mean value distance_pixel is greater than the preset judgment threshold value distance_no_move, it is judged that the camera has a posture change.

[0068] In order to facilitate comparison and obtain the judgment result: if the reprojection error mean distance_pixel is less than or equal to the preset judgment threshold distance_no_move, it is judged that the camera has not produced a posture change.

[0069] If none of the above situations apply, it is considered that the camera displacement detection and correction has failed.

[0070] The preset judgment threshold distance_no_move is obtained according to the actual situation or is an empirical value.

[0071] In one embodiment of the present application, the roadside camera saves the original external parameter calibration file and the camera internal parameter file after calibration processing. The method also includes: if it is determined that the roadside camera has a posture change, the original camera calibration external parameter file is added to the homography matrix to convert each pixel point in the image generated by the roadside camera in the pixel space to the pixel coordinate system of the first image through the homography matrix.

[0072] Specifically, if it is determined based on the above process that the camera has undergone a posture change, the current camera that has undergone the posture change is corrected in the pixel space through the solved homography matrix H. The correction method includes but is not limited to adding the homography matrix H to the original camera calibration external parameter file. Each pixel point of the image generated by the subsequent camera must be transformed in the pixel space through the homography matrix and converted to the origin.png (first image) pixel coordinate system.

[0073] In one embodiment of the present application, the first image also includes an image saved after the last correction by the roadside camera, and the method also includes: when it is not the first time to detect the camera posture change, the image saved after the last correction by the roadside camera is saved separately as the first image; when it is not the first time to detect the camera posture change, the current image in the roadside camera that is re-obtained is used as the second image.

[0074] Specifically, when it is not the first time to detect the camera pose change, imagetmp.png (the image saved after the last correction of the roadside camera) is saved as the first image (origin.png), and then the camera image during the current detection is re-captured as new.png (the second image). The operation steps are repeated later, and the two images for feature point calculation are changed to imagetmp.png (the image saved after the last correction of the roadside camera) and the new new.png (the second image) captured during the current detection. The calculated homography H needs to be updated accordingly in the external parameter file.

[0075] In one embodiment of the present application, after the preset feature point detection algorithm and the preset matching algorithm are used to perform image feature detection and matching on the first image and the second image, the method further includes: dedistorting the pixels in the point set according to the camera's internal reference file, updating the dedistorted pixel coordinates, and obtaining a set of matched feature point pairs; filtering the set of matched feature point pairs, and calculating the mean D of the relative pixel distances of the feature point pairs in the set of feature point pairs; in the process of traversing the set of feature point pairs, calculating the pixel-to-pixel distance of each point pair; if the pixel-to-pixel distance of each point pair is greater than D+d, it is determined to be a difference feature point pair and is eliminated, and the threshold of d can be adjusted.

[0076] In order to improve the quality of the image, the point pairs in the set of matched feature point pairs are filtered. The filtering algorithm used includes: removing the point pairs with too large relative pixel distance of feature points, and calculating the mean value D of the relative pixel distance of feature point pairs in the set of feature point pairs. Then traverse the point pair set, calculate the relative pixel distance of each point pair, if it is greater than D+d, it is judged as a feature point pair with too large difference and removed, where the threshold of d can be adjusted according to the actual situation.

[0077] In one embodiment of the present application, the preset feature point detection algorithm includes: performing feature point detection on the first image and the second image respectively, and detecting the feature points using a SuperPoint algorithm.

[0078] The SuperPoint (Self-Supervised Interest Point Detection and Description) algorithm is used to detect feature points.

[0079] In one embodiment of the present application, the preset matching algorithm includes: matching feature points detected by a preset feature point detection algorithm to obtain a set of feature point matching pairs, and matching feature points using a SuperGlue algorithm.

[0080] The SuperGlue (Learning Feature Matching with Graph Neural Networks) algorithm is used to match the detected feature points to obtain a set of feature point matching pairs. Then, the pixels in the point set are dedistorted according to the camera's internal reference file, and the dedistorted pixel coordinates are updated.

[0081] Preferably, the SuperPoint algorithm and the SuperGlue algorithm are used simultaneously for the detection and correction of the posture change of the roadside camera. The SuperPoint algorithm is used to solve the accurate feature points of the two images of the camera module of the roadside unit respectively, and then the feature points are accurately matched by SuperGlue.

[0082] The present application also provides a roadside camera displacement detection device 200, such as Figure 2 As shown, a schematic diagram of the structure of a roadside camera displacement detection device in an embodiment of the present application is provided. The roadside camera displacement detection device 200 at least includes: an acquisition module 210, a feature detection and matching module 220, a first calculation module 230, a second calculation module 240 and a comparison module 250, wherein:

[0083] In one embodiment of the present application, the acquisition module 210 is specifically used to: acquire a first image and a second image in a roadside camera, wherein the first image at least includes the original image obtained after the roadside camera is calibrated, and the second image includes the current image in the roadside camera when it is necessary to detect a change in camera posture.

[0084] The first image at least includes: the original image obtained by the roadside camera after calibration. That is, the original image captured by the camera after calibration. The second image includes the current image obtained by the roadside camera when the camera posture change needs to be detected for the first time.

[0085] In specific implementation, after the roadside camera is installed and external parameter calibration is completed: capture an image and save it as origin.png. At the same time, save the camera's external parameter calibration file and camera internal parameter file. And when the camera posture change needs to be detected for the first time, capture a camera image and save it as new.png.

[0086] In one embodiment of the present application, the feature detection and matching module 220 is specifically used to: use a preset feature point detection algorithm and a preset matching algorithm to perform image feature detection and matching on the first image and the second image to obtain a homography matrix of feature point pairs, and the homography matrix is ​​used to transform the feature points in the second image to obtain feature points after the homography matrix changes.

[0087] For the feature point detection algorithm and feature point matching algorithm preset for the first image and the second image, image feature detection and matching are performed on the feature points in the first image and the second image to obtain a homography matrix of the feature point pairs. By solving the homography matrix H, the feature points after the homography matrix change can be converted. Specifically, through the homography matrix, the feature points in the second image are converted to obtain the feature points after the homography matrix change.

[0088] In one embodiment of the present application, the first calculation module 230 is specifically used to calculate a first reprojection error mean after the homography matrix change based on the feature points after the homography matrix change and the corresponding feature points of the first image.

[0089] According to the solved homography matrix, the feature points calculated by the image new.png (the second image) are transformed, and then the average distance distance_pixel_1 of the pixels is calculated with the corresponding feature points calculated by the image origin.png (the first image).

[0090] It can be understood that the average distance of pixels distance_pixel_1 is also the mean value of the first reprojection error.

[0091] In one embodiment of the present application, the second calculation module 240 is specifically used to calculate the second reprojection error mean after the homography matrix is ​​not changed according to the feature points in the second image and the corresponding feature points in the first image.

[0092] The feature points calculated for the image new.png (the second image) are not transformed by the homography matrix H, and the average distance distance_pixel_2 of the pixels corresponding to the feature points calculated for the image origin.png (the first image) is calculated.

[0093] It can be understood that the average distance of pixels distance_pixel_2 is also the mean value of the second reprojection error.

[0094] In one embodiment of the present application, the comparison module 250 is specifically used to determine whether the roadside camera is displaced by comparing the first re-projection error mean value with the second re-projection error mean value.

[0095] Based on the difference between the first reprojection error mean and the second reprojection error mean, it can be further determined whether the roadside camera has been displaced. Alternatively, it can be determined that the detection and correction have failed considering other situations.

[0096] Specifically, when the posture change of the roadside camera is detected and corrected, it is reflected in the pixel coordinate system of the camera, that is, the change of pixels. The corresponding relationship between the pixel changes of the two images can be detected to determine whether the pixel coordinate system has changed. If so, the posture of the camera has changed. If not, the posture of the camera has not changed.

[0097] In one embodiment of the present application, the comparison module 250 is also used to

[0098] If the first re-projection error mean is greater than the second re-projection error mean, it is determined that the roadside camera has not changed its posture;

[0099] If the first reprojection error mean is greater than the second reprojection error mean, and both the first reprojection error mean and the second reprojection error mean are greater than a preset judgment threshold, it is determined that a posture change of the camera occurs;

[0100] If the first re-projection error mean and the second re-projection error mean are both less than or equal to the preset judgment threshold, it is determined that a posture change occurs to the camera.

[0101] In one embodiment of the present application, it further includes: a conversion module for

[0102] If it is determined that the roadside camera has not changed its posture, the original camera calibration external parameter file is added to the homography matrix so that each pixel point in the image generated by the roadside camera in the pixel space is converted to the pixel coordinate system of the first image through the homography matrix.

[0103] In one embodiment of the present application, it further includes: a continuous detection module for

[0104] When it is not the first time to detect the change of camera posture, the image saved after the last correction of the roadside camera is saved as the first image;

[0105] When it is not the first time to detect the change in camera posture, the current image in the roadside camera that is reacquired is used as the second image.

[0106] In one embodiment of the present application, it further includes: a de-distortion module for

[0107] Dedistort the pixels in the point set according to the camera's internal reference file, update the dedistorted pixel coordinates, and obtain a set of matched feature point pairs;

[0108] Filtering the set of matched feature point pairs, and calculating the mean value D of the relative pixel distances of the feature point pairs in the set of feature point pairs;

[0109] In the process of traversing the set of feature point pairs, calculating the pixel-to-pixel distance of each point pair;

[0110] If the relative pixel distance of each point pair is greater than D+d, it is determined to be a difference feature point pair and is eliminated. The threshold of d can be adjusted.

[0111] In one embodiment of the present application, the feature detection and matching module 220 is also used to

[0112] Feature point detection is performed on the first image and the second image respectively, and the SuperPoint algorithm is used to detect the feature points.

[0113] In one embodiment of the present application, the feature detection and matching module 220 is also used to

[0114] The feature points detected by the preset feature point detection algorithm are matched to obtain a set of feature point matching pairs. The SuperGlue algorithm is used to match the feature points.

[0115] It can be understood that the above-mentioned roadside camera displacement detection device can implement the various steps of the roadside camera displacement detection method provided in the aforementioned embodiment. The relevant explanations about the roadside camera displacement detection method are applicable to the roadside camera displacement detection device and will not be repeated here.

[0116] Figure 3 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. Figure 3 At the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and a memory. The memory may include a memory, such as a high-speed random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk storage. Of course, the electronic device may also include hardware required for other services.

[0117] The processor, network interface and memory can be interconnected through an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0118] The memory is used to store the program. Specifically, the program may include a program code, and the program code includes a computer operation instruction. The memory may include a memory and a non-volatile memory, and provides instructions and data to the processor.

[0119] The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it, forming a roadside camera displacement detection device at the logical level. The processor executes the program stored in the memory and is specifically used to perform the following operations:

[0120] Acquire a first image and a second image in a roadside camera, wherein the first image at least includes an original image obtained after the roadside camera has been calibrated, and the second image includes a current image in the roadside camera when a change in camera posture needs to be detected;

[0121] Using a preset feature point detection algorithm and a preset matching algorithm, image feature detection and matching are performed on the first image and the second image to obtain a homography matrix of feature point pairs, wherein the homography matrix is ​​used to transform feature points in the second image to obtain feature points after being changed by the homography matrix;

[0122] Calculating a first reprojection error mean after the homography matrix change according to the feature points after the homography matrix change and the corresponding feature points of the first image;

[0123] Calculating a second reprojection error mean after the homography matrix is ​​not changed according to the feature points in the second image and the corresponding feature points in the first image;

[0124] By comparing the first re-projection error mean value with the second re-projection error mean value, it is determined whether the roadside camera is displaced.

[0125] The above application Figure 1 The method performed by the roadside camera displacement detection device disclosed in the illustrated embodiment can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor or instructions in the form of software. The above processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in conjunction with the embodiments of the present application can be directly embodied as being executed by a hardware decoding processor, or executed by a combination of hardware and software modules in a decoding processor. The software module can be located in a storage medium mature in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0126] The electronic device may also perform Figure 1 The method is implemented by the roadside camera displacement detection device, and the roadside camera displacement detection device is implemented in Figure 1 The functions of the illustrated embodiment will not be described in detail in the embodiments of the present application.

[0127] The present application also provides a computer-readable storage medium, which stores one or more programs, wherein the one or more programs include instructions, which, when executed by an electronic device including multiple application programs, enable the electronic device to execute Figure 1 The method performed by the roadside camera displacement detection device in the embodiment shown is specifically used to perform:

[0128] Acquire a first image and a second image in a roadside camera, wherein the first image at least includes an original image obtained after the roadside camera has been calibrated, and the second image includes a current image in the roadside camera when a change in camera posture needs to be detected;

[0129] Using a preset feature point detection algorithm and a preset matching algorithm, image feature detection and matching are performed on the first image and the second image to obtain a homography matrix of feature point pairs, wherein the homography matrix is ​​used to transform feature points in the second image to obtain feature points after being changed by the homography matrix;

[0130] Calculating a first reprojection error mean after the homography matrix change according to the feature points after the homography matrix change and the corresponding feature points of the first image;

[0131] Calculating a second reprojection error mean after the homography matrix is ​​not changed according to the feature points in the second image and the corresponding feature points in the first image;

[0132] By comparing the first re-projection error mean value with the second re-projection error mean value, it is determined whether the roadside camera is displaced.

[0133] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0134] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0135] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0136] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0137] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0138] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0139] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0140] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0141] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0142] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A roadside camera displacement detection method, wherein: The method comprises: Acquire a first image and a second image in a roadside camera, wherein the first image at least includes an original image obtained after the roadside camera has been calibrated, and the second image includes a current image in the roadside camera when a change in camera posture needs to be detected; Using a preset feature point detection algorithm and a preset matching algorithm, image feature detection and matching are performed on the first image and the second image to obtain a homography matrix of feature point pairs, wherein the homography matrix is ​​used to transform feature points in the second image to obtain feature points after being changed by the homography matrix; Calculate a first reprojection error mean after the homography matrix is ​​changed according to the feature points after the homography matrix is ​​changed and the corresponding feature points of the first image; Calculating a second reprojection error mean after the homography matrix is ​​not changed according to the feature points in the second image and the corresponding feature points in the first image; By comparing the first re-projection error mean value with the second re-projection error mean value, it is determined whether the roadside camera is displaced.

2. The method of claim 1, wherein: The determining whether the roadside camera is displaced by comparing the first reprojection error mean value with the second reprojection error mean value includes: In the case where the first reprojection error mean is greater than the second reprojection error mean, the first reprojection error mean and the second reprojection error mean are subtracted to obtain a first difference. If the first difference is less than a preset judgment threshold, it is determined that the camera has not changed its posture. If the first difference is greater than the preset threshold, the first reprojection error mean is significantly different from the second reprojection error mean but the corrected error is larger, it is determined that the detection correction has failed. When the first reprojection error mean is less than or equal to the second reprojection error mean, subtract the first reprojection error mean from the second reprojection error mean to obtain a second difference; if the second difference is less than a preset judgment threshold, the first reprojection error mean is not much different from the second reprojection error mean, and it is judged that the camera has not changed its posture; If the second difference is greater than the preset threshold, the first reprojection error mean is significantly different from the second reprojection error mean, the camera detection correction is successful, and the camera corrected reprojection error becomes smaller, and it is judged that the camera posture has changed.

3. The method of claim 2, wherein: After the roadside camera is calibrated, the original external parameter calibration file and the camera internal parameter file of the camera are saved. The method further includes: If it is determined that the roadside camera has changed its posture, the original camera calibration external parameter file is added to the homography matrix so that each pixel point in the image generated by the roadside camera is converted into the pixel coordinate system of the first image in the pixel space through the homography matrix.

4. The method of claim 2, wherein: The first image also includes an image saved after the last correction by the roadside camera, and the method further includes: When it is not the first time to detect the change of camera posture, the image saved after the last correction of the roadside camera is saved as the first image; When it is not the first time to detect the change in camera posture, the current image in the roadside camera that is reacquired is used as the second image.

5. The method of claim 1, wherein: After the preset feature point detection algorithm and the preset matching algorithm are used to perform image feature detection and matching on the first image and the second image, the method further includes: Dedistort the pixels in the point set according to the camera's internal reference file, update the dedistorted pixel coordinates, and obtain a set of matched feature point pairs; Filtering the set of matched feature point pairs, and calculating the mean value D of the relative pixel distances of the feature point pairs in the set of feature point pairs; In the process of traversing the set of feature point pairs, calculating the pixel-to-pixel distance of each point pair; If the relative pixel distance of each point pair is greater than D+d, it is determined to be a difference feature point pair and is eliminated. The threshold of d can be adjusted.

6. The method of claim 1, wherein: The preset feature point detection algorithm includes: Feature point detection is performed on the first image and the second image respectively, and the SuperPoint algorithm is used to detect the feature points.

7. The method of claim 6, wherein: The preset matching algorithm includes: The feature points detected by the preset feature point detection algorithm are matched to obtain a set of feature point matching pairs. The SuperGlue algorithm is used to match the feature points.

8. A roadside camera displacement detection device, wherein: The device comprises: An acquisition module, used to acquire a first image and a second image in a roadside camera, wherein the first image at least includes an original image obtained after the roadside camera has been calibrated, and the second image needs to detect a current image in the roadside camera when the camera posture changes; A feature detection and matching module, used to perform image feature detection and matching on the first image and the second image using a preset feature point detection algorithm and a preset matching algorithm, and obtain a homography matrix of feature point pairs, wherein the homography matrix is ​​used to transform feature points in the second image to obtain feature points after being changed by the homography matrix; A first calculation module, used for calculating a first reprojection error mean after the homography matrix is ​​changed according to the feature points after the homography matrix is ​​changed and the corresponding feature points of the first image; A second calculation module, used for calculating a second reprojection error mean value after the homography matrix is ​​not changed according to the feature points in the second image and the corresponding feature points in the first image; A comparison module is used to determine whether the roadside camera is displaced by comparing the first re-projection error mean value with the second re-projection error mean value.

9. An electronic device, comprising: processor; as well as A memory arranged to store computer executable instructions, which when executed cause the processor to perform the method of any one of claims 1 to 7.

10. A computer-readable storage medium storing one or more programs, which, when executed by an electronic device including a plurality of application programs, causes the electronic device to execute any one of the methods of claims 1 to 7.

Images