Camera offset detection method and device, electronic equipment and computer readable storage medium

Abstract

The invention provides a camera offset detection method and device, electronic equipment and a computer readable storage medium. The method comprises the following steps: respectively acquiring two original images at the same moment by using two adjacent cameras on a vehicle; performing image conversion on the two original images to obtain two aerial view images; and if the pixel deviation value between the two aerial view images exceeds a target threshold value, determining that the two adjacent cameras are deviated. According to the invention, the offset of the camera on the vehicle can be found in time, and the real-time performance and accuracy of camera offset detection are improved.

Description

Technical Field

[0001] This application relates to the field of autonomous driving technology, and in particular to a camera offset detection method, device, electronic device, and computer-readable storage medium. Background Technology

[0002] Currently, during vehicle use, various reasons may cause the onboard camera to shift (such as a shift in the camera's orientation or a shift in its installation position), which may lead to deviations in object recognition. For example, when a vehicle is performing automatic parking, it needs to use images captured by the camera to perceive the environment. Due to the camera's own shift, the detected parking signs or obstacles may be inaccurate.

[0003] Because camera misalignment can be difficult to detect visually, professional testing equipment is required to determine if the camera is misaligned. This means that users are unaware of the problem after the camera has shifted, potentially leading to safety hazards when using intelligent driving functions. Therefore, finding a more convenient way to determine if a camera is misaligned without using professional testing equipment has become an urgent problem to solve. Summary of the Invention

[0004] This application provides a camera offset detection method, apparatus, electronic device, and computer-readable storage medium, which can improve the timeliness and accuracy of camera offset detection.

[0005] The technical solution of this application embodiment is implemented as follows:

[0006] This application provides a camera offset detection method, the method comprising:

[0007] Two raw images are captured simultaneously using two adjacent cameras on the vehicle.

[0008] Two bird's-eye view images are obtained by performing image conversion on the two original images;

[0009] If the pixel offset between the two bird's-eye view images exceeds the target threshold, it is determined that the two adjacent cameras have shifted.

[0010] This application embodiment provides a camera offset detection device, the device comprising:

[0011] The acquisition module is used to acquire two raw images simultaneously using two adjacent cameras on the vehicle.

[0012] The conversion module is used to convert the two original images to obtain two bird's-eye view images;

[0013] The detection module is used to determine that the two adjacent cameras have shifted if the pixel offset value between the two bird's-eye view images exceeds a target threshold.

[0014] Optionally, the detection module is further configured to, after performing image conversion on the two original images to obtain two bird's-eye view images, determine the target region of the two bird's-eye view images; the target region is located in the overlapping region between the two bird's-eye view images; and perform phase correlation matching and / or pixel feature matching on the region images corresponding to the target region in the two bird's-eye view images respectively to determine the pixel offset value between the two bird's-eye view images.

[0015] Optionally, the detection module is further configured to perform phase correlation matching on the region images corresponding to the target region in the two bird's-eye view images respectively to determine a first pixel offset value; perform pixel feature matching on the region images corresponding to the target region in the two bird's-eye view images respectively to determine a second pixel offset value; and determine the pixel offset value between the two bird's-eye view images based on the average of the first pixel offset value and the second pixel offset value.

[0016] Optionally, the detection module is further configured to determine the structural similarity between the regional images corresponding to the target region in the two bird's-eye view images; and if the structural similarity is greater than or equal to a preset similarity threshold, to perform phase correlation matching and / or pixel feature matching on the regional images corresponding to the target region in the two bird's-eye view images to determine the pixel offset value between the two bird's-eye view images.

[0017] Optionally, the detection module is further configured to update a pixel offset queue based on the pixel offset value between the two bird's-eye view images; the pixel offset queue contains multiple pixel offset values; the multiple pixel offset values ​​are determined based on multiple pairs of original images captured by the two adjacent cameras at multiple identical times; the average queue offset value is determined based on the multiple pixel offset values; and the average queue offset value is used to determine whether the two adjacent cameras have shifted.

[0018] Optionally, the detection module is further configured to accumulate an offset count when the average offset of the queue is greater than or equal to a preset offset threshold; and to determine that the two adjacent cameras have shifted when the offset count is greater than or equal to a preset count threshold.

[0019] Optionally, the detection module is further configured to determine abnormal pixel offset values ​​based on the difference between each pixel offset value in the pixel offset queue and the average offset value of the queue; the abnormal pixel offset value represents the pixel offset value of an abnormal value; and delete the abnormal pixel offset value from the pixel offset queue.

[0020] Optionally, the detection module is further configured to determine the queue offset variance corresponding to the pixel offset queue based on the plurality of pixel offset values; and to determine the pixel offset values ​​whose difference from the mean of the queue offset is greater than or equal to a preset multiple of the queue offset variance as the abnormal pixel offset values.

[0021] Optionally, the pixel offset value includes: the pixel offset value in the first coordinate dimension and the pixel offset value in the second coordinate dimension; the queue offset mean includes: the queue offset mean in the first coordinate dimension and the queue offset mean in the second coordinate dimension; the queue offset variance includes: the queue offset variance in the first coordinate dimension and the queue offset variance in the second coordinate dimension; the detection module is further configured to determine the pixel offset value as the abnormal pixel offset when the difference between the pixel offset value and the queue offset mean in the first coordinate dimension is greater than or equal to a preset multiple of the queue offset variance in the first coordinate dimension, or when the difference between the pixel offset value and the queue offset mean in the second coordinate dimension is greater than or equal to a preset multiple of the queue offset variance in the second coordinate dimension.

[0022] Optionally, the camera offset detection device further includes a prompting module and / or a calibration module; the prompting module is used to generate a camera offset prompting message when it is determined that the two adjacent cameras have offset, and / or to recalibrate the camera parameters of at least one of the two adjacent cameras.

[0023] This application provides an electronic device, the electronic device comprising:

[0024] Memory is used to store executable instructions for a computer;

[0025] The processor, when executing computer-executable instructions stored in the memory, implements the camera offset detection method provided in the embodiments of this application.

[0026] This application provides a vehicle that includes the camera offset detection device provided in this application, or the electronic device provided in this application.

[0027] This application provides a computer-readable storage medium storing a computer program or computer-executable instructions, which, when executed by a processor, implements the camera offset detection method provided in this application.

[0028] This application provides a computer program product, including a computer program or computer executable instructions. When the computer program or computer executable instructions are executed by a processor, they implement the camera offset detection method provided in this application.

[0029] The embodiments of this application have the following beneficial effects:

[0030] By using two adjacent cameras on the vehicle, two raw images are captured simultaneously. These two raw images are then transformed into two bird's-eye view images. If the pixel offset between the two bird's-eye view images exceeds a target threshold, it is determined that the two adjacent cameras have shifted, achieving real-time camera shift detection. This allows for timely identification of camera shift status and improves the real-time performance of camera shift detection. Therefore, it is more convenient to determine whether a camera is shifted without relying on specialized detection equipment. Attached Figure Description

[0031] Figure 1 This is a schematic flowchart of an optional camera offset detection method provided in an embodiment of this application;

[0032] Figure 2 This is a schematic diagram of the overlapping area provided in the embodiments of this application;

[0033] Figure 3 This is a schematic flowchart of an optional camera offset detection method provided in an embodiment of this application;

[0034] Figure 4 This is a schematic diagram of the target area in the overlapping region provided in the embodiments of this application;

[0035] Figure 5 This is a schematic flowchart illustrating the application of the camera offset detection method provided in this application to a real-world scenario.

[0036] Figure 6 This is a schematic diagram of an optional structure of the camera offset detection device provided in the embodiments of this application;

[0037] Figure 7 This is a schematic diagram of an optional structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0038] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0039] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0040] In the following description, the terms "first, second, third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0041] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0042] Unless otherwise defined, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in the embodiments of this application is for descriptive purposes only and is not intended to limit the scope of this application.

[0043] In the implementation of this application, the collection and processing of relevant data should strictly comply with the requirements of relevant national laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of laws and regulations and the authorization of the personal information subject.

[0044] Currently, vehicle cameras are typically calibrated at the factory, based on their installation position and orientation, setting both intrinsic and extrinsic parameters. However, cameras may shift during use. For example, collisions or changes in air suspension height can alter the camera's installation position and orientation, causing it to shift. Furthermore, since camera calibration is usually based on the assumption that the vehicle is on a flat surface, situations involving inclines, declines, speed bumps, or severe road roughness can deviate from this assumption, leading to camera shift.

[0045] Especially in automated parking scenarios, when a vehicle is automatically parking, it uses onboard surround-view cameras to capture images of the parking environment. These images are then fused together for parking perception and recognition to detect parking signs or obstacles in the actual environment. If the cameras are misaligned, the position of the target object in the fused image will be inaccurate, leading to parking failure or a minor collision.

[0046] Current technologies cannot detect camera shift in real time; they typically only alert the user when parking becomes impossible, resulting in poor timeliness. Furthermore, in real parking environments, three-dimensional objects exhibit varying degrees of deformation or stretching in images captured from different camera perspectives. When using image processing methods such as image comparison or matching for camera shift detection, these environmental factors significantly interfere with the image processing, reducing the accuracy of camera shift detection.

[0047] This application provides a camera offset detection method, apparatus, electronic device, and computer-readable storage medium, which can filter out the influence of environmental factors in the real environment and achieve real-time camera offset detection, improving the real-time performance and accuracy of camera offset detection. Therefore, it is more convenient to determine whether a camera is offset without using specialized detection equipment.

[0048] The camera offset detection method provided in this application can be applied to electronic devices. In some embodiments, the electronic device may include a terminal or a server. For example, the terminal may include an in-vehicle computer, a Body Control Module (BCM), or other vehicle-mounted terminals; the server may include an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, network services, and big data and artificial intelligence platforms. The specific choice depends on the actual situation, and this application does not limit the choice. When the electronic device is implemented as a server, at least two cameras configured in the vehicle send at least two raw image streams to the server. The server converts the two raw images captured simultaneously by two adjacent cameras into two bird's-eye view images and determines the pixel offset value between the two bird's-eye view images. If the pixel offset value between the two bird's-eye view images exceeds a target threshold, it is determined that the two adjacent cameras have offset. Further, the server can send the camera offset detection result to the terminal.

[0049] refer to Figure 1 , Figure 1 This is a schematic diagram of an optional flow of the camera offset detection method provided in an embodiment of this application. Figure 1As shown, the camera offset detection method provided in this application embodiment can be implemented by executing the process S101-S103, as follows:

[0050] S101. Using two adjacent cameras on the vehicle, two original images are captured at the same time.

[0051] In this embodiment, at least two cameras are configured or installed on the vehicle. Each of the at least two cameras acquires images based on its installation position and orientation, resulting in at least two original images from at least two different viewpoints. During camera offset detection, two adjacent cameras from the at least two cameras acquire images simultaneously, resulting in two original images acquired by each of the two adjacent cameras at that same time.

[0052] For example, at least two cameras may include surround-view fisheye cameras on the vehicle body. Four fisheye cameras are respectively located at the front, rear, left, and right sides of the vehicle, providing a 360° view of the surrounding environment to check for obstacles around the vehicle. Alternatively, at least two cameras may also include conventional cameras deployed at at least two locations on the vehicle body, etc. The specific choice depends on the actual situation, and this application embodiment does not limit the scope.

[0053] In this embodiment, the two adjacent cameras among the at least two cameras can include any two adjacent cameras among the at least two cameras. Thus, the camera offset detection method of this embodiment can perform offset detection on any two adjacent cameras among the at least two cameras; alternatively, it can perform offset detection on all cameras among the at least two cameras by traversing the adjacent pairs among the at least two cameras. In the case of traversing at least two cameras, the same process is used to perform camera offset detection on each pair of adjacent cameras traversed.

[0054] S102. Perform image conversion on the two original images to obtain two bird's-eye view images.

[0055] In some embodiments, for each of two adjacent cameras, an inverse perspective transformation can be used to convert the original image corresponding to that camera into a bird's-eye view image based on pre-calibrated intrinsic and extrinsic parameters, thereby obtaining two bird's-eye view images corresponding to the two adjacent cameras. Exemplarily, the intrinsic parameters may include parameters such as the camera's focal length, distortion coefficient, and resolution, while the extrinsic parameters may include the rotation and translation matrices from the camera to the vehicle body. Exemplarily, the camera's intrinsic and extrinsic parameters may be pre-recorded in the vehicle's system.

[0056] Since adjacent cameras capture images from their respective viewpoints, converting the two original images from these cameras into two bird's-eye view images allows them to be unified into the same coordinate system, enabling the determination of pixel offsets between them within that same coordinate system. For example, the two bird's-eye view images can belong to the same world coordinate system, or they can both belong to the camera coordinate system corresponding to one of the two adjacent cameras. This allows the acquisition of overlapping areas (or shared viewing areas) in the images captured by the two adjacent cameras. Here, the overlapping area represents the overlapping shooting range of the two adjacent cameras. It can be understood that the subjects in the overlapping area are the same, thus allowing subsequent image matching and pixel offset calculations to be performed using the overlapping area in the bird's-eye view images.

[0057] For example, taking at least two cameras, including four cameras deployed at the front, rear, left, and right of the vehicle body, as an example, the original images corresponding to these four cameras are converted into bird's-eye view images, and the determined overlapping areas can be as follows: Figure 2 As shown in the diagram. The dashed area 10 represents the overlapping area between the adjacent front camera and the left camera; the dashed area 20 represents the overlapping area between the adjacent front camera and the right camera; the dashed area 30 represents the overlapping area between the adjacent rear camera and the left camera; and the dashed area 40 represents the overlapping area between the adjacent rear camera and the right camera.

[0058] S103. If the pixel offset between two bird's-eye view images exceeds the target threshold, it is determined that the two adjacent cameras have shifted.

[0059] In this embodiment, the pixel offset value between two bird's-eye view images corresponding to two adjacent cameras can be determined. The pixel offset value represents the difference in pixel position of matching pixels in the two bird's-eye view images within the same coordinate system. Based on the degree of pixel position difference, it can be determined whether an offset has occurred between the two adjacent cameras.

[0060] In some embodiments, if the pixel offset value between two bird's-eye view images exceeds a target threshold, it is determined that two adjacent cameras have shifted. If the pixel offset value between two bird's-eye view images does not exceed the target threshold, images can continue to be acquired for the next offset detection.

[0061] Understandably, by using two adjacent cameras on the vehicle to simultaneously capture two raw images, and then performing image transformation on these two raw images to obtain two bird's-eye view images, if the pixel offset between the two bird's-eye view images exceeds a target threshold, it is determined that the two adjacent cameras have shifted. This achieves real-time camera shift detection, enabling timely identification of camera shift status and improving the real-time performance of camera shift detection. In summary, this improves the real-time performance and accuracy of camera shift detection. Therefore, it allows for a more convenient determination of whether a camera is shifted without the need for specialized detection equipment.

[0062] In some embodiments, after performing image conversion on the two original images in S102 to obtain two bird's-eye view images, before executing S103, it is also possible to perform image conversion on the two original images as described above. Figure 3 The process shown in S201-S202 determines the pixel offset value between two bird's-eye view images, and determines whether two adjacent cameras have shifted based on the pixel offset value between the two bird's-eye view images, as follows:

[0063] S201. Determine the target area from the two bird's-eye view images.

[0064] In some embodiments, the target area is located in the overlapping area between two bird's-eye view images. Exemplarily, the target area may include a region of interest (ROI). The position and size of the target area can be determined in advance based on the shooting range of two adjacent cameras, ensuring that the target area is located in the overlapping area between the two bird's-eye view images. Exemplarily, based on... Figure 2 ,like Figure 4 As shown, the target area may include ROI: 100 in overlapping area 1, ROI: 200 in overlapping area 2, ROI: 300 in overlapping area 3, and ROI: 400 in overlapping area 4.

[0065] S202. Perform phase correlation matching and / or pixel feature matching on the corresponding regional images of the target area in the two bird's-eye view images respectively to determine the pixel offset value between the two bird's-eye view images.

[0066] For example, phase correlation matching can be performed on the corresponding regional images of the target region in two bird's-eye view images to determine the pixel offset between the two bird's-eye view images. Alternatively, pixel feature matching can be performed on the corresponding regional images of the target region in two bird's-eye view images to determine the pixel offset between the two bird's-eye view images. Alternatively, phase correlation matching can be performed on the corresponding regional images of the target region in two bird's-eye view images to determine a first pixel offset; pixel feature matching can be performed on the corresponding regional images of the target region in two bird's-eye view images to determine a second pixel offset; and the pixel offset between the two bird's-eye view images can be determined based on the average of the first pixel offset and the second pixel offset.

[0067] In this embodiment, phase correlation matching is a method that transforms two bird's-eye view images to the frequency domain, determines matching image points in the two bird's-eye view images by matching pixels near frequency peaks in the frequency domain, and then calculates the distance between the matching image points to obtain the pixel offset value. Pixel feature matching is a method that extracts and matches corner features of pixels in two bird's-eye view images to determine matching image points in the two bird's-eye view images, and then calculates the distance between the matching image points to obtain the pixel offset value. In practical applications, at least one of phase correlation matching and pixel feature matching can be selected for pixel matching according to the actual situation. The specific selection depends on the actual situation, and this embodiment does not limit the choice.

[0068] In some embodiments, the process of determining the pixel offset value between two bird's-eye view images based on the average of the first pixel offset value and the second pixel offset value can be represented by formula (1), as follows:

[0069]

[0070] In formula (1), dis phasecorrelate Dis represents the first pixel offset value corresponding to the pixel in the target region. orbmatching The value represents the second pixel offset corresponding to a pixel in the target region, and the distance represents the pixel offset between the two bird's-eye view images obtained by calculating the average of the first and second pixel offsets corresponding to pixels in the target region.

[0071] Understandably, by determining the pixel offset between two bird's-eye view images through phase correlation matching and pixel feature matching of the target area, the pixel position difference of the matched pixels in the same coordinate system in the two bird's-eye view images corresponding to two adjacent cameras can be detected. Thus, it is more convenient to determine whether the camera has an offset without the use of professional detection equipment.

[0072] In some embodiments, before performing phase correlation matching and / or pixel feature matching on the corresponding regional images of the target region in the two bird's-eye view images, a structural similarity check can be performed on the two bird's-eye view images. If the structural similarity check passes, phase correlation matching and / or pixel feature matching can be performed on the corresponding regional images of the target region in the two bird's-eye view images to determine the pixel offset between the two bird's-eye view images.

[0073] Here, the deformation of the same three-dimensional object in a real environment differs depending on the camera's perspective. For example, with adjacent front and left cameras, the same three-dimensional object in the real environment, such as a person's leg, is stretched in different directions in the front and left cameras, resulting in significant differences between the two bird's-eye view images.

[0074] In this embodiment, a structural similarity check is performed on two bird's-eye view images to determine the structural similarity check result. If the structural similarity check result is satisfactory, it indicates that the similarity between the two bird's-eye view images is high and the interference from environmental factors is small. Therefore, it can be ensured that the pixel offset calculated from the two bird's-eye view images is caused by camera offset, rather than by interference objects (such as three-dimensional objects) in the images. This allows for further determination of whether two adjacent cameras are offset based on the pixel offset.

[0075] In some embodiments, a structural similarity index (SSIM) can be calculated between two bird's-eye view images. For example, the SSIM is calculated based on the overlapping region between the two bird's-eye view images. If the SSIM is greater than or equal to a preset similarity threshold, the structural similarity check is passed. This allows phase correlation matching and / or pixel feature matching to be performed on the corresponding region images of the target region in the two bird's-eye view images to determine the pixel offset between the two images.

[0076] If the structural similarity index is less than a preset similarity threshold, the structural similarity check is deemed unsuccessful. It should be noted that if the structural similarity check fails, the pixel offset between the two bird's-eye view images is not calculated. The electronic device then continues to perform a structural similarity check on the two new bird's-eye view images obtained from the next transformation.

[0077] It is understandable that by screening bird's-eye view images for calculating pixel offset values ​​through structural similarity checks, the interference of adverse factors such as three-dimensional objects in the bird's-eye view images on the calculation of pixel offset values ​​can be avoided or mitigated, thereby improving the accuracy of pixel offset value calculation and thus improving the accuracy of camera offset detection based on pixel offset values.

[0078] In some embodiments, two adjacent cameras continuously acquire images in real time, with each camera capturing one raw image stream. Based on the two raw image streams corresponding to the two adjacent cameras, multiple pairs of raw images captured by the two cameras at multiple simultaneous moments can be determined. Each pair of raw images includes raw images captured by the two adjacent cameras at the same simultaneous moment. Processing each pair of raw images using the method provided in this application, exemplarily, involves bird's-eye view image conversion and pixel offset value calculation to obtain the pixel offset value corresponding to each pair of raw images, thus obtaining multiple pixel offset values. A pixel offset queue is used to record these multiple pixel offset values, thereby allowing determination of whether the two adjacent cameras have shifted based on these multiple pixel offset values.

[0079] In some embodiments, after determining the pixel offset value between two bird's-eye view images, a pixel offset queue may contain multiple pixel offset values; the multiple pixel offset values ​​are determined based on multiple pairs of original images captured by two adjacent cameras at multiple identical times; the average queue offset value corresponding to the pixel offset queue is determined based on the multiple pixel offset values; and the average queue offset value is used to determine whether two adjacent cameras have shifted.

[0080] In some embodiments, a pixel offset queue of a preset length can be pre-set, the preset length of the pixel offset queue being equivalent to a preset number of pixel offset values ​​that can be recorded therein. The pixel offset values ​​between two corresponding bird's-eye view images at each same time are recorded in the pixel offset queue, thereby updating the pixel offset queue. For example, the pixel offset queue can be updated in a first-in, first-out manner, maintaining the recording of the most recently calculated preset number of pixel offset values. By calculating the average of the preset number of pixel offsets currently included in the pixel offset queue, the queue offset mean can be determined, and then based on the queue offset mean, it can be determined whether two adjacent cameras have shifted.

[0081] In some embodiments, if the average queue offset is greater than or equal to a preset offset threshold, it can be determined that two adjacent cameras have shifted.

[0082] Alternatively, in some embodiments, an offset count may be accumulated once if the average offset of the queue is greater than or equal to a preset offset threshold; if the offset count is greater than or equal to a preset count threshold, it may be determined that two adjacent cameras have shifted.

[0083] For example, the offset count is initially 0. Each time the pixel offset queue is updated, the average offset of the queue is calculated. If the average offset is greater than or equal to a preset offset threshold, the offset count is incremented by 1. If the offset count is greater than or equal to the preset count threshold, such as greater than 300, it is determined that two adjacent cameras have offset. It should be noted that the preset count threshold can be set according to actual conditions, and is not specifically limited here.

[0084] In some embodiments, after determining the mean offset of the pixel offset queue based on the pixel offset values ​​in the pixel offset queue, abnormal pixel offset values ​​can also be determined based on the difference between each pixel offset value in the pixel offset queue and the mean offset of the queue; abnormal pixel offset values ​​represent pixel offset values ​​with abnormal values; abnormal pixel offset values ​​are deleted from the pixel offset queue.

[0085] Here, the pixel offset values ​​calculated in the above process may contain some outliers that fall outside the normal offset value range. To reduce the impact of outliers on the calculation of the average offset in the queue, we can determine whether a pixel offset value is an outlier by comparing the difference between each pixel offset value in the pixel offset queue and the average offset in the queue. If the pixel offset is not an outlier, it is deleted from the pixel offset queue; otherwise, it is retained in the pixel offset queue.

[0086] For example, the queue offset variance corresponding to the pixel offset queue can be determined based on multiple pixel offset values ​​in the pixel offset queue; for a pixel offset value in the pixel offset queue, if the difference between the pixel offset value and the mean of the queue offset is greater than or equal to a preset multiple of the queue offset variance, the pixel offset value is determined to be an abnormal pixel offset value.

[0087] Here, outliers or singular values ​​in the pixel offset queue can be identified by calculating the mean and variance of the pixel offset queue, and then removed from the pixel offset queue.

[0088] In some embodiments, the pixel offset value includes: a pixel offset value in the first coordinate dimension and a pixel offset value in the second coordinate dimension; the queue offset mean includes: the queue offset mean in the first coordinate dimension and the queue offset mean in the second coordinate dimension; the queue offset variance includes: the queue offset variance in the first coordinate dimension and the queue offset variance in the second coordinate dimension; for a pixel offset value in the pixel offset queue, if the difference between the pixel offset value and the queue offset mean in the first coordinate dimension is greater than or equal to a preset multiple of the queue offset variance in the first coordinate dimension, or if the difference between the pixel offset value and the queue offset mean in the second coordinate dimension is greater than or equal to a preset multiple of the queue offset variance in the second coordinate dimension, the pixel offset value is determined to be an abnormal pixel offset value.

[0089] For example, |diff.x| represents the difference between the pixel offset value and the mean queue offset in the first coordinate dimension (e.g., the x-axis); |diff.y| represents the difference between the pixel offset value and the mean queue offset in the second coordinate dimension (e.g., the y-axis); std.x represents the variance of the queue offset in the first coordinate dimension; std.y represents the variance of the queue offset in the second coordinate dimension; taking a preset multiple of 2 as an example (other values ​​can also be used, which are not limited here), the pixel offset queue can be maintained as follows:

[0090] For a pixel offset value in the pixel offset queue, if the difference |diff.x| between the pixel offset value and the queue mean offset in the first coordinate dimension is greater than or equal to twice the queue offset variance std.x in the first coordinate dimension, or if the difference |diff.y| between the pixel offset value and the queue mean offset in the second coordinate dimension is greater than or equal to twice the queue offset variance std.y in the second coordinate dimension, the pixel offset value is deleted from the pixel offset queue, thereby updating the pixel offset queue. If the difference |diff.x| between the pixel offset value and the queue mean offset in the first coordinate dimension is less than twice the queue offset variance std.x in the first coordinate dimension, or if the difference |diff.y| between the pixel offset value and the queue mean offset in the second coordinate dimension is less than twice the queue offset variance std.y in the second coordinate dimension, the pixel offset value is retained in the pixel offset queue.

[0091] In some embodiments, after removing abnormal pixel offset values ​​from the pixel offset queue, the queue offset mean of the pixel offset queue is recalculated to determine whether two adjacent cameras have been offset based on the queue offset mean.

[0092] In some embodiments, if it is determined that two adjacent cameras are offset, a camera offset prompt message is generated, and / or, the camera parameters of at least one of the two adjacent cameras are recalibrated.

[0093] In other words, when the camera offset detection method described in this application determines that two adjacent cameras have shifted, a prompt message can be generated and displayed to the user, indicating that the camera has shifted and needs to be recalibrated. Alternatively, at least one of the two adjacent cameras can be calibrated online to re-determine the intrinsic and extrinsic parameters of at least one of the two adjacent cameras.

[0094] In some embodiments, after recalibrating the camera parameters of at least one of two adjacent cameras, the intrinsic and / or extrinsic parameters recorded in the vehicle system can be adjusted accordingly based on the intrinsic and / or extrinsic parameters of the camera obtained through recalibration.

[0095] For example, see Figure 5 , Figure 5 This is a schematic diagram illustrating the application of the camera offset detection method according to an embodiment of this application in a real-world scenario. Figure 5 As shown, four surround-view fisheye cameras installed on the vehicle body acquire four surround-view fisheye images (front, rear, left, and right). Based on the surround-view fisheye camera parameters (intrinsic and extrinsic parameters), the fisheye images are converted into four bird's-eye images using an inverse perspective transformation method. These four bird's-eye images are then input into a camera offset detection algorithm. The algorithm calculates the pixel offset between each pair of adjacent bird's-eye images according to the method described in this embodiment, and then determines whether each pair of adjacent cameras has shifted based on the pixel offset between them. If shift is detected, online calibration is initiated or the user is prompted for after-sales calibration; otherwise, a new round of camera offset detection is performed based on the four new surround-view fisheye images, continuing online monitoring.

[0096] It is understood that the embodiments of this application are based on structural similarity screening scenarios, and calculate the camera offset using a phase correlation method, a corner feature extraction and matching method, and a mean-variance statistical method. This improves the timeliness of surround-view camera offset detection, enabling timely identification of camera offset status and subsequent measures, thereby improving the success rate of automatic parking and the user experience.

[0097] This application provides a camera offset detection device, such as... Figure 6 As shown, the camera offset detection device 1 may include:

[0098] The acquisition module 11 is used to acquire two raw images at the same time using two adjacent cameras on the vehicle.

[0099] The conversion module 12 is used to convert the two original images to obtain two bird's-eye view images;

[0100] The detection module 13 is used to determine that the two adjacent cameras have shifted if the pixel offset value between the two bird's-eye view images exceeds the target threshold.

[0101] In some embodiments, the detection module 13 is further configured to, after performing image conversion on the two original images to obtain two bird's-eye view images, determine the target region of the two bird's-eye view images; the target region is located in the overlapping region between the two bird's-eye view images; perform phase correlation matching and / or pixel feature matching on the region images corresponding to the target region in the two bird's-eye view images respectively, and determine the pixel offset value between the two bird's-eye view images.

[0102] In some embodiments, the detection module 13 is further configured to perform phase correlation matching on the region images corresponding to the target region in the two bird's-eye view images respectively to determine a first pixel offset value; perform pixel feature matching on the region images corresponding to the target region in the two bird's-eye view images respectively to determine a second pixel offset value; and determine the pixel offset value between the two bird's-eye view images based on the average of the first pixel offset value and the second pixel offset value.

[0103] In some embodiments, the detection module 13 is further configured to determine the structural similarity between the regional images corresponding to the target region in the two bird's-eye view images; and, if the structural similarity is greater than or equal to a preset similarity threshold, to perform phase correlation matching and / or pixel feature matching on the regional images corresponding to the target region in the two bird's-eye view images to determine the pixel offset value between the two bird's-eye view images.

[0104] In some embodiments, the detection module 13 is further configured to update a pixel offset queue based on the pixel offset value between the two bird's-eye view images; the pixel offset queue contains multiple pixel offset values; the multiple pixel offset values ​​are determined based on multiple pairs of original images captured by the two adjacent cameras at multiple times at the same time; the average queue offset value corresponding to the pixel offset queue is determined based on the multiple pixel offset values; and the average queue offset value is used to determine whether the two adjacent cameras have shifted.

[0105] In some embodiments, the detection module 13 is further configured to update a pixel offset queue based on the pixel offset value between the two bird's-eye view images; the pixel offset queue contains multiple pixel offset values; the multiple pixel offset values ​​are determined based on multiple pairs of original images captured by the two adjacent cameras at multiple times at the same time; the average queue offset value corresponding to the pixel offset queue is determined based on the multiple pixel offset values; and the average queue offset value is used to determine whether the two adjacent cameras have shifted.

[0106] In some embodiments, the detection module 13 is further configured to accumulate an offset count when the average offset of the queue is greater than or equal to a preset offset threshold; and to determine that the two adjacent cameras have shifted when the offset count is greater than or equal to a preset count threshold.

[0107] In some embodiments, the detection module 13 is further configured to determine abnormal pixel offset values ​​based on the difference between each pixel offset value in the pixel offset queue and the average offset value of the queue; the abnormal pixel offset value represents the pixel offset value of an abnormal value; and delete the abnormal pixel offset value from the pixel offset queue.

[0108] In some embodiments, the pixel offset value includes: the pixel offset value in the first coordinate dimension and the pixel offset value in the second coordinate dimension; the queue offset mean includes: the queue offset mean in the first coordinate dimension and the queue offset mean in the second coordinate dimension; the queue offset variance includes: the queue offset variance in the first coordinate dimension and the queue offset variance in the second coordinate dimension; the detection module 13 is further configured to determine the pixel offset value as the abnormal pixel offset when the difference between the pixel offset value and the queue offset mean in the first coordinate dimension is greater than or equal to a preset multiple of the queue offset variance in the first coordinate dimension, or when the difference between the pixel offset value and the queue offset mean in the second coordinate dimension is greater than or equal to a preset multiple of the queue offset variance in the second coordinate dimension.

[0109] In some embodiments, the camera offset detection device further includes a prompting module and / or a calibration module; the prompting module is used to generate a camera offset prompting message when it is determined that the two adjacent cameras have offset, and / or to recalibrate the camera parameters of at least one of the two adjacent cameras.

[0110] It should be noted that the descriptions of the above device embodiments are similar to those of the above method embodiments, and have similar beneficial effects. For technical details not disclosed in the device embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0111] This application provides an electronic device, such as... Figure 7 As shown, the electronic device 2 may include: a memory 22 and a processor 23. The memory 22 and the processor 23 are connected via a communication bus 24; the memory 22 stores executable instructions; the processor 22, when executing the executable instructions stored in the memory 22, implements the camera offset detection method provided in this application embodiment.

[0112] This application embodiment provides a vehicle, the vehicle including as follows: Figure 6 The aforementioned camera offset detection device, or includes, as shown above, the camera offset detection device, or ... Figure 7 The aforementioned electronic device is shown.

[0113] This application provides a computer-readable storage medium storing executable instructions, wherein the executable instructions are stored and when executed by a processor, the processor will execute any of the camera offset detection methods provided in this application.

[0114] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0115] In some embodiments, executable instructions may take the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0116] As an example, executable instructions may, but do not necessarily, correspond to files in the file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., a file that stores one or more modules, subroutines, or code sections).

[0117] As an example, executable instructions can be deployed to execute on a single computing device, or on multiple computing devices located in one location, or on multiple computing devices distributed across multiple locations and interconnected via a communication network.

[0118] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A method for detecting camera offset, characterized in that, The method includes: Two raw images are captured simultaneously using two adjacent cameras on the vehicle. Two bird's-eye view images are obtained by performing image conversion on the two original images; If the pixel offset between the two bird's-eye view images exceeds the target threshold, it is determined that the two adjacent cameras have shifted.

2. The method according to claim 1, characterized in that, After performing image conversion on the two original images to obtain two bird's-eye view images, the method further includes: Determine the target region from the two bird's-eye view images; the target region is located in the overlapping area between the two bird's-eye view images; Phase correlation matching and / or pixel feature matching are performed on the corresponding region images of the target region in the two bird's-eye view images to determine the pixel offset value between the two bird's-eye view images.

3. The method according to claim 2, characterized in that, Phase correlation matching and pixel feature matching are performed on the corresponding region images of the target region in the two bird's-eye view images respectively to determine the pixel offset value between the two bird's-eye view images, including: Phase correlation matching is performed on the region images corresponding to the target region in the two bird's-eye view images respectively to determine the first pixel offset value; Pixel feature matching is performed on the region images corresponding to the target region in the two bird's-eye views to determine the second pixel offset value; The pixel offset value between the two bird's-eye view images is determined based on the average of the first pixel offset value and the second pixel offset value.

4. The method according to claim 2, characterized in that, The step of performing phase correlation matching and / or pixel feature matching on the corresponding region images of the target region in the two bird's-eye view images to determine the pixel offset value between the two bird's-eye view images includes: Determine the structural similarity between the region images corresponding to the target region in the two bird's-eye view images respectively; If the structural similarity is greater than or equal to a preset similarity threshold, phase correlation matching and / or pixel feature matching are performed on the corresponding regional images of the target region in the two bird's-eye view images to determine the pixel offset value between the two bird's-eye view images.

5. The method according to any one of claims 1-4, characterized in that, The method further includes: The pixel offset queue is updated based on the pixel offset value between the two bird's-eye view images; the pixel offset queue contains multiple pixel offset values; the multiple pixel offset values ​​are determined based on multiple pairs of original images captured by the two adjacent cameras at multiple identical times; Based on the multiple pixel offset values, determine the average queue offset corresponding to the pixel offset queue; Based on the average queue offset, determine whether two adjacent cameras have shifted.

6. The method according to claim 5, characterized in that, Determining whether two adjacent cameras have shifted based on the average queue offset includes: If the average queue offset is greater than or equal to a preset offset threshold, an offset count is accumulated once. If the offset count is greater than or equal to a preset count threshold, it is determined that the two adjacent cameras have shifted.

7. The method according to claim 5, characterized in that, The method further includes: Abnormal pixel offset values ​​are determined based on the difference between each pixel offset value in the pixel offset queue and the average offset value of the queue; the abnormal pixel offset values ​​represent pixel offset values ​​with abnormal values. Remove the abnormal pixel offset value from the pixel offset queue.

8. The method according to claim 7, characterized in that, The step of determining abnormal pixel offset values ​​based on the difference between each pixel offset value in the pixel offset queue and the average offset value of the queue includes: Based on the plurality of pixel offset values, determine the queue offset variance corresponding to the pixel offset queue; Pixel offset values ​​whose difference from the mean of the queue offset is greater than or equal to a preset multiple of the variance of the queue offset are identified as abnormal pixel offset values.

9. The method according to claim 8, characterized in that, The pixel offset value includes: the pixel offset value in the first coordinate dimension and the pixel offset value in the second coordinate dimension; the queue offset mean includes: the queue offset mean in the first coordinate dimension and the queue offset mean in the second coordinate dimension; the queue offset variance includes: the queue offset variance in the first coordinate dimension and the queue offset variance in the second coordinate dimension; determining pixel offset values ​​whose difference from the queue offset mean is greater than or equal to a preset multiple of the queue offset variance as abnormal pixel offset values ​​includes: If the difference between the pixel offset value and the mean queue offset in the first coordinate dimension is greater than or equal to a preset multiple of the variance of the queue offset in the first coordinate dimension, or if the difference between the pixel offset value and the mean queue offset in the second coordinate dimension is greater than or equal to a preset multiple of the variance of the queue offset in the second coordinate dimension, the pixel offset value is determined to be the abnormal pixel offset.

10. The method according to any one of claims 1-4, or any one of claims 6-9, characterized in that, The method further includes: If it is determined that the two adjacent cameras are offset, a camera offset prompt message is generated, and / or, the camera parameters of at least one of the two adjacent cameras are recalibrated.

11. A camera offset detection device, characterized in that, The device includes: The acquisition module is used to acquire two raw images simultaneously using two adjacent cameras on the vehicle. The conversion module is used to convert the two original images to obtain two bird's-eye view images; The detection module is used to determine that the two adjacent cameras have shifted if the pixel offset value between the two bird's-eye view images exceeds a target threshold.

12. An electronic device, characterized in that, The electronic device includes: Memory is used to store executable instructions for a computer; A processor, when executing computer-executable instructions stored in the memory, implements the method according to any one of claims 1 to 10.

13. A vehicle, characterized in that, The vehicle includes the camera offset detection device as described in claim 11, or the electronic device as described in claim 12.

14. A computer-readable storage medium storing computer-executable instructions, characterized in that, When the computer-executable instructions or computer program are executed by a processor, they implement the method described in any one of claims 1 to 10.

15. A computer program product comprising computer-executable instructions, characterized in that, When the computer-executable instructions or computer program are executed by a processor, they implement the method according to any one of claims 1 to 10.

Images