Method for checking vehicle-mounted camera and vehicle

Abstract

This application provides a method for verifying an in-vehicle camera and a vehicle. The method includes: acquiring an image sequence captured by the in-vehicle camera and motion information of the vehicle during vehicle operation; determining a residual optical flow based on the image sequence and motion information, wherein the residual optical flow characterizes the optical flow generated by the translational motion of the vehicle determined based on the image sequence; and verifying the in-vehicle camera based on the residual optical flow to obtain a target verification result, wherein the target verification result includes at least whether the installation direction of the in-vehicle camera is correct and the extrinsic parameter adjustment amount of the in-vehicle camera, the extrinsic parameter adjustment amount representing the adjustment amount of the rotation parameters of the in-vehicle camera. This application solves the technical problem of low accuracy in verifying in-vehicle cameras in related technologies.

Description

Technical Field

[0001] This application relates to the field of vehicle technology, and more specifically, to a method for verifying an in-vehicle camera and a vehicle. Background Technology

[0002] Currently, in-vehicle surround view and driver assistance systems generally rely on offline factory calibration and manual verification. During assembly, repair, or replacement, cameras are prone to issues such as incorrect installation orientation (e.g., confusion between front / back and left / right), misconnected wiring harness ports, or slight deviations in external parameters due to vibration or temperature drift. Solutions in related technologies often rely on specially designed calibration boards, fixed fixtures, or manual verification based solely on physical port markings. These methods are difficult to implement for online self-checking during actual vehicle operation and lack intelligent judgment capabilities regarding installation orientation. In complex environments such as low-texture, strong light, rain, and fog, calibration methods based on single-frame image features or template matching are prone to failure and struggle to distinguish between optical flow changes caused by pure rotation and translation, making it difficult to accurately identify the camera's true orientation. This results in low accuracy for calibrating in-vehicle cameras in related technologies.

[0003] There is currently no good solution to the above problems. Summary of the Invention

[0004] This application provides a method for verifying vehicle-mounted cameras and a vehicle, so as to at least solve the technical problem of low accuracy in verifying vehicle-mounted cameras in related technologies.

[0005] According to one aspect of the embodiments of this application, a method for verifying an in-vehicle camera is provided, comprising: acquiring an image sequence captured by the in-vehicle camera and motion information of the vehicle during vehicle operation; determining a residual optical flow based on the image sequence and motion information, wherein the residual optical flow is used to characterize the optical flow generated by the translational motion of the vehicle determined based on the image sequence; and verifying the in-vehicle camera based on the residual optical flow to obtain a target verification result, wherein the target verification result includes at least whether the installation direction of the in-vehicle camera is correct and the external parameter adjustment amount of the in-vehicle camera, the external parameter adjustment amount being used to represent the adjustment amount of the rotation parameters of the in-vehicle camera.

[0006] In the above embodiments of this application, determining the residual optical flow based on the image sequence and motion information includes: determining the optical flow field of two adjacent frames in the image sequence based on a preset optical flow calculation model to obtain a dense optical flow field; determining the pixel motion vector generated by the vehicle's rotational motion based on the motion information; removing the pixel motion vector from the dense optical flow field to obtain the residual optical flow; preferably, determining the pixel motion vector generated by the vehicle's rotational motion based on the motion information includes: performing an integral operation on the angular velocity parameter in the motion information using the vehicle's inertial measurement unit to obtain a rotation matrix; and determining the pixel motion vector based on the rotation matrix.

[0007] In the above embodiments of this application, the optical flow field of two adjacent frames in an image sequence is determined based on a preset optical flow calculation model to obtain a dense optical flow field. This includes: downsampling two adjacent frames in the image sequence to obtain two adjacent downsampled images; using a convolutional neural network with shared weights in the preset optical flow calculation model to extract multi-scale features of the two adjacent downsampled images to obtain feature maps of the two adjacent frames; and determining the dense optical flow field based on the feature maps of the two adjacent frames and the preset optical flow calculation model.

[0008] In the above embodiments of this application, the dense optical flow field is determined based on two adjacent feature maps and a preset optical flow calculation model, including: obtaining the pixel displacement offset between two adjacent feature maps, wherein the pixel displacement offset is used to represent the pixel position offset distance of the target pixel between the two adjacent feature maps; constructing an association volume based on the two adjacent feature maps and the pixel displacement offset, wherein the association volume is used to characterize the similarity between corresponding pixels in the two adjacent feature maps; and using the gated loop unit in the preset optical flow calculation model, determining the optical flow changes caused by the vehicle's motion based on the association volume to obtain the dense optical flow field.

[0009] In the above embodiments of this application, the vehicle-mounted camera is verified based on residual optical flow to obtain target verification results, including: constructing the expected optical flow corresponding to each expected direction of the vehicle to obtain multiple expected optical flow templates, wherein the expected direction is the direction pointing from the vehicle as the starting point to the surrounding area of ​​the vehicle; determining multiple matching degrees between the residual optical flow and the multiple expected optical flow templates; determining the target verification result based on the multiple matching degrees; preferably, determining multiple matching degrees between the residual optical flow and the multiple expected optical flow templates includes: obtaining the direction histogram similarity, optical flow field focus consistency, divergence and / or curl feature consistency, and dynamic target interference between the residual optical flow and the multiple expected optical flow templates; determining multiple matching degrees based on the direction histogram similarity, optical flow field focus consistency, divergence and / or curl feature consistency, and dynamic target interference.

[0010] In the above embodiments of this application, the target verification result is determined based on multiple matching degrees, including: when multiple matching degrees are all less than or equal to a preset matching degree, the target verification result is determined to be that the installation direction of the vehicle camera is incorrect; when the target matching degree among multiple matching degrees is greater than the preset matching degree, the target verification result is determined to be that the installation direction of the vehicle camera is correct, and the external parameter adjustment amount is determined.

[0011] In the above embodiments of this application, determining the extrinsic parameter adjustment amount includes: determining the desired camera in the desired direction corresponding to the target matching degree; obtaining the nominal extrinsic parameters of the desired camera, wherein the nominal extrinsic parameters are used to characterize the rotation parameters of the vehicle camera after factory calibration; determining the model optical flow corresponding to the desired camera based on the nominal extrinsic parameters and motion information; determining the extrinsic parameter adjustment amount based on the residual optical flow and the model optical flow; preferably, determining the residual optical flow and the model optical flow, and determining the extrinsic parameter adjustment amount includes: constructing a cost function based on the difference between the residual optical flow and the model optical flow; and using a preset iterative algorithm to iteratively solve the cost function to obtain the extrinsic parameter adjustment amount.

[0012] In the above embodiments of this application, the method further includes: disabling target functions related to the vehicle camera when the target verification result indicates that the installation direction of the vehicle camera is incorrect, wherein the target functions include at least one of the following: surround view stitching and automatic parking assistance.

[0013] In the above embodiments of this application, the method further includes: adjusting the rotation parameters of the vehicle camera based on the external parameter adjustment amount when the external parameter adjustment amount is less than the preset adjustment amount; and generating a prompt message when the external parameter adjustment amount is greater than or equal to the preset adjustment amount, wherein the prompt message is used to prompt the user to perform offline calibration of the vehicle camera.

[0014] According to another aspect of the embodiments of this application, a verification device for an in-vehicle camera is also provided, comprising: an acquisition module, configured to acquire an image sequence captured by the in-vehicle camera and motion information of the vehicle during vehicle operation; a determination module, configured to determine a residual optical flow based on the image sequence and motion information, wherein the residual optical flow is used to characterize the optical flow generated by the translational motion of the vehicle determined based on the image sequence; and a verification module, configured to verify the in-vehicle camera based on the residual optical flow to obtain a target verification result, wherein the target verification result includes at least whether the installation direction of the in-vehicle camera is correct, and the external parameter adjustment amount of the in-vehicle camera, the external parameter adjustment amount being used to represent the adjustment amount of the rotation parameters of the in-vehicle camera.

[0015] According to another aspect of the embodiments of this application, a vehicle is also provided, including: a memory storing an executable program; and a processor for running the program, wherein the program executes the methods in various embodiments of this application when it runs.

[0016] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored executable program, wherein, when the executable program is running, it controls the device where the computer-readable storage medium is located to perform the methods of various embodiments of this application.

[0017] According to another aspect of the embodiments of this application, a computer program product is also provided, including a computer program that, when executed by a processor, implements the methods of various embodiments of this application.

[0018] According to another aspect of the embodiments of this application, a computer program product is also provided, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the methods in various embodiments of this application.

[0019] According to another aspect of the embodiments of this application, a computer program is also provided, which, when executed by a processor, implements the methods of the various embodiments of this application.

[0020] In this embodiment of the application, during the vehicle's movement, firstly, an image sequence captured by the vehicle-mounted camera and the vehicle's motion information are acquired; then, based on the image sequence and motion information, a residual optical flow is determined, wherein the residual optical flow is used to characterize the optical flow generated by the vehicle's translational motion, determined based on the image sequence; finally, based on the residual optical flow, the vehicle-mounted camera is verified to obtain a target verification result, wherein the target verification result includes at least whether the installation direction of the vehicle-mounted camera is correct, and the external parameter adjustment amount of the vehicle-mounted camera, the external parameter adjustment amount being used to represent the adjustment amount of the rotation parameters of the vehicle-mounted camera. This application acquires image sequences and vehicle motion information simultaneously during vehicle operation, models the motion information, and removes pixel motion caused by vehicle rotation. This extracts residual optical flow that reflects only vehicle translation, effectively removing interference components and making the optical flow signal more realistically represent the relative motion relationship between the scene and the vehicle. By using residual optical flow to verify the vehicle-mounted camera, the correctness of the camera's installation orientation can be accurately identified, and the adjustment amount of the vehicle-mounted camera's external parameters can be determined. This upgrades the verification process from qualitative judgment to quantitative compensation, enabling dynamic correction of assembly deviations and temperature drift. This process relies on the dynamic physical laws in real driving scenarios for adaptive verification, which can significantly improve the robustness and accuracy of orientation recognition and external parameter calibration, thereby solving the technical problem of low accuracy in verifying vehicle-mounted cameras in related technologies. Attached Figure Description

[0021] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0022] Figure 1 This is a flowchart of a verification method for an in-vehicle camera according to an embodiment of the present invention;

[0023] Figure 2This is a schematic diagram of a vehicle-mounted camera verification device according to an embodiment of the present invention. Detailed Implementation

[0024] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0025] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0026] According to an embodiment of this application, an embodiment of a verification method for an in-vehicle camera is provided. The steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowcharts, in some cases the steps shown or described may be performed in a different order than that shown here.

[0027] This embodiment provides a method for verifying vehicle-mounted cameras. Figure 1 This is a flowchart of a verification method for an in-vehicle camera according to an embodiment of this application, as follows: Figure 1 As shown, the process includes the following steps:

[0028] Step S102: During the vehicle's movement, acquire the image sequence captured by the vehicle-mounted camera and the vehicle's motion information.

[0029] The aforementioned vehicle-mounted cameras can refer to cameras installed around a vehicle in multiple directions, such as front, rear, left, and right, to collect image data of the vehicle's surrounding environment and obtain image sequences. Multiple cameras may require orientation verification and extrinsic parameter calibration. The installation orientation may be misaligned due to assembly errors, maintenance, replacement, or mechanical vibration. For example, if a front-view camera is installed as a rear-view camera, this application needs to automatically identify its true orientation.

[0030] The aforementioned image sequence can refer to multiple frames of image data continuously collected by an in-vehicle camera during vehicle operation and arranged in chronological order. In this application, the image sequence can be used to calculate pixel motion information between adjacent frames as the basic input for generating the optical flow field.

[0031] The aforementioned motion information can refer to dynamic parameters synchronously collected by on-board sensors during vehicle operation, including linear velocity, angular velocity, acceleration, and rotation angle, which are used to characterize the rigid body motion state of the vehicle itself. In this application, the motion information is used to model the pixel motion components caused by the vehicle body rotation, thereby separating the residual optical flow dominated only by the vehicle translational motion from the observed optical flow.

[0032] As an optional implementation, during vehicle operation, a continuous sequence of images can be acquired in real time via an onboard camera. Simultaneously, vehicle linear velocity and steering angle are obtained from the controller area network bus, and triaxial angular velocity and acceleration are obtained from the inertial measurement unit. Synchronization of image frames and motion data is achieved through hardware timestamps or software interpolation. Image preprocessing, including distortion correction and brightness normalization, ensures input consistency. Motion data is filtered to eliminate high-frequency noise while retaining the effective physical components of translation and rotation. This setup requires no additional calibration equipment; sufficient motion information can be obtained through natural vehicle movement, significantly improving deployability and robustness in vehicle operating environments and laying a reliable data foundation for online calibration-free self-calibration.

[0033] Step S104: Determine the residual optical flow based on the image sequence and motion information.

[0034] Among them, residual optical flow is used to characterize the optical flow generated by the translational motion of the vehicle, which is determined based on the image sequence.

[0035] The aforementioned residual optical flow can refer to the optical flow that remains after removing the pixel displacement component caused by the vehicle's rotational motion from the observed optical flow field determined by the image sequence. It is mainly caused by the vehicle's translational motion and reflects the image pixel displacement caused by the vehicle's translational motion relative to the static environment. The spatial distribution characteristics of the residual optical flow, such as divergence, convergence, and lateral flow, are related to the actual orientation of the vehicle-mounted camera.

[0036] As an optional implementation, residual optical flow can be determined based on image sequences and motion information. For example, dense optical flow algorithms can be used to calculate the observed optical flow field between adjacent frames in an image sequence, reflecting the overall motion of pixels. Subsequently, the vehicle rotation matrix obtained by inertial measurement unit integration can be combined with the pinhole imaging model to deduce the pixel displacement component caused only by vehicle rotation, and this pixel displacement component can be subtracted from the observed optical flow, ultimately retaining the residual optical flow dominated only by vehicle translational motion. The above process effectively eliminates global parallax interference caused by rotation, allowing the optical flow field to focus on reflecting the local flow field structure of vehicle forward, backward, or lateral movement. This can significantly improve the accuracy of direction determination, establish a strong correspondence between the optical flow pattern and the camera installation orientation (e.g., forward divergence, backward convergence), reduce interference from lighting, texture changes, or dynamic targets, and provide clear input for subsequent direction recognition and extrinsic parameter fine-tuning based on optical flow template matching.

[0037] Step S106: Based on the residual optical flow, the vehicle-mounted camera is verified to obtain the target verification result.

[0038] The target verification results include at least whether the installation direction of the vehicle camera is correct, and the external parameter adjustment amount of the vehicle camera. The external parameter adjustment amount is used to represent the adjustment amount of the rotation parameters of the vehicle camera.

[0039] The aforementioned target verification result can refer to the camera judgment result output after analyzing the residual optical flow, and should include at least the following: whether the installation direction of the vehicle-mounted camera is correct, i.e., whether the actual orientation of the current camera is consistent with the nominal orientation configured in its system; and the extrinsic parameter adjustment amount of the vehicle-mounted camera, which is the small adjustment value required to adjust the rotation parameters, used to adjust the camera's extrinsic parameters online to align the image coordinate system with the vehicle coordinate system. The target verification result can be used to trigger system responses such as error prevention alarms, automatic calibration, or function degradation.

[0040] As an optional implementation, vehicle-mounted cameras can be calibrated based on residual optical flow. For example, desired optical flow templates corresponding to multiple directions (front, rear, left, right) can be constructed. Theoretical optical flow distributions can be generated using a pinhole imaging model based on the vehicle's motion direction and nominal extrinsic parameters. Subsequently, the similarity of the direction histogram, the consistency of optical flow focus, and the matching degree of divergence or curl features between the residual optical flow and each desired optical flow template are calculated, and a dynamic target interference penalty term is fused to comprehensively score and determine the matching degree with each desired optical flow template. If the matching degree exceeds the corresponding threshold, the installation direction can be confirmed as correct, and an improvement target is constructed based on the difference between the residual optical flow and the model optical flow in the corresponding desired direction. The rotational extrinsic parameter fine-tuning amount is solved iteratively. The above process jointly models the optical flow geometric features and the motion physics model, achieving integrated direction discrimination and extrinsic parameter estimation, avoiding reliance on calibration boards or manual intervention. In vehicle usage scenarios, errors such as reverse installation and misconnection can be automatically identified, while small rotational deviations are quantified and compensated, significantly improving the accuracy of surround view stitching and parking perception, and enhancing system reliability.

[0041] In this embodiment of the application, during the vehicle's movement, firstly, an image sequence captured by the vehicle-mounted camera and the vehicle's motion information are acquired; then, based on the image sequence and motion information, a residual optical flow is determined, wherein the residual optical flow is used to characterize the optical flow generated by the vehicle's translational motion, determined based on the image sequence; finally, based on the residual optical flow, the vehicle-mounted camera is verified to obtain a target verification result, wherein the target verification result includes at least whether the installation direction of the vehicle-mounted camera is correct, and the external parameter adjustment amount of the vehicle-mounted camera, the external parameter adjustment amount being used to represent the adjustment amount of the rotation parameters of the vehicle-mounted camera. This application acquires image sequences and vehicle motion information simultaneously during vehicle operation, models the motion information, and removes pixel motion caused by vehicle rotation. This extracts residual optical flow that reflects only vehicle translation, effectively removing interference components and making the optical flow signal more realistically represent the relative motion relationship between the scene and the vehicle. By using residual optical flow to verify the vehicle-mounted camera, the correctness of the camera's installation orientation can be accurately identified, and the adjustment amount of the vehicle-mounted camera's external parameters can be determined. This upgrades the verification process from qualitative judgment to quantitative compensation, enabling dynamic correction of assembly deviations and temperature drift. This process relies on the dynamic physical laws in real driving scenarios for adaptive verification, which can significantly improve the robustness and accuracy of orientation recognition and external parameter calibration, thereby solving the technical problem of low accuracy in verifying vehicle-mounted cameras in related technologies.

[0042] In the above embodiments of this application, determining the residual optical flow based on the image sequence and motion information includes: determining the optical flow field of two adjacent frames in the image sequence based on a preset optical flow calculation model to obtain a dense optical flow field; determining the pixel motion vector generated by the vehicle's rotational motion based on the motion information; removing the pixel motion vector from the dense optical flow field to obtain the residual optical flow; preferably, determining the pixel motion vector generated by the vehicle's rotational motion based on the motion information includes: performing an integral operation on the angular velocity parameter in the motion information using the vehicle's inertial measurement unit to obtain a rotation matrix; and determining the pixel motion vector based on the rotation matrix.

[0043] The aforementioned preset optical flow calculation model can refer to a dense optical flow estimation algorithm model that is pre-trained and deployed on an in-vehicle computing platform. The preset optical flow calculation model can extract multi-scale features from two adjacent frames of images, construct associated volumes, and accurately predict the motion vectors of each pixel through multiple iterations, outputting a dense optical flow field.

[0044] The aforementioned dense optical flow field can refer to the calculation of the corresponding two-dimensional motion vector in adjacent frames for each pixel of each frame in an image sequence, forming a full-pixel-level motion vector field consistent with the image resolution.

[0045] The aforementioned pixel motion vector can refer to the displacement vector of each pixel in the image coordinate system caused by the vehicle's own rotational motion, i.e., the attitude change caused by the angular velocity. The magnitude and direction of the pixel motion vector can be determined by the camera intrinsic parameters, the vehicle's angular velocity, and the pixel's spatial position. The pixel motion vector is generated by the relationship between the vehicle's rotation and the fixed installation of the camera. In this application, the pixel motion vector needs to be subtracted from the dense optical flow field to separate the effective information related to the vehicle's translation.

[0046] The aforementioned rotation matrix can refer to an orthogonal matrix that describes the attitude change of a vehicle in a short period of time, calculated by integrating the angular velocity information collected by the inertial measurement unit over time. The rotation matrix can characterize the rotation relationship between the vehicle coordinate system and the inertial coordinate system, and is used to accurately map the angular motion of the vehicle onto the image plane, thereby deriving the theoretical pixel displacement caused by rotation on the image sequence, i.e., the pixel motion vector, which can be used as the input of the physical model for realizing optical flow rotation compensation.

[0047] As an optional implementation, a dense optical flow field of adjacent image frames can be calculated using a preset optical flow model to obtain the two-dimensional displacement vector of each pixel, reflecting the overall motion of the image. Simultaneously, angular velocity data collected by the inertial measurement unit can be used to obtain a short-time rotation matrix through numerical integration. Combined with a pinhole imaging model within the camera, the pixel motion vector caused solely by vehicle rotation can be inferred. Subsequently, this rotational motion vector is subtracted pixel-by-pixel from the dense optical flow, retaining the residual optical flow dominated solely by vehicle translational motion. This process achieves physical decoupling of the motion components, eliminating global parallax interference caused by vehicle steering, pitch, and other rotations, allowing the optical flow field to accurately reflect the vehicle's forward, backward, or lateral movement. This setup significantly improves the sensitivity and robustness of the optical flow to real translational motion, providing clear input for subsequent direction determination, avoiding direction recognition errors due to rotation misjudgments, and facilitating online self-calibration that relies on natural driving without the need for a calibration board.

[0048] In the above embodiments of this application, the optical flow field of two adjacent frames in an image sequence is determined based on a preset optical flow calculation model to obtain a dense optical flow field. This includes: downsampling two adjacent frames in the image sequence to obtain two adjacent downsampled images; using a convolutional neural network with shared weights in the preset optical flow calculation model to extract multi-scale features of the two adjacent downsampled images to obtain feature maps of the two adjacent frames; and determining the dense optical flow field based on the feature maps of the two adjacent frames and the preset optical flow calculation model.

[0049] The aforementioned convolutional neural network (CNN) can refer to a shared-weight multilayer CNN structure used in a pre-defined optical flow calculation model to extract spatial features from images. This CNN can consist of multiple convolutional layers and activation functions, employing a parameter-sharing mechanism to perform feature extraction in parallel on adjacent downsampled images, outputting multi-scale feature maps. CNNs can capture multi-level visual information from local edges and textures to global structures in images. Due to the use of shared weights, they can efficiently process image pairs while maintaining motion consistency.

[0050] As an optional implementation, adjacent frames in an image sequence can be downsampled to reduce computational load and improve processing efficiency. Subsequently, a convolutional neural network with shared weights can be used to extract multi-scale feature maps from the two downsampled frames in layers, capturing global motion trends from low-resolution coarse features to high-resolution fine features to preserve local details. Based on the feature maps of adjacent frames, a correlation volume can be constructed to measure the matching similarity between pixels, and a dense optical flow field can be obtained by iteratively improving the initial optical flow. The above design balances accuracy and real-time performance. Multi-scale feature extraction can enhance robustness to scenes with large displacements and texture loss, while shared weights reduce the number of parameters, providing high-precision, high-density motion field input for subsequent rotation compensation and residual optical flow extraction, which helps to realize the underlying perception foundation for calibration-free online calibration.

[0051] In the above embodiments of this application, the dense optical flow field is determined based on two adjacent feature maps and a preset optical flow calculation model, including: obtaining the pixel displacement offset between two adjacent feature maps, wherein the pixel displacement offset is used to represent the pixel position offset distance of the target pixel between the two adjacent feature maps; constructing an association volume based on the two adjacent feature maps and the pixel displacement offset, wherein the association volume is used to characterize the similarity between corresponding pixels in the two adjacent feature maps; and using the gated loop unit in the preset optical flow calculation model, determining the optical flow changes caused by the vehicle's motion based on the association volume to obtain the dense optical flow field.

[0052] The aforementioned target pixel can refer to a specific pixel point used to calculate motion displacement in the feature maps of two adjacent frames. In this application, the target pixel can be a discrete location in the image space. The feature vector of the target pixel in the feature map of the previous frame is matched with the feature vector of the corresponding position in the feature map of the next frame to infer the motion direction and distance of the pixel in the image sequence.

[0053] The aforementioned associated volume can refer to a data structure used to quantify the feature similarity of pixels in the feature maps of two adjacent frames under different displacement offsets.

[0054] The aforementioned gated loop unit can refer to a recurrent neural network module in a preset optical flow calculation model used to iteratively improve optical flow estimation. The gated loop unit can gradually and accurately predict the initial optical flow based on the matching information provided by the associated volume, thereby obtaining a dense optical flow field.

[0055] As an optional implementation, the candidate displacement offset of each pixel in the feature maps of two adjacent frames can be calculated to construct an associated volume. The voxel value of the associated volume is used to measure pixel similarity by feature dot product. Subsequently, a gated recurrent unit can be used to iteratively fuse spatial context and motion consistency constraints with the associated volume as input, gradually adjusting the initial optical flow estimation and outputting a high-precision dense optical flow field. The above process combines matching similarity with motion dynamic modeling, which can significantly improve the stability of optical flow in weak texture and low contrast scenes. In the vehicle environment, it can effectively cope with problems such as illumination changes and rain and fog interference, ensuring that the optical flow field is continuous, dense, and physically reasonable, providing a reliable foundation for subsequent rotation compensation and residual optical flow extraction.

[0056] In the above embodiments of this application, the vehicle-mounted camera is verified based on residual optical flow to obtain target verification results, including: constructing the expected optical flow corresponding to each expected direction of the vehicle to obtain multiple expected optical flow templates, wherein the expected direction is the direction pointing from the vehicle as the starting point to the surrounding area of ​​the vehicle; determining multiple matching degrees between the residual optical flow and the multiple expected optical flow templates; determining the target verification result based on the multiple matching degrees; preferably, determining multiple matching degrees between the residual optical flow and the multiple expected optical flow templates includes: obtaining the direction histogram similarity, optical flow field focus consistency, divergence and / or curl feature consistency, and dynamic target interference between the residual optical flow and the multiple expected optical flow templates; determining multiple matching degrees based on the direction histogram similarity, optical flow field focus consistency, divergence and / or curl feature consistency, and dynamic target interference.

[0057] The aforementioned multiple desired optical flow templates can refer to theoretical optical flow distribution templates pre-calculated and stored based on the vehicle's kinematics model and camera extrinsic parameters for multiple standard installation directions around the vehicle, such as front, rear, left, and right. Each desired optical flow template can correspond to different assumed camera orientations. For example, when the vehicle is moving forward, the front camera should exhibit optical flow diverging outward from the image center; when the rear camera is reversing, the optical flow should converge towards the image center; and when the left or right camera is moving straight, the optical flow should exhibit unidirectional horizontal flow.

[0058] The aforementioned orientation histogram similarity refers to statistically analyzing the motion directions of each pixel in the residual optical flow field to generate an orientation angle distribution histogram, and then measuring its similarity to the orientation histograms corresponding to each desired optical flow template. A higher orientation histogram similarity indicates that the orientation distribution of the residual optical flow is closer to the theoretical pattern of a certain desired direction.

[0059] The aforementioned optical flow field focus consistency refers to the degree of agreement between the position of the optical flow focus in the residual optical flow field and the theoretical position of the optical flow focus in the desired direction. The optical flow focus can be the center point where various vectors converge or diverge in the optical flow field. Higher optical flow field focus consistency indicates a more accurate match between the vehicle's motion direction and the camera's orientation.

[0060] The aforementioned consistency of divergence characteristics refers to the degree of matching between the divergence of pixel motion in the residual optical flow field and the theoretical divergence pattern of the desired direction. Divergence reflects the expansion or contraction trend of the optical flow field. For example, a forward-looking camera should exhibit high divergence when moving forward, meaning the image center spreads outwards; a rear-looking camera should exhibit high negative divergence when reversing, meaning it converges towards the center. By calculating the local or global divergence distribution of the residual optical flow and comparing it with the desired divergence pattern of the corresponding template, the robustness of direction discrimination can be enhanced.

[0061] The aforementioned consistency of curl characteristics refers to the degree of matching between the curl of pixel motion in the residual optical flow field and the theoretical curl pattern of the desired direction. Curl reflects the local rotational trend of the optical flow field. Ideally, when the vehicle is traveling straight and the camera is correctly installed, the curl should be close to zero, i.e., without rotational distortion. Slight curl characteristics may occur when the camera moves laterally in left or right view, requiring comparison with the theoretical model. By calculating the mean square error or correlation between the curl field of the residual optical flow and the curl field of the template, it is determined whether there is a misjudgment of direction. For example, abnormal curl may occur when a forward view is misjudged as a left view.

[0062] The aforementioned dynamic target interference refers to the noise and misleading effect on consistency judgment caused by abnormal optical flow generated by non-static objects in the image, such as pedestrians, other vehicles, animals, and moving obstacles. The motion of these targets does not conform to the overall vehicle motion model, and the expected distribution of optical flow vectors and residual optical flow is inconsistent, which will significantly reduce the direction matching score. In this application, dynamic target interference is used as a penalty item. After identifying and removing dynamic pixels through motion segmentation algorithms, the degree of pollution quantitative indicators, such as the proportion of dynamic pixels and the number of optical flow anomalies, are still retained in the statistics to reduce the overall matching score and prevent misjudgment. The stronger the dynamic target interference, the lower the matching score, thereby improving the system's robustness to environmental noise.

[0063] As an optional implementation, theoretical desired optical flow templates can be constructed for multiple desired directions (front, rear, left, right, etc.) based on the vehicle's motion direction and nominal extrinsic parameters. For example, the forward-looking template shows central divergence, the rear-looking template shows convergence, and the left and right templates show lateral dominant flow fields. Corresponding desired direction histograms, optical flow focal positions, divergence, and curl features are defined. Subsequently, multi-dimensional similarity can be calculated between the measured residual optical flow and each desired optical flow template. The direction histogram can use Bach distance to measure distribution consistency, the optical flow focal position to assess center offset, and the divergence or curl features to verify whether the flow pattern matches. Dynamic target interference is introduced as a penalty term to suppress interference from pedestrians, vehicles, etc. A comprehensive weighted average is then obtained for the matching degree of each direction, with higher scores corresponding to the actual installation direction. This multi-feature fusion mechanism combines physical priors with statistical features, significantly improving the robustness and anti-interference capability of direction discrimination, avoiding misjudgments of single features in complex scenarios, and achieving high-confidence automatic direction recognition without calibration boards. This provides a reliable basis for subsequent extrinsic parameter fine-tuning and system error prevention.

[0064] In the above embodiments of this application, the target verification result is determined based on multiple matching degrees, including: when multiple matching degrees are all less than or equal to a preset matching degree, the target verification result is determined to be that the installation direction of the vehicle camera is incorrect; when the target matching degree among multiple matching degrees is greater than the preset matching degree, the target verification result is determined to be that the installation direction of the vehicle camera is correct, and the external parameter adjustment amount is determined.

[0065] The aforementioned preset matching degree can refer to the threshold used to determine whether the installation direction of the vehicle camera is correct, or it can be the lower limit of the comprehensive similarity score set based on a large amount of real road scene data and simulation verification.

[0066] As an optional implementation, when the matching degree between the residual optical flow and multiple desired optical flow templates is less than or equal to the preset matching degree, it can be determined that the camera installation direction is inconsistent with expectations, possibly due to reverse installation or incorrect port connection. If there is a target matching degree greater than the preset matching degree, it can be confirmed that the direction is the true installation orientation, and the extrinsic parameter fine-tuning process is triggered. At this time, based on the theoretical motion model corresponding to the template, the desired model optical flow is generated, and an error function is constructed with the measured residual optical flow. The fine-tuning amount of the rotational extrinsic parameter is iteratively solved. The above process achieves intelligent separation of error detection and normal calibration through threshold discrimination, avoiding blind improvement when the direction is incorrect and preventing erroneous extrinsic parameters from contaminating the system. This helps to achieve accurate identification and isolation of direction errors, preventing splicing errors or sensing failures caused by miscalibration; ensuring that extrinsic parameter fine-tuning is only performed under the premise of correct direction, improving calibration reliability and security, providing a decision basis for error prevention alarms and function degradation, and realizing a closed-loop logic of first judging errors and then correcting them.

[0067] In the above embodiments of this application, determining the extrinsic parameter adjustment amount includes: determining the desired camera in the desired direction corresponding to the target matching degree; obtaining the nominal extrinsic parameters of the desired camera, wherein the nominal extrinsic parameters are used to characterize the rotation parameters of the vehicle camera after factory calibration; determining the model optical flow corresponding to the desired camera based on the nominal extrinsic parameters and motion information; determining the extrinsic parameter adjustment amount based on the residual optical flow and the model optical flow; preferably, determining the residual optical flow and the model optical flow, and determining the extrinsic parameter adjustment amount includes: constructing a cost function based on the difference between the residual optical flow and the model optical flow; and using a preset iterative algorithm to iteratively solve the cost function to obtain the extrinsic parameter adjustment amount.

[0068] The aforementioned desired camera refers to the type of camera with a high overall score during the camera orientation determination stage, corresponding to the target matching degree. The desired camera's orientation has been inferred as a more probable correct direction. The desired camera is a theoretical virtual model of a camera selected based on the matching results of residual optical flow and desired template, and is a good match for the current vehicle motion state.

[0069] The aforementioned model optical flow refers to the theoretically observable optical flow field derived from the factory-calibrated rotation parameters of the vehicle camera and the vehicle's motion information through a pinhole imaging model or geometric projection relationship. The model optical flow is determined by the rigid body translational motion of the vehicle and the nominal extrinsic parameters, reflecting the expected displacement of static scene pixels in the image under ideal installation and calibration conditions.

[0070] The aforementioned cost function can refer to a mathematical objective function used to quantify the difference between the observed residual optical flow and the theoretical model optical flow. The design objective is to find a better external parameter rotation adjustment by minimizing this cost function.

[0071] The aforementioned preset iterative algorithm can refer to a numerical improvement method for solving the aforementioned nonlinear cost function. In this application, the Gauss-Newton iterative algorithm is preferred.

[0072] As an optional implementation, after confirming the correct camera installation orientation, the nominal extrinsic parameters corresponding to that orientation can be retrieved. Combined with the current vehicle linear and angular velocities, a theoretical model optical flow is generated through forward derivation using a pinhole imaging model. Subsequently, the difference between the measured residual optical flow and the model optical flow can be calculated pixel-by-pixel to construct a cost function, minimizing the optical flow error between the residual and model optical flows. A Gaussian-Newton iterative algorithm can be used to gradient-improve the rotation parameters, gradually approximating the optimal extrinsic parameter adjustment amount. The above process adjusts the rotation parameters while fixing the translation amount, ensuring stable improvement and compliance with engineering constraints. This facilitates extrinsic parameter self-calibration, compensates for assembly deviations and temperature drift effects, and continuously improves perception accuracy without manual intervention. Extrinsic parameter errors can be dynamically corrected without relying on a calibration board, significantly improving the positioning consistency of surround-view stitching and parking assistance, and enhancing the long-term reliability and robustness of the system.

[0073] In the above embodiments of this application, the method further includes: disabling target functions related to the vehicle camera when the target verification result indicates that the installation direction of the vehicle camera is incorrect, wherein the target functions include at least one of the following: surround view stitching and automatic parking assistance.

[0074] The aforementioned target functions can refer to advanced driver assistance systems or surround-view perception functions that can operate normally when the vehicle-mounted camera is correctly installed and has accurate external parameters. In this application, this may specifically include, but is not limited to, surround-view stitching, i.e., real-time stitching of images from four cameras to generate a distortion-free, seamless environmental bird's-eye view, and automatic parking assistance, including functions such as automatic parking space recognition, path planning, and trajectory tracking.

[0075] As an optional implementation, when a camera is determined to be installed in the wrong direction (e.g., a front-view camera mistakenly installed as a rear-view camera), and the confidence level is high, error prevention logic can be triggered promptly. This retrieves the associated functional modules of the camera, such as surround-view stitching and automatic parking assist, and proactively disables advanced functions that rely on directional information. The disabling action can be achieved through software flags or function enable signals, without interrupting the camera's power supply. It only blocks the use of data in the fusion algorithm, ensuring the system can still provide basic functions, such as single-screen display of the reversing image. This mechanism prevents serious safety risks such as stitching misalignment, parking trajectory deviation, or collision misjudgment caused by erroneous image input before the directional error is corrected. It helps to implement a safety-first strategy of error isolation, avoiding systemic functional failures caused by directional misjudgment, improving user trust and driving safety. Simultaneously, it provides users with clear fault prompts, guiding them to perform offline calibration, forming a closed-loop error prevention system of detection, disabling, and alarm repair, significantly reducing after-sales return rates and safety liability risks.

[0076] In the above embodiments of this application, the method further includes: adjusting the rotation parameters of the vehicle camera based on the external parameter adjustment amount when the external parameter adjustment amount is less than the preset adjustment amount; and generating a prompt message when the external parameter adjustment amount is greater than or equal to the preset adjustment amount, wherein the prompt message is used to prompt the user to perform offline calibration of the vehicle camera.

[0077] The aforementioned preset adjustment amount can refer to the allowable online adjustment threshold of the external parameter rotation parameter set to ensure driving safety and calibration reliability.

[0078] As an optional implementation, after the external parameter adjustment amount is obtained through iterative improvement, the amplitude of the external parameter adjustment amount can be compared with the preset adjustment amount. If the external parameter adjustment amount is less than the preset adjustment amount, it indicates that the deviation is a normal temperature drift or a small assembly error. The external parameter adjustment amount can be automatically written into the runtime external parameter cache, overwriting the original nominal parameter, and realizing online self-calibration without manual intervention. If the external parameter adjustment amount is greater than or equal to the preset adjustment amount, it can be judged as a significant installation deviation, such as forced correction after the lens is installed backward or mechanical loosening. In this case, automatic update is not performed, but a clear prompt message is generated, such as "If the camera external parameter deviation is too large, please go to the service station for professional calibration," and the user is notified through the central control screen, application, or cloud push. This hierarchical handling strategy of self-healing for small deviations and alarming for large deviations ensures the accuracy of daily operation and avoids the risk of false calibration caused by over-compensation. It can improve the system's intelligence level, reduce production line and after-sales calibration costs, and ensure that the calibration results are always within a safe and reliable range, avoiding the failure of the sensing system due to erroneous self-calibration.

[0079] The technical solution proposed in this application is described below with reference to an optional embodiment. This application proposes a method and system for camera orientation error prevention and automatic calibration based on optical flow and vehicle motion information. It relates to vehicle-mounted perception and calibration technology. Specifically, it utilizes optical flow fields and vehicle motion information, such as wheel encoders, inertial measurement units, vehicle speed and turning angle signals, to automatically determine the camera installation orientation online, such as front, rear, left, and right, and automatically detects and fine-tunes the camera's extrinsic parameters. This application is applicable to surround-view, parking, and driver assistance camera systems. The proposed production-oriented online method can be implemented without a calibration board, automatically determining the actual installation orientation of each camera; estimating and fine-tuning the deviation from the nominal extrinsic parameters; triggering error prevention alarms or function degradation when the orientation is incorrect, and providing error correction suggestions; and possessing engineering-usable robustness and real-time performance under real road conditions and various weather conditions.

[0080] The method proposed in this application mainly includes the following: data acquisition and preprocessing, synchronously acquiring camera image sequences and vehicle motion information, including linear velocity and angular velocity, and performing distortion correction, noise suppression, and brightness normalization on the images; optical flow estimation and rotation compensation, calculating the dense optical flow between adjacent frames, and using the angular velocity information of the inertial measurement unit to calculate and subtract the pixel motion caused by the pure rotation of the vehicle body, obtaining the residual optical flow dominated by translational motion; direction assumption and discrimination, generating expected optical flow templates under different installation direction assumptions based on the vehicle's linear velocity and the camera's nominal extrinsic parameters. By calculating the similarity score between the observed residual optical flow and each expected template, the actual direction and confidence level of the camera are comprehensively determined. The external parameter deviation estimation and automatic fine-tuning, under the condition that the direction determination is valid, constructs a cost function based on the difference between the residual optical flow and the model optical flow, and improves the rotation part of the camera's external parameters with small perturbations to obtain the external parameter fine-tuning amount; error prevention strategy and linkage, when the fine-tuning amount is within a reasonable threshold and passes consistency verification, the runtime external parameters are updated; when the camera direction is detected to be inconsistent with the system configuration, an error prevention alarm is triggered and function degradation is executed. This application achieves the following beneficial effects through the above technical solution: First, it enables online confirmation of direction and external parameter deviation without calibration board, reducing assembly and maintenance error prevention costs. Second, it can output robustly under real road conditions, reducing bird's-eye view stitching or perception false detection caused by direction errors. It combines improvement goals with physical constraints to meet automotive-grade real-time requirements. It can be integrated with mass production processes, such as end-of-line self-inspection, automatic confirmation after after-sales replacement, and self-inspection after over-the-air download updates.

[0081] This application relates to vehicle-mounted sensing and calibration technology, aiming to automatically determine the installation orientation of cameras, such as front, rear, left, and right, and to automatically detect and fine-tune the camera's extrinsic parameters. The method includes camera orientation determination, extrinsic parameter fine-tuning, system error prevention, and linkage processes. Camera orientation determination aims to accurately identify the actual installation orientation of each camera. First, data acquisition and synchronization are performed, acquiring video streams from the cameras under test, and simultaneously obtaining vehicle motion information, including the vehicle's linear velocity, from the vehicle bus and inertial measurement unit. angular velocity =( , , The system measures rotation angle and acceleration, and synchronizes image data and motion information in time. After data synchronization, image preprocessing is performed. To correct lens distortion, distortion correction is first applied to the image; then, through 3D processing... A 3-core Gaussian filter is used to suppress noise, and the brightness is then normalized to the [0,1] range to reduce the impact of illumination variations. Next, optical flow calculations and rotation compensation can be performed on adjacent frames. (at time t) and (At time t+1), the observed optical flow field of adjacent frames is calculated using the dense optical flow algorithm. This refers to a dense optical flow field. To balance computational speed and accuracy, the original image can be downsampled to 960×540 resolution before being input into a pre-defined optical flow calculation model. This model extracts multi-scale features from the two frames using a convolutional neural network with shared weights, thus retaining a high-resolution feature map at 1 / 8 scale. and And construct the associated volume as follows:

[0082] ;

[0083] in,( , The displacement is used as the offset, and the dot product measures pixel similarity; subsequently, the initial optical flow is predicted based on the associated volume. The system is then iteratively improved through a gated loop unit to output a converged dense optical flow field.

[0084] ;

[0085] in, , Represents the pixel (x,y) from arrive The number of pixels of displacement in the horizontal and vertical directions.

[0086] Subsequently, the short-time rotation matrix can be obtained by integrating the angular velocity of the inertial measurement unit. ( t), and then calculate the pixel motion caused by pure rotation. This refers to the pixel motion vector. The rotation component is subtracted from the observed optical flow to weaken the depth-independent rotation element, resulting in a residual optical flow dominated by translation.

[0087] ;

[0088] To further improve stability, motion segmentation techniques can be used to remove dynamic targets such as pedestrians and vehicles from the image. The identification of dynamic targets is based on a motion model and optical flow matching method, utilizing the vehicle's own motion parameters, such as linear velocity. With angular velocity The nominal extrinsic parameters from the camera to the vehicle and the pinhole imaging model are used to generate the expected optical flow of static scene points on the image. If the scene points are far apart, they can be simplified to directional templates to reduce computational complexity. Then, the residual optical flow is compared. With desired optical flow The difference is used to distinguish pixel attributes, if:

[0089] ;

[0090] If a point is identified as a dynamic point, meaning it is in motion, or an unreliable point, such as one where occlusion or noise causes inaccurate optical flow, it will be excluded from subsequent improvements; otherwise, it will be identified as a static point, retaining the static background and road surface area to improve the stability of subsequent calculations.

[0091] To obtain residual optical flow dominated by translation After excluding dynamically unreliable points, basic statistical analysis is performed only within the retained static pixel regions to calculate the distribution of residual optical flow in each direction to obtain the direction histogram H( Simultaneously, the divergence of the optical flow is extracted to measure its diffusion or convergence characteristics, and the curl is used to measure its rotational characteristics. The focal point of the optical flow (FOE), i.e., the central starting point of the optical flow, is also located, which is closely related to the vehicle's direction of motion. Based on this, for the four possible directions of motion—front, rear, left, and right—the system combines the vehicle's linear velocity v and camera extrinsic parameters. Based on the pinhole imaging model or long-distance approximation, templates for the desired optical flow direction in each direction are constructed. This refers to the theoretically expected optical flow distribution in that direction. For example, when the front-facing camera moves forward, the optical flow diverges from the center and converges towards the center when reversing, while the left and right-facing cameras correspond to a lateral flow pattern. To quantify the degree of matching between the observed optical flow and the expected template in each direction, a comprehensive scoring function S(H) is designed:

[0092] ;

[0093] in, It refers to the weights, and sim is the histogram of observation directions. Histogram of desired direction Similarity metrics, such as Bach distance and histogram intersection, are used to measure similarity. The higher the similarity, the higher the score for this item. It is a weight that measures the degree of fit between the observed optical flow focus (FOE) position and the expected optical flow focus (FOE) position in that direction. For example, the optical flow focus (FOE) should be close to the center of the image when moving forward. λ is the weight, measuring whether the divergence or curl characteristics of the observed optical flow are consistent with the desired pattern. For example, the divergence should be high when moving forward, and the curl should be low when moving left or right. λ is the weight; dynamic targets, such as moving vehicles or pedestrians, will disrupt the consistent pattern of the optical flow, so this term is a penalty term. The stronger the dynamic interference, the lower the score of this term. The optimal direction is obtained, that is, the direction with the higher overall score among all candidate directions:

[0094] ;

[0095] Meanwhile, to improve the robustness of the judgment, a sliding time window, such as 2 seconds, is introduced. Within the window, the scores of each direction are accumulated, and a voting mechanism is used to count the number of optimal results, and the final H is output. And confidence level; when the vehicle travels straight at a constant speed for more than 2 seconds and the higher confidence level is not less than 0.8, the direction determination is confirmed to be valid, thereby filtering out cases of unstable motion or insufficient confidence level and ensuring the reliability of the results.

[0096] Next, the extrinsic parameter fine-tuning process can be performed. Based on the correct direction determination, this stage involves accurately calibrating the camera's extrinsic rotation angle. When the vehicle's direction of motion is clear and the absolute value of its linear velocity is not less than 3 m / s, to reduce interference from dynamic targets or noise on the improvement, a set of pixels in the static background area will be selected first. This serves as input for subsequent improvements, ensuring the high reliability of the data used in the calculations. The core of this process is fine-tuning the rotational extrinsic parameter R from the camera to the vehicle body. Specifically, this involves introducing small perturbations δR to the three attitude angles included in the rotational extrinsic parameter R: yaw, pitch, and roll. The camera's mounting position parameter t remains unchanged or is subject to only weak constraints, as position errors are typically smaller than rotational errors in engineering practice, or have a lower adjustment priority. To achieve this fine-tuning, a corresponding cost function can be constructed on a static pixel set. Above, by minimizing the observed residual optical flow The optical flow of the model generated by the small perturbation δR and the vehicle motion parameters (v,w) Improvements can be made by addressing the differences between them.

[0097] ;

[0098] Where ρ is a robust kernel function, such as the Huber kernel or L1 kernel, used to suppress outliers, such as pixels misclassified as static points or errors in optical flow calculation, to prevent a few bad pixels from dominating the improvement results; at the same time, a regularization term R(δR)=η is added. ‖δR‖², η is taken as 10 - To prevent excessive adjustments to the minute perturbation δR, the adjustment range must be on the order of 3, ensuring compliance with the underlying assumptions of minute perturbations. The specific improvement process can employ a Gauss-Newton iterative algorithm to solve the aforementioned nonlinear least squares problem. First, the initial value of the minute perturbation δR is set as the nominal extrinsic parameter, serving as the starting point for the improvement; then, for each pixel... Calculate the residual :

[0099] ;

[0100] And solve for the Jacobian matrix of the residual with respect to the small perturbation δR. That is, the partial derivative matrix of the residual with respect to each parameter of the small perturbation δR; then, by integrating the Jacobian matrix of each pixel and the residual, a linear equation system J is constructed. T J ΔδR=-J T r, the parameter increment is obtained by solving. After δR, update δ =δ +ΔδR; Repeat the above steps until the change in residual or small perturbation δR is less than the set threshold, at which point the iteration terminates.

[0101] To further ensure the rationality of the improvement results, additional constraints are introduced. On the one hand, joint improvements are made under different working conditions such as straight-line movement and turning to eliminate the coupling between depth and scale. Since the contribution of different motion modes to optical flow is related to depth and scale, joint improvements can effectively decouple them. On the other hand, box constraints are applied to the small perturbation δR, such as limiting the absolute value of roll angle to no more than 3°, pitch angle to no more than 3°, and yaw angle to no more than 5°, to ensure that the fine-tuning amount is within a reasonable range allowed by engineering practice. Through the above process, robust fine-tuning of the camera rotation extrinsic parameters is finally achieved, improving the accuracy and stability of the relevant vision system. Then, improvements can be made under two typical motion conditions: straight-line movement and small turning. If the difference in the small perturbation δR obtained from the two improvements is less than the set threshold of 0.5°, the result is considered reliable. The verified fine-tuned extrinsic parameters take effect in a runtime overwrite manner without modifying the factory calibration file, and the version and timestamp are recorded for problem traceability.

[0102] Next, the system's error prevention and linkage process is implemented, allowing the aforementioned results to be applied to system-level error prevention and control. For each camera in the system, such as front, rear, left, and right, independent direction determination is performed, generating a mapping table from physical port to the determined direction. This table is then automatically compared with the system's preset port-to-nominal direction configuration table. If inconsistencies occur, such as port 1 being determined to be left-viewing but configured as front-viewing, the system will issue specific alarm information such as port misconnection or camera installation flip, thus completing the automatic configuration comparison and providing error correction suggestions. For direction error prevention scenarios, when the determined direction of a camera is inconsistent with the nominal direction and the confidence level is higher than the 0.8 threshold, the system can set a diagnostic fault code and issue an alarm through the human-machine interface or the cloud, while simultaneously disabling advanced functions that depend on that camera's direction, such as surround-view stitching and automatic parking. In the calibration effectiveness management stage, if the external parameter deviation quantification result is within the threshold and the consistency verification passes, the system can automatically update the runtime external parameters. If the deviation exceeds the threshold, it will prompt for offline professional calibration.

[0103] Through this implementation method, this application can automatically and accurately determine the installation direction of the camera in complex driving environments, and at the same time compensate for external parameter deviations online, which greatly improves the automation level and robustness of the system.

[0104] According to another aspect of the present invention, a vehicle-mounted camera verification device is also provided. This device can execute the vehicle-mounted camera verification method of the above embodiments. The specific implementation method and preferred application scenarios are the same as those of the above embodiments, and will not be described in detail here.

[0105] Figure 2 This is a schematic diagram of a verification device for an in-vehicle camera according to an embodiment of this application, as shown below. Figure 2 As shown, the device includes the following: an acquisition module 202, a determination module 204, and a verification module 206.

[0106] The acquisition module 202 is used to acquire image sequences and vehicle motion information captured by the vehicle-mounted camera during vehicle operation; the determination module 204 is used to determine the residual optical flow based on the image sequence and motion information, wherein the residual optical flow is used to characterize the optical flow generated by the translational motion of the vehicle based on the image sequence; the verification module 206 is used to verify the vehicle-mounted camera based on the residual optical flow to obtain the target verification result, wherein the target verification result includes at least whether the installation direction of the vehicle-mounted camera is correct, and the external parameter adjustment amount of the vehicle-mounted camera, wherein the external parameter adjustment amount is used to represent the adjustment amount of the rotation parameters of the vehicle-mounted camera.

[0107] The determining module is further configured to determine the optical flow field of two adjacent frames in an image sequence based on a preset optical flow calculation model, thereby obtaining a dense optical flow field; determine the pixel motion vector generated by the vehicle's rotational motion based on motion information; and remove the pixel motion vector from the dense optical flow field to obtain residual optical flow. Preferably, determining the pixel motion vector generated by the vehicle's rotational motion based on motion information includes: using the vehicle's inertial measurement unit to perform an integral operation on the angular velocity parameter in the motion information to obtain a rotation matrix; and determining the pixel motion vector based on the rotation matrix.

[0108] The determination module is also used to downsample two adjacent frames in the image sequence to obtain two adjacent downsampled images; to extract multi-scale features of the two adjacent downsampled images using a convolutional neural network with shared weights in the preset optical flow calculation model to obtain feature maps of the two adjacent frames; and to determine the dense optical flow field based on the feature maps of the two adjacent frames and the preset optical flow calculation model.

[0109] The determination module is further used to obtain the pixel displacement offset between two adjacent feature maps, where the pixel displacement offset represents the pixel position offset distance of the target pixel between two adjacent feature maps; based on the two adjacent feature maps and the pixel displacement offset, an association volume is constructed, where the association volume is used to characterize the similarity between corresponding pixels in the two adjacent feature maps; using the gated loop unit in the preset optical flow calculation model, the optical flow changes caused by the vehicle's motion are determined based on the association volume to obtain a dense optical flow field.

[0110] The verification module is further used to construct the desired optical flow corresponding to each desired direction of the vehicle, obtaining multiple desired optical flow templates, wherein the desired direction is the direction pointing from the vehicle as the starting point to the surrounding area; determine multiple matching degrees between the residual optical flow and the multiple desired optical flow templates; and determine the target verification result based on the multiple matching degrees. Preferably, determining multiple matching degrees between the residual optical flow and the multiple desired optical flow templates includes: obtaining the direction histogram similarity, optical flow field focus consistency, divergence and / or curl feature consistency, and dynamic target interference between the residual optical flow and the multiple desired optical flow templates; and determining multiple matching degrees based on the direction histogram similarity, optical flow field focus consistency, divergence and / or curl feature consistency, and dynamic target interference.

[0111] The verification module is also used to determine that the installation direction of the vehicle camera is incorrect when multiple matching degrees are less than or equal to the preset matching degree; and to determine that the installation direction of the vehicle camera is correct when the target matching degree is greater than the preset matching degree among multiple matching degrees, and to determine the external parameter adjustment amount.

[0112] The verification module is further used to determine the desired camera in the desired direction corresponding to the target matching degree; obtain the nominal extrinsic parameters of the desired camera, wherein the nominal extrinsic parameters are used to characterize the rotation parameters of the vehicle camera after factory calibration; determine the model optical flow corresponding to the desired camera based on the nominal extrinsic parameters and motion information; determine the extrinsic parameter adjustment amount based on the residual optical flow and the model optical flow; preferably, determining the residual optical flow and the model optical flow, and determining the extrinsic parameter adjustment amount, includes: constructing a cost function based on the difference between the residual optical flow and the model optical flow; and using a preset iterative algorithm to iteratively solve the cost function to obtain the extrinsic parameter adjustment amount.

[0113] The verification module is also used to disable target functions related to the vehicle camera when the target verification result indicates that the installation direction of the vehicle camera is incorrect. The target functions include at least one of the following: surround view stitching and automatic parking assistance.

[0114] The verification module is also used to adjust the rotation parameters of the vehicle camera based on the external parameter adjustment amount when the external parameter adjustment amount is less than the preset adjustment amount; and to generate a prompt message when the external parameter adjustment amount is greater than or equal to the preset adjustment amount, wherein the prompt message is used to prompt the user to perform offline calibration of the vehicle camera.

[0115] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0116] Embodiments of this application also provide a vehicle, including: a memory storing an executable program; and a processor for running the program, wherein the program executes the methods described in various embodiments of this application when it runs.

[0117] Embodiments of this application also provide a computer-readable storage medium including a stored executable program, wherein, when the executable program is running, it controls the device where the computer-readable storage medium is located to perform the methods of various embodiments of this application.

[0118] Embodiments of this application also provide a computer program product, including a computer program that, when executed by a processor, implements the methods of various embodiments of this application.

[0119] Embodiments of this application also provide a computer program product, including a non-volatile computer-readable storage medium for storing a computer program that, when executed by a processor, implements the methods in various embodiments of this application.

[0120] Embodiments of this application also provide a computer program that, when executed by a processor, implements the methods described in the various embodiments of this application.

[0121] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0122] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0123] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0124] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0125] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0126] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A verification method for a vehicle-mounted camera, characterized in that, include: During the vehicle's operation, image sequences captured by the onboard camera and the vehicle's motion information are acquired. Based on the image sequence and the motion information, a residual optical flow is determined, wherein the residual optical flow is used to characterize the optical flow generated by the translational motion of the vehicle, determined based on the image sequence; Based on the residual optical flow, the vehicle-mounted camera is verified to obtain a target verification result. The target verification result includes at least whether the installation direction of the vehicle-mounted camera is correct and the external parameter adjustment amount of the vehicle-mounted camera. The external parameter adjustment amount is used to represent the adjustment amount of the rotation parameters of the vehicle-mounted camera.

2. The verification method for vehicle-mounted cameras according to claim 1, characterized in that, Determining the residual optical flow based on the image sequence and the motion information includes: Based on a preset optical flow calculation model, the optical flow field of two adjacent frames in the image sequence is determined to obtain a dense optical flow field. Based on the motion information, determine the pixel motion vector generated by the rotational motion of the vehicle; The residual optical flow is obtained by removing the pixel motion vector from the dense optical flow field; Preferably, determining the pixel motion vector generated by the rotational motion of the vehicle based on the motion information includes: using the vehicle's inertial measurement unit to perform an integral operation on the angular velocity parameter in the motion information to obtain a rotation matrix; and determining the pixel motion vector based on the rotation matrix.

3. The verification method for vehicle-mounted cameras according to claim 2, characterized in that, Based on a preset optical flow calculation model, the optical flow fields of two adjacent frames in the image sequence are determined to obtain a dense optical flow field, including: The image sequence is downsampled to obtain two adjacent downsampled images; Using the convolutional neural network with shared weights in the preset optical flow calculation model, multi-scale features of the downsampled images of two adjacent frames are extracted to obtain feature maps of two adjacent frames. The dense optical flow field is determined based on the feature maps of the two adjacent frames and the preset optical flow calculation model.

4. The verification method for vehicle-mounted cameras according to claim 3, characterized in that, Based on the feature maps of the two adjacent frames and the preset optical flow calculation model, the dense optical flow field is determined, including: Obtain the pixel displacement offset between two adjacent feature maps, wherein the pixel displacement offset is used to represent the pixel position offset distance of the target pixel between the two adjacent feature maps; Based on the feature maps of two adjacent frames and the pixel displacement offset, an association volume is constructed, wherein the association volume is used to characterize the similarity between corresponding pixels in the feature maps of two adjacent frames; Using the gated loop unit in the preset optical flow calculation model, the optical flow changes caused by the motion of the vehicle are determined based on the associated volume, and the dense optical flow field is obtained.

5. The verification method for vehicle-mounted cameras according to claim 1, characterized in that, Based on the residual optical flow, the vehicle-mounted camera is verified to obtain the target verification result, including: Construct the desired optical flow corresponding to each desired direction of the vehicle to obtain multiple desired optical flow templates, wherein the desired direction is the direction pointing from the vehicle to the surrounding area of ​​the vehicle; Determine multiple matching degrees between the residual optical flow and the plurality of desired optical flow templates; Based on the multiple matching degrees, the target verification result is determined; Preferably, determining multiple matching degrees between the residual optical flow and the plurality of desired optical flow templates includes: acquiring the orientation histogram similarity, optical flow field focal consistency, divergence and / or curl feature consistency, and dynamic target interference between the residual optical flow and the plurality of desired optical flow templates; and determining the plurality of matching degrees based on the orientation histogram similarity, the optical flow field focal consistency, the divergence and / or curl feature consistency, and the dynamic target interference.

6. The verification method for vehicle-mounted cameras according to claim 5, characterized in that, Based on the multiple matching degrees, the target verification result is determined, including: If all the matching degrees are less than or equal to the preset matching degree, the target verification result is determined to be that the installation direction of the vehicle camera is incorrect. If the target matching degree among the multiple matching degrees is greater than the preset matching degree, the target verification result is determined to be that the installation direction of the vehicle camera is correct, and the external parameter adjustment amount is determined.

7. The verification method for a vehicle-mounted camera according to claim 6, characterized in that, Determining the external parameter adjustment amount includes: Determine the desired camera with the desired direction corresponding to the target matching degree; Obtain the nominal extrinsic parameters of the desired camera, wherein the nominal extrinsic parameters are used to characterize the rotation parameters of the vehicle camera after factory calibration; Based on the nominal extrinsic parameters and the motion information, determine the model optical flow corresponding to the desired camera; Based on the residual optical flow and the model optical flow, the external parameter adjustment amount is determined; Preferably, determining the residual optical flow and the model optical flow, and determining the extrinsic parameter adjustment amount, includes: constructing a cost function based on the difference between the residual optical flow and the model optical flow; and using a preset iterative algorithm to iteratively solve the cost function to obtain the extrinsic parameter adjustment amount.

8. The verification method for a vehicle-mounted camera according to any one of claims 1 to 7, characterized in that, The method further includes: If the target verification result indicates that the installation orientation of the vehicle camera is incorrect, the target function related to the vehicle camera is disabled, wherein the target function includes at least one of the following: surround view stitching and automatic parking assistance.

9. The verification method for a vehicle-mounted camera according to any one of claims 1 to 7, characterized in that, The method further includes: If the external parameter adjustment amount is less than the preset adjustment amount, the rotation parameters of the vehicle camera are adjusted based on the external parameter adjustment amount; If the external parameter adjustment amount is greater than or equal to the preset adjustment amount, a prompt message is generated, wherein the prompt message is used to prompt the user to perform offline calibration of the vehicle camera.

10. A vehicle, characterized in that, include: Memory, which stores executable programs; A processor for running the program, wherein the program executes the verification method for the vehicle-mounted camera according to any one of claims 1 to 9 when it runs.

Images