Sensor parameter calibration method and related apparatus

Abstract

The embodiment of the application provides a sensor parameter calibration method and related device, wherein the method comprises the following steps: constructing a joint state vector; determining that the current scene is in a degenerate scene according to the joint state vector; generating a vehicle driving guide instruction according to a degenerate type; generating a vehicle driving guide picture and controlling a display device to display the vehicle driving guide picture; obtaining first sensor data; and screening first target data from the first sensor data. According to the state information of the sensor on the vehicle, it is determined that the current scene is in a degenerate scene, then the user is guided to perform different specific driving control operations according to the degenerate type, and further, the data of the sensor is observed, and a specific data screening operation is performed to screen high-quality first target data, which is beneficial to improve the accuracy of sensor parameter calibration.

Description

Technical Field

[0001] This application belongs to the field of vehicle control, specifically relating to a sensor parameter calibration method and related devices. Background Technology

[0002] In autonomous driving systems, the accuracy of spatiotemporal parameters from multiple sensors is crucial. However, existing online sensor parameter calibration relies on natural vehicle driving data. In low-excitation scenarios such as constant speed driving on a highway, certain degrees of freedom of extrinsic parameters (such as lateral translation and time delay) are unobservable, causing filter parameters to diverge or drift. Furthermore, if drastic maneuvers are performed for calibration, traditional algorithms will suffer from severe object edge depth mismatch due to drastic parallax changes, leading to optimization failure. Summary of the Invention

[0003] This application provides a sensor parameter calibration method and related apparatus. Based on the status information of the sensors on the vehicle, it is determined that the current scene is a degradation scene. Then, the user is guided to perform different specific driving control operations according to the degradation type. Furthermore, the sensor data is observed and specific data filtering operations are performed to filter out high-quality first target data, which helps to improve the accuracy of sensor parameter calibration.

[0004] In a first aspect, embodiments of this application provide a sensor parameter calibration method, the method comprising: Acquire the status information of the sensors on the target vehicle; Construct a joint state vector based on the state information; Determine whether the current scene is in a degenerate scene based on the joint state vector; If the current scene is determined to be a degraded scene based on the joint state vector, then the current degradation type is determined. Based on the degradation type, generate vehicle driving guidance instructions; Based on the vehicle driving guidance instructions, a vehicle driving guidance screen is generated, and the display device is controlled to display the vehicle driving guidance screen; When it is detected that the user executes the vehicle control operation corresponding to the vehicle driving guidance command, the data of multiple first sensors on multiple sensors on the target vehicle are acquired. The first target data is selected from the multiple first sensor data.

[0005] In one possible example, determining whether the current scene is in a degenerate scene based on the joint state vector includes: The state update increment is determined based on the joint state vector; If it is determined that the filter parameters do not converge based on the state update increment, then the first information matrix is ​​determined based on the Jacobian matrix corresponding to the state update increment. Determine the first minimum singular value of the first information matrix; If it is determined that the first minimum singular value is less than the first preset threshold, then the current scene is determined to be a degraded scene; If it is determined that the first minimum singular value is greater than or equal to the first preset threshold, then it is determined that the current scene is not in a degenerate scene; If the filter parameters converge based on the state update increment, then the current optimal state estimate is determined based on the state update increment and the joint state vector. Render the point cloud reprojection based on the current optimal state estimate and determine the reprojection error; If the parameters are determined to be inaccurate based on the reprojection error, then the current scene is determined to be in a degraded scene based on the first minimum singular value and the first preset threshold.

[0006] In one possible example, determining the state update increment based on the joint state vector includes: Construct a photometric error model; The residual vector is determined based on the photometric error model and the joint state vector. Determine the Jacobian matrix based on the joint state vector; The state update increment is determined based on the Jacobi matrix and the residual vector.

[0007] In one possible example, determining the current degradation type includes: Determine the right singular vector corresponding to the first minimum singular value; Determine the values ​​of the elements in the right singular vector that correspond to the horizontal translation to obtain the first value; Determine the values ​​of the element components corresponding to the time delay in the right singular vector to obtain the second value; If it is determined that the first value is greater than the second preset threshold, then the current degradation type is determined to be unobservable spatial parameters; If the second value is determined to be greater than the third preset threshold, then the current degradation type is determined to be unobservable time parameter.

[0008] In one possible example, generating vehicle driving guidance instructions based on the degradation type includes: When the degradation type is that the time parameter is unobservable, a first vehicle driving guidance instruction is generated. The first vehicle driving guidance instruction is used to guide the user to perform a first vehicle control operation to change the longitudinal acceleration of the target vehicle. When the degradation type is unobservable spatial parameters, a second vehicle driving guidance instruction is generated. The second vehicle driving guidance instruction is used to guide the user to perform a second vehicle control operation to change the angular velocity of the target vehicle.

[0009] In one possible example, the step of filtering the first target data from the plurality of first sensor data includes: LiDAR point cloud projection is performed based on the data from the plurality of first sensors; Perform depth continuity verification on multiple projection points to eliminate abnormal projection points; A spatiotemporal selection operation is performed on multiple projection points to obtain a target feature point set, which is then used as the first target data.

[0010] In one possible example, the process of performing depth continuity verification on multiple projection points to remove abnormal projection points includes: Determine the depth gradient in the neighborhood of each projection point; Remove projection points from a set of projection points whose depth gradient is greater than a preset gradient threshold.

[0011] Secondly, embodiments of this application provide a sensor parameter calibration device, which includes an acquisition unit, a determination unit, a judgment unit, a generation unit, and a filtering unit; wherein, The acquisition unit is used to acquire the status information of the sensors on the target vehicle; it is also used to acquire multiple first sensor data of multiple sensors on the target vehicle when it detects that the user has executed a vehicle control operation corresponding to the vehicle driving guidance command. The determining unit is configured to construct a joint state vector based on the state information; and to determine the current degradation type if the current scene is determined to be in a degradation scene based on the joint state vector. The judgment unit is used to determine whether the current scene is in a degraded scene based on the joint state vector. The generation unit is configured to generate vehicle driving guidance instructions based on the degradation type; and to generate a vehicle driving guidance screen based on the vehicle driving guidance instructions, and to control a display device to display the vehicle driving guidance screen. The filtering unit is used to filter out the first target data from the plurality of first sensor data.

[0012] A third aspect of this application provides an electronic device including: a processor and a memory; and one or more programs stored in the memory and configured to be executed by the processor, the programs including instructions for some or all of the steps as described in the first aspect.

[0013] A fourth aspect of this application provides a computer-readable storage medium for storing a computer program that causes a computer to perform some or all of the steps described in the first aspect of this application.

[0014] A fifth aspect of this application provides a computer program product, comprising a non-transitory computer-readable storage medium storing a computer program operable to cause a computer to perform some or all of the steps described in the first aspect of this application. This computer program product may be a software installation package.

[0015] As can be seen, in this embodiment, the state information of the sensors on the target vehicle is first obtained. Then, a joint state vector is constructed based on the state information. Next, the joint state vector is used to determine whether the current scene is in a degraded scenario. Further, if the joint state vector indicates that the current scene is in a degraded scenario, the current degradation type is determined. Further, based on the degradation type, a vehicle driving guidance command is generated. Then, based on the vehicle driving guidance command, a vehicle driving guidance screen is generated, and the display device is controlled to display the vehicle driving guidance screen. Next, when it is detected that the user executes the vehicle control operation corresponding to the vehicle driving guidance command, multiple first sensor data from multiple sensors on the target vehicle are obtained. Finally, first target data is filtered from the multiple first sensor data. By determining that the current scene is in a degraded scenario based on the sensor state information on the vehicle, different specific driving control operations are guided to the user based on the degradation type. Furthermore, by observing the sensor data and performing specific data filtering operations, high-quality first target data is filtered out, which helps improve the accuracy of sensor parameter calibration. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a schematic diagram of the architecture of a sensor parameter calibration system provided in an embodiment of this application.

[0018] Figure 2 This is a schematic flowchart of a sensor parameter calibration method provided in an embodiment of this application; Figure 3 This is a schematic diagram of a process for determining a state update increment provided in an embodiment of this application; Figure 4 This is a schematic diagram of a process for determining the degradation type provided in an embodiment of this application; Figure 5 This is a schematic diagram of a process for filtering target data provided in an embodiment of this application; Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application; Figure 7 This is a block diagram of the functional units of a sensor parameter calibration device provided in an embodiment of this application. Detailed Implementation

[0019] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0020] The terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0021] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0022] In the embodiments of this application, "and / or" describes the relationship between associated objects, indicating that three relationships can exist. For example, A and / or B can represent the following three situations: A exists alone; A and B exist simultaneously; B exists alone. Among them, A and B can be singular or plural.

[0023] In this embodiment, the symbol " / " can indicate that the preceding and following objects are in an "or" relationship. Alternatively, the symbol " / " can also represent a division sign, i.e., performing a division operation. For example, A / B can mean A divided by B.

[0024] In the embodiments of this application, "at least one item" or its similar expression refers to any combination of these items, including any combination of a single item or a plurality of items. "One or more" means one or more, while "multiple" means two or more. For example, "at least one item" of a, b, or c can represent the following seven cases: a, b, c; a and b; a and c; b and c; a, b, and c. Each of a, b, and c can be an element or a set containing one or more elements.

[0025] In the embodiments of this application, "equal to" can be used with "greater than" and is applicable to technical solutions used when "greater than" is used; it can also be used with "less than" and is applicable to technical solutions used when "less than" is used. When "equal to" is used with "greater than", it is not used with "less than"; when "equal to" is used with "less than", it is not used with "greater than".

[0026] To better understand the solutions of the embodiments of this application, the electronic devices, related concepts and background that may be involved in the embodiments of this application will be introduced below.

[0027] The electronic device described in this application embodiment is a device with wireless communication capabilities, and may be referred to as a terminal, user equipment (UE), mobile station (MS), mobile terminal (MT), access terminal device, vehicle-mounted terminal device, industrial control terminal device, UE unit, UE station, mobile station, remote station, remote terminal device, mobile device, UE terminal device, wireless communication device, UE agent, or UE device, etc. The terminal device can be fixed or mobile. It should be noted that the terminal device can support at least one wireless communication technology, such as LTE, New Radio (NR), Wideband Code Division Multiple Access (WCDMA), etc. For example, terminal devices can be mobile phones, tablets, desktop computers, laptops, all-in-one computers, in-vehicle terminals, virtual reality (VR) terminal devices, augmented reality (AR) terminal devices, wireless terminals in industrial control, wireless terminals in self-driving, wireless terminals in remote medical surgery, wireless terminals in smart grids, wireless terminals in transportation safety, wireless terminals in smart cities, wireless terminals in smart homes, cellular phones, cordless phones, session initiation protocol (SIP) phones, wireless local loop (WLL) stations, personal digital assistants (PDAs), handheld devices with wireless communication capabilities, electronic devices or other processing devices connected to a wireless modem, wearable devices, terminal devices in future mobile communication networks, or terminal devices in future evolved public land mobile networks (PLMNs), etc.

[0028] Please see Figure 1 , Figure 1This is a schematic diagram of the architecture of a sensor parameter calibration system provided in an embodiment of this application. The sensor parameter calibration system 1 includes a vehicle control system 10, multiple sensors 20, and a display device 30. The vehicle control system 10 is connected to the multiple sensors 20 and the display device 30 respectively.

[0029] The sensors include, but are not limited to, lidar, cameras, inertial measurement units (IMUs), velocity sensors, gyroscopes, and accelerometers.

[0030] The display device can be used to display vehicle driving guidance screens to guide users to perform specific vehicle driving operations.

[0031] In one possible example, the vehicle control system 10 first acquires the status information of the sensors 20 on the target vehicle. Then, the vehicle control system 10 constructs a joint state vector based on the status information. Next, the vehicle control system 10 determines whether the current scene is in a degradation scene based on the joint state vector. Further, if the vehicle control system 10 determines that the current scene is in a degradation scene based on the joint state vector, it determines the current degradation type. Further still, the vehicle control system 10 generates a vehicle driving guidance command based on the degradation type. Then, the vehicle control system 10 generates a vehicle driving guidance screen based on the vehicle driving guidance command and controls the display device 30 to display the vehicle driving guidance screen. Next, when the vehicle control system 10 detects that the user executes the vehicle control operation corresponding to the vehicle driving guidance command, it acquires multiple first sensor data from multiple sensors on the target vehicle. Finally, the vehicle control system 10 filters out the first target data from the multiple first sensor data. If the current scene is determined to be a degradation scene based on the status information of the sensor 20 on the vehicle, then the user is guided to perform different specific driving control operations according to the degradation type. Furthermore, the data of the sensor 20 is observed and specific data filtering operations are performed to filter out high-quality first target data, which helps to improve the accuracy of sensor parameter calibration.

[0032] Please see Figure 2 , Figure 2 This is a flowchart illustrating a sensor parameter calibration method provided in an embodiment of this application. The method includes: Step S201: Obtain the status information of the sensors on the target vehicle.

[0033] Step S202: Construct a joint state vector based on the state information.

[0034] The joint state vector x is defined as: x = [R, p, v, bg, ba, g, tau] TWhere R is the attitude of the IMU in the global coordinate system, p is the position, v is the velocity, bg and ba are the zero biases of the gyroscope and accelerometer, respectively, g is the gravity vector, and tau is the inverse of the camera exposure time. Optionally, x may also include the camera-LiDAR time synchronization deviation, Deltat, i.e., x is defined as: x = [R, p, v, bg, ba, g, tau, Deltat] T .

[0035] Step S203: Determine whether the current scene is in a degraded scene based on the joint state vector.

[0036] If the current scene is a degraded scene, it means that there is a certain state component with extremely low information content in certain directions, that is, the direction is unobservable.

[0037] Step S204: If the current scene is determined to be in a degraded scene based on the joint state vector, then the current degradation type is determined.

[0038] The degradation types include unobservable time parameters and unobservable spatial parameters. For example, unobservable time parameters can refer to unobservable time delays, while unobservable spatial parameters can refer to unobservable lateral translations and rotations.

[0039] Step S205: Generate vehicle driving guidance instructions based on the degradation type.

[0040] Optionally, generating vehicle driving guidance instructions based on the degradation type includes the following steps: when the degradation type is that the time parameter is unobservable, a first vehicle driving guidance instruction is generated, which is used to guide the user to perform a first vehicle control operation to change the longitudinal acceleration of the target vehicle; when the degradation type is that the spatial parameter is unobservable, a second vehicle driving guidance instruction is generated, which is used to guide the user to perform a second vehicle control operation to change the angular velocity of the target vehicle.

[0041] Step S206: Generate a vehicle driving guidance screen according to the vehicle driving guidance instruction, and control the display device to display the vehicle driving guidance screen.

[0042] Different vehicle driving guidance commands generate different vehicle driving guidance screens.

[0043] For example, when time deviation is unobservable, a pulse braking / rapid acceleration command is generated to guide the driver to create a sudden change in longitudinal acceleration; when rotation or lateral translation is unobservable, a screen including a virtual S-shaped trajectory is generated to guide the driver to perform continuous S-shaped turns, creating angular velocity excitation. Specifically, when the driver follows the serpentine trajectory, the vehicle will continuously turn left and right. At this time, the vehicle's yaw rate will exhibit an alternating positive and negative near-sine wave shape. When turning right, the angular velocity is positive, and when turning left, the angular velocity is negative. This alternating waveform is the excitation signal injected into the vehicle chassis and control system.

[0044] Step S207: When it is detected that the user executes the vehicle control operation corresponding to the vehicle driving guidance command, multiple first sensor data from multiple sensors on the target vehicle are acquired.

[0045] When a user executes a vehicle control operation corresponding to a vehicle driving guidance command, the corresponding spatiotemporal parameters are in an observable state, and data from multiple sensors are collected in real time.

[0046] The data from multiple sensors includes IMU data, LiDAR point cloud data, and camera data.

[0047] Step S208: Select the first target data from the plurality of first sensor data.

[0048] In particular, in scenarios with large parallax, such as S-shaped turns, to avoid the extrinsic parameters being affected by radar points hitting the foreground and pixels taking the background, depth consistency detection can be performed on multiple first sensors. In order to avoid feature degradation, spatiotemporal feature dual selection is then performed to filter out the first target data.

[0049] Optionally, if it is determined from the joint state vector that the current scene is not in a degraded scene, i.e., in a normal scene, then multiple second sensor data from multiple sensors on the target vehicle in the normal scene are obtained, and second target data is selected from the multiple second sensor data.

[0050] As can be seen, in this embodiment, the state information of the sensors on the target vehicle is first obtained. Then, a joint state vector is constructed based on the state information. Next, the joint state vector is used to determine whether the current scene is in a degraded scenario. Further, if the joint state vector indicates that the current scene is in a degraded scenario, the current degradation type is determined. Further, based on the degradation type, a vehicle driving guidance command is generated. Then, based on the vehicle driving guidance command, a vehicle driving guidance screen is generated, and the display device is controlled to display the vehicle driving guidance screen. Next, when it is detected that the user executes the vehicle control operation corresponding to the vehicle driving guidance command, multiple first sensor data from multiple sensors on the target vehicle are obtained. Finally, first target data is filtered from the multiple first sensor data. By determining that the current scene is in a degraded scenario based on the sensor state information on the vehicle, different specific driving control operations are guided to the user based on the degradation type. Furthermore, by observing the sensor data and performing specific data filtering operations, high-quality first target data is filtered out, which helps improve the accuracy of sensor parameter calibration.

[0051] In one possible example, regarding the determination of whether the current scene is in a degraded scene based on the joint state vector, the above method may include the following steps: determining a state update increment based on the joint state vector; if it is determined that the filter parameters are not convergent based on the state update increment, then determining a first information matrix based on the Jacobian matrix corresponding to the state update increment; determining a first minimum singular value of the first information matrix; if it is determined that the first minimum singular value is less than a first preset threshold, then determining that the current scene is in a degraded scene; if it is determined that the first minimum singular value is greater than or equal to the first preset threshold, then determining that the current scene is not in a degraded scene. If it is determined that the filter parameters are convergent based on the state update increment, then determining a current optimal state estimate based on the state update increment and the joint state vector; performing point cloud reprojection based on the current optimal state estimate, and determining the reprojection error; if it is determined that the parameters are inaccurate based on the reprojection error, then determining whether the current scene is in a degraded scene based on the first minimum singular value and the first preset threshold.

[0052] Please see Figure 3 Regarding determining the state update increment based on the joint state vector, the above method may include the following steps: Step S301: Construct a photometric error model.

[0053] To address scenarios with dramatic lighting changes, such as tunnel entry / exit and alternating light and dark conditions, the inverse tau of camera exposure time is incorporated into the joint state vector. The photometric error model is based on direct image alignment. Images captured by the camera contain multiple two-dimensional pixels, and the corresponding three-dimensional spatial points are the visual map points. For the visual map points... The reference image patch is projected onto the current frame through an affine transformation, and the photometric residual is calculated. ,in, and These are the inverse exposure times for the current frame and the reference frame, respectively. For visual map points The corresponding local affine transformation matrix can be obtained by linearizing the local homography matrix transformation from the reference frame to the current frame. and These are the grayscale values ​​of the current frame and the reference frame, respectively. Represents visual map points in the reference frame The corresponding pixel coordinates.

[0054] The photometric error model also incorporates rolling shutter compensation. Specifically, for CMOS cameras using rolling shutters, pixels in different rows are exposed at different times. Let the image's... The actual exposure time of the row pixels is: .

[0055] in, The start time of the frame. This represents the exposure time interval between adjacent rows. This represents the time synchronization deviation between the LiDAR and the camera. At this moment, the sensor's pose can be interpolated using IMU integration: , This is a Lie algebraic perturbation consisting of the IMU's angular and linear velocities. Therefore, position correction is required when projecting LiDAR points onto the image. ,in, The 3D coordinates of the lidar before correction. This is the estimated linear velocity at the current moment.

[0056] Step S302: Determine the residual vector based on the photometric error model and the joint state vector.

[0057] In this context, a single visual map point corresponds to a photometric residual, and the photometric residuals corresponding to multiple visual map points form a residual vector.

[0058] Step S303: Determine the Jacobian matrix based on the joint state vector.

[0059] Among them, for photometric residual Jacobi for tau is: Each luminance residual corresponds to a Jacobian, and multiple Jacobians form the Jacobian matrix H.

[0060] Step S304: Determine the state update increment based on the Jacobi matrix and the residual vector.

[0061] In the IESKF framework, parameter convergence is determined by the norm of the iterative update. Specifically, in the... After the next iteration, calculate the state update increment. , , Indicates the first The state update increment is calculated after the next iteration. Indicates the first The state update increment is calculated after the next iteration. This is the weight matrix. Indicates the first The residual vector of the next iteration Indicates the first Jacobi matrix of the next iteration Indicates the first The Jacobian matrix transpose is used in the next iteration. When the state update increment is less than the fourth preset threshold (e.g., the fourth preset threshold = 1e-3), the filter parameters are considered to have converged, and the iteration stops. The fourth preset threshold can be set manually or by system default, and is not limited here.

[0062] The information matrix is ​​H. T H. Where H is the Jacobian matrix, H T H is the transpose of the Jacobian matrix.

[0063] Let the information matrix M = H H. Perform singular value decomposition on M: M=UΣU .

[0064] Where the singular values ​​on the diagonal of Σ are σ1≥σ2≥ ≥σn≥0. The smallest singular value σmin is σn.

[0065] The first preset threshold can be set manually or by system default, and no restriction is made here.

[0066] The information matrix is ​​a measure of the information provided by the measurement data for each state component. When the minimum singular value or minimum eigenvalue of the information matrix is ​​close to zero, it indicates that the information content in the direction of some state components is extremely low, that is, the direction of the state component is considered unobservable.

[0067] Wherein, the current optimal state estimate = state update increment + joint state vector.

[0068] Among them, the optimal state estimate obtained by real-time calculation using IESKF is used to calculate the LiDAR global map points. Projected onto the current camera image plane. The projection formula is: ,in The pose of the IMU in the global coordinate system (including rotation R and translation p). The extrinsic parameters of the camera relative to the IMU. For camera projection functions. For pinhole cameras: ,in, Let these be the coordinates of the point in the camera coordinate system. and These are the principal point coordinates of the camera. This is the normalized focal length of the camera in the horizontal direction of the image. This is the normalized focal length of the camera in the vertical direction of the image.

[0069] When the spatiotemporal parameters are inaccurate, the reprojection position of LiDAR points will deviate from the true position of the corresponding object in the image, forming a visible ghost. The calibration status can be intuitively fed back through color coding. Calculation of reprojection error , Indicates the reprojected pixel coordinates. Represents the actual pixel coordinates, when When (the fifth preset threshold) is reached, the projection point is marked in red; when The time marker is green. Drivers or operators can observe the projection overlap in real time: red indicates inaccurate parameters, and green indicates accurate parameters. The fifth preset threshold can be set manually or by system default; no specific limit is imposed here. For example, the fifth preset threshold could be 5 pixels.

[0070] As can be seen, in the embodiments of this application, when the filter parameters do not converge, the minimum singular value of the information matrix can be used to determine whether the current scene is degraded. When the filter parameters converge, the reprojection error is further determined, the accuracy of the parameters is determined based on the reprojection error, and the current scene is further determined based on the minimum singular value of the information matrix, which helps to improve the accuracy of the degraded scene judgment.

[0071] Please see Figure 4 In determining the current degradation type, the above method may include the following steps: Step S401: Determine the right singular vector corresponding to the first minimum singular value.

[0072] Among them, the right singular vector vmin corresponding to the first minimum singular value is the last column of V.

[0073] Step S402: Determine the values ​​of the elements in the right singular vector that correspond to the horizontal translation to obtain the first value.

[0074] Step S403: Determine the value of the element component corresponding to the time delay in the right singular vector to obtain the second value.

[0075] Step S404: If it is determined that the first value is greater than the second preset threshold, then the current degradation type is determined to be unobservable spatial parameters.

[0076] The second preset threshold can be set manually or by system default, and is not limited here. For example, the second preset threshold is 0.8.

[0077] Step S405: If it is determined that the second value is greater than the third preset threshold, then the current degradation type is determined to be unobservable time parameter.

[0078] The third preset threshold can be set manually or by system default, and is not limited here. For example, the third preset threshold is 0.7.

[0079] Specifically, the values ​​of the corresponding rotated element components in the right singular vector can be determined to obtain a third value. If the third value is greater than a seventh preset threshold, the current degradation type is determined to be unobservable spatial parameters. The seventh preset threshold can be set manually or by system default; no specific limitation is made here. For example, the seventh preset threshold could be 0.8.

[0080] As can be seen, in this example, determining the degradation type by the values ​​of the elements of the right singular vector corresponding to the first minimum singular value helps improve the accuracy of degradation type determination.

[0081] Please see Figure 5 The method for selecting first target data from the plurality of first sensor data may include the following steps: Step S501: Project the lidar point cloud based on the data from the plurality of first sensors; Step S502: Perform depth continuity verification on multiple projection points to remove abnormal projection points; Optionally, the step of performing depth continuity verification on multiple projection points to remove abnormal projection points includes: determining the depth gradient in the neighborhood of each projection point; and removing projection points among the multiple projection points whose depth gradient is greater than a preset gradient threshold.

[0082] Specifically, a neighborhood window is determined using the center of the image projection points, for example, an N×N pixel rectangular window. Other projection points within the neighborhood window are searched. For each projection point and other projection points in its neighborhood, the depth gradient is determined by fitting a plane or calculating the difference.

[0083] The preset gradient threshold can be set manually or by system default, and no restriction is made here.

[0084] Step S503: Perform spatiotemporal selection operation on multiple projection points to obtain a target feature point set, and use the target feature point set as the first target data.

[0085] The spatiotemporal selection operation comprises two levels: spatial uniformity and field-of-view uniformity. For spatial uniformity, a 3D voxel hash map is used, with root voxels managed by a hash table. Each root voxel contains an octree structure storing local planes at different scales. Only representative feature points are retained within each voxel to ensure uniform feature distribution in 3D space and avoid excessive feature concentration in a particular local area. For field-of-view uniformity, the current image is divided into a uniform grid. Within each grid, only the projection point with the highest gray-level gradient response and passing depth continuity verification is retained. The gray-level gradient response is scored using corner points: for each LiDAR projection point's position on the image, a small window (e.g., 7x7 pixels) is taken centered on that point, and the image structure tensor is calculated. ,in, , are the gray-level gradients of the image in the horizontal and vertical directions, respectively. Eigenvalue decomposition yields The Shi-Tomasi score is defined as score = min( , ),in, , Let H be the eigenvalues ​​of the image structure tensor. This spatiotemporal dual screening ensures that the row rank of the Jacobian matrix H is full rank, thereby guaranteeing the stability of the state solution.

[0086] As can be seen, in this example, the sensor data undergoes both depth-based continuous verification and spatiotemporal dual selection to obtain the first target data, which helps to improve the accuracy of data screening and sensor parameter calibration.

[0087] Please see Figure 6 , Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application, such as... Figure 6 As shown, the electronic device 600 includes a processor 601, a memory 603, a communication interface 602, and one or more programs, wherein the one or more programs are stored in the memory 603, and the one or more programs are configured to be executed by the processor 601 with instructions for the following steps: Acquire the status information of the sensors on the target vehicle; Construct a joint state vector based on the state information; Determine whether the current scene is in a degenerate scene based on the joint state vector; If the current scene is determined to be a degraded scene based on the joint state vector, then the current degradation type is determined. Based on the degradation type, generate vehicle driving guidance instructions; Based on the vehicle driving guidance instructions, a vehicle driving guidance screen is generated, and the display device is controlled to display the vehicle driving guidance screen; When it is detected that the user executes the vehicle control operation corresponding to the vehicle driving guidance command, the data of multiple first sensors on multiple sensors on the target vehicle are acquired. First target data is selected from the multiple first sensor data.

[0088] As can be seen, in this embodiment, the electronic device 600 first acquires the state information of the sensors on the target vehicle, then constructs a joint state vector based on the state information, and then determines whether the current scene is in a degraded scene based on the joint state vector. Further, if the joint state vector indicates that the current scene is in a degraded scene, the current degradation type is determined. Further, based on the degradation type, a vehicle driving guidance command is generated, followed by a vehicle driving guidance screen, and the display device is controlled to display the vehicle driving guidance screen. Then, when it is detected that the user executes the vehicle control operation corresponding to the vehicle driving guidance command, multiple first sensor data from multiple sensors on the target vehicle are acquired. Finally, first target data is filtered from the multiple first sensor data. By determining that the current scene is in a degraded scene based on the sensor state information on the vehicle, the user is guided to execute different specific driving control operations according to the degradation type. Furthermore, by observing the sensor data and performing specific data filtering operations, high-quality first target data is filtered out, which helps improve the accuracy of sensor parameter calibration.

[0089] In one possible example, regarding the step of determining whether the current scene is in a degenerate scene based on the joint state vector, the above procedure includes instructions for performing the following steps: The state update increment is determined based on the joint state vector; If it is determined that the filter parameters do not converge based on the state update increment, then the first information matrix is ​​determined based on the Jacobian matrix corresponding to the state update increment. Determine the first minimum singular value of the first information matrix; If it is determined that the first minimum singular value is less than the first preset threshold, then the current scene is determined to be a degraded scene; If it is determined that the first minimum singular value is greater than or equal to the first preset threshold, then it is determined that the current scene is not in a degenerate scene; If the filter parameters converge based on the state update increment, then the current optimal state estimate is determined based on the state update increment and the joint state vector. Render the point cloud reprojection based on the current optimal state estimate and determine the reprojection error; If the parameters are determined to be inaccurate based on the reprojection error, then the current scene is determined to be in a degraded scene based on the first minimum singular value and the first preset threshold.

[0090] In one possible example, regarding the determination of the state update increment based on the joint state vector, the above procedure includes instructions for performing the following steps: Construct a photometric error model; The residual vector is determined based on the photometric error model and the joint state vector. Determine the Jacobian matrix based on the joint state vector; The state update increment is determined based on the Jacobi matrix and the residual vector.

[0091] In one possible example, regarding the determination of the current degradation type, the above procedure includes instructions for performing the following steps: Determine the right singular vector corresponding to the first minimum singular value; Determine the values ​​of the elements in the right singular vector that correspond to the horizontal translation to obtain the first value; Determine the values ​​of the element components corresponding to the time delay in the right singular vector to obtain the second value; If it is determined that the first value is greater than the second preset threshold, then the current degradation type is determined to be unobservable spatial parameters; If the second value is determined to be greater than the third preset threshold, then the current degradation type is determined to be unobservable time parameter.

[0092] In one possible example, in generating vehicle driving guidance instructions based on the degradation type, the above procedure also includes instructions for performing the following steps: When the degradation type is that the time parameter is unobservable, a first vehicle driving guidance instruction is generated. The first vehicle driving guidance instruction is used to guide the user to perform a first vehicle control operation to change the longitudinal acceleration of the target vehicle. When the degradation type is unobservable spatial parameters, a second vehicle driving guidance instruction is generated. The second vehicle driving guidance instruction is used to guide the user to perform a second vehicle control operation to change the angular velocity of the target vehicle.

[0093] In one possible example, regarding the filtering of first target data from the plurality of first sensor data, the above procedure further includes instructions for performing the following steps: LiDAR point cloud projection is performed based on the data from the plurality of first sensors; Perform depth continuity verification on multiple projection points to eliminate abnormal projection points; A spatiotemporal selection operation is performed on multiple projection points to obtain a target feature point set, which is then used as the first target data.

[0094] In one possible example, regarding the depth continuity check of multiple projection points to eliminate abnormal projection points, the above procedure also includes instructions for performing the following steps: Determine the depth gradient in the neighborhood of each projection point; Remove projection points from a set of projection points whose depth gradient is greater than a preset gradient threshold.

[0095] The above mainly describes the solutions of the embodiments of this application from the perspective of the method execution process. It is understood that, in order to achieve the above functions, the electronic device 600 includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should readily recognize that, in conjunction with the units and algorithm steps of the various examples described in the embodiments provided herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed by hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0096] This application embodiment can divide the electronic device 600 into functional units according to the above method example. For example, each function can be divided into a separate functional unit, or two or more functions can be integrated into one processing unit. The integrated unit can be implemented in hardware or as a software functional unit. It should be noted that the unit division in this application embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods.

[0097] When dividing each function into modules according to its corresponding function. Figure 7 This is a functional unit block diagram of a sensor parameter calibration device provided in an embodiment of this application, such as... Figure 7 As shown, the sensor parameter calibration device includes an acquisition unit 701, a determination unit 702, a judgment unit 703, a generation unit 704, and a filtering unit 705; wherein, The acquisition unit 701 is used to acquire the status information of the sensors on the target vehicle; it is also used to acquire multiple first sensor data of multiple sensors on the target vehicle when it detects that the user has executed a vehicle control operation corresponding to the vehicle driving guidance command. The determining unit 702 is configured to construct a joint state vector based on the state information; and to determine the current degradation type if it is determined from the joint state vector that the current scene is in a degradation scene. The judgment unit 703 is used to determine whether the current scene is in a degraded scene based on the joint state vector. The generation unit 704 is configured to generate a vehicle driving guidance instruction based on the degradation type; and to generate a vehicle driving guidance screen based on the vehicle driving guidance instruction, and to control a display device to display the vehicle driving guidance screen. The filtering unit 705 is used to filter out the first target data from the plurality of first sensor data.

[0098] As can be seen from the embodiments of this application, the sensor parameter calibration device first acquires the state information of the sensors on the target vehicle, then constructs a joint state vector based on the state information, and then determines whether the current scene is in a degraded scene based on the joint state vector. Further, if the joint state vector determines that the current scene is in a degraded scene, the current degradation type is determined. Further, based on the degradation type, a vehicle driving guidance command is generated, followed by a vehicle driving guidance screen, and the display device is controlled to display the vehicle driving guidance screen. Then, when it is detected that the user executes the vehicle control operation corresponding to the vehicle driving guidance command, multiple first sensor data from multiple sensors on the target vehicle are acquired. Finally, first target data is filtered from the multiple first sensor data. By determining that the current scene is in a degraded scene based on the state information of the sensors on the vehicle, and by guiding the user to execute different specific driving control operations according to the degradation type, and further observing the sensor data and performing specific data filtering operations to filter out high-quality first target data, the accuracy of sensor parameter calibration is improved.

[0099] In one possible example, regarding the determination of whether the current scene is in a degenerate scene based on the joint state vector, the determination unit 703 is specifically used for: The state update increment is determined based on the joint state vector; If it is determined that the filter parameters do not converge based on the state update increment, then the first information matrix is ​​determined based on the Jacobian matrix corresponding to the state update increment. Determine the first minimum singular value of the first information matrix; If it is determined that the first minimum singular value is less than the first preset threshold, then the current scene is determined to be a degraded scene; If it is determined that the first minimum singular value is greater than or equal to the first preset threshold, then it is determined that the current scene is not in a degenerate scene; If the filter parameters converge based on the state update increment, then the current optimal state estimate is determined based on the state update increment and the joint state vector. Render the point cloud reprojection based on the current optimal state estimate and determine the reprojection error; If the parameters are determined to be inaccurate based on the reprojection error, then the current scene is determined to be in a degraded scene based on the first minimum singular value and the first preset threshold.

[0100] In one possible example, regarding the determination of the state update increment based on the joint state vector, the determining unit 702 is specifically used for: Construct a photometric error model; The residual vector is determined based on the photometric error model and the joint state vector. Determine the Jacobian matrix based on the joint state vector; The state update increment is determined based on the Jacobi matrix and the residual vector.

[0101] In one possible example, the determining unit 702 is specifically used for determining the current degradation type in order to: Determine the right singular vector corresponding to the first minimum singular value; Determine the values ​​of the elements in the right singular vector that correspond to the horizontal translation to obtain the first value; Determine the values ​​of the element components corresponding to the time delay in the right singular vector to obtain the second value; If it is determined that the first value is greater than the second preset threshold, then the current degradation type is determined to be unobservable spatial parameters; If the second value is determined to be greater than the third preset threshold, then the current degradation type is determined to be unobservable time parameter.

[0102] In one possible example, regarding the generation of vehicle driving guidance instructions based on the degradation type, the generation unit 704 is further specifically configured to: When the degradation type is that the time parameter is unobservable, a first vehicle driving guidance instruction is generated. The first vehicle driving guidance instruction is used to guide the user to perform a first vehicle control operation to change the longitudinal acceleration of the target vehicle. When the degradation type is unobservable spatial parameters, a second vehicle driving guidance instruction is generated. The second vehicle driving guidance instruction is used to guide the user to perform a second vehicle control operation to change the angular velocity of the target vehicle.

[0103] In one possible example, in terms of filtering the first target data from the plurality of first sensor data, the filtering unit 705 is further specifically configured to: LiDAR point cloud projection is performed based on the data from the plurality of first sensors; Perform depth continuity verification on multiple projection points to eliminate abnormal projection points; A spatiotemporal selection operation is performed on multiple projection points to obtain a target feature point set, which is then used as the first target data.

[0104] In one possible example, in the process of performing depth continuity verification on multiple projection points to eliminate abnormal projection points, the filtering unit 705 is further specifically used for: Determine the depth gradient in the neighborhood of each projection point; Remove projection points from a set of projection points whose depth gradient is greater than a preset gradient threshold.

[0105] It should be noted that all relevant content of each step involved in the above method embodiments can be referenced from the functional description of the corresponding functional module, and will not be repeated here.

[0106] The electronic device provided in this embodiment is used to perform the above-described sensor parameter calibration method, and therefore can achieve the same effect as the above-described implementation method.

[0107] When using integrated units, the electronic device may include a processing module, a storage module, and a communication module. The processing module can be used to control and manage the actions of the electronic device; for example, it can support the electronic device in executing the steps performed by the aforementioned functional units. The storage module can support the electronic device in executing stored program code and data. The communication module can support communication between the electronic device and other devices.

[0108] The processing module can be a processor or a controller. It can implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. The processor can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of digital signal processing (DSP) and a microprocessor, etc. The storage module can be a memory. The communication module can specifically be a radio frequency circuit, a Bluetooth chip, a Wi-Fi chip, or other devices that interact with other electronic devices.

[0109] This application also provides a computer storage medium storing a computer program for electronic data interchange, which causes a computer to perform some or all of the steps of any of the methods described in the above method embodiments, wherein the computer includes an electronic device.

[0110] This application also provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program operable to cause a computer to perform some or all of the steps of any of the methods described in the above method embodiments. The computer program product may be a software installation package, and the computer includes a control platform.

[0111] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0112] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0113] In the several embodiments provided in this application, it should be understood that the disclosed apparatus can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical or other forms.

[0114] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0115] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0116] If the aforementioned integrated units are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned memory includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0117] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage device, which may include: a flash drive, a read-only memory, a random access memory, a magnetic disk, or an optical disk, etc.

[0118] The embodiments of this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for calibrating sensor parameters, characterized in that, include: Acquire the status information of the sensors on the target vehicle; Construct a joint state vector based on the state information; Determine whether the current scene is in a degenerate scene based on the joint state vector; If the current scene is determined to be a degraded scene based on the joint state vector, then the current degradation type is determined. Based on the degradation type, generate vehicle driving guidance instructions; According to the vehicle driving guidance command, a vehicle driving guidance screen is generated, and the display device is controlled to display the vehicle driving guidance screen. The degradation type includes unobservable time parameters and unobservable spatial parameters. When the user executes the vehicle control operation corresponding to the vehicle driving guidance command, both time and space parameters are observable. When it is detected that the user executes the vehicle control operation corresponding to the vehicle driving guidance command, the data of multiple first sensors on multiple sensors on the target vehicle are acquired. The first target data is selected from the multiple first sensor data.

2. The method according to claim 1, characterized in that, The step of determining whether the current scene is in a degenerate scene based on the joint state vector includes: The state update increment is determined based on the joint state vector; If the filter parameters are determined not to converge based on the state update increment, then the first information matrix is ​​determined based on the Jacobian matrix corresponding to the state update increment; and the first minimum singular value of the first information matrix is ​​determined. If it is determined that the first minimum singular value is less than the first preset threshold, then the current scene is determined to be a degraded scene; If it is determined that the first minimum singular value is greater than or equal to the first preset threshold, then it is determined that the current scene is not in a degenerate scene; If the filter parameters converge based on the state update increment, then the current optimal state estimate is determined based on the state update increment and the joint state vector; the point cloud is reprojected based on the current optimal state estimate, and the reprojection error is determined. If the parameters are determined to be inaccurate based on the reprojection error, then the current scene is determined to be in a degraded scene based on the first minimum singular value and the first preset threshold.

3. The method according to claim 2, characterized in that, Determining the state update increment based on the joint state vector includes: Construct a photometric error model; The residual vector is determined based on the photometric error model and the joint state vector. Determine the Jacobian matrix based on the joint state vector; The state update increment is determined based on the Jacobi matrix and the residual vector.

4. The method according to claim 3, characterized in that, Determining the current degradation type includes: Determine the right singular vector corresponding to the first minimum singular value; Determine the values ​​of the elements in the right singular vector that correspond to the horizontal translation to obtain the first value; Determine the values ​​of the element components corresponding to the time delay in the right singular vector to obtain the second value; If it is determined that the first value is greater than the second preset threshold, then the current degradation type is determined to be unobservable spatial parameters; If the second value is determined to be greater than the third preset threshold, then the current degradation type is determined to be unobservable time parameter.

5. The method according to claim 4, characterized in that, The step of generating vehicle driving guidance instructions based on the degradation type includes: When the degradation type is that the time parameter is unobservable, a first vehicle driving guidance instruction is generated. The first vehicle driving guidance instruction is used to guide the user to perform a first vehicle control operation to change the longitudinal acceleration of the target vehicle. When the degradation type is unobservable spatial parameters, a second vehicle driving guidance instruction is generated. The second vehicle driving guidance instruction is used to guide the user to perform a second vehicle control operation to change the angular velocity of the target vehicle.

6. The method according to claim 1, characterized in that, The step of filtering the first target data from the plurality of first sensor data includes: LiDAR point cloud projection is performed based on the data from the plurality of first sensors; Perform depth continuity verification on multiple projection points to eliminate abnormal projection points; A spatiotemporal selection operation is performed on multiple projection points to obtain a target feature point set, which is then used as the first target data.

7. The method according to claim 6, characterized in that, The process of performing depth continuity verification on multiple projection points to remove abnormal projection points includes: Determine the depth gradient in the neighborhood of each projection point; Remove projection points from a set of projection points whose depth gradient is greater than a preset gradient threshold.

8. A sensor parameter calibration device, characterized in that, The sensor parameter calibration device includes an acquisition unit, a determination unit, a judgment unit, a generation unit, and a filtering unit; wherein, The acquisition unit is used to acquire the status information of the sensors on the target vehicle; it is also used to acquire multiple first sensor data of multiple sensors on the target vehicle when it detects that the user has executed a vehicle control operation corresponding to the vehicle driving guidance command. The determining unit is configured to construct a joint state vector based on the state information; and to determine the current degradation type if the current scene is determined to be in a degradation scene based on the joint state vector. The judgment unit is used to determine whether the current scene is in a degraded scene based on the joint state vector; The generation unit is used to generate vehicle driving guidance instructions according to the degradation type; it is also used to generate vehicle driving guidance screen according to the vehicle driving guidance instructions, and control the display device to display the vehicle driving guidance screen. The degradation type includes unobservable time parameters and unobservable spatial parameters. When the user executes the vehicle control operation corresponding to the vehicle driving guidance instructions, both time and space parameters are observable. The filtering unit is used to filter out the first target data from the plurality of first sensor data.

9. An electronic device, characterized in that, It includes a processor and a memory, the memory being used to store one or more programs and configured to be executed by the processor, the programs including instructions for performing the steps of the method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, A computer program for storing electronic data interchange, wherein the computer program causes a computer to perform the method as described in any one of claims 1-7.

Images