A data preprocessing method based on virtual sensors

Abstract

This invention discloses a data preprocessing method based on virtual sensors. First, a virtual sensor with identical intrinsic and extrinsic parameters to the sensors used in the training dataset is constructed. Then, the sensor coordinate system used during inference is transformed into the virtual sensor coordinate system using a homography matrix. This invention transforms the sensor coordinate system used during inference into a virtual sensor coordinate system, quickly ensuring spatial consistency between sensors while eliminating dependence on brand and model of inference sensors. Furthermore, it improves model performance and generalization ability while meeting the real-time response requirements of scenarios such as autonomous driving and industrial robots. This addresses the problem of traditional data preprocessing techniques neglecting the fact that computer vision models are generally trained on large-scale public datasets, and the sensors used in the training dataset often have inconsistent endo and extrinsic parameters with those used during inference, ultimately leading to insufficient model performance and generalization ability.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology and relates to a data preprocessing method, specifically a data preprocessing method based on virtual sensors. Background Technology

[0002] Traditional data preprocessing techniques, as a key step before inference in machine vision models, can be used to ensure the consistency of data from different sensors in space and time, providing reliable input for subsequent tasks. However, they overlook the fact that computer vision models are generally trained using large-scale public datasets. The sensors used in the training dataset often have inconsistent intrinsic and extrinsic parameters with those used during inference, ultimately leading to insufficient model performance and generalization ability. Summary of the Invention

[0003] Compared to traditional data preprocessing techniques, this invention provides a data preprocessing method based on virtual sensors. This method addresses the problem that traditional data preprocessing techniques neglect the fact that computer vision models are generally trained on large-scale public datasets, and the sensors used in the training dataset often have inconsistent endo and extrinsic parameters with the sensors used during inference, ultimately leading to insufficient model performance and generalization ability.

[0004] The objective of this invention is achieved through the following technical solution:

[0005] A data preprocessing method based on virtual sensors includes the following steps:

[0006] Step 1: Initialize the intrinsic and extrinsic parameters of the virtual sensor and the sensor used during inference:

[0007] Step 11: Initialize the intrinsic and extrinsic parameters of the virtual sensor:

[0008] Let the virtual sensor intrinsic parameter matrix be denoted as A. i The extrinsic parameter matrix is ​​denoted as (R i ,T i ), where: A i Includes the focal length, principal point coordinates, and distortion coefficients of the virtual sensor, R i T is the relative rotation matrix between different virtual sensor coordinate systems. i This represents the relative displacement vector between different virtual sensor coordinate systems;

[0009] Steps 1 and 2: Initialize the intrinsic and extrinsic parameters of the sensors used during inference:

[0010] Let A denote the sensor intrinsic parameter matrix used during inference. j The extrinsic parameter matrix is ​​denoted as (R j ,T j ), where: A jIncludes the focal length, principal point coordinates, and distortion coefficients of the sensors used during inference, R j T is the relative rotation matrix between different sensor coordinate systems used during inference. j This refers to the relative displacement vector between different sensor coordinate systems used during inference;

[0011] Step 2: Homography Matrix Calculation

[0012] Step 2.1, Segmentation of the visual task plane:

[0013] The visual task focuses on a virtual plane P that is tangent to the local ground in the data. v The virtual plane P v In the ground coordinate system G = (x, y, z), this is obtained by setting z = 0;

[0014] Step 22: Calculate the normalized coordinates:

[0015] In the virtual plane P v Select four different feature points x i =(x i ,y i ,0) T , where x i y i These are the position coordinates of the feature points, where i = 1, 2, 3, 4. These coordinates are then projected onto the images of the virtual sensor and the sensor used during inference, respectively, to obtain normalized coordinates. and Where k = 1, 2, 3, 4, u i and v i This indicates that the feature points, after scaling and translation, are in plane P. v Position coordinates on;

[0016] Steps 2 and 3: Calculate the elements in the homography matrix:

[0017] Calculate the 3×3 homography matrix H using the least squares method. i,j ,in:

[0018]

[0019] Step 3: Coordinate system transformation before model inference:

[0020] The sensor coordinate system A used during inference i Transform to virtual sensor coordinate system A j middle:

[0021] H i,j A i =A j .

[0022] Compared with the prior art, the present invention has the following advantages:

[0023] This invention transforms the sensor coordinate system used during inference into a virtual sensor coordinate system through a data preprocessing method. This quickly ensures spatial consistency between sensors, eliminates the dependence on brand and model of inference sensors, and improves model performance and generalization ability while meeting the real-time response requirements of scenarios such as autonomous driving and industrial robots. Attached Figure Description

[0024] Figure 1 This is an overall flowchart of the data preprocessing method based on virtual sensors according to the present invention.

[0025] Figure 2 This is a schematic diagram illustrating the sensor transformation to a virtual sensor used in the inference process of the data preprocessing method based on a virtual sensor in this invention. Detailed Implementation

[0026] The technical solution of the present invention will be further described below with reference to the accompanying drawings, but it is not limited thereto. Any modifications or equivalent substitutions to the technical solution of the present invention that do not depart from the spirit and scope of the technical solution of the present invention should be covered within the protection scope of the present invention.

[0027] This invention provides a data preprocessing method based on virtual sensors. By constructing a virtual sensor with the same intrinsic and extrinsic parameters as the sensors used in the training dataset, the sensor data coordinate system used during inference is transformed into the virtual sensor data coordinate system according to the homography matrix. This quickly ensures spatial consistency between sensors and improves the model's performance and generalization ability. Figure 1 As shown, the method specifically includes the following steps:

[0028] Step 1: Initialize the intrinsic and extrinsic parameters of the virtual sensor and the sensor used during inference:

[0029] Step 11: Initialize the intrinsic and extrinsic parameters of the virtual sensor:

[0030] The intrinsic and extrinsic parameters of the virtual sensor are the average values ​​of the intrinsic and extrinsic parameters of the sensor in the training dataset. Let A denote the intrinsic parameter matrix of the virtual sensor. i A i Includes the focal length, principal point coordinates, and distortion coefficients of the virtual sensor; the extrinsic parameter matrix is ​​denoted as (R... i ,T i ), R i T is the relative rotation matrix between different virtual sensor coordinate systems. i This represents the relative displacement vector between different virtual sensor coordinate systems.

[0031] Steps 1 and 2: Initialize the intrinsic and extrinsic parameters of the sensors used during inference:

[0032] The parameters of the sensors used during inference are obtained through calibration, typically calculated by taking images of a calibration board at different angles and positions using the sensors. The intrinsic parameter matrix of the sensors used during inference is denoted as A. j A j Includes the focal length, principal point coordinates, and distortion coefficients of the sensors used during inference; the extrinsic parameter matrix is ​​denoted as (R... j ,T j ), R j T is the relative rotation matrix between different sensor coordinate systems used during inference. j This refers to the relative displacement vector between different sensor coordinate systems used during inference.

[0033] Step 2: Homography Matrix Calculation

[0034] Step 2.1, Segmentation of the visual task plane:

[0035] The visual task focuses on a virtual plane P that is tangent to the ground in the data. v The virtual plane P v In the ground coordinate system G = (x, y, z), it is obtained by setting z = 0.

[0036] Step 22: Calculate the normalized coordinates:

[0037] homography matrix Since h9 = 1 and has 8 degrees of freedom, at least 4 feature points are needed for the solution in the virtual plane P. v Select four different feature points x i =(x i ,y i ,0) T , where x i y i These are the position coordinates of the feature points, where i = 1, 2, 3, 4. They are then projected onto the images of the virtual sensor and the sensor used during inference, respectively, to obtain normalized coordinates. and in and This indicates that the feature points, after scaling and translation, are in plane P. v The position coordinates on the [top].

[0038] Steps 2 and 3: Calculate the elements in the homography matrix:

[0039] Calculate the 3×3 homography matrix H using the least squares method. i,j ,in:

[0040]

[0041] It represents the rotation and translation relationship between the virtual sensor and the sensor coordinate system used during inference.

[0042] Step 3: Coordinate system transformation before model inference:

[0043] The sensor coordinate system A used during inference i Transform to virtual sensor coordinate system A j middle:

[0044] H i,j A i =A j .

[0045] Example:

[0046] This example uses real-world urban road scene data for experimentation. The entire road section is 1.2 kilometers long, and a total of 1180 image frames are generated. A calibration board was used to calibrate the intrinsic and extrinsic parameters of the sensor used during inference. The 1180 image frames used in the experiment were manually labeled. Several popular object detection models were used in the experiment. The dataset used to train the models was the publicly available Kitti dataset. The intrinsic and extrinsic parameters of the sensor in the Kitti dataset were used as the intrinsic and extrinsic parameters of the virtual sensor. The coordinates of the sensor data used during inference were transformed to the coordinates of the virtual sensor data. Finally, the superiority of this invention is demonstrated by comparing the object detection accuracy before and after using this invention.

[0047] In this embodiment, the overall process of the data preprocessing method based on virtual sensors is as follows: Figure 1 As shown, the transformation from the sensor used during inference to a virtual sensor is illustrated in the diagram. Figure 2 As shown, the specific steps are as follows:

[0048] Step 1: Initialize the virtual sensor and the sensor's intrinsic and extrinsic parameters during inference:

[0049] Step 11: Initialize the intrinsic and extrinsic parameters of the virtual sensor:

[0050] The intrinsic and extrinsic parameters of the virtual sensor are the average values ​​of the intrinsic and extrinsic parameters of the sensors in the training dataset. High-precision calibrated sensor intrinsic and extrinsic parameters are selected from the publicly available dataset Kitti. The virtual sensor intrinsic parameter matrix is ​​denoted as A. i A i Includes the focal length, principal point coordinates, and distortion coefficients of the virtual sensor; the extrinsic parameter matrix is ​​denoted as (R... i ,T i ), R i T is the relative rotation matrix between different virtual sensor coordinate systems. i This represents the relative displacement vector between different virtual sensor coordinate systems.

[0051] Camera intrinsic matrix LiDAR to camera extrinsic matrix

[0052] Steps 1 and 2: Initialize the sensor using its intrinsic and extrinsic parameters during inference:

[0053] The parameters of the sensors used during inference are obtained through calibration, specifically by calibrating the sensors used during inference in the experiment using a calibration board. The intrinsic parameter matrix of the sensors used during inference is denoted as A. j The extrinsic parameter matrix is ​​denoted as (R j ,T j A j Includes the focal length, principal point coordinates, and distortion coefficients of the sensors used during inference; the extrinsic parameter matrix is ​​denoted as (R... j ,T j ), R j T is the relative rotation matrix between different sensor coordinate systems used during inference. j This refers to the relative displacement vector between different sensor coordinate systems used during inference.

[0054] Camera intrinsic matrix LiDAR to camera extrinsic matrix

[0055] Step 2: Homography Matrix Calculation

[0056] Step 2.1, Segmentation of the visual task plane:

[0057] The object detection task focuses on a virtual plane P that is tangent to the ground in the image. v In the ground coordinate system G=(x,y,z), it is obtained by setting z=0.

[0058] Step 22: Calculate the normalized coordinates:

[0059] homography matrix Since h9 = 1 and has 8 degrees of freedom, at least 4 feature points are needed for the solution in the virtual plane P. v Select four different feature points x i =(x i ,y i ,0) T , where x i y i These are the position coordinates of the feature points, where i = 1, 2, 3, 4. These coordinates are then projected onto the images from the virtual sensor and the sensor used during inference, respectively, to obtain normalized coordinates. and in and This indicates that the feature points, after scaling and translation, are in plane P. v The position coordinates on the [top].

[0060] Steps 2 and 3: Calculate the elements in the homography matrix:

[0061] Calculate the H of the 3×3 homography matrix using the least squares method. i,j ,in:

[0062]

[0063] It represents the rotation and translation relationship between the virtual sensor and the sensor coordinate system used during inference.

[0064] Step 3: Coordinate system transformation before model inference:

[0065] The sensor coordinate system A used during inference i Transform to virtual sensor coordinate system A j middle:

[0066] H i,j A i =A j .

[0067] This embodiment experiments and verifies the method in a real urban road scene, using a total of 1180 frames of images and 3 types of labels (people, vehicles, and motorcycles) for object detection. The embodiment uses four mainstream deep learning model backbones and the same object detection task head, conducting experiments with and without this invention. 148 experimental frames containing more than ten instances of scenes not present in the training dataset are designated as hard examples, and mAP (mean accuracy) is used as the quantification metric.

[0068] Table 1 shows the implementation results in the real urban road dataset.

[0069]

[0070] The results in Table 1 show that the overall detection accuracy of the model using the present invention for data preprocessing is significantly improved in the experiment. It performs better when facing complex scenarios that have not appeared in the training dataset, and the average inference time only increases by 8%, which proves that the present invention effectively improves the performance and generalization ability of the model.

Claims

1. A data preprocessing method based on virtual sensors, characterized in that... The method includes the following steps: Step 1: Initialize the intrinsic and extrinsic parameters of the virtual sensor and the sensor used during inference: Step 11: Initialize the intrinsic and extrinsic parameters of the virtual sensor: Let the virtual sensor intrinsic parameter matrix be denoted as A. i The extrinsic parameter matrix is ​​denoted as (R i ,T i ), where: A i Includes the focal length, principal point coordinates, and distortion coefficients of the virtual sensor; the extrinsic parameter matrix is ​​denoted as (R... i ,T i ), R i T is the relative rotation matrix between different virtual sensor coordinate systems. i This represents the relative displacement vector between different virtual sensor coordinate systems; Steps 1 and 2: Initialize the intrinsic and extrinsic parameters of the sensors used during inference: Let A denote the sensor intrinsic parameter matrix used during inference. j The extrinsic parameter matrix is ​​denoted as (R j ,T j ), where: A j Includes the focal length, principal point coordinates, and distortion coefficients of the sensors used during inference; the extrinsic parameter matrix is ​​denoted as (R... j ,T j ), R j T is the relative rotation matrix between different sensor coordinate systems used during inference. j This refers to the relative displacement vector between different sensor coordinate systems used during inference; Step 2: Homography Matrix Calculation Step 2.1, Segmentation of the visual task plane: The visual task focuses on a virtual plane P that is tangent to the local ground in the data. v The virtual plane P v In the ground coordinate system G = (x, y, z), this is obtained by setting z = 0; Step 22: Calculate the normalized coordinates: In the virtual plane P v Select four different feature points x i =(x i ,y i ,0) T , where x i y i These are the position coordinates of the feature points, where i = 1, 2, 3, 4. These coordinates are then projected onto the images of the virtual sensor and the sensor used during inference, respectively, to obtain normalized coordinates. and Where k = 1, 2, 3, 4, u i and v i This indicates that the feature points, after scaling and translation, are in plane P. v Position coordinates on; Steps 2 and 3: Calculate the elements in the homography matrix: Calculate the 3×3 homography matrix H using the least squares method. i,j ,in: Step 3: Transformation of the data coordinate system before model inference: The coordinate system A of the sensor data used during inference i Coordinate system A transformed to virtual sensor data j middle: H i,j A i =A j 。 2. The data preprocessing method based on virtual sensors according to claim 1, characterized in that... In step one, the intrinsic and extrinsic parameters of the virtual sensor are the average values ​​of the intrinsic and extrinsic parameters of the sensor in the training dataset.

3. The data preprocessing method based on virtual sensors according to claim 1, characterized in that... In steps one and two, the parameters of the sensor used during inference are obtained through calibration, calculated by taking images of the calibration plate at different angles and positions using the sensor.

4. The data preprocessing method based on virtual sensors according to claim 1, characterized in that... In step two,

Images