Data preprocessing method based on virtual sensor

Abstract

The invention discloses a data preprocessing method based on a virtual sensor, and the method comprises the steps: firstly constructing a virtual sensor which is consistent with a sensor used in a training data set in internal and external parameters, and then converting a sensor coordinate system used in reasoning into a virtual sensor coordinate system according to a homography matrix; according to the method, the sensor coordinate system used during reasoning is converted into the virtual sensor coordinate system, the spatial consistency between the sensors is quickly ensured, the dependence of the reasoning sensors on brands and models is eliminated, the real-time response requirements of scenes such as automatic driving and industrial robots are met, the model performance and generalization ability are improved, and the real-time response requirements of the scenes such as the automatic driving and the industrial robots are met. The problem that the traditional data preprocessing technology neglects that a computer vision model generally adopts a large-scale public data set for training, and the internal and external parameters of a sensor used in the training data set are often inconsistent with those of a sensor used during reasoning, so that the model performance and generalization ability are insufficient is solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision and relates to a data preprocessing method, in particular to a data preprocessing method based on a virtual sensor. Background Art

[0002] Traditional data preprocessing technology is a key step before machine vision model reasoning. It can be used to ensure the consistency of data from different sensors in space and time, and provide reliable input for subsequent tasks. However, it ignores the fact that computer vision models are generally trained using large-scale public data sets, and the sensors used in the training data sets are often inconsistent with the internal and external parameters of the sensors used during reasoning, which ultimately leads to insufficient model performance and generalization capabilities. Summary of the invention

[0003] Compared with traditional data preprocessing technology, the present invention provides a data preprocessing method based on virtual sensors. This method solves the problem that traditional data preprocessing technology ignores the fact that computer vision models are generally trained using large-scale public data sets, and the sensors used in the training data sets are often inconsistent with the internal and external parameters of the sensors used during reasoning, which ultimately leads to insufficient model performance and generalization ability.

[0004] The objective of the present invention is achieved through the following technical solutions:

[0005] A data preprocessing method based on a virtual sensor comprises the following steps:

[0006] Step 1: Initialize the internal and external parameters of the virtual sensor and the sensor used during inference:

[0007] Step 1. Initialize the internal and external parameters of the virtual sensor:

[0008] The virtual sensor internal parameter matrix is ​​recorded as A i , the external parameter matrix is ​​recorded as (R i ,T i ), where: A i Contains the focal length, principal point coordinates and distortion coefficients of the virtual sensor, R i is the relative rotation matrix between different virtual sensor coordinate systems, T i is the relative displacement vector between different virtual sensor coordinate systems;

[0009] Step 1 and 2: Initialize the internal and external parameters of the sensor used during inference:

[0010] The sensor internal parameter matrix used in inference is denoted as A j , the external parameter matrix is ​​recorded as (R j ,T j ), where: A jContains the focal length, principal point coordinates, and distortion coefficients of the sensor used during inference, R j is the relative rotation matrix between different sensor coordinate systems used in inference, T j is the relative displacement vector between different sensor coordinate systems used in inference;

[0011] Step 2: Calculate the homography matrix:

[0012] Step 21: Segmentation of the visual task plane:

[0013] The visual task focuses on a virtual plane P that is tangent to the local ground in the data. v , the virtual plane P v Obtained by setting z = 0 in the ground coordinate system G = (x, y, z);

[0014] Step 22: Calculate normalized coordinates:

[0015] In the virtual plane P v Select four different feature points x i =(x i ,y i ,0) T , where x i ,y i is the position coordinate of the feature point and i = 1, 2, 3, 4, which are then projected onto the images of the virtual sensor and the sensor used during inference to obtain the normalized coordinates and where k = 1, 2, 3, 4, u i and v i Indicates that the feature points are scaled and translated on the plane P v The position coordinates on ;

[0016] Step 2: Calculate the elements in the homography matrix:

[0017] Calculate the 3×3 homography matrix H using the least squares method i,j ,in:

[0018]

[0019] Step 3: Transformation of coordinate system before model reasoning:

[0020] The sensor coordinate system A used during inference i Transform to virtual sensor coordinate system A j middle:

[0021] H i,j A i =A j .

[0022] Compared with the prior art, the present invention has the following advantages:

[0023] The present invention transforms the sensor coordinate system used in reasoning into a virtual sensor coordinate system through a data preprocessing method, which quickly ensures the spatial consistency between sensors while getting rid of the dependence of reasoning sensors on brands and models. It also improves model performance and generalization capabilities while meeting the real-time response requirements of scenarios such as autonomous driving and industrial robots. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 It is an overall flow chart of the data preprocessing method based on virtual sensors of the present invention.

[0025] Figure 2 It is a schematic diagram of transforming a sensor used in reasoning of the data preprocessing method based on a virtual sensor of the present invention into a virtual sensor. DETAILED DESCRIPTION

[0026] The technical solution of the present invention is further described below in conjunction with the accompanying drawings, but is not limited thereto. Any modification or equivalent replacement of the technical solution of the present invention without departing from the spirit and scope of the technical solution of the present invention should be included in the protection scope of the present invention.

[0027] The present invention provides a data preprocessing method based on virtual sensors. By constructing a virtual sensor with internal and external parameters consistent with those of the sensor used in the training data set, the sensor data coordinate system used in reasoning is transformed into the virtual sensor data coordinate system according to the homography matrix, which quickly ensures the spatial consistency between sensors and improves the performance and generalization ability of the model. Figure 1 As shown, the method specifically comprises the following steps:

[0028] Step 1: Initialize the internal and external parameters of the virtual sensor and the sensor used during inference:

[0029] Step 1. Initialize the internal and external parameters of the virtual sensor:

[0030] The internal and external parameters of the virtual sensor are the average values ​​of the internal and external parameters of the sensor in the training data set. The virtual sensor internal parameter matrix is ​​denoted as A i , A i Contains the focal length, principal point coordinates and distortion coefficients of the virtual sensor; the external parameter matrix is ​​recorded as (R i ,T i ), R i is the relative rotation matrix between different virtual sensor coordinate systems, T i is the relative displacement vector between different virtual sensor coordinate systems.

[0031] Step 1 and 2: Initialize the internal and external parameters of the sensor used during inference:

[0032] The parameters of the sensor used in inference are obtained by calibration, which is generally calculated by taking images of the calibration plate at different angles and positions. The sensor internal parameter matrix used in inference is denoted as A j , A j Contains the focal length, principal point coordinates, and distortion coefficients of the sensor used during inference; the external parameter matrix is ​​recorded as (R j ,T j ), R j is the relative rotation matrix between different sensor coordinate systems used in inference, T j is the relative displacement vector between different sensor coordinate systems used during inference.

[0033] Step 2: Calculate the homography matrix:

[0034] Step 21: Segmentation of the visual task plane:

[0035] The visual task focuses on a virtual plane P that is tangent to the ground in the data. v , the virtual plane P v It is obtained by setting z=0 in the ground coordinate system G=(x, y, z).

[0036] Step 22: Calculate normalized coordinates:

[0037] Homography matrix Medium 9 = 1 and has 8 degrees of freedom, then at least 4 feature points are required for solution. v Select four different feature points x i =(x i ,y i ,0) T , where x i ,y i is the position coordinate of the feature point and i = 1, 2, 3, 4. Then it is projected onto the image of the virtual sensor and the sensor used in inference to obtain the normalized coordinates and in and Indicates that the feature points are scaled and translated on the plane P v The position coordinates on .

[0038] Step 2: Calculate the elements in the homography matrix:

[0039] Calculate the 3×3 homography matrix H using the least squares method i,j ,in:

[0040]

[0041] It represents the rotation and translation relationship between the virtual sensor and the sensor coordinate system used during inference.

[0042] Step 3: Coordinate system transformation before model reasoning:

[0043] The sensor coordinate system A used during inference i Transform to virtual sensor coordinate system A j middle:

[0044] H i,j A i =A j .

[0045] Example:

[0046] This example uses real urban road scene data for experiments. The entire road is 1.2 kilometers long, with a total of 1,180 frames of images. In the experiment, a calibration plate was used to calibrate the internal and external parameters of the sensor used in reasoning. The 1,180 frames of images used in the experiment were manually labeled. In the experiment, multiple popular target detection models were used. The data set for training the model uses the public data set Kitti, and the internal and external parameters of the sensor in the Kitti data set are used as the internal and external parameters of the virtual sensor. The coordinates of the sensor data used in reasoning are transformed to the coordinates of the virtual sensor data. Finally, the superiority of the present invention is illustrated by comparing the target detection accuracy before and after the use of the present invention.

[0047] In this embodiment, the overall process of the data preprocessing method based on the virtual sensor is as follows: Figure 1 As shown in Figure 2, the sensor used in reasoning is transformed into a virtual sensor as shown in Figure 2. Figure 2 As shown, the specific steps are as follows:

[0048] Step 1: Initialize the internal and external parameters of the virtual sensor and the sensor during inference:

[0049] Step 1. Initialize the internal and external parameters of the virtual sensor:

[0050] The internal and external parameters of the virtual sensor are the average values ​​of the internal and external parameters of the sensors in the training data set. The internal and external parameters of the sensors that have been calibrated with high precision in the public data set Kitti are selected. The internal parameter matrix of the virtual sensor is denoted as A i , A i Contains the focal length, principal point coordinates and distortion coefficients of the virtual sensor; the external parameter matrix is ​​recorded as (R i ,T i ), R i is the relative rotation matrix between different virtual sensor coordinate systems, T i is the relative displacement vector between different virtual sensor coordinate systems.

[0051] Camera intrinsic matrix The external parameter matrix from the lidar to the camera

[0052] Step 1 and 2: Initialize the internal and external parameters of the sensor during inference:

[0053] The parameters of the sensor used during inference are obtained by calibration, and the internal parameter matrix of the sensor used during inference is recorded as A j , the external parameter matrix is ​​recorded as (R j ,T j ). j Contains the focal length, principal point coordinates, and distortion coefficients of the sensor used during inference; the external parameter matrix is ​​recorded as (R j ,T j ), R j is the relative rotation matrix between different sensor coordinate systems used in inference, T j is the relative displacement vector between different sensor coordinate systems used during inference.

[0054] Camera intrinsic matrix The external parameter matrix from the lidar to the camera

[0055] Step 2: Calculate the homography matrix:

[0056] Step 21: Segmentation of the visual task plane:

[0057] The target detection task focuses on the virtual plane P that is tangent to the ground in the image. v , obtained by setting z = 0 in the ground coordinate system G = (x, y, z).

[0058] Step 22: Calculate normalized coordinates:

[0059] Homography matrix Medium 9 = 1 and has 8 degrees of freedom, then at least 4 feature points are required for solution. v Select four different feature points x i =(x i ,y i ,0) T , where x i ,y i is the position coordinate of the feature point and i = 1, 2, 3, 4. Then it is projected onto the image of the virtual sensor and the sensor used during inference to obtain the normalized coordinates and in and Indicates that the feature points are scaled and translated on the plane P vThe position coordinates on .

[0060] Step 2: Calculate the elements in the homography matrix:

[0061] Calculate the homography matrix 3×3 H according to the least squares method i,j ,in:

[0062]

[0063] It represents the rotation and translation relationship between the virtual sensor and the sensor coordinate system used during inference.

[0064] Step 3: Transformation of coordinate system before model reasoning:

[0065] The sensor coordinate system A used during inference i Transform to virtual sensor coordinate system A j middle:

[0066] H i,j A i =A j .

[0067] This embodiment experiments and verifies the method in a real urban road scene, using a total of 1180 frames of images and 3 types of labels (people, cars, motorcycles) for target detection tasks. The embodiment uses 4 mainstream deep learning model backbones and the same target detection task head, and conducts experiments with and without the present invention. 148 frames of experimental images containing scenes that have not appeared in the training data set and more than ten instances are recorded as difficult cases, and mAP (mean average precision) is used as a quantitative indicator.

[0068] Table 1 Implementation results on real urban road dataset

[0069]

[0070] The results in Table 1 show that the overall detection accuracy of the model using the present invention for data preprocessing in the experiment is significantly improved, and it performs better when facing complex scenes that have not appeared in the training data set, and the average inference time is only increased by 8%, which proves that the present invention effectively improves the performance and generalization ability of the model.

Claims

1. A data preprocessing method based on virtual sensor, characterized in that The method comprises the following steps: Step 1: Initialize the internal and external parameters of the virtual sensor and the sensor used during inference: Step 1. Initialize the internal and external parameters of the virtual sensor: The virtual sensor internal parameter matrix is ​​recorded as A i , the external parameter matrix is ​​recorded as (R i ,T i ), where: A i Contains the focal length, principal point coordinates and distortion coefficients of the virtual sensor; the external parameter matrix is ​​recorded as (R i ,T i ), R i is the relative rotation matrix between different virtual sensor coordinate systems, T i is the relative displacement vector between different virtual sensor coordinate systems; Step 1 and 2: Initialize the internal and external parameters of the sensor used during inference: The sensor internal parameter matrix used in inference is denoted as A j , the external parameter matrix is ​​recorded as (R j ,T j ), where: A j Contains the focal length, principal point coordinates, and distortion coefficients of the sensor used during inference; the external parameter matrix is ​​recorded as (R j ,T j ), R j is the relative rotation matrix between different sensor coordinate systems used in inference, T j is the relative displacement vector between different sensor coordinate systems used in inference; Step 2: Calculate the homography matrix: Step 21: Segmentation of the visual task plane: The visual task focuses on a virtual plane P that is tangent to the local ground in the data. v , the virtual plane P v Obtained by setting z = 0 in the ground coordinate system G = (x, y, z); Step 22: Calculate normalized coordinates: In the virtual plane P v Select four different feature points x i =(x i ,y i ,0) T , where x i ,y i is the position coordinate of the feature point and i = 1, 2, 3, 4, which are then projected onto the images of the virtual sensor and the sensor used during inference to obtain the normalized coordinates and where k = 1, 2, 3, 4, u i and v i Indicates that the feature points are scaled and translated on the plane P v The position coordinates on ; Step 2: Calculate the elements in the homography matrix: Calculate the 3×3 homography matrix H using the least squares method i,j ,in: Step 3: Transformation of data coordinate system before model inference: The coordinate system A of the sensor data used in inference i Transform to the coordinate system A of the virtual sensor data j middle: H i,j A i =A j 。 2. The data preprocessing method based on virtual sensor according to claim 1 is characterized in that In the step 11, the internal and external parameters of the virtual sensor are the average values ​​of the internal and external parameters of the sensor in the training data set.

3. The data preprocessing method based on virtual sensor according to claim 1 is characterized in that In the steps 1 and 2, the parameters of the sensor used in reasoning are obtained by calibration, and are calculated by taking images of the calibration plate at different angles and positions by the sensor.

4. The data preprocessing method based on virtual sensor according to claim 1 is characterized in that In the step 22,

Images