Automobile monitoring image simulation system and method

Abstract

The invention discloses an automobile monitoring image simulation system and method in the technical field of photoelectric detection, and the system comprises a display module which is used for projecting an automobile multi-view image to display equipment; the mapping module is used for controlling the real camera to collect the automobile multi-view image displayed by the display equipment, establishing a mapping relation between the real camera and the virtual camera, and obtaining a virtual view camera image according to the mapping relation; and the image processing module is used for inputting the virtual view angle camera image into a pre-trained image processing model to obtain a fitted lane line and human body posture estimation, and superposing the fitted lane line and human body posture estimation on the virtual view angle camera image to obtain a target image. The technical problem that in the prior art, due to the fact that data generated through computer simulation does not conform to the actual working condition, the accuracy of experimental samples needed by testing is lowered, and the result obtained through testing is further distorted can be solved.

Description

Technical Field

[0001] The invention belongs to the technical field of photoelectric detection technology and relates to an automobile monitoring image simulation system and method. Background Art

[0002] In the actual road environment, driverless cars need to face various complex and sudden situations, such as traffic jams, pedestrians crossing, emergency braking, etc., and usually make autonomous driving decisions, vehicle tracking, pedestrian behavior prediction and other responses through images fed back by car monitoring. By testing the response of driverless cars to complex and sudden vehicle conditions, the decision-making ability, reaction speed and control accuracy of the driverless system can be verified, and technical optimization can be carried out for existing problems. However, the experimental samples required for testing are usually obtained through simulation testing, field testing and road testing. Among them, the scene of field testing is single and cannot approach the real scene; the cost of road testing is high and the operation is complicated; simulation testing simulates various driving scenes and road conditions through computers to conduct a large number of tests and data analysis. However, the existing simulation tests have the problem that the data generated by computer simulation does not match the actual working conditions, resulting in a decrease in the accuracy of the experimental samples required for the test, and further distorting the test results. Summary of the invention

[0003] The purpose of the present invention is to overcome the deficiencies in the prior art and to provide a vehicle monitoring image simulation system and method, which can solve the technical problem in the prior art that the data generated by computer simulation does not match the actual working conditions, resulting in a decrease in the accuracy of the experimental samples required for the test, and further distorting the test results.

[0004] In order to solve the above technical problems, the present invention is implemented by adopting the following technical solutions: In a first aspect, the present invention provides a vehicle monitoring image simulation system, comprising: a vehicle panoramic multi-view virtual imaging model, a display device and a real camera, wherein the vehicle panoramic multi-view virtual imaging model comprises a virtual camera, and further comprises: A simulation module, used for simulating and generating a multi-view image of a car through the panoramic multi-view virtual imaging model of the car; A display module, used for projecting the multi-view image of the car onto the display device; A mapping module, used to control the real camera to collect the multi-view images of the car displayed by the display device, establish a mapping relationship between the real camera and the virtual camera, and obtain the virtual view camera image according to the mapping relationship; The image processing module is used to input the virtual view camera image into a pre-trained image processing model to obtain fitted lane lines and human body posture estimates, and superimpose the fitted lane lines and human body posture estimates on the virtual view camera image to obtain a target image.

[0005] Furthermore, the virtual cameras are distributed in the same position as the real cameras.

[0006] In a second aspect, the present invention provides a simulation method of the automobile monitoring image simulation system according to the first aspect, comprising: Generate a multi-view image of the car through simulation of the car panoramic multi-view virtual imaging model; Projecting the multi-view image of the car onto the display device; Controlling the real camera to collect the multi-view images of the car displayed by the display device, establishing a mapping relationship between the real camera and the virtual camera, and obtaining the virtual view camera image according to the mapping relationship; The virtual view camera image is input into a pre-trained image processing model to obtain fitted lane lines and human body posture estimation, and the fitted lane lines and human body posture estimation are superimposed on the virtual view camera image to obtain a target image.

[0007] Furthermore, before controlling the real camera to capture the multi-view image of the car displayed by the display device: The multi-view image of the car displayed by the display device is processed by Zhang Zhengyou's camera calibration method to calibrate the intrinsic parameters of the real camera.

[0008] Furthermore, it also includes: Based on the calibrated real camera intrinsic parameters, the multi-view image of the car displayed by the display device is processed by the image dedistortion function of OpenCV to complete the image dedistortion and obtain a distortion-free multi-view image of the car.

[0009] Furthermore, establishing a mapping relationship between the real camera and the virtual camera includes: Based on the real image captured by the real camera and the display image displayed by the display device, solving the change matrix of the real camera and the display device, and obtaining the corresponding relationship between the real camera and the display device based on the change matrix; Based on the direct projection relationship between the display device and the virtual camera and the corresponding relationship between the real camera and the display device, a mapping relationship between the real camera and the virtual camera is established.

[0010] Furthermore, before inputting the virtual view camera image into the pre-trained image processing model: Perform pixel-level annotation on lane lines in the virtual perspective camera image to form a binary image, and perform key point annotation on vehicles and pedestrians in the virtual perspective camera image; The virtual view camera image is resized and normalized to meet the input requirements of the LaneNet and PoseNet models.

[0011] Furthermore, the training process of the image processing model includes: Inputting the labeled lane line data set into the image processing model, performing binary segmentation on the labeled lane line data set using a cross entropy loss function, performing instance segmentation on the labeled lane line data set using a clustering loss function, performing perspective transformation on the segmentation result using H-Net, and fitting continuous lane lines; The vehicle and pedestrian data sets with marked key points are input into the image processing model, the mean square error loss function constructed based on the marked key points and the predicted key points is minimized, the predicted key points are output, the positions of the predicted key points are connected into a skeleton structure, and a human posture estimation is generated.

[0012] Compared with the prior art, the present invention has the following beneficial effects: A pre-built panoramic multi-view virtual imaging model of the car is used to generate a multi-view image of the car and project it onto the display. A real camera is used to capture the display image, and then the image change from the real camera to the virtual camera is established. The virtual avatar that has a mapping relationship with the real camera is input into the image processing module. Finally, the target image for testing the reaction of the driverless car to complex and sudden vehicle conditions is obtained, which improves the accuracy of the experimental samples required for the test and provides accuracy for verifying the decision-making ability, reaction speed and control precision of the driverless system. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 A schematic diagram of the positions of cars and virtual cameras in a car panoramic multi-view virtual imaging model of a car monitoring image simulation system provided in the first embodiment of the present invention; Figure 2 A mapping relationship diagram of a vehicle monitoring image simulation system provided by the first embodiment of the present invention; Figure 3 A schematic diagram of a flow chart of a vehicle monitoring image simulation method provided in the second embodiment of the present invention; Figure 4 A control logic diagram of a vehicle monitoring image simulation method provided by the second embodiment of the present invention; Figure 5 A checkerboard target image for calibration of a vehicle monitoring image simulation method provided in the second embodiment of the present invention. DETAILED DESCRIPTION

[0014] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of the present invention.

[0015] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the system or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and the like are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, features defined as "first", "second", and the like may explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise specified, "multiple" means two or more.

[0016] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood by specific circumstances. Example

[0017] This embodiment provides a vehicle monitoring image simulation system, as shown in the attached Figure 1 and attached Figure 2 As shown, it includes: a car panoramic multi-view virtual imaging model, a display device and a real camera, the car panoramic multi-view virtual imaging model includes a virtual camera, and also includes: The car panoramic multi-view virtual imaging model is a system that uses computer graphics and image processing technology to generate panoramic images of the inside and outside of the car. This model can be used for a variety of purposes and is an existing mature technology, which will not be described in detail here. In this embodiment, the display device is a display. A simulation module, used for simulating and generating a multi-view image of a car through the panoramic multi-view virtual imaging model of the car; A display module, used for projecting the multi-view image of the car onto the display device; A mapping module, used to control the real camera to collect the multi-view images of the car displayed by the display device, establish a mapping relationship between the real camera and the virtual camera, and obtain the virtual view camera image according to the mapping relationship; The image processing module is used to input the virtual view camera image into a pre-trained image processing model to obtain fitted lane lines and human body posture estimates, and superimpose the fitted lane lines and human body posture estimates on the virtual view camera image to obtain a target image. Example

[0018] The present invention provides a simulation and testing method for the automobile monitoring image simulation system according to Embodiment 1, as shown in the attached Figure 3 , Attachment Figure 4 and attached Figure 5 As shown, including: Generate a multi-view image of the car through simulation of the car panoramic multi-view virtual imaging model; Projecting the multi-view image of the car onto the display device; Processing the multi-view image of the car displayed by the display device by using Zhang Zhengyou's camera calibration method, calibrating the intrinsic parameters of the real camera, and based on the calibrated intrinsic parameters of the real camera, processing the multi-view image of the car displayed by the display device by using the image dedistortion function of OpenCV to complete image dedistortion and obtain a distortion-free multi-view image of the car; Regarding the Zhang Zhengyou camera calibration method, the Zhang Zhengyou camera calibration method is a technology widely used in camera calibration. It was proposed by Dr. Zhang Zhengyou in his 1999 paper "Flexible Camera Calibration By Viewing a PlaneFrom Unknown Orientations". The method is widely used because of its simple operation and high precision. In practical applications, the Zhang Zhengyou calibration method is usually used in combination with a computer vision library such as OpenCV. The calibration process is automatically completed by writing corresponding codes, and the camera parameters are obtained; that is, the present application processes the multi-view image of the car displayed by the display device through the Zhang Zhengyou camera calibration method, and processes the multi-view image of the car displayed by the display device through the image dedistortion function of OpenCV: Let the coordinates of the multi-view image of the car be , the unit is pixel, and its normalized coordinate is , in millimeters, , is the coordinate of the characteristic point in the target coordinate system. According to the pre-built pinhole imaging model, the target point The corresponding image point It can be expressed as, , in, is a non-zero coefficient, is the camera intrinsic parameter matrix. and is the coordinate of the principal point on the image plane, and Respectively in and The proportionality factor in the direction, is the tilt factor of the two axes of the image, , t They are the rotation matrix and translation vector between the target coordinate system and the camera coordinate system, are the homogeneous coordinates of the distorted image points, To normalize the homogeneous coordinates of the image points, the lens distortion model can be expressed as, , , in, , and is the radial distortion coefficient of the lens. Usually, the internal parameters of the camera need to be accurately calibrated before measurement, mainly including ; After the real camera collects multi-view images of the car, it is substituted into the camera dedistortion model to obtain the distortion-free image; Control the real camera to collect the multi-view images of the car displayed by the display device, and establish a mapping relationship between the real camera and the virtual camera: The virtual camera configuration parameters are set according to the simulation environment, including the virtual camera focal length , resolution , display configuration parameters include resolution , the pixel size is s , unit is mm; First, the checkerboard target is projected onto the display, and the real camera captures the display image. According to the number of corner points and pixel size in the display, the coordinate sequence of the display checkerboard target point is obtained. q , the coordinates of the checkerboard image points collected by the real camera are p , the mapping relationship between the two is p= ; Based on the real image captured by the real camera and the display image displayed by the display device, the change matrix of the real camera and the display device is solved: , Among them, the For real images, To display the image, is the change matrix; The display image of the display device and the virtual image of the virtual camera are directly projected, without other operations such as projection transformation, which is expressed as , Based on the direct projection relationship between the display device and the virtual camera and the corresponding relationship between the real camera and the display device , establish the mapping relationship between the real camera and the virtual camera ; The virtual camera image is obtained according to the mapping relationship. The image of the virtual camera imaging angle can be restored through the video captured by the real camera. At this time, the relevant imaging system also captures the image, completing the data collection and processing. The restored virtual camera image and the configuration parameters of each virtual camera, such as internal and external parameters, are used for target recognition and measurement. The lane lines in the virtual perspective camera image are annotated at the pixel level to form a binary image (the lane lines are white and the background is black), and the key points of the vehicles and pedestrians in the virtual perspective camera image are annotated. Each object is annotated with at least key points such as the head, shoulders, elbows, knees, etc.; Resizing and normalizing the virtual view camera image to meet the input requirements of the LaneNet and PoseNet models; The virtual view camera image is input into a pre-trained image processing model to obtain fitted lane lines and human body posture estimation, and the fitted lane lines and human body posture estimation are superimposed on the virtual view camera image to obtain a target image.

[0019] The training process of the image processing model includes: Inputting the labeled lane line data set into the image processing model, performing binary segmentation on the labeled lane line data set using a cross entropy loss function, performing instance segmentation on the labeled lane line data set using a clustering loss function, performing perspective transformation on the segmentation result using H-Net, and fitting continuous lane lines; The vehicle and pedestrian data sets with marked key points are input into the image processing model, the mean square error loss function constructed based on the marked key points and the predicted key points is minimized, the predicted key points are output, the positions of the predicted key points are connected into a skeleton structure, and a human posture estimation is generated.

[0020] The image processing model includes LaneNet, H-Net and PoseNet. LaneNet is a network model for lane detection. It realizes end-to-end lane detection by converting the lane detection problem into an instance segmentation problem. LaneNet contains two branches: semantic segmentation branch and pixel vector representation branch. The semantic segmentation branch is responsible for distinguishing lane pixels from the background, while the instance segmentation branch assigns lane pixels to different lane instances. Finally, the results of the two branches are combined through a clustering algorithm to realize instance segmentation of lanes. H-Net is a small neural network whose purpose is to predict the perspective transformation parameters of the input image. This perspective transformation parameters are used to transform the image from a regular perspective to a bird's-eye view perspective (BEV). In this perspective, lane detection and fitting become simpler and more accurate. In lane detection tasks, perspective transformation is usually used to transform lanes from a perspective image to a virtual orthogonal perspective, which simplifies the geometry of lanes and makes them easier to detect and fit by the model.

[0021] By training H-Net, the model can learn the optimal perspective transformation parameters for each input image, thereby improving the quality of lane line fitting. During the training process, the goal of H-Net is to minimize the error between the transformed lane line pixels and the actual lane line position. This is usually achieved through the least squares method or other regression loss functions. In this way, H-Net can customize the perspective transformation for each image, thereby improving the accuracy and robustness of lane line detection.

[0022] PoseNet is a network model for real-time human pose estimation that can predict the key point locations of the human body in an image or video. It uses a convolutional neural network (CNN) to predict the pose of a human body in an image, including the pose of a single person or multiple people. PoseNet can process a single image and estimate the human pose in real time, supporting the estimation of multiple people and multiple key points.

[0023] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A vehicle monitoring image simulation system, comprising a vehicle panoramic multi-view virtual imaging model, a display device and a real camera, wherein the vehicle panoramic multi-view virtual imaging model comprises a virtual camera, characterized in that: Also includes: A simulation module, used for simulating and generating a multi-view image of a car through the panoramic multi-view virtual imaging model of the car; A display module, used for projecting the multi-view image of the car onto the display device; A mapping module, used to control the real camera to collect the multi-view images of the car displayed by the display device, establish a mapping relationship between the real camera and the virtual camera, and obtain the virtual view camera image according to the mapping relationship; The image processing module is used to input the virtual view camera image into a pre-trained image processing model to obtain fitted lane lines and human body posture estimates, and superimpose the fitted lane lines and human body posture estimates on the virtual view camera image to obtain a target image.

2. The vehicle monitoring image simulation system according to claim 1, characterized in that: The virtual cameras are distributed in the same position as the real cameras.

3. A simulation method of the automobile monitoring image simulation system according to claim 1, characterized in that: include: Generate a multi-view image of the car through simulation of the car panoramic multi-view virtual imaging model; Projecting the multi-view image of the car onto the display device; Controlling the real camera to collect the multi-view images of the car displayed by the display device, establishing a mapping relationship between the real camera and the virtual camera, and obtaining the virtual view camera image according to the mapping relationship; The virtual view camera image is input into a pre-trained image processing model to obtain fitted lane lines and human body posture estimation, and the fitted lane lines and human body posture estimation are superimposed on the virtual view camera image to obtain a target image.

4. The vehicle monitoring image simulation method according to claim 3, characterized in that: Before controlling the real camera to collect the multi-view image of the car displayed by the display device: The multi-view image of the car displayed by the display device is processed by Zhang Zhengyou's camera calibration method to calibrate the intrinsic parameters of the real camera.

5. The vehicle monitoring image simulation method according to claim 4, characterized in that: Also includes: Based on the calibrated real camera intrinsic parameters, the multi-view image of the car displayed by the display device is processed by the image dedistortion function of OpenCV to complete the image dedistortion and obtain a distortion-free multi-view image of the car.

6. The vehicle monitoring image simulation method according to claim 5, characterized in that: Establishing the mapping relationship between the real camera and the virtual camera includes: Based on the real image captured by the real camera and the display image displayed by the display device, solving the change matrix of the real camera and the display device, and obtaining the corresponding relationship between the real camera and the display device based on the change matrix; Based on the direct projection relationship between the display device and the virtual camera and the corresponding relationship between the real camera and the display device, a mapping relationship between the real camera and the virtual camera is established.

7. The vehicle monitoring image simulation method according to claim 3, characterized in that: Before feeding the virtual view camera image into the pre-trained image processing model: Perform pixel-level annotation on lane lines in the virtual perspective camera image to form a binary image, and perform key point annotation on vehicles and pedestrians in the virtual perspective camera image; The virtual view camera image is resized and normalized to meet the input requirements of the LaneNet and PoseNet models.

8. The vehicle monitoring image simulation method according to claim 7, characterized in that: The training process of the image processing model includes: Inputting the labeled lane line data set into the image processing model, performing binary segmentation on the labeled lane line data set using a cross entropy loss function, performing instance segmentation on the labeled lane line data set using a clustering loss function, performing perspective transformation on the segmentation result using H-Net, and fitting continuous lane lines; The vehicle and pedestrian data sets with marked key points are input into the image processing model, the mean square error loss function constructed based on the marked key points and the predicted key points is minimized, the predicted key points are output, the positions of the predicted key points are connected into a skeleton structure, and a human posture estimation is generated.

Images