Panoramic looking-around model construction method, vehicle-mounted device and storage medium

Abstract

The invention relates to the field of intelligent driving, and provides a panoramic looking-around model construction method, and the method comprises the steps: recognizing a target object in an environment image obtained from a camera device of a vehicle, and determining a first coordinate of the target object in the environment image; converting the environment image into a bird's-eye view image of a bird's-eye view angle, and determining a second coordinate of the target object in the bird's-eye view image according to the first coordinate; determining a third coordinate of the target object in a third coordinate system corresponding to the camera device according to the second coordinate; determining a fourth coordinate of the target object in a fourth coordinate system corresponding to the vehicle according to the third coordinate, and determining an initial distance between the target object and the vehicle according to the fourth coordinate; and determining a target distance based on a preset correction parameter and the initial distance, and constructing a target panoramic looking-around model of the vehicle according to the target distance and the environment image. By means of the method, the safety of driving travel of the user can be improved.

Description

Technical Field

[0001] The present application belongs to the field of intelligent driving and relates to image processing technology, and specifically to a method for constructing a panoramic surround view model, a vehicle-mounted device and a storage medium. Background Art

[0002] The panoramic view system installed in the vehicle can obtain images around the vehicle through cameras installed in the front, back, left, and right directions of the vehicle, and build a panoramic view three-dimensional model based on these images to simulate the environment around the vehicle, thereby helping users better understand the distance between the vehicle and surrounding objects. However, objects in the three-dimensional models constructed in the related art are prone to distortion or distortion, making it difficult for users to accurately identify objects around the vehicle and judge the distance between the vehicle and the objects based on the three-dimensional model, resulting in increased driving risks for users. Summary of the invention

[0003] In view of the above, it is necessary to propose a panoramic view model construction method, a vehicle-mounted device and a storage medium, which can solve the problem of increased driving risk for users due to distortion or distortion of objects in the constructed vehicle panoramic view model.

[0004] An embodiment of the present application provides a method for constructing a panoramic surround view model, the method comprising: identifying a target object in an environmental image acquired from a camera device of a vehicle, and determining a first coordinate of the target object in a first coordinate system corresponding to the environmental image; converting the environmental image into a bird's-eye view image from a bird's-eye view perspective, and determining a second coordinate of the target object in a second coordinate system corresponding to the bird's-eye view image according to the first coordinate; determining a third coordinate of the target object in a third coordinate system corresponding to the camera device according to the second coordinate; determining a fourth coordinate of the target object in a fourth coordinate system corresponding to the vehicle according to the third coordinate, and determining an initial distance between the target object and the vehicle according to the fourth coordinate; determining a target distance based on a preset correction parameter and the initial distance, and constructing a target panoramic surround view model of the vehicle according to the target distance and the environmental image.

[0005] In one embodiment, the identifying of the target object in the environmental image captured by the camera device of the vehicle and determining the first coordinates of the target object in a first coordinate system corresponding to the environmental image include: using a preset feature recognition algorithm to identify the target object in the environmental image and generating a rectangular bounding box of the target object in the environmental image; determining the coordinates of the target corner points of the rectangular bounding box in the first coordinate system, and using the coordinates of the target corner points as the first coordinates.

[0006] In one embodiment, the preset feature recognition algorithm includes: one or more of a feature recognition algorithm based on a machine learning model and a feature recognition algorithm based on a deep learning model.

[0007] In one embodiment, the target corner points include the lower left corner point and the lower right corner point of the rectangular bounding box; the second coordinates include: the coordinates of the first projection point of the lower left corner point in the overhead image in the second coordinate system, and the coordinates of the second projection point of the lower right corner point in the overhead image in the second coordinate system.

[0008] In one embodiment, determining the third coordinates of the target object in the third coordinate system corresponding to the camera device based on the second coordinates includes: determining the midpoint coordinates between the coordinates of the first projection point in the second coordinate system and the coordinates of the second projection point in the second coordinate system; and determining the coordinates of the midpoint coordinates in the third coordinate system as the third coordinates based on a first conversion relationship between the second coordinate system and the third coordinate system.

[0009] In one embodiment, determining the fourth coordinate of the target object in a fourth coordinate system corresponding to the vehicle based on the third coordinate includes: determining the installation distance between the camera device and the center of the vehicle in the bird's-eye view, the installation distance including a horizontal distance and a vertical distance; determining a second conversion relationship between the third coordinate system and the fourth coordinate system based on the installation distance, and converting the third coordinate to the fourth coordinate based on the second conversion relationship, wherein the origin of the fourth coordinate system is located at the center of the vehicle.

[0010] In one embodiment, determining the initial distance between the target object and the vehicle according to the fourth coordinate includes: determining the Euclidean distance between the fourth coordinate and the center of the vehicle as the initial distance.

[0011] In one embodiment, the target distance is determined based on a preset correction parameter and the initial distance, and a target panoramic view model of the vehicle is constructed according to the target distance and the environmental image, including: determining the target distance according to the difference between the initial distance and the correction parameter; constructing a three-dimensional bowl-shaped grid model with the target distance as the length of the bottom radius, and projecting the environmental images in four directions of the vehicle onto the three-dimensional bowl-shaped grid model to obtain the target panoramic view model.

[0012] An embodiment of the present application provides a panoramic surround view model construction device, which includes: an identification module, which is used to identify a target object in an environmental image acquired from a camera device of a vehicle, and determine a first coordinate of the target object in a first coordinate system corresponding to the environmental image; a determination module, which is used to convert the environmental image into a bird's-eye view image from a bird's-eye view perspective, and determine a second coordinate of the target object in a second coordinate system corresponding to the bird's-eye view image according to the first coordinate; determine a third coordinate of the target object in a third coordinate system corresponding to the camera device according to the second coordinate; determine a fourth coordinate of the target object in a fourth coordinate system corresponding to the vehicle according to the third coordinate, and determine an initial distance between the target object and the vehicle according to the fourth coordinate; and a construction module, which determines a target distance based on a preset correction parameter and the initial distance, and constructs a target panoramic surround view model of the vehicle according to the target distance and the environmental image.

[0013] An embodiment of the present application provides a vehicle-mounted device, which includes: a memory and at least one processor, and the processor is used to implement the panoramic surround view model construction method when executing a computer program stored in the memory.

[0014] An embodiment of the present application provides a vehicle, which includes at least one camera device and the vehicle-mounted device.

[0015] An embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the panoramic surround view model construction method is implemented.

[0016] In summary, the panoramic view model construction method described in the present application determines the first coordinate of the target object in the first coordinate system corresponding to the environmental image by identifying the target object in the environmental image obtained from the camera device of the vehicle; based on the preset perspective projection matrix, the environmental image is converted into a bird's-eye view image of a bird's-eye view perspective, and the second coordinate of the target object in the second coordinate system corresponding to the bird's-eye view image is determined according to the first coordinate; the third coordinate of the target object in the third coordinate system corresponding to the camera device is determined according to the second coordinate; the fourth coordinate of the target object in the fourth coordinate system corresponding to the vehicle is determined according to the third coordinate, and the initial distance between the target object and the vehicle is determined according to the fourth coordinate; the target distance is determined based on the preset correction parameter and the initial distance, and the target panoramic view model of the vehicle is constructed according to the target distance and the environmental image. It can avoid the distortion or distortion of objects in the constructed vehicle panoramic view model, obtain a vehicle panoramic view model that is closer to the real environment, and ensure the driving safety of users when driving according to the vehicle panoramic view model. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a structural diagram of a vehicle-mounted device provided in one embodiment of the present application.

[0018] Figure 2 This is an example diagram of the installation position of a camera device in a vehicle provided in one embodiment of the present application.

[0019] Figure 3 It is a flowchart of a method for constructing a panoramic surround view model provided in one embodiment of the present application.

[0020] Figure 4 This is an example diagram of a top view of a vehicle provided in one embodiment of the present application.

[0021] Figure 5 This is an example diagram of the first coordinates of a target object provided by an embodiment of the present application.

[0022] Figure 6 This is an example diagram of an environment image and a corresponding bird's-eye view image provided by an embodiment of the present application.

[0023] Figure 7 It is an example diagram of the second coordinate system and the third coordinate system provided in one embodiment of the present application.

[0024] Figure 8 This is an example diagram of a panoramic surround view from a bird's-eye view of a vehicle provided in one embodiment of the present application.

[0025] Fig. 9 It is an example diagram of the third coordinate system and the fourth coordinate system provided in one embodiment of the present application.

[0026] Fig.10 This is an example diagram of a panoramic surround view model provided by an embodiment of the present application.

[0027] Fig.11 It is a structural diagram of a panoramic surround view model building device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0028] In order to more clearly understand the above-mentioned purposes, features and advantages of the present application, the present application is described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0029] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein in the specification of this application are only for the purpose of describing the embodiments in one embodiment and are not intended to limit this application.

[0030] It should be noted that in this application, "at least one" means one or more, and "more than one" means two or more than two. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The terms "first", "second", "third", "fourth", etc. (if any) in the specification, claims and drawings of this application are used to distinguish similar objects, rather than to describe a specific order or sequence.

[0031] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a concrete way. The following embodiments and features in the embodiments may be combined with each other without conflict.

[0032] In one embodiment, the panoramic view system installed in the vehicle can obtain images around the vehicle through cameras installed in the front, back, left, and right directions of the vehicle, and build a panoramic view three-dimensional model based on these images to simulate the environment around the vehicle, thereby helping users better understand the distance between the vehicle and surrounding objects. However, objects in the three-dimensional models built in the related art are prone to distortion or distortion, making it difficult for users to accurately identify objects around the vehicle and judge the distance between the vehicle and the objects based on the three-dimensional model, resulting in increased driving risks for users.

[0033] To solve the above problems, an embodiment of the present application provides a method for constructing a panoramic surround view model, which determines the first coordinate of the target object in the first coordinate system corresponding to the environmental image by identifying the target object in the environmental image obtained from the camera device of the vehicle; converts the environmental image to a bird's-eye view image of a bird's-eye view based on a preset perspective projection matrix, and determines the second coordinate of the target object in the second coordinate system corresponding to the bird's-eye view image according to the first coordinate; determines the third coordinate of the target object in the third coordinate system corresponding to the camera device according to the second coordinate; determines the fourth coordinate of the target object in the fourth coordinate system corresponding to the vehicle according to the third coordinate, and determines the initial distance between the target object and the vehicle according to the fourth coordinate; determines the target distance based on the preset correction parameter and the initial distance, and constructs the target panoramic surround view model of the vehicle according to the target distance and the environmental image. It can avoid the distortion or distortion of objects in the constructed vehicle panoramic surround view model, obtain a vehicle panoramic surround view model that is closer to the real environment, and ensure the driving safety of users when driving according to the vehicle panoramic surround view model.

[0034] Figure 1 This is a schematic diagram of the structure of a vehicle-mounted device provided in an embodiment of the present application. The embodiment of the present application does not impose any limitation on the specific type of the vehicle-mounted device.

[0035] like Figure 1 As shown, the vehicle-mounted device 10 can be installed in a vehicle 1, and the vehicle-mounted device 10 can include a communication module 101, a memory 102, a processor 103, an input / output (I / O) interface 104, and a bus 105. The processor 103 is coupled to the communication interface 101, the memory 102, and the I / O interface 104 through the bus 105.

[0036] The communication module 101 may include a wired communication module and / or a wireless communication module. The wired communication module may provide one or more wired communication solutions such as universal serial bus (USB), controller area network bus (CAN), local interconnect network (LIN), FlexRay, etc. The wireless communication module may provide one or more wireless communication solutions such as wireless fidelity (Wi-Fi), bluetooth (BT), mobile communication network, frequency modulation (FM), near field communication technology (NFC), infrared technology (IR), etc.

[0037] The memory 102 may include one or more random access memories (RAM) and one or more non-volatile memories (NVM). The random access memory can be directly read and written by the processor 103, and can be used to store executable programs (such as machine instructions) of the operating system or other running programs, and can also be used to store user and application data. The random access memory may include static random-access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), etc.

[0038] The non-volatile memory may also store executable programs and user and application data, etc., and may be pre-loaded into the random access memory for direct reading and writing by the processor 110. The non-volatile memory may include a disk storage device and a flash memory.

[0039] The memory 102 is used to store one or more computer programs. The one or more computer programs are configured to be executed by the processor 103. The one or more computer programs include multiple instructions. When the multiple instructions are executed by the processor 103, the panoramic surround view model construction method executed on the vehicle-mounted device 10 can be implemented.

[0040] In other embodiments, the vehicle-mounted device 10 further includes an external memory interface for connecting to an external memory to expand the storage capacity of the vehicle-mounted device 10 .

[0041] The processor 103 may include one or more processing units, for example, the processor 103 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0042] The processor 103 provides computing and control capabilities. For example, the processor 103 is used to execute a computer program stored in the memory 102 to implement the above-mentioned panoramic surround view model construction method.

[0043] The I / O interface 104 is used to provide a channel for user input or output. For example, the I / O interface 104 can be used to connect various input and output devices, such as a mouse, keyboard, touch device, display screen, etc., so that the user can enter information or visualize information.

[0044] The I / O interface 104 may also be used to provide a channel for data transmission with the camera device 106 . For example, the I / O interface 104 may be used to obtain an environmental image of the vehicle from the camera device 106 .

[0045] The camera device 106 includes at least one camera device installed in the vehicle 1, which is used to capture an image of the environment in which the vehicle 1 is located. The camera device 106 may be a fisheye camera, an infrared camera, etc. Figure 2 , which is an example diagram of the installation position of the camera device in a vehicle provided by an embodiment of the present application. The camera device includes four camera devices installed in the front, back, left, and right directions of the vehicle, and the union of the four field of view ranges of the four camera devices can cover the range within 360 degrees around the vehicle.

[0046] The bus 105 is at least used to provide a channel for mutual communication among the communication module 101 , the memory 102 , the processor 103 , and the I / O interface 104 in the vehicle-mounted device 10 .

[0047] It is understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the vehicle-mounted device 10. In other embodiments of the present application, the vehicle-mounted device 10 may include more or fewer components than shown in the figure, or combine certain components, or separate certain components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0048] Figure 3 : is a flow chart of a method for constructing a panoramic surround view model provided by an embodiment of the present application. The method for constructing a panoramic surround view model is applied to a vehicle-mounted device, for example Figure 1 The vehicle-mounted device 10 specifically includes the following steps. According to different requirements, the order of the steps in the flowchart can be changed, and some can be omitted.

[0049] S31, identifying a target object in an environment image acquired from a camera device of a vehicle, and determining a first coordinate of the target object in a first coordinate system corresponding to the environment image.

[0050] In one embodiment, at least one camera device is installed in the vehicle, and the vehicle-mounted device can receive the unique identification of each camera device and the installation position of each camera device relative to the vehicle input by the user, so as to distinguish the cameras installed at different positions. The installation position of each camera device relative to the vehicle can include the installation distance of each camera device from the center of the vehicle, and the installation distance includes a horizontal distance and a vertical distance, wherein the center of the vehicle represents the center of the top view of the vehicle.

[0051] For example Figure 4 As shown, a rectangular coordinate system O1X1Y1 corresponding to the vehicle is established with the center of the top view of the vehicle (i.e., the center of the dotted bounding box) as the origin, and the direction of the front of the vehicle in the top view is defined as the front of the vehicle. A camera device is installed in each of the four directions of the front, back, left and right of the vehicle. Among them, the camera device installed in the front of the vehicle is identified as cameraF, the camera device installed at the rear of the vehicle is identified as cameraB, the camera device installed on the left of the vehicle is identified as cameraL, and the camera device installed on the right of the vehicle is identified as cameraR. The horizontal distance of any camera device compared to the center of the vehicle includes the distance between the camera device and the Y1 axis, and the vertical distance of any camera device compared to the center of the vehicle includes the distance between the camera device and the X1 axis. In subsequent embodiments, the following will be used. Figure 4 The camera F and camera B are shown as being installed on the Y1 axis for example. In other embodiments, if camera F or camera B is not installed on the Y1 axis, a certain translation compensation may be performed during the coordinate transformation to implement the subsequent process.

[0052] In one embodiment, before identifying the target object in the environmental image obtained from the camera device of the vehicle, the method further includes preprocessing the environmental image, and the preprocessing includes but is not limited to: resizing, for example, resizing the environmental image to the size required for input by the feature recognition algorithm; image optimization, for example, improving the texture clarity of the environmental image based on an interpolation algorithm (for example, a bilinear interpolation algorithm); grayscale processing, for example, converting the environmental image into a grayscale image using a weighted average method; filtering processing, for example, using a preset filter (for example, a mean filter, a median filter, etc.) to smooth the environmental image to remove noise; distortion correction, for example, when the camera device is a fisheye camera, the object in the environmental image may be distorted, and the environmental image can be distorted based on the camera parameters of the fisheye camera (for example, focal length, principal point coordinates, distortion coefficient, etc.) and a pre-selected correction model (for example, a Pinhole model, a fisheye model, etc.). By preprocessing the environmental image, the accuracy of identifying the target object in the environmental image can be improved.

[0053] In one embodiment, after obtaining the distortion-corrected environmental image corresponding to any camera device in the vehicle, in order to facilitate the subsequent step of obtaining a panoramic surround view of the vehicle from a bird's-eye view to construct a panoramic surround model of the vehicle, all the camera devices of the vehicle can be jointly calibrated based on the checkerboard calibration method, thereby converting all environmental images taken by all the camera devices of the vehicle into the same coordinate system.

[0054] Specifically, the process of jointly calibrating all the camera devices of the vehicle based on the checkerboard calibration method includes: obtaining a checkerboard image of the ground in the corresponding direction taken by each camera device; performing internal parameter calibration and external parameter calibration on each camera device to obtain initial camera parameters such as internal parameters and external parameters of each camera device; extracting target feature points in the checkerboard image in each direction, and matching feature points between the camera devices based on the target feature points; optimizing and updating the initial camera parameters of each camera device based on feature point matching information using methods such as minimizing ghosting errors, so that the updated camera parameters can more accurately align the checkerboard images taken by the corresponding camera devices; evaluating the accuracy of the updated camera parameters, and if the evaluation result indicates that the accuracy of the updated camera parameters is less than a preset accuracy threshold, repeating the above steps until camera parameters with an accuracy greater than or equal to the accuracy threshold are obtained, thereby completing the joint calibration of all the camera devices.

[0055] In one embodiment, the world distance corresponding to the length of each pixel point in the environment image may also be determined in the above-mentioned joint calibration process. For example, it may be determined that the world distance corresponding to the length of each pixel point in the environment image is 1 cm.

[0056] In one embodiment, for any camera device of a vehicle (e.g., cameraF), the identifying of a target object in an environment image captured by the camera device of the vehicle and determining a first coordinate of the target object in a first coordinate system corresponding to the environment image includes: using a preset feature recognition algorithm to identify the target object in the environment image and generating a rectangular bounding box of the target object in the environment image; determining the coordinates of the target corner points of the rectangular bounding box in the first coordinate system, and using the coordinates of the target corner points as the first coordinates. The target object includes an object with height, such as a vehicle, a wall, a column, a step, a human body, and the like.

[0057] In one embodiment, the preset feature recognition algorithm includes: one or more of a feature recognition algorithm based on a machine learning model and a feature recognition algorithm based on a deep learning model.

[0058] In one embodiment, the feature recognition algorithm based on the machine learning model may use algorithms such as linear regression algorithm, support vector regression algorithm, ridge regression algorithm, decision tree algorithm, etc. The feature recognition algorithm based on the machine learning model can learn how to classify pixels in the environment image through a supervised learning method, can realize feature recognition of the environment image without relying on display programming, and can improve the efficiency of feature recognition.

[0059] In one embodiment, the feature recognition algorithm based on the deep learning model can use a variety of neural network structures, such as convolutional neural networks, recurrent neural networks, long short-term memory networks, etc. The feature recognition algorithm based on the deep learning model can extract low-level features of the environment image based on the deep learning model, obtain a more abstract high-level representation of the environment image based on the low-level features, determine the distributed features in the environment image based on the high-level representation, and realize feature recognition of the environment image. It can be applicable to feature recognition of environment images containing complex objects, and improve the accuracy and efficiency of feature recognition.

[0060] In one example, using a fully convolutional neural network model to identify target objects in an environmental image includes: using multiple convolutional layers to filter the environmental image through convolution operations to obtain feature images of multiple scales of the environmental image; using an activation layer to perform a nonlinear transformation on the output of the convolution layer using an activation function (such as ReLU, sigmoid, tanh, etc.) to increase the expression ability of the network; using a pooling layer to reduce the size of the feature image to reduce the number of parameters and the computational complexity of the network; using an upsampling layer to perform a difference or deconvolution operation on the output of the pooling layer to restore the size of the feature image to obtain a high-resolution feature representation of the environmental image; using a skip connection mechanism to connect different layers in the network to better capture feature information of different scales in the environmental image; using the output layer to perform feature classification based on a semantic segmentation algorithm to obtain the object category to which the pixel points in the environmental image belong, thereby determining the target object in the environmental image.

[0061] In one embodiment, in order to determine the specific position of the target object in the environment image, a rectangular bounding box of the target object can be first generated in the environment image, then the coordinates of the target corner points of the rectangular bounding box of the target object are determined in the first coordinate system corresponding to the environment image, and finally the coordinates of the target corner points are used as the first coordinates of the target object in the first coordinate system corresponding to the environment image. The target corner points include the lower left corner point and the lower right corner point of the rectangular bounding box.

[0062] For example Figure 5 As shown in FIG. 1 , an example diagram of the first coordinates of the target object provided in an embodiment of the present application is shown. In which, a first coordinate system O2X2Y2 is established with the position of the upper left corner of the environment image captured by camera F as the origin, and a target object (e.g. Figure 5 The rectangular enclosing box shown by the dotted line of the vehicle rear view shown in the figure, determines the coordinates (x1, y1) of the lower left corner point and the coordinates (x2, y2) of the lower right corner point of the rectangular enclosing box shown by the dotted line in the first coordinate system, and takes the coordinates (x1, y1) and the coordinates (x2, y2) as the first coordinates.

[0063] Step S32: convert the environment image into a bird's-eye view image from a bird's-eye view perspective, and determine the second coordinates of the target object in a second coordinate system corresponding to the bird's-eye view image according to the first coordinates.

[0064] In one embodiment, when converting the environment image to a bird's-eye view image from a bird's-eye view perspective, a projection transformation of the environment image to the bird's-eye view image can be achieved using a preset perspective projection matrix based on the principle of building a two-dimensional panoramic surround view monitor (AVM).

[0065] Specifically, taking cameraF as an example, the method for obtaining the perspective projection matrix may include: based on the checkerboard calibration method, using cameraF to capture a first image of a checkerboard on the ground within the field of view of cameraF, and using an auxiliary camera to capture a second image of the checkerboard, wherein the auxiliary camera is located above the center of the vehicle and the shooting angle of the auxiliary camera is parallel to the plane where the vehicle is located; determining the environmental coordinates of multiple calibrated feature points in the first image in the coordinate system where the first image is located, and determining in the second image the bird's-eye view coordinates of the points corresponding to the multiple calibrated feature points in the coordinate system where the second image is located; calculating a homography matrix for transforming the environmental coordinates corresponding to the multiple calibrated feature points to corresponding bird's-eye view coordinates, and using the homography matrix as the projection perspective matrix.

[0066] In one embodiment, the perspective projection matrix is ​​used to perform projection transformation on each pixel point in the environment image to obtain a bird's-eye view image corresponding to the environment image (for example, Figure 6 As shown), and the coordinates of the projection point corresponding to each pixel point in the environment image in the overhead image in the second coordinate system where the overhead image is located, wherein the second coordinate system can be Figure 7 In addition, since the world distance corresponding to the length of each pixel in the environment image is a known parameter, the world distance corresponding to the length of each pixel in the bird's-eye view image can be obtained by using the perspective projection matrix to transform the known parameter accordingly. Figure 6 As shown, when the environmental image is converted into a bird's-eye view image from a bird's-eye view perspective, due to the change in perspective, the image closer to the top in the environmental image is stretched (including lengthening and widening) to a greater extent, and the image closer to the bottom in the environmental image is stretched to a smaller extent.

[0067] In one embodiment, when each pixel in the environment image is projected and transformed using the perspective projection matrix, the rectangular bounding box of the target object can be projected and transformed to obtain the bounding box of the target object in the bird's-eye view in the bird's-eye view image (for example, Figure 6 The second coordinates of the target object in the second coordinate system corresponding to the overhead image include: the first projection point of the lower left corner point in the overhead image (e.g. Figure 6 The coordinates of the lower right corner point in the second coordinate system (shown in FIG. 1 ) are the second projection points of the lower right corner point in the bird's-eye view image (for example, Figure 6 (shown) coordinates in the second coordinate system.

[0068] In one embodiment, since all the camera devices in the vehicle have been jointly calibrated in step S31, a panoramic surround view of the vehicle from a bird's-eye view perspective can be obtained based on the joint calibration according to the projection perspective matrix between the environment image and the bird's-eye view image. Figure 8 As shown, it is an example diagram of a panoramic surround view from a bird's-eye view of a vehicle provided in an embodiment of the present application.

[0069] Step S33: determining a third coordinate of the target object in a third coordinate system corresponding to the camera device according to the second coordinate.

[0070] In one embodiment, determining the third coordinates of the target object in the third coordinate system corresponding to the camera device based on the second coordinates includes: determining the midpoint coordinates (for example, (x3, y3)) between the coordinates of the first projection point in the second coordinate system and the coordinates of the second projection point in the second coordinate system; and determining the coordinates of the midpoint coordinates in the third coordinate system as the third coordinates based on a first conversion relationship between the second coordinate system and the third coordinate system.

[0071] In one embodiment, for the convenience of calculation, when the midpoint coordinates are used to calculate the distance between the target object and the corresponding camera device, the second coordinates of the target object in the bird's-eye view image may be converted into third coordinates in a third coordinate system corresponding to the camera device.

[0072] Specifically, take cameraF as an example, Figure 7 As shown, since the center of the field of view of the camera device is located on the central vertical line of the environment image, and the lower boundary of the bird's-eye view image is the boundary of the front of the vehicle, which is also the installation position of camera F. Therefore, the center position of the lower boundary of each bird's-eye view image is the location of each camera device, and the third coordinate system O4X4Y4 where the camera device is located can be established with the center position of the lower boundary of the bird's-eye view image as the origin.

[0073] In one embodiment, since both the second coordinate system and the third coordinate system are rectangular coordinate systems, the first conversion relationship between the second coordinate system and the third coordinate system can be determined by determining the positional relationship between the two origins of the second coordinate system and the third coordinate system. Specifically, the length W and width H of the bird's-eye view image can be determined according to the length and width of the environment image using the projection transformation matrix, wherein the length of the bird's-eye view image can be the side where the origin of the second coordinate system is located (e.g. Figure 7 The width of the bird's-eye view image can be the length of two parallel boundaries of the bird's-eye view image (e.g. Figure 7The method comprises the following steps: determining the distance between the upper boundary and the lower boundary of the bird's-eye view image in the second coordinate system; determining the positional relationship between the two origins of the second coordinate system and the third coordinate system according to the length and width of the bird's-eye view image, including: the coordinates of the origin of the third coordinate system in the second coordinate system are (W / 2, -H); determining the first transformation relationship according to the positional relationship between the two origins of the second coordinate system and the third coordinate system, the first transformation relationship including: if the coordinates of any point in the second coordinate system are (x4, y4), then the coordinates of any point in the third coordinate system are (x5, y5) = (-x4 + W / 2, H - |y4|).

[0074] In one embodiment, the coordinates (-x3+W / 2,H-|y3|) of the midpoint coordinates (x3, y3) in the third coordinate system can be determined as the third coordinates according to a first conversion relationship between the second coordinate system and the third coordinate system.

[0075] In one embodiment, since the world distance corresponding to the length of each pixel in the environmental image and the bird's-eye view image are known parameters, the world distance between the target object and the camera device can be determined based on the third coordinate. Specifically, the world distance between the target object and the camera device can be the Euclidean distance between the third coordinate and the origin of the third coordinate system.

[0076] In one embodiment, for any camera device, since the environment image may contain multiple target objects, in order to avoid distortion or distortion of the target panoramic view model constructed in subsequent steps, the method also includes: determining the distance between each target object in the environment image and the corresponding camera device, and constructing the target panoramic view model based on the target object corresponding to the minimum distance.

[0077] Step S34: determining a fourth coordinate of the target object in a fourth coordinate system corresponding to the vehicle according to the third coordinate, and determining an initial distance between the target object and the vehicle according to the fourth coordinate.

[0078] In one embodiment, determining the fourth coordinate of the target object in a fourth coordinate system corresponding to the vehicle based on the third coordinate includes: determining the installation distance between the camera device and the center of the vehicle in the bird's-eye view, the installation distance including a horizontal distance and a vertical distance; determining a second conversion relationship between the third coordinate system and the fourth coordinate system based on the installation distance, and converting the third coordinate to the fourth coordinate based on the second conversion relationship, wherein the origin of the fourth coordinate system is located at the center of the vehicle.

[0079] In one embodiment, taking cameraL as an example, for example Fig. 9, which is an example diagram of the third coordinate system and the fourth coordinate system provided in the embodiment of the present application. Among them, O1X1Y1 represents the fourth coordinate system with the center of the vehicle as the origin, and O4X4Y4 represents the third coordinate system corresponding to cameraL.

[0080] In one embodiment, the method for determining the second conversion relationship is similar to the method for determining the first conversion relationship. Since the horizontal distance and vertical distance between the camera device (e.g., cameraL) and the center of the vehicle in the bird's-eye view are known parameters (refer to step S31), the second conversion relationship may include a translation transformation of the third coordinate based on the horizontal distance and the vertical distance. For details, refer to the method for determining the first conversion relationship.

[0081] In one embodiment, determining the initial distance between the target object and the vehicle according to the fourth coordinate includes: determining the Euclidean distance between the fourth coordinate and the center of the vehicle as the initial distance.

[0082] Step S35 , determining the target distance based on the preset correction parameter and the initial distance, and constructing a target panoramic surround view model of the vehicle according to the target distance and the environment image.

[0083] In one embodiment, the target distance is determined based on the preset correction parameter and the initial distance, and the target panoramic view model of the vehicle is constructed according to the target distance and the environmental image, including: determining the target distance according to the difference between the initial distance and the correction parameter; constructing a three-dimensional bowl-shaped grid model with the target distance as the length of the bottom radius, and projecting the environmental images in four directions of the vehicle onto the three-dimensional bowl-shaped grid model to obtain the target panoramic view model. The bottom of the three-dimensional bowl-shaped grid model is centered on the center of the vehicle.

[0084] For example Fig.10 The example diagram of the panoramic view model provided by the embodiment of the present application is shown in FIG. The panoramic view model uses a three-dimensional bowl-shaped grid model, and the panoramic view model of the vehicle can be obtained by projecting the environment images in four directions of the vehicle onto the three-dimensional bowl-shaped grid model. Fig.10 The model on the left is a model that uses the initial distance as the length of the bottom radius of the three-dimensional bowl-shaped mesh model. Compared with the model on the right, it can be seen that the target object of the cylinder in the left model is distorted at the bottom of the bowl. To avoid the above problem, the initial distance can be corrected so that the target object can be projected as a whole to the inside of the bowl wall during projection, for example Fig.10 The model on the right is shown.

[0085] In one embodiment, the method for determining the correction parameter includes: determining the category of the target object; determining the width of the target object based on the category, and using the width of the target object as the correction parameter. For example, when the category of the target object is a human body, the average width of a human body can be used as the correction parameter. In another embodiment, a parameter input by a user can also be used as the correction parameter.

[0086] In one embodiment, the difference between the initial distance and the correction parameter is used as the target distance, so that the target object can be projected to the outside of the three-dimensional bowl-shaped mesh model with the target distance as the bottom radius, thereby projecting the target object onto the bowl wall of the three-dimensional bowl-shaped mesh model to avoid distortion and distortion of the target object during projection. During vehicle driving, by continuously determining the target object and the correction parameters corresponding to the target object, it can be ensured that the panoramic view model of the vehicle at each moment is not distorted, thereby ensuring the driving safety of the user.

[0087] In one embodiment, the method may further include: if the target panoramic view model is distorted, updating the target panoramic view model, including: using the updated correction parameters to reduce the bottom radius of the three-dimensional bowl-shaped grid model. The determination result of whether the target panoramic view model is distorted by the user driving the vehicle may be received.

[0088] In one embodiment, the panoramic view model construction method provided by the embodiment of the present application determines the first coordinate of the target object in the first coordinate system corresponding to the environmental image by identifying the target object in the environmental image obtained from the camera device of the vehicle; based on the preset perspective projection matrix, the environmental image is converted into a bird's-eye view image of a bird's-eye view perspective, and the second coordinate of the target object in the second coordinate system corresponding to the bird's-eye view image is determined according to the first coordinate; the third coordinate of the target object in the third coordinate system corresponding to the camera device is determined according to the second coordinate; the fourth coordinate of the target object in the fourth coordinate system corresponding to the vehicle is determined according to the third coordinate, and the initial distance between the target object and the vehicle is determined according to the fourth coordinate; the target distance is determined based on the preset correction parameter and the initial distance, and the target panoramic view model of the vehicle is constructed according to the target distance and the environmental image. It can avoid the distortion or distortion of objects in the constructed vehicle panoramic view model, obtain a vehicle panoramic view model that is closer to the real environment, ensure the driving safety of users when driving according to the vehicle panoramic view model, and reduce the property loss of users caused by vehicle collisions.

[0089] Fig.11 It is a structural diagram of a panoramic surround view model building device provided in one embodiment of the present application.

[0090] In some embodiments, the panoramic view model building device 40 may include a plurality of functional modules composed of computer program segments. The computer programs of the various program segments in the panoramic view model building device 40 may be stored in a memory of the vehicle-mounted device and executed by at least one processor to execute (see Figure 3 Description) Function of building a panoramic surround view model.

[0091] In this embodiment, the panoramic view model building device 40 can be divided into multiple functional modules according to the functions it performs. The functional modules may include: an identification module 401, a determination module 402, and a construction module 403. The module referred to in this application refers to a series of computer program segments that can be executed by at least one processor and can complete fixed functions, which are stored in a memory. In this embodiment, the functional implementation method of each module in the panoramic view model building device 40 can refer to the above definition of the panoramic view model building method, and will not be repeated here.

[0092] The recognition module 401 is used to recognize a target object in an environment image acquired from a camera device of a vehicle, and determine a first coordinate of the target object in a first coordinate system corresponding to the environment image.

[0093] The determination module 402 is used to convert the environmental image into a bird's-eye view image from a bird's-eye view perspective, determine the second coordinate of the target object in a second coordinate system corresponding to the bird's-eye view image according to the first coordinate; determine the third coordinate of the target object in a third coordinate system corresponding to the camera device according to the second coordinate; determine the fourth coordinate of the target object in a fourth coordinate system corresponding to the vehicle according to the third coordinate, and determine the initial distance between the target object and the vehicle according to the fourth coordinate.

[0094] The construction module 403 determines the target distance based on the preset correction parameters and the initial distance, and constructs the target panoramic surround model of the vehicle according to the target distance and the environment image.

[0095] An embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. The computer program includes program instructions. The method implemented when the program instructions are executed can refer to the methods in the above-mentioned embodiments of the present application.

[0096] The computer-readable storage medium may be an internal memory of the vehicle-mounted device described in the above embodiment, such as a hard disk or memory of the vehicle-mounted device. The computer-readable storage medium may also be an external storage device of the vehicle-mounted device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the vehicle-mounted device.

[0097] In some embodiments, the computer-readable storage medium may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function, etc.; the data storage area may store data created according to the use of the vehicle-mounted device, etc.

[0098] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0099] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0100] In the embodiments provided in the present application, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0101] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0102] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A method for constructing a panoramic surround view model. It is characterized in that The method comprises: Identify a target object in an environment image acquired from a camera device of the vehicle, and determine a first coordinate of the target object in a first coordinate system corresponding to the environment image; Converting the environment image into a bird's-eye view image from a bird's-eye view perspective, and determining a second coordinate of the target object in a second coordinate system corresponding to the bird's-eye view image according to the first coordinate; Determine, according to the second coordinate, a third coordinate of the target object in a third coordinate system corresponding to the camera device; Determine a fourth coordinate of the target object in a fourth coordinate system corresponding to the vehicle according to the third coordinate, and determine an initial distance between the target object and the vehicle according to the fourth coordinate; The target distance is determined based on a preset correction parameter and the initial distance, and a target panoramic surround view model of the vehicle is constructed according to the target distance and the environment image.

2. The method for constructing a panoramic surround view model according to claim 1, It is characterized in that The identifying a target object in an environment image captured by a camera device of the vehicle and determining a first coordinate of the target object in a first coordinate system corresponding to the environment image includes: Identify the target object in the environment image using a preset feature recognition algorithm, and generate a rectangular bounding box of the target object in the environment image; The coordinates of the target corner point of the rectangular bounding box are determined in the first coordinate system, and the coordinates of the target corner point are used as the first coordinates.

3. The method for constructing a panoramic surround view model according to claim 2, It is characterized in that The preset feature recognition algorithm includes: one or more of a feature recognition algorithm based on a machine learning model and a feature recognition algorithm based on a deep learning model.

4. The method for constructing a panoramic surround view model according to claim 2, It is characterized in that The target corner points include the lower left corner point and the lower right corner point of the rectangular bounding box; the second coordinates include: the coordinates of the first projection point of the lower left corner point in the overhead image in the second coordinate system, and the coordinates of the second projection point of the lower right corner point in the overhead image in the second coordinate system.

5. The method for constructing a panoramic surround view model according to claim 4, It is characterized in that Determining the third coordinate of the target object in the third coordinate system corresponding to the camera device according to the second coordinate includes: Determine the coordinates of a midpoint between the coordinates of the first projection point in the second coordinate system and the coordinates of the second projection point in the second coordinate system; According to a first conversion relationship between the second coordinate system and the third coordinate system, the coordinates of the midpoint in the third coordinate system are determined as the third coordinates.

6. The method for constructing a panoramic surround view model according to claim 1, It is characterized in that Determining the fourth coordinate of the target object in a fourth coordinate system corresponding to the vehicle according to the third coordinate comprises: Determining an installation distance between the camera device and the center of the vehicle in the bird's-eye view, the installation distance including a horizontal distance and a vertical distance; A second conversion relationship between the third coordinate system and the fourth coordinate system is determined according to the installation distance, and the third coordinate is converted to the fourth coordinate according to the second conversion relationship, wherein the origin of the fourth coordinate system is located at the center of the vehicle.

7. The method for constructing a panoramic surround view model according to claim 1, It is characterized in that Determining the initial distance between the target object and the vehicle according to the fourth coordinate includes: A Euclidean distance between the fourth coordinate and the center of the vehicle is determined as the initial distance.

8. The method for constructing a panoramic surround view model according to claim 1, It is characterized in that The determining the target distance based on the preset correction parameter and the initial distance, and constructing the target panoramic surround view model of the vehicle according to the target distance and the environment image, comprises: Determining the target distance according to the difference between the initial distance and the correction parameter; A three-dimensional bowl-shaped grid model is constructed with the target distance as the length of the bottom radius, and the environment images in four directions of the vehicle are projected onto the three-dimensional bowl-shaped grid model to obtain the target panoramic surround model.

9. A vehicle-mounted device, It is characterized in that The vehicle-mounted device includes a memory and at least one processor, wherein the memory stores at least one instruction, and when the at least one instruction is executed by the at least one processor, the panoramic surround view model construction method as described in any one of claims 1 to 8 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the method for constructing a panoramic surround view model as described in any one of claims 1 to 8 is implemented.

Images