A vehicle-mounted camera pitch angle calibration method, device, equipment and storage medium

Abstract

The application discloses a kind of vehicle-mounted camera pitch angle calibration method, device, equipment and storage medium.It includes: obtaining the original image dynamically collected by vehicle-mounted camera and the initial pitch angle of static calibration;The original image is input into vanishing point detection algorithm model to obtain vanishing point coordinates, and the pitch angle change value is generated according to the vanishing point coordinates;According to the initial pitch angle of the pitch angle change value update acquisition dynamic calibration update pitch angle, and the vehicle-mounted camera is calibrated using update pitch angle.By inputting the original image dynamically collected by vehicle-mounted camera into vanishing point detection algorithm model to obtain vanishing point coordinates, the accuracy of calibration is improved using vanishing point detection algorithm, the pitch angle change value can be calculated according to the vanishing point coordinates, and the initial pitch angle of static calibration is updated to obtain the update pitch angle of dynamic calibration, and finally the calibration of vehicle-mounted camera is completed by update pitch angle, without manual measurement, automatic calibration during vehicle driving can be completed.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of camera calibration, and particularly relates to a vehicle-mounted camera pitch angle calibration method, device, equipment and storage medium. BACKGROUND

[0002] In sensors of an automatic driving vehicle, a front camera as a main visual sensor plays an important sensing role. In a complex road surface environment, due to the existence of up and down slopes of roads and frequent bumps in a driving process, a static calibration parameter of the camera often has an error with a real-time camera external parameter. The camera external parameter is related to a pose of the camera installed on the vehicle, and the pitch angle is an important parameter of the camera pose. Therefore, it is necessary to dynamically calibrate the pitch angle of the front camera.

[0003] In the prior art, a group of horizontal parallel lines are used to calibrate the pitch angle of the camera. The calibration algorithm of the prior art needs to measure an actual distance between two reference lines, which increases a human workload and cannot achieve automatic calibration in a driving process of the vehicle. In addition, manual measurement is prone to measurement errors, and thus the pitch angle calibration is inaccurate. SUMMARY

[0004] The present application provides a vehicle-mounted camera pitch angle calibration method, device, equipment and storage medium, so as to dynamically calibrate the pitch angle of the vehicle-mounted camera.

[0005] According to an aspect of the present application, a vehicle-mounted camera pitch angle calibration method is provided, which comprises the following steps.

[0006] An original image dynamically collected by a vehicle-mounted camera and an initial pitch angle calibrated statically are acquired.

[0007] The original image is input into a vanishing point detection algorithm model to acquire vanishing point coordinates, and a pitch angle change value is generated according to the vanishing point coordinates.

[0008] The initial pitch angle is updated according to the pitch angle change value to acquire an updated pitch angle calibrated dynamically, and the vehicle-mounted camera is calibrated by using the updated pitch angle.

[0009] Optionally, the original image is input into a vanishing point detection algorithm model to obtain vanishing point coordinates, including: splitting the original image to generate image slices, wherein each image slice has an image number, and the image number corresponds to the position of the image slice in the original image; calculating the embedding vector of each image slice after projection through the vanishing point detection algorithm model, and determining the image number corresponding to the embedding vector, wherein the vanishing point detection algorithm model includes a linear projection layer module, a self-attention encoder module and a multi-layer perception module; splicing each embedding vector according to the image number to generate a splicing matrix; obtaining a learnable vector according to the splicing matrix, and splicing the splicing matrix and the learnable vector to generate a learning matrix, wherein the learnable vector and the splicing matrix have the same dimension; and generating vanishing point coordinates according to the learning matrix through the vanishing point detection algorithm model.

[0010] Optionally, the embedding vector of each image slice after projection is calculated through the vanishing point detection algorithm model, including: obtaining an initial vector corresponding to each image slice; and inputting each initial vector into the linear projection layer module in sequence to obtain the embedding vector, wherein the linear projection layer contains a first correspondence relationship between the initial vector and the embedding vector.

[0011] Optionally, the vanishing point coordinates are generated according to the learning matrix through the vanishing point detection algorithm model, including: inputting the learning matrix into the self-attention encoder module to obtain image features output by the self-attention encoder module, wherein the self-attention encoder module includes an image feature extraction method corresponding to the learning matrix; inputting the image features into the multi-layer perception module to obtain a vanishing vector matched with the image features output by the multi-layer perception module, wherein the multi-layer perception module includes a second correspondence relationship between the image features and the vanishing vector; and generating the vanishing point coordinates according to the vanishing vector.

[0012] Optionally, before the image features are input into the multi-layer perception module, it further includes: obtaining an image feature sample set, wherein each image feature sample and its corresponding vanishing vector are included in the image feature sample set; and performing deep learning on the second correspondence relationship between each image feature sample and the vanishing vector to establish the multi-layer perception module.

[0013] Optionally, the pitch angle change value is generated according to the vanishing point coordinates, including: obtaining the center coordinates and the camera focal length of the original image, and calculating the vertical coordinate difference between the vanishing point coordinates and the center coordinates; and determining the pitch angle change value according to the vertical coordinate difference and the camera focal length.

[0014] Optionally, after the initial pitch angle is updated according to the pitch angle change value to obtain a dynamic updated pitch angle, it further includes: obtaining an initial camera external parameter of the vehicle-mounted camera, calculating a final camera external parameter according to the updated pitch angle and the initial camera external parameter; and calibrating the vehicle-mounted camera according to the final camera external parameter.

[0015] According to another aspect of the present application, there is provided a vehicle-mounted camera pitch angle calibration device, comprising:

[0016] an original image and initial pitch angle acquisition module configured to acquire an original image dynamically captured by a vehicle-mounted camera and an initial pitch angle calibrated statically;

[0017] a pitch angle change value generation module configured to input the original image into a vanishing point detection algorithm model to acquire vanishing point coordinates and generate a pitch angle change value according to the vanishing point coordinates;

[0018] a vehicle-mounted camera calibration module configured to update the initial pitch angle according to the pitch angle change value to acquire a dynamic updated pitch angle and calibrate the vehicle-mounted camera using the updated pitch angle.

[0019] According to another aspect of the present application, there is provided an electronic device, comprising:

[0020] at least one processor; and

[0021] a memory communicatively connected to the at least one processor; wherein,

[0022] the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to perform a vehicle-mounted camera pitch angle calibration method according to any one of the embodiments of the present application.

[0023] According to another aspect of the present application, there is provided a computer readable storage medium storing computer instructions for enabling a processor to perform a vehicle-mounted camera pitch angle calibration method according to any one of the embodiments of the present application.

[0024] The technical solution of the embodiments of the present application improves the accuracy of calibration by inputting the original image dynamically captured by the vehicle-mounted camera into the vanishing point detection algorithm model to acquire the vanishing point coordinates, using the vanishing point detection algorithm, calculating the pitch angle change value according to the vanishing point coordinates, updating the initial pitch angle calibrated statically to acquire the updated pitch angle calibrated dynamically, and finally calibrating the vehicle-mounted camera using the updated pitch angle without manual measurement, i.e. automatic calibration during vehicle driving.

[0025] It should be understood that the content described in this section is not intended to identify key or important features of the embodiments of the present application, nor to limit the scope of the present application. Other features of the present application will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0026] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed in the embodiment description. Obviously, the drawings in the following description only show some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0027] Figure 1 is a flow chart of a vehicle-mounted camera pitch angle calibration method according to the first embodiment of the present application;

[0028] Figure 2 is a flow chart of another vehicle-mounted camera pitch angle calibration method according to the first embodiment of the present application;

[0029] Figure 3 is a schematic diagram of an original image segmentation process according to the first embodiment of the present application;

[0030] Figure 4 is an encoder structure schematic diagram of a self-attention encoder module according to the first embodiment of the present application;

[0031] Figure 5 is a flow chart of another vehicle-mounted camera pitch angle calibration method according to the second embodiment of the present application;

[0032] Figure 6 is a structure schematic diagram of a vehicle-mounted camera pitch angle calibration device according to the third embodiment of the present application;

[0033] Figure 7 is a structure schematic diagram of an electronic device for implementing a vehicle-mounted camera pitch angle calibration method according to the present application. DETAILED DESCRIPTION

[0034] In order to make the technical personnel in the art better understand the present application scheme, the following will combine the drawings in the embodiments of the present application, and the technical solutions in the embodiments of the present application will be described clearly and completely. Obviously, the described embodiments are only some of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should belong to the scope of protection of the present application.

[0035] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and in the above description of the drawings are used to distinguish similar objects and are not necessarily used to describe a particular sequential or chronological order. It should be understood that the data thus used can be interchanged under appropriate circumstances so that the embodiments of the application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a list of steps or units is not necessarily limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to such processes, methods, products or devices.

[0036] Embodiment one

[0037] Figure 1 A flowchart of a vehicle-mounted camera pitch angle calibration method is provided for the first embodiment of the present application. The present embodiment can be applied to dynamically calibrate the pitch angle of a vehicle-mounted camera. The method can be executed by a vehicle-mounted camera pitch angle calibration device, which can be realized in the form of hardware and / or software, and can be configured in a computer. As shown in Figure 1 The method comprises:

[0038] S110, acquiring the original image dynamically collected by the vehicle-mounted camera and the initial pitch angle calibrated statically.

[0039] The vehicle-mounted camera refers to a device arranged on a vehicle and used for shooting images around the vehicle. The vehicle-mounted camera can be arranged at the front of the vehicle, for example, at the front position of the vehicle, to collect images of the front road in the driving direction of the vehicle. Or the vehicle-mounted camera can be arranged at the back of the vehicle, for example, at the rear position of the vehicle, to collect images of the front road in the opposite direction of the driving direction of the vehicle. Dynamic collection refers to real-time shooting of the vehicle-mounted camera during the driving of the vehicle. The original image refers to an image shot by the vehicle-mounted camera during the driving of the vehicle. The original image can be in a red green blue (RGB) format. The pitch angle refers to the angle between the optical axis of the vehicle-mounted camera and the horizontal plane. The initial pitch angle refers to the pitch angle of the vehicle-mounted camera in static calibration. Static calibration refers to calibration of the vehicle when the vehicle is static. However, in a complex road environment, due to the existence of up and down slopes of the road and frequent bumps during driving, the static calibration parameters of the camera often differ from the dynamic real-time calibration parameters. If the pitch angle of the vehicle-mounted camera is not accurately determined, the processing result of the video image shot by the vehicle-mounted camera will have errors, which will affect the estimation of the vehicle state on the road by the driving system, for example, the estimation of the distance of the vehicle, thereby affecting the decision of the driving system and further affecting the normal driving of the vehicle. Therefore, the controller will rely on the original image dynamically collected by the vehicle-mounted camera and the initial pitch angle in static calibration to dynamically calibrate the vehicle-mounted camera.

[0040] In S120, the original image is input into a vanishing point detection algorithm model to obtain vanishing point coordinates, and a pitch angle change value is generated according to the vanishing point coordinates.

[0041] The vanishing point refers to the vanishing point of the road, that is, the position where the parallel lines of two roads intersect. The vanishing point coordinates refer to the coordinates of the position. The vanishing point detection algorithm model includes a linear projection layer module, a self-attention encoder module and a multi-layer perception module. The linear projection layer module is a module that linearly projects the input vector to output an embedded vector. The self-attention encoder module is a module that builds a network model and extracts image features based on a self-attention extreme value visual transformer encoder. The multi-layer perception module is a module that generates vanishing point coordinates according to the input image features.

[0042] Figure 2 A flowchart of a vehicle-mounted camera pitch angle calibration method is provided for the first embodiment of the application. S120 mainly includes the following steps S121 to S127:

[0043] In S121, the original image is divided to generate image slices.

[0044] Specifically, the controller divides the original image to generate a plurality of equal-sized picture blocks as image slices. The image slices have image numbers, and the image numbers correspond to the positions of the image slices in the original image.Figure 3 is a schematic diagram of an original image segmentation process, Figure 3 In the figure, the square on the left is the original image before segmentation, and the nine picture blocks on the right are generated after the original image is segmented. The numbers 1-9 are the numbers of the image slices, corresponding to the positions of the image slices in the original image.

[0045] S122, calculate the embedding vectors of each image slice after projection by the vanishing point detection algorithm model, and determine the image numbers corresponding to the embedding vectors.

[0046] Optionally, calculating the embedding vectors of each image slice after projection by the vanishing point detection algorithm model comprises: obtaining initial vectors corresponding to each image slice; inputting each initial vector into a linear projection layer module in turn to obtain an embedding vector, wherein the linear projection layer contains a first correspondence relationship between the initial vectors and the embedding vectors.

[0047] Specifically, the controller calculates the embedding vectors of each image slice after projection by the vanishing point detection algorithm model. The controller expands each image slice into a 1-dimensional vector to generate an initial vector corresponding to each image slice, and the initial vector is represented by seq1. seq1=N*L, where N represents the image number and L represents the length. L=3*Hp*Wp, where Hp represents the pixel height of each image slice and Wp represents the pixel width of each image slice. Then, each initial vector is input into a linear projection layer module in order of the image number, and the embedding vector after projection is obtained. The embedding vector is represented by seq2. seq2=seq1*E, where E represents a learnable parameter matrix in the linear projection layer module, and seq1∈R N×L ,E∈R L×D ,seq2∈R N ×D .

[0048] S123, according to the image number, the embedding vectors are spliced to generate a splicing matrix.

[0049] Specifically, the embedding vectors can be spliced in order of the image number to generate a splicing matrix. The image number is represented by P n , and the splicing matrix is represented by seq3, i.e. seq3={[P1,seq21],[P2,seq22],…,[P N ,seq2 N ]}, seq2 n ∈R D ,P n ∈R 1 ,seq3∈R N×(D+1) ,n=1…N.

[0050] S124, obtain the learnable vector according to the splicing matrix, and splice the splicing matrix and the learnable vector to generate a learning matrix.

[0051] Specifically, the learnable vector can be obtained according to the splicing matrix seq3, the learnable vector has the same dimension as the splicing matrix, that is, the learnable vector is a (D+1) dimensional vector v t , and the splicing matrix and the learnable vector are spliced to generate a learning matrix seq4, seq4=[v t , seq3], v t ∈R D+1 , seq4∈R (N+1)×(D+1) .

[0052] S125, generate vanishing point coordinates according to the learning matrix through a vanishing point detection algorithm model.

[0053] Optionally, the vanishing point coordinates are generated according to the learning matrix through the vanishing point detection algorithm model, including: inputting the learning matrix into a self-attention encoder module to obtain image features output by the self-attention encoder module, wherein the self-attention encoder module includes an image feature extraction manner corresponding to the learning matrix; inputting the image features into a multi-layer perceptron module to obtain a vanishing vector matched with the image features output by the multi-layer perceptron module, wherein the multi-layer perceptron module includes a second correspondence relationship between the image features and the vanishing vector; and generating the vanishing point coordinates according to the vanishing vector.

[0054] Specifically, the spliced learning matrix is input into the self-attention encoder module, and since the self-attention encoder module includes the image feature extraction manner corresponding to the learning matrix, the image features seq5 can be output by the self-attention encoder module, seq5∈R (N+1)×(D+1) . Figure 4 An encoder structure schematic diagram of a self-attention encoder module provided in the embodiment is shown in the following figure, Figure 4 wherein, seq4 is the input learning matrix, seq5 is the output image features, and the encoder includes a normalization layer, a multi-head attention layer, a normalization layer and a multi-layer perceptron.

[0055] Further, the image features are input into the multi-layer perceptron module, since the multi-layer perceptron module includes the second correspondence relationship between the image features and the vanishing vector, the vanishing vector matched with the image features output by the multi-layer perceptron module can be obtained, and since the vector output by the multi-layer perceptron module has a shape, for example, 2*n, the vector can be coordinateized to generate the vanishing point coordinates.

[0056] Optionally, before the image features are input into the multi-layer perception module, the method further comprises: obtaining an image feature sample set, wherein the image feature sample set comprises each image feature sample and a corresponding vanishing vector of the image feature sample; and performing deep learning on a second correspondence between each image feature sample and the vanishing vector to establish the multi-layer perception module.

[0057] Specifically, the multi-layer perception module is a trained neural network structure, the controller obtains the image feature sample and the vanishing vector of the image feature sample labeled by a user, the user refers to a staff or a technician performing vehicle-mounted camera calibration, the multi-layer perception module performs deep learning on the second correspondence between each image feature sample and the vanishing vector, and parameters in the multi-layer perception module are updated according to a gradient of a current loss at each time of learning to generate the final multi-layer perception module.

[0058] S126, obtaining a center coordinate of the original image and a camera focal length, and calculating a vertical coordinate difference between the vanishing point coordinate and the center coordinate.

[0059] Specifically, the center coordinate refers to a coordinate located at the center of the image in a static state, because the vanishing point of the road in the static state is located at the center of the image, that is, when the width of the image is w pixels and the height of the image is h pixels, the center coordinate is The vanishing point coordinate can be represented by (v x ,v y ), and the vertical coordinate difference between the vanishing point coordinate and the center coordinate is a change of the road vanishing point in the vertical direction of the image, that is, the vertical coordinate difference between the vanishing point coordinate and the center coordinate is

[0060] S127, determining a pitch angle change value according to the vertical coordinate difference and the camera focal length.

[0061] Specifically, the pitch angle change value can be determined according to the vertical coordinate difference between the vanishing point coordinate and the center coordinate and the camera focal length, that is, the pitch angle change value is represented by the following formula (1):

[0062]

[0063] wherein, Δpitch represents the pitch angle change value, f represents the camera focal length, and Δv represents the vertical coordinate difference between the vanishing point coordinate and the center coordinate.

[0064] S130, updating the initial pitch angle according to the pitch angle change value to obtain an updated pitch angle of dynamic calibration, and calibrating the vehicle-mounted camera by using the updated pitch angle.

[0065] Specifically, the pitch angle change value and the initial pitch angle can be added as an updated pitch angle of dynamic calibration, and the updated pitch angle is the final pitch angle calibration result of the vehicle-mounted camera. Through the updated pitch angle, the vehicle-mounted camera can be calibrated to avoid position deviation of images collected by the vehicle-mounted camera in a bumpy situation.

[0066] The technical scheme of the embodiment of the application improves the accuracy of calibration by inputting the original image dynamically collected by the vehicle-mounted camera into a vanishing point detection algorithm model to obtain vanishing point coordinates, and calculating the pitch angle change value according to the vanishing point coordinates, and updating the initial pitch angle of static calibration to obtain the updated pitch angle of dynamic calibration. Finally, the vehicle-mounted camera is calibrated through the updated pitch angle, and the automatic calibration during vehicle driving can be completed without manual measurement.

[0067] Embodiment two

[0068] Figure 5 A flowchart of a vehicle-mounted camera pitch angle calibration method provided by the second embodiment of the application is shown in the figure. The embodiment adds a process of calculating the final camera external parameter on the basis of the first embodiment. The specific content of steps S210-S230 is substantially the same as that of steps S110-S130 in the first embodiment, and therefore will not be described again in this embodiment. As shown in the figure, the method comprises the following steps. Figure 5

[0069] S210, obtaining an original image dynamically collected by a vehicle-mounted camera and an initial pitch angle of static calibration.

[0070] S220, inputting the original image into a vanishing point detection algorithm model to obtain vanishing point coordinates, and generating a pitch angle change value according to the vanishing point coordinates.

[0071] Optionally, inputting the original image into the vanishing point detection algorithm model to obtain the vanishing point coordinates comprises: cutting the original image to generate image slices, wherein each image slice has an image number, and the image number corresponds to the position of the image slice in the original image; calculating the embedded vector after projection of each image slice through the vanishing point detection algorithm model to determine the image number corresponding to the embedded vector, wherein the vanishing point detection algorithm model comprises a linear projection layer module, a self-attention encoder module and a multi-layer perception module; splicing the embedded vectors according to the image numbers to generate a splicing matrix; obtaining a learnable vector according to the splicing matrix, and splicing the splicing matrix and the learnable vector to generate a learning matrix, wherein the learnable vector and the splicing matrix have the same dimension; and generating the vanishing point coordinates according to the learning matrix through the vanishing point detection algorithm model.

[0072] S230, updating the initial pitch angle according to the pitch angle change value to obtain an updated pitch angle of dynamic calibration, and calibrating the vehicle-mounted camera by using the updated pitch angle.​

[0073] S240, acquiring an initial camera external parameter of the vehicle-mounted camera, and calculating a final camera external parameter according to the updated pitch angle and the initial camera external parameter.

[0074] The camera external parameter refers to a parameter of the vehicle-mounted camera in a world coordinate system, such as the position and rotation direction of the camera, and the external parameter will change with the movement of the camera, compared with the invariable internal parameter. The initial camera external parameter refers to the static calibration of the vehicle-mounted camera external parameter. During calibration, a checkerboard can be pasted on a plane as a calibration object. By adjusting the direction of the calibration object or the camera, some photos of the calibration object in different directions are taken, and the corner points of the checkerboard are extracted by a corner point extraction algorithm to estimate the external parameter in the ideal non-distortion condition. The camera external parameter is related to the pose of the camera mounted on the vehicle, and the pitch angle is an important parameter of the camera pose. Therefore, after the updated pitch angle is acquired, the final camera external parameter can be calculated according to the geometric relationship between the updated pitch angle and the initial camera external parameter.

[0075] S250, calibrating the vehicle-mounted camera according to the final camera external parameter.

[0076] Specifically, in a complex road environment, due to the existence of up and down slopes on the road and the frequent bumps during driving, the static calibration parameters of the camera often have errors with the real-time camera external parameter, which changes the projection relationship between the image and the three-dimensional space, causes the position deviation of the visual perception result on the two-dimensional image projected into the three-dimensional space, affects the normal work and accurate judgment of the downstream technology stack, and is not conducive to the driving safety of autonomous driving. Therefore, it is necessary to dynamically calibrate the external parameter of the camera to obtain a more accurate real-time camera external parameter for the autonomous driving technology. Therefore, after the final camera external parameter is calculated according to the updated pitch angle and the initial camera external parameter, the final camera external parameter is the final external parameter calibration result of the vehicle-mounted camera.

[0077] The technical scheme of the embodiment of the application inputs the original image dynamically collected by the vehicle-mounted camera into a vanishing point detection algorithm model to obtain vanishing point coordinates, uses the vanishing point detection algorithm to improve the accuracy of calibration, calculates the pitch angle change value according to the vanishing point coordinates, updates the initial pitch angle of static calibration to obtain the updated pitch angle of dynamic calibration, and finally completes the calibration of the vehicle-mounted camera through the updated pitch angle. Without manual measurement, automatic calibration during vehicle driving can be completed.

[0078] Embodiment three

[0079] Figure 6 A structure schematic diagram of a vehicle-mounted camera pitch angle calibration device provided by the embodiment three of the application is shown in FIG. 3. Figure 6As shown, the device comprises: an original image and initial pitch angle acquisition module 310, configured to acquire an original image dynamically collected by a vehicle-mounted camera and an initial pitch angle calibrated statically; a pitch angle change value generation module 320, configured to input the original image into a vanishing point detection algorithm model to acquire vanishing point coordinates, and generate a pitch angle change value according to the vanishing point coordinates; and a vehicle-mounted camera calibration module 330, configured to update the initial pitch angle according to the pitch angle change value to acquire a dynamic updated pitch angle, and calibrate the vehicle-mounted camera by using the updated pitch angle.

[0080] Optionally, the pitch angle change value generation module 320 specifically comprises: an image slice generation unit, configured to split the original image to generate image slices, wherein each image slice has an image number, and the image number corresponds to the position of the image slice in the original image; an embedding vector calculation unit, configured to calculate embedding vectors of the image slices after projection by using the vanishing point detection algorithm model, and determine the image number corresponding to the embedding vectors, wherein the vanishing point detection algorithm model comprises a linear projection layer module, a self-attention encoder module and a multi-layer perception module; a splicing matrix generation unit, configured to splice the embedding vectors according to the image numbers to generate a splicing matrix; a learning matrix generation unit, configured to acquire a learnable vector according to the splicing matrix, splice the splicing matrix and the learnable vector to generate a learning matrix, wherein the learnable vector and the splicing matrix have the same dimension; and a vanishing point coordinate generation unit, configured to generate vanishing point coordinates according to the learning matrix by using the vanishing point detection algorithm model.

[0081] Optionally, the embedding vector calculation unit is specifically configured to: acquire initial vectors corresponding to the image slices; and input the initial vectors into the linear projection layer module in sequence to obtain the embedding vectors, wherein the linear projection layer comprises a first correspondence relationship between the initial vectors and the embedding vectors.

[0082] Optionally, the vanishing point coordinate generation unit is specifically configured to: input the learning matrix into the self-attention encoder module to acquire image features output by the self-attention encoder module, wherein the self-attention encoder module comprises an image feature extraction manner corresponding to the learning matrix; input the image features into the multi-layer perception module to acquire vanishing vectors matched with the image features output by the multi-layer perception module, wherein the multi-layer perception module comprises a second correspondence relationship between the image features and the vanishing vectors; and generate the vanishing point coordinates according to the vanishing vectors.

[0083] Optionally, the vanishing point coordinate generation unit is specifically configured to: acquire an image feature sample set before inputting the image features into the multi-layer perception module, wherein the image feature sample set comprises image feature samples and corresponding vanishing vectors; and perform deep learning on the second correspondence relationship between the image feature samples and the vanishing vectors to establish the multi-layer perception module.

[0084] Optionally, the pitch angle change value generation module 320 further comprises a pitch angle change value determination unit, configured to acquire the center coordinate and the camera focal length of the original image, and calculate the vertical coordinate difference between the vanishing point coordinate and the center coordinate; and determine the pitch angle change value according to the vertical coordinate difference and the camera focal length.

[0085] Optionally, the vehicle-mounted camera pitch angle calibration device further comprises a final camera external parameter calibration module, configured to acquire the initial camera external parameter of the vehicle-mounted camera, calculate the final camera external parameter according to the updated pitch angle and the initial camera external parameter, and calibrate the vehicle-mounted camera according to the final camera external parameter.

[0086] The technical scheme of the embodiment of the application, by inputting the original image dynamically collected by the vehicle-mounted camera into the vanishing point detection algorithm model to acquire the vanishing point coordinate, using the vanishing point detection algorithm, improves the accuracy of calibration, according to the vanishing point coordinate, the pitch angle change value can be calculated, and the initial pitch angle of the static calibration is updated to acquire the updated pitch angle of the dynamic calibration, and the initial camera external parameter can be updated according to the updated pitch angle, and finally the calibration of the vehicle-mounted camera is completed through the final camera external parameter and the updated pitch angle, without manual measurement, automatic calibration during vehicle driving can be completed.

[0087] The vehicle-mounted camera pitch angle calibration device provided by the embodiment of the application can execute the vehicle-mounted camera pitch angle calibration method provided by any embodiment of the application, and has the corresponding function modules and beneficial effects of the execution method.

[0088] Embodiment Four

[0089] Figure 7 A structural schematic diagram of an electronic device 10 that can be used to implement embodiments of the application is shown. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smart phones, wearable devices (e.g., headsets, glasses, watches, etc.), and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not intended to limit the implementations of the applications described and / or claimed in this document.

[0090] As Figure 7As shown, the electronic device 10 includes at least one processor 11, and a memory, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., communicatively connected to the at least one processor 11, where the memory stores a computer program executable by the at least one processor. The processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or loaded into the random access memory (RAM) 13 from the storage unit 18. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0091] Various components in the electronic device 10 are connected to the I / O interface 15, including an input unit 16, such as a keyboard, a mouse, etc., an output unit 17, such as various types of displays, a speaker, etc., a storage unit 18, such as a magnetic disk, an optical disk, etc., and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0092] The processor 11 can be various general and / or special-purpose processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 performs various methods and processes described above, such as a vehicle-mounted camera pitch angle calibration method.

[0093] In some embodiments, a vehicle-mounted camera pitch angle calibration method can be implemented as a computer program tangibly embodied in a computer readable storage medium, such as the storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of a vehicle-mounted camera pitch angle calibration method described above can be performed. Alternatively, in other embodiments, the processor 11 can be configured to perform a vehicle-mounted camera pitch angle calibration method by any other appropriate means, such as by means of firmware.

[0094] The various embodiments of the systems and techniques described above can be implemented in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a load programmable logic device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0095] Computer programs used to implement the processes of the application can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the computer program, when executed, can cause instructions defined in the flow charts and / or block diagrams to be implemented. The computer program can be executed entirely on a machine, partially on a machine, partially on a machine as a standalone software package and partially on a remote machine or entirely on a remote machine or server.

[0096] In the context of the present application, a computer-readable storage medium can be a tangible medium that can contain or store computer programs for use by or in connection with an instruction execution system, apparatus, or device. Computer-readable storage media can include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. More specific examples of the machine-readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disk drives, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0097] To provide for interaction with a user, the systems and techniques described here can be implemented on an electronic device having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the electronic device. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0098] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0099] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. A server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system, to solve the defects of large management difficulty and weak business scalability in traditional physical host and VPS service.

[0100] It should be understood that the various forms of flow shown above can be re-ordered, added to, or deleted from without departing from the scope of the present disclosure. For example, the steps recited in the present disclosure can be executed in parallel, executed in sequence, or executed in a different order, as long as the desired results of the present disclosure are achieved, and the present disclosure is not limited herein.

[0101] The specific embodiments described above are not intended to be limiting, and persons skilled in the art will appreciate that various modifications, combinations, sub-combinations and alternatives can be made to the specific embodiments without departing from the spirit and principles of the disclosure. Accordingly, the disclosure is not limited to the specific embodiments described above, but only by the scope of the appended claims.

Claims

1. A vehicle camera pitch angle calibration method, characterized in that, The method comprises the following steps: acquiring an original image dynamically collected by a vehicle-mounted camera and an initial pitch angle calibrated statically; inputting the original image into a vanishing point detection algorithm model to acquire a vanishing point coordinate, and generating a pitch angle change value according to the vanishing point coordinate; updating the initial pitch angle according to the pitch angle change value to acquire an updated pitch angle calibrated dynamically, and calibrating the vehicle-mounted camera by using the updated pitch angle; wherein the step of inputting the original image into the vanishing point detection algorithm model to acquire the vanishing point coordinate comprises the following steps: cutting the original image to generate image slices, wherein each image slice has an image number corresponding to the position of the image slice in the original image; calculating the embedded vector of each image slice after projection by using the vanishing point detection algorithm model, and determining the image number corresponding to the embedded vector, wherein the vanishing point detection algorithm model comprises a linear projection layer module, a self-attention encoder module and a multi-layer perception module; splicing each embedded vector according to the image number to generate a splicing matrix; acquiring a learnable vector according to the splicing matrix, splicing the splicing matrix and the learnable vector to generate a learning matrix, wherein the learnable vector and the splicing matrix have the same dimension; generating the vanishing point coordinate according to the learning matrix by using the vanishing point detection algorithm model; wherein the step of calculating the embedded vector of each image slice after projection by using the vanishing point detection algorithm model comprises the following steps: acquiring an initial vector corresponding to each image slice, wherein the initial vector is represented by seq1, seq1=N*L, N represents the image number, L represents the length, L=3*Hp*Wp, Hp represents the pixel height of each image slice, and Wp represents the pixel width of each image slice; inputting each initial vector into the linear projection layer module in sequence to obtain the embedded vector, wherein the linear projection layer comprises a first correspondence relationship between the initial vector and the embedded vector, and the embedded vector is represented by seq2, seq2=seq1*E, E represents a learnable parameter matrix located in the linear projection layer module.

2. The method of claim 1, wherein, the step of generating the vanishing point coordinate according to the learning matrix by using the vanishing point detection algorithm model comprises the following steps: inputting the learning matrix into the self-attention encoder module to acquire image features output by the self-attention encoder module, wherein the self-attention encoder module comprises an image feature extraction method corresponding to the learning matrix; inputting the image features into the multi-layer perception module to acquire a vanishing vector matched with the image features output by the multi-layer perception module, wherein the multi-layer perception module comprises a second correspondence relationship between the image features and the vanishing vector; generating the vanishing point coordinate according to the vanishing vector.

3. The method of claim 2, wherein, Before the step of inputting the image features into the multi-layer perception module, the method further comprises the following steps: acquiring an image feature sample set, wherein the image feature sample set comprises each image feature sample and its corresponding vanishing vector; Deep learning is performed on a second correspondence relationship between each of the image feature samples and the vanishing vector to establish the multi-layer perception module.

4. The method of claim 1, wherein, The generation of the pitch angle change value according to the vanishing point coordinates comprises: obtaining the center coordinates and the camera focal length of the original image, calculating the vertical coordinate difference between the vanishing point coordinates and the center coordinates; determining the pitch angle change value according to the vertical coordinate difference and the camera focal length.

5. The method of claim 1, wherein, After the dynamic updated pitch angle is obtained by updating the initial pitch angle according to the pitch angle change value, the method further comprises: obtaining the initial camera external parameter of the vehicle-mounted camera, calculating the final camera external parameter according to the updated pitch angle and the initial camera external parameter; calibrating the vehicle-mounted camera according to the final camera external parameter.

6. A vehicle camera pitch angle calibration apparatus characterized by comprising: Comprise: an original image and initial pitch angle obtaining module, configured to obtain an original image dynamically collected by a vehicle-mounted camera and an initial pitch angle obtained by static calibration; a pitch angle change value generating module, configured to input the original image into a vanishing point detection algorithm model to obtain vanishing point coordinates, and generate a pitch angle change value according to the vanishing point coordinates; a vehicle-mounted camera calibration module, configured to update the initial pitch angle according to the pitch angle change value to obtain a dynamic updated pitch angle, and calibrate the vehicle-mounted camera by using the updated pitch angle; wherein the pitch angle change value generating module specifically comprises: an image slice generating unit, configured to split the original image to generate image slices, wherein each of the image slices has an image number corresponding to the position of the image slice in the original image; an embedded vector calculating unit, configured to calculate embedded vectors of each of the image slices after projection by using the vanishing point detection algorithm model, and determine the image number corresponding to the embedded vectors, wherein the vanishing point detection algorithm model comprises a linear projection layer module, a self-attention encoder module and a multi-layer perception module; a splicing matrix generating unit, configured to splice each of the embedded vectors according to the image number to generate a splicing matrix; a learning matrix generating unit, configured to obtain a learnable vector according to the splicing matrix, splice the splicing matrix and the learnable vector to generate a learning matrix, wherein the learnable vector and the splicing matrix have the same dimension; a vanishing point coordinate generating unit, configured to generate the vanishing point coordinates according to the learning matrix by using the vanishing point detection algorithm model; wherein the embedded vector calculating unit is specifically configured to: obtain an initial vector corresponding to each of the image slices, wherein the initial vector is represented by seq1, seq1=N*L, N represents the image number, L represents the length, L=3*Hp*Wp, Hp represents the pixel height of each image slice, and Wp represents the pixel width of each image slice; input each of the initial vectors into the linear projection layer module in sequence to obtain the embedded vectors, wherein the linear projection layer comprises a first correspondence relationship between the initial vectors and the embedded vectors, and the embedded vectors are represented by seq2, seq2=seq1*E, and E represents a learnable parameter matrix located in the linear projection layer module.

7. An electronic device, comprising: The electronic device comprises: at least one processor; and a memory connected with the at least one processor; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-5.

8. A computer storage medium, characterized in that The computer storage medium stores computer instructions for causing the processor to implement the method of any one of claims 1-5 when executed.

Images