Data processing method, electronic device, vehicle, and storage medium

Abstract

The application discloses a data processing method, an electronic device, a vehicle and a computer readable storage medium, which obtains a first environment image and a second environment image of a target environment taken from different heights of a vehicle through an electronic device in the vehicle, and then obtains target point cloud data of the target environment according to first point cloud data corresponding to the first environment image and second point cloud data corresponding to the second environment image. The application performs three-dimensional point cloud reconstruction on the environment images obtained at different heights of the vehicle. Since the environment images at different heights can describe the environment images from different angles, more details in the target environment are obtained, so that the point cloud data corresponding to the environment images at different heights is fused to obtain environment feature information, the accuracy of the environment feature information is improved, and the accuracy of long-distance perception through the environment feature information is further improved.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of intelligent driving, and in particular, to a data processing method, an electronic device, a vehicle, and a computer readable storage medium. BACKGROUND

[0002] Based on the rapid development of the intelligent driving industry, people have increasingly high requirements for the capabilities of intelligent driving. Long-range perception can provide more decision-making time for intelligent driving systems, so as to better provide route planning and control the driving of the vehicle, thereby achieving the effect of improving user safety and comfort.

[0003] Currently, long-range perception is often performed by using a wide baseline camera, but the three-dimensional reconstruction accuracy of the wide baseline camera is poor. SUMMARY

[0004] The present disclosure provides a data processing method, an electronic device, a vehicle, and a computer readable storage medium. The method can perform three-dimensional point cloud reconstruction according to images obtained from different directions during the driving of the vehicle, improve the accuracy of three-dimensional point cloud reconstruction, and achieve the effect of improving the accuracy of long-range perception of the vehicle.

[0005] In a first aspect, a data processing method is provided. The method includes:

[0006] obtaining a first environment image and a second environment image, the first environment image and the second environment image being images of a target environment taken from different directions, respectively;

[0007] obtaining environment feature information of the target environment according to the first environment image and the second environment image.

[0008] In a second aspect, an electronic device is provided. The electronic device includes a memory and a processor. The memory stores a computer program. When the processor executes the computer program, the steps of the method provided in the first aspect are implemented.

[0009] In a third aspect, an image acquisition system of a vehicle is provided. The image acquisition system includes a first image acquisition component and a second image acquisition component. The first image acquisition component and the second image acquisition component are configured to take a target environment from different directions, respectively, to obtain environment feature parameters of the target environment.

[0010] In a fourth aspect, a vehicle is provided. The vehicle includes the electronic device provided in the second aspect.

[0011] In a fifth aspect, a computer readable storage medium is provided. The computer readable storage medium stores a computer program. When the computer program is executed by a processor, the steps of the method provided in the first aspect are implemented.

[0012] The application provides a data processing method, an electronic device, a vehicle and a computer readable storage medium. The method obtains a first environment image and a second environment image of a target environment captured from different heights of a vehicle through an electronic device in the vehicle, and then obtains target point cloud data of the target environment according to first point cloud data corresponding to the first environment image and second point cloud data corresponding to the second environment image. The application performs three-dimensional point cloud reconstruction on the environment images obtained at different heights of the vehicle. Since the environment images at different heights can describe the environment images from different angles, more details in the target environment are obtained. Therefore, the point cloud data corresponding to the environment images in different directions is fused to obtain environment feature information, which can improve the accuracy of the environment feature information and further improve the accuracy of long-distance perception through the environment feature information. BRIEF DESCRIPTION OF DRAWINGS

[0013] Other features, objects and advantages of the application will become more apparent from the following detailed description of non-limiting embodiments made with reference to the accompanying drawings:

[0014] Figure 1 An application scenario diagram of a data processing method provided by the application is shown in the following figure:

[0015] Figure 2 A step flowchart of a data processing method provided by the application is shown in the following figure:

[0016] Figure 3 A step flowchart of another data processing method provided by the application is shown in the following figure:

[0017] Figure 4 A step flowchart of another data processing method provided by the application is shown in the following figure:

[0018] Figure 5 A step flowchart of another data processing method provided by the application is shown in the following figure:

[0019] Figure 6 A step flowchart of another data processing method provided by the application is shown in the following figure:

[0020] Figure 7 A step flowchart of another data processing method provided by the application is shown in the following figure:

[0021] Figure 8 A step flowchart of another data processing method provided by the application is shown in the following figure:

[0022] Figure 9 A step flowchart of another data processing method provided by the application is shown in the following figure:

[0023] Figure 10 A step flowchart of another data processing method provided by the application is shown in the following figure:

[0024] Figure 11 A step flow chart of another data processing method provided by the present application;

[0025] Figure 12 A step flow chart of another data processing method provided by the present application;

[0026] Figure 13 A step flow chart of another data processing method provided by the present application;

[0027] Figure 14 A step flow chart of another data processing method provided by the present application;

[0028] Figure 15 A structural schematic diagram of an electronic device provided by the present application;

[0029] Figure 16 A structural schematic diagram of a computer system provided by the present application. DETAILED DESCRIPTION

[0030] The present application will be further described below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are intended to be merely illustrative of the present application and not in limitation thereof. It should also be noted that only the parts related to the present application are shown in the accompanying drawings for the purpose of description.

[0031] It should be noted that the embodiments and features in the embodiments of the present application can be combined with each other without conflict. The present application will be described in detail below with reference to the accompanying drawings and embodiments.

[0032] Reference should be made to Figure 1 , Figure 1An application scenario diagram of the long-distance sensing method provided by the embodiments of the present application is shown in the figure. At least two first cameras 100 are arranged at the positions of the front headlights of the vehicle, and two second cameras 200 are arranged at the positions of the roof of the vehicle, and the first cameras and the second cameras have a height difference. An electronic device (not shown in the figure) is arranged inside the vehicle. The two first cameras 100 are used to collect first environment images, and the two second cameras 200 are used to collect second environment images. The two first cameras 100 and the two second cameras 200 are in communication connection with the electronic device, and can send the collected first environment images and second environment images to the electronic device through a network, so that the electronic device performs three-dimensional point cloud reconstruction on the first environment images and the second environment images, obtains first point cloud data corresponding to the first environment images and second point cloud data corresponding to the second environment images, and finally performs point cloud fusion processing on the two, to obtain environment feature information, so as to plan the driving route of the vehicle in the process of driving the vehicle, so as to achieve the purpose of ensuring the safe driving of the vehicle and improving the driving experience of the driver.

[0033] In order to solve the problem of poor three-dimensional point cloud reconstruction accuracy existing in the long-distance sensing of the vehicle in the driving process at present, the present application provides a new long-distance sensing method, and the steps of the method are described as follows. Figure 2 Figure 2 A long-distance sensing method is provided for the present application, and the electronic device in the vehicle is taken as an example to illustrate the application of the long-distance method. The method comprises the following steps:

[0034] Step S20, acquiring first environment images and second environment images, the first environment images and the second environment images being images obtained by photographing a target environment from different directions;

[0035] The target environment can be the environment in which the vehicle is driving, and the vehicle can be any one of the traffic tools that need to plan the path in the driving process. For example, the first environment image can be the image collected by the first camera, and the second environment image can be the image collected by the second camera. Since the number of the first camera and the second camera is not limited to one, the number of the first environment image and the second environment image is also not limited. The first camera is a first binocular camera, and the second camera is a second binocular camera.

[0036] When there are multiple first cameras and multiple second cameras, the multiple first cameras can be arranged at different positions on a horizontal line, and the multiple second cameras can also be arranged at different positions on a horizontal line. For example, Figure 1 ​As shown, if there are two first cameras and two second cameras, one first camera can be arranged on the right front headlamp of the vehicle, and one first camera can be arranged on the left front headlamp of the vehicle. One second camera can be arranged on the right front corner of the roof, and one second camera can be arranged on the left front corner of the roof.

[0037] Optionally, between the first camera and the second camera, a third camera, a fourth camera, a fifth camera, etc. can be arranged; the third camera, the fourth camera, and the fifth camera can be arranged on the side of the front windshield of the vehicle, and the number of the third camera, the fourth camera, and the fifth camera can include multiple, which is not limited in the present application, and can be arranged according to the use environment of the vehicle, the preference of the driver, the driving demand, etc.

[0038] It should be noted here that since the present application plans the driving route of the vehicle during driving, the first camera, the second camera, the third camera, the fourth camera, etc. need to be arranged on the part of the center line of the vehicle including the front windshield, so that the images of the target environment in front of the vehicle during driving can be better collected, and the driving route of the vehicle can be more reasonably planned.

[0039] Optionally, the plurality of first environment images can be captured by the first cameras arranged on the left side and the right side of the vehicle; the plurality of second environment images can be captured by the second cameras arranged on the left side and the right side of the vehicle. After the first camera and the second camera capture the corresponding environment images, the unique identifier of the camera and the shooting time can be added to the corresponding environment images for targeted processing by the electronic device.

[0040] For example, the first environment image can be a plurality of images captured by the first camera within a preset time period in its shooting range. The second environment image can be a plurality of images captured by the second camera within the same preset time period in its shooting range. The preset time period is, for example, 1 second, 2 seconds, 5 seconds, 1 minute, 2 minutes, 3 minutes, etc. The first camera and the second camera can continuously capture a plurality of first environment images and a plurality of second environment images within the preset time period, or can capture a plurality of first environment images and a plurality of second environment images according to a preset shooting interval within the preset time period, as long as it can support subsequent long-distance perception, which is not limited in the present application.

[0041] After the electronic device of the present application receives the first environment image and the second environment image sent by the first camera and the second camera based on the network, the first environment image and the second environment image need to be processed to obtain the environmental feature information of the target environment according to the first environment image and the second environment image. The processing flow of the electronic device for each environment image is described below.

[0042] Step S30, obtaining the environmental feature information of the target environment according to the first and second environment images.

[0043] First, the electronic device obtains the first and second environment images for processing, and the process is shown in the following Figure 3 :

[0044] Step S301, obtaining the corresponding first point cloud data according to the first environment images, and obtaining the corresponding second point cloud data according to the second environment images;

[0045] The first point cloud data can be obtained by matching the first feature points of each first environment image through the descriptors of the first feature points; and the second point cloud data can be obtained by matching the second feature points of each second environment image through the descriptors of the second feature points.

[0046] The steps of obtaining the first feature points of each first environment image and the second feature points of each second environment image are described as follows: Figure 4

[0047] Step S401, obtaining the first brightness map of the first environment image and the second brightness map of the second environment image;

[0048] First, the spatial transformation is performed on each first and second environment image, that is, the first and second environment images are transformed from RGB space to YUV space. The specific operation of spatial transformation is to extract the brightness information in the three components of the first and second environment images and put it into the Y component; then extract the hue and color saturation information in the three components and put it into the U and V components. This processing is essentially a processing of the gray values of the first and second environment images to obtain the corresponding first and second brightness images.

[0049] The purpose of spatial transformation of the first and second environment images is to greatly remove the redundant information in the image through the initial image in YUV format. Since the human eye is more sensitive to two-point information and less sensitive to color, the spatial transformation of the first and second environment images can balance the image effect and compression rate, and facilitate more efficient three-dimensional point cloud reconstruction in the subsequent process.

[0050] Alternatively, after the electronic device of the vehicle receives a plurality of first environment images and a plurality of second environment images collected by the first and second cameras through the network, it can sequentially process each environment image, or batch process each environment image, or process the corresponding time of the shooting time based on the time of shooting the environment image, which is not limited in the present application.​

[0051] If the first camera includes two, the first left camera and the first right camera, since the first left camera and the first right camera are arranged on the same horizontal line of the vehicle, the first environment images taken at the same time point by the first left camera and the first right camera will have overlapping scenes, so the first environment images taken at the same time point by the first left camera and the first right camera are processed at the same time, and the identification of the same features in the first environment images is more accurate, so the accuracy of the three-dimensional point cloud reconstruction can be improved. Similarly, if the second camera also includes a second left camera and a second right camera, the electronic device can also process the second environment images taken at the same time point by the second left camera and the second right camera at the same time, and the identification of the same features will also be more accurate, which can further improve the accuracy of the three-dimensional point cloud reconstruction.

[0052] For example, the electronic device can first process the first environment image taken by the first left camera at 10:00 and the first environment image taken by the first right camera at 10:00, and then process the first environment image taken by the first left camera at 10:01 and the first environment image taken by the first right camera at 10:01, and so on. Correspondingly, the electronic device can also process the second environment image taken by the second left camera in the second camera at 10:00 and the second environment image taken by the second right camera at 10:00, the second environment image taken by the second left camera in the second camera at 10:01 and the second environment image taken by the second right camera at 10:01, and so on, in turn, after processing the multiple first environment images taken by the first camera.

[0053] Step S402, determining the difference information between the first brightness maps of each first environment image;

[0054] The difference information between the first brightness maps can be determined according to the steps as shown in Figure 5

[0055] Step S501, decomposing the first brightness map into a first base layer and a first detail layer;

[0056] First, the first brightness map can be smoothed, that is, filtered, which is to remove the noise in the first brightness image, so as to obtain a better quality initial image, and provide more reliable data support for subsequent three-dimensional point cloud reconstruction, and obtain more accurate three-dimensional reconstruction point cloud. For example, the first brightness image can be smoothed using a bilateral filtering method to obtain a first base layer corresponding to each first brightness image.

[0057] ​The high frequency part and the low frequency part in the first luminance image can be separated by the smoothing processing. Since the first luminance image will lose many details (the details are the high frequency part of the initial image) after the smoothing processing, the image after the smoothing processing is taken as the first base layer corresponding to the first luminance image, which is equivalent to obtaining the image of the low frequency part in the original initial image. Separating the low frequency and the high frequency of the initial image is essentially the processing of the pixel gray value of the image, so the obtained first base layer also belongs to the gray image.

[0058] Based on the above, the low frequency part in the first luminance image is obtained after the smoothing processing of the first luminance image. Then, the pixel gray value of the original first luminance image is subtracted from the low frequency part (i.e. the first base layer), so as to obtain the high frequency part (i.e. the first detail layer corresponding to the first luminance image).

[0059] In step S502, the first base layer is globally mapped to obtain a first global adaptive output layer.

[0060] The essence of the global mapping is to compress the luminance range of each pixel point in the first base layer.

[0061] The present application can be that the first base layer of the first luminance image is globally mapped by a mapping function to obtain the first global adaptive output layer corresponding to the first luminance image.

[0062] The mapping function is:

[0063]

[0064] L g1 is the first global adaptive output layer of the base layer; L base max1 is the maximum value data of the gray value of each pixel point in the base image of the first luminance image; is the first logarithmic average luminance, which is set as:

[0065]

[0066] N1 is the total number of pixels in the first luminance image; δ is a minimum value (the minimum value is used to prevent singular point when the value of the pixel is zero), for example, 0.0001; x, y are the coordinates of each pixel point in the first luminance image; L base1 is the gray value of each pixel point in the first luminance image.

[0067] In step S503, the first luminance difference value of the two first global adaptive output layers is calculated.

[0068] The first luminance difference value is obtained by subtracting the average luminance of the adaptive luminance space corresponding to the two first environment images captured by the first left camera and the first right camera in the first camera at the same time point, for example.

[0069] At step S403, the first adaptive luminance space is obtained by processing the first luminance map according to the difference information between the first luminance maps.

[0070] Optionally, the first adaptive luminance space is obtained according to the first global adaptive output layer, the first luminance difference value, and the first detail layer.

[0071] As shown in Figure 7 , Figure 7 An optional method embodiment for obtaining the first adaptive luminance space is provided in the present application, and the method embodiment includes the following steps:

[0072] At step S701, the first target number of layers is determined according to the first luminance difference value and the first global adaptive output layer.

[0073] The first adaptive luminance space is obtained by layering the first global adaptive output layer from different pixel information layers, that is, a plurality of images are obtained by processing the first adaptive luminance space, and the plurality of images represent different image information. Therefore, the accuracy of three-dimensional point cloud reconstruction can be improved by performing three-dimensional point cloud reconstruction according to the images after layering processing.

[0074] Therefore, before obtaining the first adaptive luminance space, the first target number of layers for layering processing of the first luminance image needs to be determined.

[0075] The first target number of layers can be calculated according to the following formula:

[0076]

[0077] N1 is the first target number of layers; D1 is a preset first luminance difference value; T1 is a luminance difference value of adjacent images in a preset adaptive luminance space layer, and T1 is usually assigned a value of 7.

[0078] At step S702, the first luminance superposition value corresponding to each layer is determined based on the first target number of layers and the first global adaptive output layer.

[0079] The first luminance superposition value corresponding to each layer is obtained by the following formula based on the first target number of layers obtained by the above method:

[0080]

[0081] A1 is a first brightness superposition value, D1 is a preset first brightness difference value, which is determined by a first target layer number, the electronic device is provided with corresponding relationship information between the first target layer number and γ, so when the electronic device determines the first target layer number, γ can be determined from the corresponding relationship information. Since the value of γ can be multiple, multiple first brightness superposition values can be obtained.

[0082] For example, if the first target layer number is 3, and the brightness of the first environment image is greater than the brightness of the first environment image collected by the first right camera of the first camera at the same time, the value of γ when processing the first environment image collected by the first left camera in the first camera is (1, 2, 3), and the value of γ when processing the first environment image collected by the first right camera is (-1, -2, -3); the brightness of the first environment image is less than the brightness of the first environment image collected by the first right camera of the first camera at the same time, the value of γ when processing the first environment image collected by the first left camera is (-1, -2, -3), and the value of γ when processing the first environment image collected by the first right camera is (1, 2, 3).

[0083] Step S703, the first brightness superposition value, the first global adaptive output layer and the first detail layer corresponding to each layer are subjected to gray value increasing or decreasing processing to obtain a first adaptive brightness space.

[0084] In this application, a plurality of first adaptive brightness spaces corresponding to the first brightness image can be obtained by the following formula:

[0085]

[0086] L1 is a first adaptive brightness space corresponding to the first brightness image; Lg1 is a first global adaptive output layer; Ldeatail1 is a first detail layer. The formula is to add or subtract the gray value of the corresponding pixel points in different images.

[0087] Since the value of A1 includes multiple values, a plurality of first adaptive brightness spaces corresponding to the first brightness image can be obtained.

[0088] Step S404, determine the difference information between the second brightness images of each second environment image;

[0089] In which, the difference information between the second brightness images of each second environment image can be determined according to the steps as shown in Figure 6

[0090] Step S601, decompose the second brightness image into a second base layer and a second detail layer;

[0091] ​The second luminance image can be smoothed first, and the smoothing is filtering. The filtering removes noise in the second luminance image, so that a better initial image is obtained, and more reliable data is provided for subsequent three-dimensional point cloud reconstruction, and more accurate three-dimensional reconstruction point cloud is obtained. For example, the second luminance image can be smoothed by using a bilateral filtering method to obtain a second base layer corresponding to each second luminance image.

[0092] The smoothing can separate the high-frequency part and the low-frequency part in the second luminance image. Since the second luminance image after smoothing loses many details (the details are the high-frequency part of the initial image), the image after smoothing is used as the second base layer corresponding to the second luminance image, which is equivalent to obtaining an image of the low-frequency part of the original initial image. Separating the low-frequency and high-frequency of the initial image is essentially processing the pixel gray value of the image, so obtaining the second base layer also belongs to a gray image.

[0093] Based on the above, after smoothing the second luminance image, the low-frequency part in the second luminance image is obtained. Then, the pixel gray value of the original second luminance image is subtracted from the low-frequency part (i.e., the second base layer), so that the high-frequency part (i.e., the second detail layer corresponding to the second luminance image) is obtained.

[0094] In step S602, the second base layer is globally mapped to obtain a second global adaptive output layer.

[0095] The second base layer of the second luminance image can be globally mapped by using a mapping function to obtain a second global adaptive output layer corresponding to the first luminance image.

[0096] The mapping function is:

[0097]

[0098] Lg2 is the second global adaptive output layer of the base layer; Lbase max2 is the maximum value data of the gray value of each pixel point in the base image of the second luminance image. Lg2 is the second global adaptive output layer of the base layer; Lbase max2 is the maximum value data of the gray value of each pixel point in the base image of the second luminance image.

[0099]

[0100] N2 is the total number of pixels in the first luminance image; δ is a minimum value (the minimum value is used to prevent the value of the pixel from being zero and producing a singular point), for example, 0.0001; x, y are the coordinates of each pixel point in the second luminance image; L base2 is the gray value of each pixel point in the second luminance image.

[0101] Step S603, calculating a second luminance difference value of the two second global adaptive output layers.

[0102] The second luminance difference value is obtained by subtracting the average luminance of the adaptive luminance space corresponding to the two second environment images captured by the second left camera and the second right camera in the second camera at the same time point, for example.

[0103] Step S405, processing the second luminance map according to the difference information between the second luminance maps to obtain a second adaptive luminance space;

[0104] Optionally, the second adaptive luminance space is obtained according to the second global adaptive output layer, the second luminance difference value, and the second detail layer.

[0105] As shown in Figure 8 , Figure 8 An optional method embodiment for obtaining the second adaptive luminance space provided by the present application includes the following steps:

[0106] Step S801, determining a second target layer number according to the second luminance difference value and the second global adaptive output layer;

[0107] The second adaptive luminance space is obtained by layering the second global adaptive output layer from different pixel information layers, that is, a plurality of images are obtained after processing the second adaptive luminance space, and the plurality of images represent different image information. Therefore, according to the three-dimensional point cloud reconstruction of the image after layering processing, the accuracy of the three-dimensional point cloud reconstruction can be improved.

[0108] Therefore, before obtaining the second adaptive luminance space, the second target layer number for layering processing of the second luminance image needs to be determined.

[0109] The second target layer number can be calculated according to the following formula:

[0110]

[0111] N2 is the second target layer number; D2 is a preset second luminance difference value; T2 is a luminance difference value of adjacent images in a preset adaptive luminance space layer, and T2 is usually assigned a value of 7.

[0112] Step S802, determining a second luminance superposition value corresponding to each layer based on the second target layer number and the second global adaptive output layer;

[0113] After the second target layer number is obtained based on the above method, the second luminance superposition value corresponding to each layer is obtained by the following formula:

[0114]

[0115] A2 is a second brightness superposition value, D2 is a preset second brightness difference value, which is determined by a second target layer number, the electronic device is provided with corresponding relationship information between the second target layer number and g, so when the electronic device determines the second target layer number, g can be determined from the corresponding relationship information. Since the value of g can have multiple values, multiple second brightness superposition values are obtained.

[0116] In step S803, the second brightness superposition value corresponding to each layer, the second global adaptive output layer and the second detail layer are subjected to gray value increasing or decreasing processing to obtain a second adaptive brightness space.

[0117] In the present application, a plurality of second adaptive brightness spaces corresponding to the second brightness image can be obtained by the following formula:

[0118]

[0119] L2 is a second adaptive brightness space corresponding to the second brightness image; Lg2 is a second global adaptive output layer; Ldetail2 is a second detail layer. The formula is to add or subtract the gray value of the corresponding pixel point in different images.

[0120] Since the value of A2 includes multiple values, a plurality of second adaptive brightness spaces corresponding to the first brightness image can be obtained.

[0121] In step S406, a first feature point is extracted from the first adaptive brightness space, and a second feature point is extracted from the second adaptive brightness space.

[0122] In the present application, the feature point in the image can be defined as a point of interest or a point with prominent features in the image. By identifying the feature points in the same area of the environment image taken at different angles, the object in the same area can be more accurately identified, and the accuracy of subsequent three-dimensional point cloud reconstruction is improved.

[0123] In the present application, the electronic device of the vehicle processes the first environment image to obtain a first adaptive brightness space, detects feature points in each first adaptive brightness space to obtain corner points of each first adaptive brightness space. The basic idea of obtaining image corner points is that if there are enough pixels around a pixel with large value difference, the point is largely a corner point. Therefore, the present application can detect feature points in each first adaptive brightness space to obtain corner points of each first adaptive brightness space by, for example, FAST algorithm. The method of feature point detection by FAST algorithm belongs to the prior art and will not be described here.

[0124] Similarly, after the electronic device of the vehicle processes the second environment image to obtain a second adaptive brightness space, feature points are detected in each second adaptive brightness space to obtain corner points of each second adaptive brightness space.

[0125] After obtaining the corner points of each first adaptive brightness space and the corner points of each second adaptive brightness space based on the above method, many feature points may be densely distributed in the feature area of the image. In order to improve the efficiency of subsequent operation, some corner points with large difference in pixel value from the threshold value can be screened and retained, and the data amount of subsequent operation can be reduced. This process can be realized by non-maximum suppression processing, which relies on checking all corner points in the field of each corner point. If the S value of the corner point is not the maximum S value of all corner points in the field, the principle of discarding the original corner point is used to operate, wherein the S value represents the sum of the absolute values of the difference between the corner point and each pixel value on the circle used when the corner point is extracted.

[0126] The present application obtains the corresponding corner points by performing feature point detection on each first adaptive brightness map and each second adaptive brightness map, and obtains the first feature points and the second feature points corresponding to each first adaptive brightness map and each second adaptive brightness map by performing non-maximum suppression processing on each corner point. The extraction speed of the feature points is fast, which not only provides reliable data support for subsequent three-dimensional reconstruction, but also improves the efficiency of three-dimensional reconstruction.

[0127] After the electronic device of the present application obtains the first feature points by performing key point recognition on each first adaptive brightness map space and obtains the second feature points by performing key point recognition on each second adaptive brightness space, the first feature points and the second feature points need to be further matched to perform three-dimensional reconstruction according to the matched points to obtain the first point cloud data and the second point cloud data. The process of obtaining the first point cloud data and the second point cloud data will be described below. Figure 9

[0128] Step S901, matching the first feature points of each first environment image by the descriptor of the first feature points, and matching the second feature points of each second environment image by the descriptor of the second feature points.

[0129] After obtaining the first feature points and the second feature points by the above method, the present application describes each feature point by local feature description. Since local feature description is a binary descriptor algorithm, the cost of extraction is low, and the subsequent matching only needs to use simple Hamming distance to complete the exclusive OR operation between bits, so time and space can be saved, and the efficiency of subsequent three-dimensional point cloud reconstruction is further improved.

[0130] The first feature points of the first environment image will be described below.

[0131] First, taking the first feature point as the center, a field window of s*s is removed, and N groups of points are randomly selected in the window, wherein N=128, 256, 512, and the default is 256. ​

[0132] Then, assuming x, y are two end points of a certain point pair, p(x), p(y) are pixel values corresponding to the two points, then:

[0133]

[0134] The binary assignment is performed on each point pair to form a feature vector of the first feature point. The vector is a local feature description of the first feature point. The vector is generally a 128-512 bit string and only contains 1 and 0.

[0135] The expression form of the binary description is as follows:

[0136]

[0137] N represents the number of first feature points, K represents the number of weak learners, l represents the label of the first feature point, γ is a learning rate parameter, a represents the weight of each weak learner, and h(x) represents a system function. The specific representation is as follows:

[0138]

[0139] f(x) represents a feature extraction equation, and T represents a threshold. The expression of f(x) is as follows:

[0140]

[0141] I(q) and I(r) represent the gray values of the pixel q of one of the first feature points and the pixel r of the other first feature point, R(p, s) represents a rectangular frame with p as the center and s as the size, p1 and p2 represent the positions of the feature points of the reference image and the to-be-matched image, wherein the reference image and the to-be-matched image are determined according to a matching rule or user selection, and are not limited herein.

[0142] That is, after the first feature points and the second feature points are obtained, the first feature points and the second feature points are described by the binary description described above.

[0143] After the local feature descriptions of the first feature points and the local feature descriptions of the second feature points are obtained by the above method, the corresponding matching points can be obtained by feature matching by a Hamming distance method or the like.

[0144] In step S902, a first reference image and a first matching image corresponding to the first reference image are determined from each first adaptive brightness space, the first reference image and the first matching image being first adaptive brightness spaces obtained after processing of first environmental images collected by different cameras in the same direction;

[0145] In matching each first feature point, the first adaptive brightness space collected by different cameras at the same height needs to be correspondingly matched. That is, if the first camera includes the first left camera and the first right camera, the first feature points of the first adaptive brightness space obtained after processing the first environmental images shot by the first left camera and the first right camera at the same time point need to be matched. First, the first reference image needs to be determined, which can be the first adaptive brightness space corresponding to the first environmental image shot by the first left camera, or the first adaptive brightness space corresponding to the first environmental image shot by the first right camera. The determination can be random or according to a predetermined determination rule, which is not limited in the application. After one of the two corresponding first adaptive brightness images is determined as the first reference image, the other first adaptive brightness image is taken as the first matching image. The matching process of the first feature points of multiple first adaptive brightness images is the same, which is only exemplarily described herein and is not limited.

[0146] Step S903, determining a second reference image and a second matching image corresponding to the second reference image from each second adaptive brightness space, the second reference image and the second matching image being second adaptive brightness spaces obtained after processing second environmental images collected by different cameras in the same direction;

[0147] The method of determining the second reference image and the second matching image is the same as that of determining the first reference image and the first matching image, which is not limited in the application.

[0148] Step S904, performing feature matching on the descriptors of the first feature points of the first matching image and the descriptors of the first feature points of the first reference image to obtain first matching points of the first matching image, the descriptors of the first matching points of the first matching image being the same as those of the corresponding first feature points of the first reference image;

[0149] After the first reference image and the first matching image are determined according to the above method, feature matching is performed according to the descriptors of the first feature points. Since the first feature points of the first reference image and the first matching image are binary codes, matching is determined by Hamming distance. If the number of same elements in the bit positions corresponding to the feature codes of two feature points is less than a preset value, for example, 128, the two feature points can be determined as not matching points, and vice versa. After the first feature points of the first reference image and the first matching image are matched according to the above method, the first matching points of the first matching image are obtained.

[0150] Step S905, performing feature matching between the descriptor of the second feature point of the second matching image and the descriptor of the first feature point of the first reference image, to obtain a second matching point of the second matching image, the second matching point of the second matching image being the same as the descriptor of the corresponding second feature point of the second reference image.

[0151] The method for determining the second matching point is the same as the method for determining the first matching point, which is not limited herein.

[0152] Step S906, determining the first matching point as an image matching point of the first adaptive brightness space, and determining the second matching point as an image matching point of the second adaptive brightness space.

[0153] Since the electronic device needs to process multiple first environment images and multiple second environment images, and the processing methods of the multiple first environment images are the same, multiple first adaptive brightness spaces and multiple second adaptive brightness spaces are obtained, and further, first matching images corresponding to the first adaptive brightness spaces and second matching images corresponding to the second adaptive brightness spaces are obtained, and finally, the image matching points of the first adaptive brightness space and the image matching points of the second adaptive brightness space are obtained based on the above matching method.

[0154] Step S907, performing three-dimensional point cloud reconstruction on the image matching point of the first adaptive brightness space to obtain first point cloud data.

[0155] Step S908, performing three-dimensional point cloud reconstruction on the image matching point of the second adaptive brightness space to obtain second point cloud data.

[0156] The present application provides a data processing method, which obtains corresponding image matching points by matching each adaptive brightness space. Since multiple adaptive brightness spaces in the same direction represent different pixel information in the same target environment, matching pixel points of adaptive brightness spaces in the same direction can obtain more accurate pixel point information. Then, the first point cloud data and the second point cloud data obtained by matching the image matching points for three-dimensional point cloud reconstruction can improve the accuracy of three-dimensional point cloud reconstruction at the same height.

[0157] Next, the method for processing the first environment image and the second environment image by the electronic device will be described in whole as an example. Figure 10 and Figure 11

[0158] As Figure 10 and Figure 11 ​As shown, after obtaining the first and second environment images, the electronic device first performs spatial variation on the first and second environment images to obtain a first and second luminance image; then decomposes the first luminance image into a first base layer and a first detail layer, and decomposes the second luminance image into a second base layer and a second detail layer; next, performs global mapping on the first and second base layers respectively to obtain a first and second global adaptive output layer; finally, obtains a first adaptive luminance image according to the first global adaptive output layer, the first luminance difference value and the first detail layer, and obtains a second adaptive luminance image according to the second global adaptive output layer, the second luminance difference value and the second detail layer.

[0159] In another embodiment, before obtaining the first and second point cloud data according to the mutually matched first and second feature points, in order to obtain the first and second point cloud data with high precision, the application can also first screen the mutually matched first and second feature points, and then perform three-dimensional reconstruction according to the screened first and second target matching points to obtain the first and second point cloud data, as shown in Figure 12 As shown, the method specifically comprises the following steps:

[0160] In step S1201, the mutually matched first and second feature points are screened based on a first and second inlier screening model, respectively, to obtain first and second target matching points.

[0161] The first inlier screening model is obtained according to the mutually matched first feature points, and the obtaining of the first inlier screening model comprises the following steps:

[0162] First, randomly select n groups of mutually matched first feature point pairs from the mutually matched first feature points as initial samples and calculate the parameter model corresponding to the homography matrix, denoted as M.

[0163] Second, sequentially input the remaining mutually matched first feature points into the model M, and if the input mutually matched first feature points can well fit the model, count them as inlier points of the model, otherwise do not count them, and finally count the number of all inlier points of the model.

[0164] Third, repeat the above steps k times (the iteration number k is determined by the following formula), and select the model with the largest number of inlier points among the k models as the first inlier screening model, wherein all the mutually matched first feature points in the first inlier screening model are the final first target matching points.

[0165]

[0166] In the formula: 2n is the initial number of samples, where n is 2, 4, etc.; p is the probability that n samples randomly selected from all matching first feature point pairs are all interior points of the first interior point screening model; w is the probability that a point is selected from all matching image point pairs as an interior point of the first interior point screening model.

[0167] The method for obtaining the second interior point screening model is the same as the method for obtaining the first interior point screening model, and will not be described in detail here.

[0168] Step S1202: Transform the three-dimensional coordinates of each first target matching point and each second target matching point to the global coordinate system to obtain the first point cloud data and the second point cloud data.

[0169] In this process, after the first and second matching feature points are filtered by the first and second interior point filtering models respectively to obtain the first target matching point and the second target matching point, the three-dimensional coordinates corresponding to the first target matching point need to be calculated using the triangulation principle based on the triangular relationship between the cameras capturing the first environmental image; and the three-dimensional coordinates corresponding to the second target matching point need to be calculated using the triangulation principle based on the triangular relationship between the cameras capturing the second environmental image. Then, using the camera calibration parameters of each camera, the three-dimensional coordinates of the first and second target matching points are transformed into a unified global coordinate system using matrix transformation to obtain the first point cloud data and the second point cloud data, so that they can be fused in the same coordinate system to obtain environmental feature information.

[0170] Furthermore, after obtaining the first and second point cloud data, it is also necessary to obtain environmental feature information of the target environment based on the first and second point cloud data to improve the accuracy of 3D point cloud reconstruction in different directions, and provide more reliable data support for long-distance perception of vehicles during driving. The following section will combine... Figure 13 The target point cloud data of the target environment is obtained based on the first and second point cloud data, and is explained below:

[0171] Step S302: Obtain environmental feature information of the target environment based on the first point cloud data and the second point cloud data.

[0172] The target point cloud data can be obtained by fusing the first point cloud data and the second point cloud data; it can also be obtained by filtering the first point cloud data and the second point cloud data and retaining one of them; or it can be obtained by converting the first point cloud data and the second point cloud data. This application does not limit the specific method.

[0173] like Figure 13 As shown, Figure 13An optional method embodiment for obtaining environmental feature information provided in the present application includes the following steps:

[0174] In step S1301, coordinate transformation is performed on one of the first point cloud data and the second point cloud data to obtain third point cloud data.

[0175] Optionally, coordinate transformation is performed on the one of the first point cloud data and the second point cloud data with less data to obtain the third point cloud data.

[0176] Before coordinate transformation is performed on one of the first point cloud data and the second point cloud data, first coordinate information of the first point cloud data and second coordinate information of the second point cloud data need to be obtained. Specifically, the first coordinate information of the first point cloud data is obtained by matrix transformation of three-dimensional coordinates of the first target matching point to a unified global coordinate system using camera calibration parameters of the first camera. Similarly, the second coordinate information of the second point cloud data is obtained by matrix transformation of three-dimensional coordinates of the second target matching point to a unified global coordinate system using camera calibration parameters of the second camera.

[0177] Then, the coordinate transformation relationship is determined, specifically including the following steps:

[0178] The coordinate transformation relationship can be obtained according to the following method:

[0179] First, the ICP algorithm is used for fine registration, that is, the shortest distance is calculated to find the nearest point (pi, qi), and then the optimal matching parameters R and T are calculated by iteration to minimize the error function. The error function is:

[0180] The error function is:

[0181]

[0182] Where n is the number of nearest point pairs, is one of the first coordinate information, is the corresponding nearest point in the second coordinate information, R is a rotation matrix, and T is a translation variable.

[0183] Finally, the coordinate transformation relationship matrix is output, which includes the rotation matrix R and the translation variable T.

[0184] Optionally, the third point cloud data can be obtained by coordinate transformation according to the following steps:

[0185] Coordinate transformation is performed on the one of the first point cloud data and the second point cloud data with less data to obtain the third point cloud data.

[0186] Wherein, based on the coordinate conversion relationship, the one with less point cloud data in the first point cloud data and the second point cloud data is subjected to coordinate conversion to obtain third point cloud data.

[0187] After the coordinate conversion, the two point cloud data represent a feature, at this time, the third point cloud data and the one with more point cloud data are subjected to point cloud fusion to obtain the environmental feature information, that is, step S1302 is executed.

[0188] In step S1302, the environmental feature information is obtained according to the third point cloud data and the other one of the first point cloud data and the second point cloud data, and the other one of the first point cloud data and the second point cloud data corresponds to the same coordinate system as the third point cloud data.

[0189] Optionally, the application provides an optional embodiment for obtaining environmental feature information, which comprises the following steps:

[0190] In step 1401, the points of the third point cloud data, the other one of the first point cloud data and the second point cloud data are subjected to feature matching.

[0191] Wherein, the other one refers to the one with more point cloud data in the first point cloud data and the second point cloud data, for example, the second point cloud data, and the points of the third point cloud data are subjected to feature matching with the points of the second point cloud data, so that the coincident point cloud data in the second point cloud data and the third point cloud data can be filtered out for subsequent fusion processing.

[0192] In step 1402, the third point cloud data and the other one of the first point cloud data and the second point cloud data are subjected to fusion processing according to the mutually matched points to obtain the environmental feature information.

[0193] Wherein, the fusion process can be to obtain new points by averaging the mutually matched points, and then to retain the points without matching to obtain target point cloud data. The mutually matched points can also be processed in other ways, which are not limited in the application.

[0194] The data processing method provided by the application can improve the accuracy of three-dimensional reconstruction in the vertical direction by performing point cloud fusion processing on three-dimensional point clouds reconstructed in different directions, and further improve the performance of long-distance sensing of the vehicle.

[0195] The application provides a data processing method, an electronic device, a vehicle and a computer readable storage medium. The method comprises the following steps: acquiring, by an electronic device in a vehicle, a first environment image and a second environment image obtained by photographing a target environment from different directions; and obtaining target point cloud data of the target environment according to first point cloud data corresponding to the first environment image and second point cloud data corresponding to the second environment image. The application performs three-dimensional point cloud reconstruction on the environment images obtained from different directions of the vehicle. Since the environment images from different directions can describe the environment images from different angles, more details in the target environment can be obtained. Therefore, the point cloud data corresponding to the environment images from different directions is fused to obtain the target point cloud data, which can improve the accuracy of the target point cloud data and further improve the accuracy of long-distance sensing based on the target point cloud data.

[0196] Further reference Figure 15 which shows an exemplary structural block diagram of the electronic device 1500 for performing the data processing method according to an embodiment of the application.

[0197] The electronic device comprises an acquisition module 1501 and an obtaining module 1502.

[0198] The acquisition module 1501 is configured to acquire a first environment image and a second environment image, wherein the first environment image and the second environment image are images obtained by photographing a target environment from different directions.

[0199] The obtaining module 1502 is configured to obtain environment feature information of the target environment according to the first environment image and the second environment image.

[0200] It should be understood that the modules described in the electronic device 1500 correspond to the respective steps in the method described with reference to Figure 2 The operations and features described above with respect to the method also apply to the electronic device 1500 and the units contained therein, and will not be described here. The electronic device 1500 can be pre-embedded in a browser or other secure application of the electronic device, or can be loaded into the browser or the secure application thereof of the electronic device by downloading or the like. The corresponding units in the electronic device 1500 can cooperate with each other to realize the solutions of the embodiments of the application.

[0201] In another embodiment, the application further provides an image acquisition system of a vehicle, comprising a first image acquisition component and a second image acquisition component, wherein the first image acquisition component and the second image acquisition component are configured to photograph a target environment from different directions to obtain environment feature parameters of the target environment. Optionally, the first image acquisition component is a first binocular camera, and the second image acquisition component is a second binocular camera.

[0202] The application also provides a vehicle comprising the electronic device described above, which can perform the following operations:

[0203] obtaining a first environment image and a second environment image, the first environment image and the second environment image being images of a target environment taken from different directions respectively;

[0204] obtaining environment feature information of the target environment according to the first environment image and the second environment image.

[0205] Reference is made below to Figure 16 which shows a structural diagram of a computer system 1600 suitable for use in implementing the vehicle of the embodiments of the application.

[0206] As shown in Figure 16 , the computer system 1600 comprises a central processing unit (CPU) 1601, which can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 1602 or programs loaded from a storage portion 1608 into a random access memory (RAM) 1603. Various programs and data required for the operation of the system 1600 are also stored in the RAM 1603. The CPU 1601, the ROM 1602, and the RAM 1603 are connected to each other through a bus 1604. An input / output (I / O) interface 1605 is also connected to the bus 1604.

[0207] The following components are connected to the I / O interface 1605: an input portion 1606 comprising a keyboard, a mouse, and the like; an output portion 1607 comprising a cathode ray tube (CRT), a liquid crystal display (LCD), and the like, and a speaker, and the like; a storage portion 1608 comprising a hard disk, and the like; and a communication portion 1609 comprising a network interface card such as a LAN card, a modem, and the like. The communication portion 1609 performs communication processing via a network such as the Internet. A drive 1610 is also connected to the I / O interface 1605 as necessary. A removable medium 1611 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, and the like is mounted on the drive 1610 as necessary, so that a computer program read therefrom is installed in the storage portion 1608 as necessary.

[0208] In particular, the processes described above with reference to Figure 2 may be implemented as a computer software program. For example, embodiments of the present disclosure include a computer program product comprising a computer program tangibly embodied on a machine-readable medium, the computer program comprising program code for performing the methods of Figure 2 In such embodiments, the computer program can be downloaded and installed from a network via the communication portion 1609, and / or installed from the removable medium 1611.

[0209] As another aspect, the present application also provides a computer readable storage medium, which can be the computer readable storage medium contained in the apparatus described in the above embodiments; or can exist separately and not be assembled into the apparatus. The computer readable storage medium stores one or more programs used by one or more processors to execute the formula input method described in the present application.

[0210] The above description is merely preferred embodiments of the present application and a description of the principles of the technology used. It should be understood by those skilled in the art that the scope of the application involved in the present application is not limited to the technical solutions formed by the specific combinations of the above technical features, and should also cover other technical solutions formed by any combinations of the above technical features or equivalent features without departing from the inventive concept. For example, the above features can be replaced with the technical features disclosed in the present application (but not limited to) having similar functions to form technical solutions.

Claims

1. A data processing method, characterized by, The method comprises: obtaining a first environment image and a second environment image, the first environment image and the second environment image being images taken from different directions of a target environment respectively; obtaining environment feature information of the target environment according to the first environment image and the second environment image; The method further comprises the steps of obtaining first feature points of each first environment image and second feature points of each second environment image: obtaining a first brightness map of the first environment image and a second brightness map of the second environment image; determining difference information between the first brightness maps of each first environment image; processing the first brightness maps according to the difference information between the first brightness maps to obtain a first adaptive brightness space; determining difference information between the second brightness maps of each second environment image; processing the second brightness maps according to the difference information between the second brightness maps to obtain a second adaptive brightness space; extracting the first feature points from the first adaptive brightness space and extracting the second feature points from the second adaptive brightness space; The determination of the difference information between the first brightness maps of each first environment image comprises: decomposing the first brightness maps into a first base layer and a first detail layer; performing global mapping on the first base layer to obtain a first global adaptive output layer; calculating a first brightness difference value of two first global adaptive output layers; The determination of the difference information between the second brightness maps of each second environment image comprises: decomposing the second brightness maps into a second base layer and a second detail layer; performing global mapping on the second base layer to obtain a second global adaptive output layer; calculating a second brightness difference value of two second global adaptive output layers; determining a first target layer number according to the first brightness difference value and the first global adaptive output layer, and determining a second target layer number according to the second brightness difference value and the second global adaptive output layer; determining a first brightness superposition value corresponding to each layer based on the first target layer number and the first global adaptive output layer, and determining a second brightness superposition value corresponding to each layer based on the second target layer number and the second global adaptive output layer; performing gray value increase or decrease processing on the first brightness superposition value corresponding to each layer, the first global adaptive output layer and the first detail layer to obtain the first adaptive brightness space, and performing gray value increase or decrease processing on the second brightness superposition value corresponding to each layer, the second global adaptive output layer and the second detail layer to obtain the second adaptive brightness space.

2. The method of claim 1, wherein, The environment feature information is obtained in the following manner: obtaining corresponding first point cloud data according to the first environment image, and obtaining corresponding second point cloud data according to the second environment image; obtaining environment feature information of the target environment according to the first point cloud data and the second point cloud data.

3. The method of claim 2, wherein: the first point cloud data is obtained by matching the first feature points of each first environment image through the descriptors of the first feature points of each first environment image. The second point cloud data is obtained by matching the second feature points of each of the second environment images according to the descriptors of the second feature points of each of the second environment images.

4. The method of claim 1, wherein, The first environment image is captured by a first camera on the vehicle, and the second environment image is captured by a second camera on the vehicle, and the first camera and the second camera have a height difference.

5. The method of claim 1, wherein, The processing of the first brightness maps according to the difference information between the first brightness maps to obtain a first adaptive brightness space comprises: According to the first global adaptive output layer, the first brightness difference value and the first detail layer, a first adaptive brightness space is obtained. The processing of the second brightness maps according to the difference information between the second brightness maps to obtain a second adaptive brightness space comprises: According to the second global adaptive output layer, the second brightness difference value and the second detail layer, a second adaptive brightness space is obtained.

6. The method of claim 2, wherein, The environment feature information of the target environment is obtained according to the first point cloud data and the second point cloud data, comprising: One of the first point cloud data and the second point cloud data is coordinate-converted to obtain third point cloud data; According to the third point cloud data, the other one of the first point cloud data and the second point cloud data, the environment feature information is obtained, and the other one of the first point cloud data and the second point cloud data corresponds to the same coordinate system as the third point cloud data.

7. The method of claim 6, wherein, The coordinate conversion of one of the first point cloud data and the second point cloud data to obtain third point cloud data comprises: The one of the first point cloud data and the second point cloud data with less data is coordinate-converted to obtain third point cloud data.

8. The method of claim 6, wherein, The environment feature information is obtained according to the third point cloud data, the other one of the first point cloud data and the second point cloud data, comprising: The points of the third point cloud data, the other one of the first point cloud data and the second point cloud data are feature-matched; According to the mutually matched points, the third point cloud data and the other one of the first point cloud data and the second point cloud data are fused to obtain the environment feature information. 9.An electronic device comprising a memory and a processor, the memory storing a computer program, wherein, The processor executes the computer program to realize the steps of the method of any one of claims 1 to 8.

10. An image acquisition system for a vehicle, characterized by A data processing method for realizing any one of claims 1-8, comprising a first image acquisition component and a second image acquisition component, the first image acquisition component and the second image acquisition component are configured to capture the target environment from different directions respectively to obtain the environment feature parameters of the target environment.

11. The image acquisition system of claim 10, wherein, The first image acquisition component is a first binocular camera, and the second image acquisition component is a second binocular camera.

12. A vehicle characterized by comprising: The vehicle comprises the electronic device of claim 9.

13. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to realize the steps of the method of any one of claims 1 to 8. The computer program is executed by the processor to realize the steps of the method of any one of claims 1 to 8.

Images