Image data processing method and device

Abstract

The disclosed embodiments disclose a method and apparatus for processing image data, wherein the processing method includes: processing a first image and a second image using a first neural network to obtain a homography matrix; determining, based on the homography matrix, mapped image features of features of the first image; fusing the mapped image features with features of the second image to obtain fused image features; and processing the fused image features using a second neural network to obtain a first pixel aspect ratio of the second image. The disclosed embodiments can obtain a dense and accurate pixel aspect ratio, thereby assisting in 3D scene modeling.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to a method and device for processing image data. Background Art

[0002] The plane parallax method models 3D scenes based on the differences between observing the same object or scene from two perspectives. This method relies on a specific plane and can restore the height of a pixel in the scene from the plane and the distance to the observation point, that is, the pixel height-to-depth ratio of the pixel.

[0003] Current planar disparity methods rely on optical flow estimation to match corresponding points between two viewpoints. However, optical flow methods cannot produce dense estimates and are significantly affected by noise. Determining how to obtain dense and accurate pixel-to-pixel aspect ratios based on image data is an urgent problem. Summary of the Invention

[0004] In order to solve the above technical problems, the present disclosure is proposed.

[0005] According to a first aspect of an embodiment of the present disclosure, a method for processing image data is provided, comprising:

[0006] Processing a first image and a second image using a first neural network to obtain a homography matrix, wherein the first image is taken at a first moment, the second image is taken at a second moment, and the first image and the second image have road surface elements in the same area;

[0007] determining, according to the homography matrix, a mapped image feature of a first image feature, wherein the first image feature is a feature extracted based on the first image;

[0008] fusing the mapped image features and the second image features to obtain fused image features, wherein the second image features are features extracted based on the second image;

[0009] The fused image features are processed using a second neural network to obtain a first pixel aspect ratio of the second image.

[0010] According to a second aspect of an embodiment of the present disclosure, there is provided an apparatus for processing image data, including:

[0011] a homography matrix determination module, configured to process a first image and a second image using a first neural network to obtain a homography matrix, wherein the first image is taken at a first moment, the second image is taken at a second moment, and the first image and the second image have road surface elements in the same area;

[0012] a mapping image feature determination module, configured to determine a mapping image feature of a first image feature according to the homography matrix, wherein the first image feature is a feature extracted based on the first image;

[0013] a fusion module, configured to fuse the mapped image features and the second image features to obtain fused image features, wherein the second image features are features extracted based on the second image;

[0014] The first pixel height-to-depth ratio determination module is configured to process the fused image features using a second neural network to obtain a first pixel height-to-depth ratio of the second image.

[0015] According to a third aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and the computer program is used to execute the image data processing method described in the first aspect.

[0016] According to a fourth aspect of an embodiment of the present disclosure, an electronic device is provided, the electronic device including:

[0017] processor;

[0018] a memory for storing instructions executable by the processor;

[0019] The processor is configured to read the executable instructions from the memory and execute the instructions to implement the image data processing method described in the first aspect above.

[0020] Based on the image data processing method and device provided by the above-mentioned embodiment of the present disclosure, a first neural network is used to process the first image and the second image taken by the camera and having road surface elements in a common area to obtain a homography matrix; then, the first image features are mapped by the homography matrix to obtain mapping image features, and the mapping image features are fused with the second image features to obtain fused image features; the fused image features are processed by the second neural network to determine the first pixel aspect ratio. The first pixel aspect ratio is the ratio between the height of the pixel of the target object in the second image relative to the road surface and the pixel depth, and the ratio can be used for 3D scene modeling. The image data processing method of the embodiment of the present disclosure can obtain a dense and accurate pixel aspect ratio based on the image data, which can assist in 3D scene modeling.

[0021] The technical solution of the present disclosure is further described in detail below through the accompanying drawings and examples. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The above and other purposes, features, and advantages of the present disclosure will become more apparent through a more detailed description of the embodiments of the present disclosure in conjunction with the accompanying drawings. The accompanying drawings are intended to provide a further understanding of the embodiments of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the present disclosure and are not intended to limit the present disclosure. In the drawings, the same reference numerals generally represent the same components or steps.

[0023] Figure 1 is a flowchart of a method for processing image data in one embodiment of the present disclosure;

[0024] Figure 2 is a flow chart of step S1 in one embodiment of the present disclosure;

[0025] Figure 3 is a diagram showing the working principle of the first neural network in an example of the present disclosure;

[0026] Figure 4 is a flowchart of a method for processing image data in another embodiment of the present disclosure;

[0027] Figure 5 is a schematic diagram of a flow chart after step S4 in another embodiment of the present disclosure;

[0028] Figure 6 is a schematic diagram of a flow chart after step S4 in another embodiment of the present disclosure;

[0029] Figure 7 is a flow chart of step S5″ in one embodiment of the present disclosure;

[0030] Figure 8 is a flow chart of steps S5″-6 in one embodiment of the present disclosure;

[0031] Figure 9 is a structural block diagram of an apparatus for processing image data in one embodiment of the present disclosure;

[0032] Figure 10 is a structural block diagram of a homography matrix determination module 100 in one embodiment of the present disclosure;

[0033] Figure 11 is a structural block diagram of an image data processing device in another embodiment of the present disclosure;

[0034] Figure 12 is a structural block diagram of an image data processing device in another embodiment of the present disclosure;

[0035] Figure 13 is a structural block diagram of an image data processing device in yet another embodiment of the present disclosure;

[0036] Figure 14is a structural block diagram of an overall loss value determination module 1000 in one embodiment of the present disclosure;

[0037] Figure 15 is a structural diagram of an electronic device provided by an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0038] Below, the exemplary embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all the embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the exemplary embodiments described herein.

[0039] It should be noted that the relative arrangement of components and steps, the numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure unless specifically stated otherwise.

[0040] Those skilled in the art will understand that the terms "first" and "second" in the embodiments of the present disclosure are only used to distinguish different steps, devices or modules, and do not represent any specific technical meanings, nor do they indicate a necessary logical order between them.

[0041] It should also be understood that in the embodiments of the present disclosure, “a plurality of” may refer to two or more than two, and “at least one” may refer to one, two, or more than two.

[0042] It should also be understood that any component, data or structure mentioned in the embodiments of the present disclosure can generally be understood as one or more, unless explicitly limited or otherwise indicated in the context.

[0043] In addition, the term "and / or" in this disclosure is merely a description of the association relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone. In addition, the character " / " in this disclosure generally indicates that the related objects are in an "or" relationship.

[0044] It should also be understood that the description of the various embodiments in this disclosure focuses on the differences between the various embodiments, and the same or similar aspects thereof can be referenced with each other. For the sake of brevity, they will not be described one by one.

[0045] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present disclosure, its application, or uses.

[0046] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered part of the specification.

[0047] It should be noted that like reference numerals and letters refer to like items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0048] The embodiments of the present disclosure can be applied to electronic devices such as terminal devices, computer systems, and servers, and can operate in conjunction with numerous other general-purpose or specialized computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments, and / or configurations suitable for use with terminal devices, computer systems, servers, and other electronic devices include, but are not limited to, personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network personal computers, minicomputer systems, mainframe computer systems, and distributed cloud computing technology environments including any of the above systems, among others.

[0049] Electronic devices such as terminal devices, computer systems, and servers can be described in the general context of computer system-executable instructions (such as program modules) executed by a computer system. Generally, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in a distributed cloud computing environment, where tasks are performed by remote processing devices linked via a communication network. In a distributed cloud computing environment, program modules can be located on local or remote computing system storage media, including storage devices.

[0050] Exemplary Methods

[0051] Figure 1 This is a flow chart of a method for processing image data in one embodiment of the present disclosure. This embodiment can be applied on a server, such as Figure 1 As shown, the following steps are included:

[0052] S1: Processing a first image and a second image using a first neural network to obtain a homography matrix, wherein the first image is taken at a first moment, the second image is taken at a second moment, and the first image and the second image have road surface elements in the same area.

[0053] Specifically, a camera is provided on the vehicle, and camera intrinsic parameters and camera extrinsic parameters are pre-set. When the vehicle is traveling, images are captured by the camera.

[0054] A first image captured by the same camera at a first moment and a second image captured by the same camera at a second moment are obtained. The images can be captured by shooting a video, or by capturing multiple frames of images at intervals. In this embodiment, the interval between the first moment and the second moment can be M frames, where M is an integer greater than 0.

[0055] Since images captured in a driving scene usually include a road surface, this embodiment uses the road surface in the first image and the second image as a reference plane required by the plane parallax method.

[0056] A feature extraction network is used to extract features from the first image to obtain first image features, and a feature extraction network is used to extract features from the second image to obtain second image features. The feature extraction network may belong to the first neural network or may be a network independent of the first neural network. In this embodiment, feature extraction is performed in the same downsampling manner. For example, the first image originally having 3*h*w dimensions (h and w represent the width and length of the image, respectively) is downsampled to obtain a first feature map of n*h'*w' dimensions as the first image features. The second image originally having 3*h*w dimensions is downsampled to obtain a second feature map of n*h'*w' dimensions as the second image features. Where n is the number of channels, h' can be 1 / 32, 1 / 64, and so on of h. w' is 1 / 32, 1 / 64, and so on of w. The values ​​of h' and w' may be the same or different.

[0057] The first image features and the second image features are processed using a first neural network to obtain a homography matrix for aligning the road surfaces in the first image and the second image. For example, the homography matrix is ​​calculated as follows:

[0058]

[0059] Among them, H represents the homography matrix, for example, a matrix with elements of 3X3 can be used, K represents the camera intrinsic parameter, K -1 represents the inverse matrix of K, d represents the height of the camera relative to the road surface, which can be obtained through calibration, R and t represent the camera relative rotation matrix (e.g., 3X3) and relative translation matrix (e.g., 1X3) between the first image and the second image, respectively, and N represents the road surface normal.

[0060] S2: Determine a mapped image feature of the first image feature according to the homography matrix.

[0061] Specifically, the first image feature is mapped to the perspective of the second image feature using the homography matrix to obtain a mapped image feature. The dimension of the mapped image feature is the same as that of the first image feature and the second image feature. According to the example of step S1, the dimension of the mapped image feature is n*h'*w'.

[0062] S3: Fusing the mapped image features and the second image features to obtain fused image features.

[0063] In an optional manner, the mapped image feature and the second image feature are superimposed according to the channel dimension to obtain a fused image feature. According to the example of steps S1 and S2, the dimension of the fused image feature is 2n*h'*w'.

[0064] S4: Processing the fused image features using a second neural network to obtain a first pixel aspect ratio of the second image.

[0065] Specifically, the second neural network is a pre-trained model, such as a deep learning model. The second neural network can predict the pixel aspect ratio based on the fused image features. In this embodiment, the pixel aspect ratio predicted by the second neural network based on the fused image features is used as the first pixel aspect ratio.

[0066] In this embodiment, a first neural network is used to process a first image and a second image taken by a camera and having road surface elements in a common area to obtain a homography matrix; then, the first image features are mapped by the homography matrix to obtain mapping image features, and the mapping image features are fused with the second image features to obtain fused image features; the fused image features are processed by a second neural network to determine a first pixel aspect ratio. The first pixel aspect ratio is the ratio between the height of the pixel of the target object in the second image relative to the road surface and the pixel depth, and this ratio can be used for 3D scene modeling. The image data processing method of the disclosed embodiment can obtain a dense and accurate pixel aspect ratio based on the image data, thereby assisting in 3D scene modeling.

[0067] Figure 2 This is a flow chart of step S1 in one embodiment of the present disclosure. Figure 2 As shown, in this embodiment, step S1 includes:

[0068] S2-1: Fusing the first image feature and the second image feature to obtain a third image feature.

[0069] Figure 3 This is a working principle diagram of the first neural network in an example of the present disclosure. Figure 3As shown, in this example, the feature extraction network belongs to the first neural network. After the feature extraction network extracts the first image feature and the second image feature, the first image feature and the second image feature are input into the feature fusion module for fusion. Among them, the feature fusion module can superimpose the first image feature and the second image feature according to the channel dimension to obtain a fused image feature. For example, when the dimension of the first image feature is n*h'*w' and the dimension of the second image feature is n*h'*w', the dimension of the third image feature is 2n*h'*w'.

[0070] S2-2: Use the road surface sub-network in the first neural network to process the third image feature to determine road surface normal information.

[0071] Please continue to refer to Figure 3 The feature fusion module inputs the third image feature to the road surface sub-network, which predicts the road surface normal information based on the third image feature. The road surface sub-network is a network model that predicts the road surface normal information based on the input image feature with road surface features. In this embodiment, the road surface normal information can be obtained by N = [n X ,n Y ,n z ] is expressed by the road surface equation. Among them, n X ,n Y and n z is the three-dimensional coordinate in the road coordinate system.

[0072] S2-3: Process the third image feature using the posture subnetwork in the first neural network to determine the relative posture of the camera between the first image and the second image.

[0073] Please continue to refer to Figure 3 The feature fusion module inputs the third image feature into the pose sub-network, which then predicts the camera's relative pose based on the third image feature and outputs it. The pose sub-network is a network model that predicts the camera's relative pose based on the input image features and outputs it. In this embodiment, the camera's relative pose includes a relative rotation matrix and a relative translation matrix.

[0074] S2-4: Determine a homography matrix based on the pre-stored camera height relative to the road surface, road surface normal information, and camera relative posture.

[0075] Please continue to refer to Figure 3The first neural network can determine the homography matrix using the above-mentioned calculation method of the homography matrix based on the camera's height relative to the road surface, the road surface normal information, and the camera's relative posture. It should be noted that the first neural network can also output the camera's height relative to the road surface, the road surface normal information, and the camera's relative posture, and then other modules outside the first neural network can determine the homography matrix.

[0076] In this embodiment, the road surface subnetwork and the posture subnetwork in the first neural network are used to process the fused third image features respectively. For example, the road surface subnetwork and the posture subnetwork can be used to process the third image features obtained by superimposing the first image features and the second image features in the channel dimension to obtain road surface normal information and camera relative posture. Based on the pre-stored camera height relative to the road surface, road surface normal information and camera relative posture, the homography matrix can be accurately determined.

[0077] Figure 4 FIG. 1 is a flow chart of a method for processing image data in another embodiment of the present disclosure. Figure 4 As shown, in this embodiment, after step S4, the following steps are further included:

[0078] S5: Determine a second pixel aspect ratio of the target object based on radar scanning data corresponding to the target object in the second image within an acquisition time of the second image.

[0079] Specifically, the vehicle is equipped with an onboard radar. Assuming the second image is captured at time t2, radar scan data near the vehicle at time t2 is obtained via the onboard radar. By analyzing the second image and the radar scan data near the vehicle, radar scan data corresponding to the target object can be extracted from the radar scan data near the vehicle based on the analysis results. Based on the extracted radar scan data, the position of the target object relative to the vehicle and the volume of the target object can be accurately determined. Furthermore, the true value of the pixel aspect ratio at time t2 can be generated, which is recorded as the second pixel aspect ratio.

[0080] S6: Adjust parameters of the second neural network based on the difference between the first pixel height-to-depth ratio and the second pixel height-to-depth ratio.

[0081] Specifically, based on the difference between the true value of the pixel aspect ratio (i.e., the second pixel aspect ratio) and the predicted value of the pixel aspect ratio (i.e., the first pixel aspect ratio) at time t2, the parameters of the second neural network are adjusted by back propagation.

[0082] In this embodiment, at the same time, the true value of the pixel height-to-depth ratio determined by the radar data is used as the supervision information of the second neural network. The parameters of the second neural network are reasonably adjusted through the difference between the true value and the predicted value of the pixel height-to-depth ratio to improve the prediction accuracy of the second neural network.

[0083] Figure 5 FIG. 1 is a flow chart after step S4 in another embodiment of the present disclosure. Figure 5 As shown, in this embodiment, after step S4, the following steps are further included:

[0084] S5ˊ: Reconstruct the first image using the homography matrix to obtain a first reconstructed image.

[0085] Specifically, the homography matrix is ​​used to reconstruct the first image by adopting a reverse mapping method to obtain a first reconstructed image.

[0086] S6ˊ: Adjust the matrix parameters of the homography matrix based on the pixel displacement of the road surface elements in the same area between the first reconstructed image and the second image.

[0087] Specifically, if the matrix parameters of the homography matrix are optimal, the first reconstructed image and the second image will be aligned in the road surface portion; if the matrix parameters of the homography matrix are not optimal, there will be pixel displacement between the first reconstructed image and the second image in the road surface portion.

[0088] Based on the pixel displacement of the road surface elements in the same area between the first reconstructed image and the second image, the matrix parameters of the homography matrix can be reasonably adjusted.

[0089] S7ˊ: Based on the homography matrix after adjusting the matrix parameters, adjust the parameters of the road sub-network and the attitude sub-network.

[0090] Specifically, the homography matrix is ​​determined based on the road surface normal information predicted by the road surface sub-network, the camera relative pose predicted by the pose sub-network, and the pre-stored camera relative road height. Therefore, by adjusting the homography matrix after matrix parameters, the parameters of the road surface sub-network and the pose sub-network are appropriately adjusted through backpropagation.

[0091] In this embodiment, the pixel displacement on the road surface elements in the same area between the first reconstructed image and the second image can reasonably adjust the matrix parameters of the homography matrix. The homography matrix after adjusting the matrix parameters is used as supervision information to reasonably adjust the parameters of the road surface sub-network and the posture sub-network, thereby improving the prediction accuracy of the road surface sub-network and the posture sub-network.

[0092] Figure 6 FIG. 1 is a flow chart after step S4 in another embodiment of the present disclosure. Figure 6 As shown, in this embodiment, after step S4, the following steps are further included:

[0093] S5″: determining an overall loss value based on the radar scanning data corresponding to the target object in the second image within the acquisition time of the second image, the first pixel aspect ratio, and the road surface mask of the second image.

[0094] Specifically, this embodiment treats the first neural network and the second neural network as a whole. Based on the first pixel aspect ratio and the road mask of the second image, the overall photometric loss value can be calculated using a photometric loss function. Based on the radar scan data corresponding to the target object in the second image during the acquisition time of the second image, the position of the target image relative to the vehicle and the volume of the target object can be obtained, thereby assisting in determining the overall supervisory loss value. Based on the overall photometric loss value and the supervisory loss value, the overall loss value can be determined.

[0095] S6″: Based on the overall loss value, adjust the parameters of the first neural network and the second neural network.

[0096] In this embodiment, the first neural network and the second neural network are taken as a whole. By calculating the overall loss value, the parameters of the first neural network and the second neural network can be reasonably adjusted to improve the prediction accuracy of the first neural network and the second neural network.

[0097] Figure 7 FIG. 1 is a flow chart of step S5″ in one embodiment of the present disclosure. Figure 7 As shown, in this embodiment, step S5″ includes:

[0098] S5″-1: Determine a second pixel aspect ratio of the target object based on the radar scanning data corresponding to the target object in the second image.

[0099] Specifically, the vehicle is equipped with an onboard radar. Assuming the second image is captured at time t2, radar scan data near the vehicle at time t2 is obtained via the onboard radar. By analyzing the second image and the radar scan data near the vehicle, radar scan data corresponding to the target object can be extracted from the radar scan data near the vehicle based on the analysis results. Based on the extracted radar scan data, the position of the target object relative to the vehicle and the volume of the target object can be accurately determined. Furthermore, the true value of the pixel aspect ratio at time t2 can be generated, which is recorded as the second pixel aspect ratio.

[0100] S5″-2: Determine a first loss value based on the first pixel aspect ratio and the second pixel aspect ratio. The first pixel aspect ratio may be subtracted from the second pixel aspect ratio to obtain the first loss value.

[0101] S5″-3: reconstructing the first image using the homography matrix to obtain a first reconstructed image.

[0102] Specifically, the homography matrix is ​​used to reconstruct the first image by adopting a reverse mapping method to obtain a first reconstructed image.

[0103] S5″-4: Determine a pixel displacement between the first image area and the second image area based on the first pixel aspect ratio. The first image area is the remaining image area in the first reconstructed image excluding the road surface image area, and the second image area is the remaining image area in the second image excluding the road surface image area.

[0104] Specifically, the homography matrix is ​​used to achieve basic alignment between the first and second reconstructed images in the road surface image area. (If the matrix parameters of the homography matrix are not optimal, some pixels in the road surface area between the first and second images may still be offset.) However, the first and second reconstructed images are not aligned in the remaining image areas, excluding the road surface image area. A pixel-by-pixel comparison is performed on the remaining image areas, excluding the road surface image area, to determine the pixel offset between the first and second image areas.

[0105] S5″-5: Based on the pixel displacement between the first image area and the second image area, the pixel position of the first reconstructed image is adjusted to obtain a second reconstructed image.

[0106] Specifically, based on the pixel displacement between the first and second image regions, the pixel positions of the first reconstructed image are adjusted to achieve pixel alignment between the second reconstructed image and the second image in the road surface image region. Since the first reconstructed image has already achieved basic alignment with the second image in the road surface image region, the second reconstructed image and the second image are basically aligned across the entire image.

[0107] S5″-6: Determine a second loss value based on the second reconstructed image, the second image, and the road surface mask of the second image.

[0108] Specifically, the photometric loss between the second reconstructed image and the second image may be calculated based on the second reconstructed image, the second image, and the road surface mask of the second image as the second loss value.

[0109] S5″-7: Determine an overall loss value based on the first loss value and the second loss value. The first loss value and the second loss value may be added to obtain the overall loss value.

[0110] In this embodiment, the overall loss value can be reasonably determined based on the radar scanning data corresponding to the target object in the second image during the acquisition time of the second image, the first pixel aspect ratio, and the road surface mask of the second image, so as to reasonably adjust the parameters of the first neural network and the second neural network based on the overall loss value, thereby improving the prediction accuracy of the first neural network and the second neural network.

[0111] Figure 8 FIG. 1 is a flow chart of steps S5″-6 in one embodiment of the present disclosure. Figure 8 As shown, in this embodiment, steps S5″-6 include:

[0112] S5″-6-1: Determine the full-image photometric error between the second reconstructed image and the second image.

[0113] In an optional method, based on the photometric error function, the full image photometric error is determined by the following formula:

[0114]

[0115] L_photo1=L p (It,Isw)

[0116] Among them, L p Represents the photometric loss coefficient, α represents the weight, and α is a constant, It represents the second image, Isw represents the second reconstructed image, SSIM(It, Isw) represents the structural similarity parameter between the second image and the second reconstructed image, and L_photo1 represents the full-image photometric error.

[0117] S5″-6-2: Based on the full image photometric error and the road surface mask of the second image, determine the photometric error between the second reconstructed image and the second image in the road surface image area.

[0118] In an optional manner, the photometric error between the second reconstructed image and the second image in the road surface image area is determined by the following formula:

[0119] L_photo2=mask_ground*L_photo1

[0120] Wherein, L_photo2 represents the photometric error between the second reconstructed image and the second image in the road surface image area, and mask_ground represents the road surface mask of the second image.

[0121] S5″-6-3: Determine a second loss value based on the photometric error of the entire image and the photometric error between the second reconstructed image and the second image in the road surface image area.

[0122] Specifically, the second loss value is determined by the following formula:

[0123] L_photoT=L photo 1+L_photo2

[0124] In this embodiment, based on the second reconstructed image, the second image and the road mask of the second image, the second loss value between the second reconstructed image and the second image can be reasonably determined, so as to reasonably adjust the parameters of the first neural network and the second neural network based on the second loss value, thereby improving the prediction accuracy of the first neural network and the second neural network.

[0125] Any of the image data processing methods provided in the embodiments of the present disclosure can be executed by any appropriate device with data processing capabilities, including but not limited to terminal devices and servers. Alternatively, any of the image data processing methods provided in the embodiments of the present disclosure can be executed by a processor, such as by invoking corresponding instructions stored in a memory to execute any of the image data processing methods mentioned in the embodiments of the present disclosure. This will not be further described below.

[0126] Exemplary devices

[0127] Figure 9 FIG. 1 is a structural block diagram of an image data processing device according to an embodiment of the present disclosure. Figure 9 As shown, in this embodiment, the image data processing device includes: a homography matrix determination module 100, a mapping image feature determination module 200, a fusion module 300 and a first pixel height-to-depth ratio determination module 400.

[0128] Among them, the homography matrix determination module 100 is used to use the first neural network to process the first image and the second image to obtain the homography matrix, wherein the first image is taken at the first moment, the second image is taken at the second moment, and the first image and the second image have road surface elements in the same area; the mapping image feature determination module 200 is used to determine the mapping image feature of the first image feature according to the homography matrix, wherein the first image feature is a feature extracted based on the first image; the fusion module 300 is used to fuse the mapping image feature and the second image feature to obtain a fused image feature, wherein the second image feature is a feature extracted based on the second image; the first pixel height-to-depth ratio determination module 400 is used to process the fused image feature using the second neural network to obtain the first pixel height-to-depth ratio of the second image.

[0129] Figure 10 FIG. 1 is a structural block diagram of a homography matrix determination module 100 in one embodiment of the present disclosure. Figure 10 As shown, in this embodiment, the homography matrix determination module 100 includes:

[0130] a fusion unit 101, configured to fuse the first image feature and the second image feature to obtain a third image feature;

[0131] a road surface normal information determining unit 102, configured to process the third image feature using the road surface sub-network in the first neural network to determine road surface normal information;

[0132] a camera relative pose determination unit 103, configured to process the third image feature using the pose subnetwork in the first neural network to determine the camera relative pose between the first image and the second image;

[0133] The homography matrix determining unit 104 is configured to determine the homography matrix based on pre-stored camera height relative to the road surface, the road surface normal information, and the camera relative posture.

[0134] Figure 11 FIG. 1 is a structural block diagram of an image data processing device in another embodiment of the present disclosure. Figure 11 As shown, in this embodiment, the image data processing device further includes:

[0135] a second pixel height-to-depth ratio determining module 500, configured to determine a second pixel height-to-depth ratio of the target object based on radar scanning data corresponding to the target object in the second image during an acquisition time of the second image;

[0136] The first parameter adjustment module 600 is used to adjust the parameters of the second neural network based on the difference between the first pixel height-to-depth ratio and the second pixel height-to-depth ratio.

[0137] Figure 12 FIG. 1 is a structural block diagram of an image data processing device in another embodiment of the present disclosure. Figure 12 As shown, in this embodiment, the image data processing device further includes:

[0138] An image reconstruction module 700 is configured to reconstruct the first image using the homography matrix to obtain a first reconstructed image;

[0139] a homography matrix parameter adjustment module 800, configured to adjust matrix parameters of the homography matrix based on pixel displacements of road surface elements in the same area between the first reconstructed image and the second image;

[0140] The first network parameter adjustment module 900 is configured to adjust parameters of the road surface sub-network and the posture sub-network based on the homography matrix after adjusting the matrix parameters.

[0141] Figure 13FIG. 1 is a structural block diagram of an image data processing device in another embodiment of the present disclosure. Figure 13 As shown, in this embodiment, the image data processing device further includes:

[0142] an overall loss value determining module 1000 for determining an overall loss value based on radar scan data corresponding to a target object in the second image during an acquisition time of the second image, the first pixel aspect ratio, and a road surface mask of the second image;

[0143] The second network parameter adjustment module 1100 is configured to adjust parameters of the first neural network and the second neural network based on the overall loss value.

[0144] Figure 14 FIG. 1 is a structural block diagram of an overall loss value determination module 1000 in one embodiment of the present disclosure. Figure 14 As shown, in this embodiment, the overall loss value determination module 1000 includes:

[0145] A second pixel height-to-depth ratio determining unit 1001 is configured to determine a second pixel height-to-depth ratio of the target object based on the radar scanning data;

[0146] A first loss value determining unit 1002 is configured to determine a first loss value based on the first pixel height-to-depth ratio and the second pixel height-to-depth ratio;

[0147] A first image reconstruction unit 1003 is configured to reconstruct the first image using the homography matrix to obtain a first reconstructed image;

[0148] a pixel displacement determining unit 1004 configured to determine a pixel displacement between a first image region and a second image region based on the first pixel aspect ratio, wherein the first image region is a remaining image region in the first reconstructed image excluding the road surface image region, and the second image region is a remaining image region in the second image excluding the road surface image region;

[0149] A second reconstructed image unit 1005 is configured to adjust pixel positions of the first reconstructed image based on the pixel displacement to obtain a second reconstructed image;

[0150] A second loss value determining unit 1006 is configured to determine a second loss value based on the second reconstructed image, the second image, and a road surface mask of the second image;

[0151] The overall loss value determining unit 1007 is configured to determine the overall loss value based on the first loss value and the second loss value.

[0152] In one embodiment of the present disclosure, the second loss value determination unit 1006 is specifically used to determine the full-image photometric error between the second reconstructed image and the second image; the second loss value determination unit 1006 is also used to determine the photometric error between the second reconstructed image and the second image in the road surface image area based on the full-image photometric error and the road surface mask of the second image; the second loss value determination unit 1006 is also used to determine the second loss value based on the full-image photometric error and the photometric error between the second reconstructed image and the second image in the road surface image area.

[0153] It should be noted that the specific implementation of the image data processing device of the embodiment of the present disclosure is similar to the specific implementation of the image data processing method of the embodiment of the present disclosure. Please refer to the image data processing method part for details. In order to reduce redundancy, it will not be described in detail.

[0154] Exemplary electronic devices

[0155] Below, reference Figure 15 To describe the electronic device according to the embodiment of the present disclosure. Figure 15 As shown, the electronic device includes one or more processors 10 and a memory 20 .

[0156] The processor 10 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device to perform desired functions.

[0157] The memory 20 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), a hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 10 may execute the program instructions to implement the image data processing methods of the various embodiments of the present disclosure described above and / or other desired functions. Various contents such as input signals, signal components, noise components, etc. may also be stored in the computer-readable storage medium.

[0158] In one example, the electronic device may further include an input device 30 and an output device 40, which are interconnected via a bus system and / or other connection mechanisms (not shown). The input device 30 may be, for example, a keyboard, a mouse, etc. The output device 40 may include, for example, a display, a speaker, a printer, a communication network, and remote output devices connected thereto.

[0159] Of course, to simplify, Figure 15 Only some of the components related to the present disclosure in the electronic device are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, the electronic device may further include any other appropriate components according to specific application scenarios.

[0160] Exemplary computer-readable storage media

[0161] Computer readable storage media can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can, for example, include but is not limited to a system, device or component of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0162] The basic principles of the present disclosure have been described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, and effects mentioned in this disclosure are merely illustrative and not restrictive, and should not be construed as necessarily possessed by each embodiment of the present disclosure. Furthermore, the specific details disclosed above are provided for illustrative purposes and to facilitate understanding, rather than as limitations. These details do not limit the present disclosure to necessarily being implemented using these specific details.

[0163] Each embodiment in this specification is described in a progressive manner, with each embodiment focusing on its differences from the other embodiments. References to the same or similar parts between the various embodiments are sufficient. For system embodiments, since they largely correspond to method embodiments, their description is relatively simple. For relevant parts, references to the description of the method embodiments are sufficient.

[0164] The block diagrams of the devices, devices, equipment, and systems involved in this disclosure are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these devices, devices, equipment, and systems can be connected, arranged, or configured in any manner. Words such as "include," "comprise," "have," and the like are open-ended words, meaning "including but not limited to," and can be used interchangeably therewith. The words "or" and "and" used herein refer to the words "and / or" and can be used interchangeably therewith, unless the context clearly indicates otherwise. The word "such as" used herein refers to the phrase "such as but not limited to," and can be used interchangeably therewith.

[0165] The methods and apparatus of the present disclosure may be implemented in many ways. For example, the methods and apparatus of the present disclosure may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above order of steps for the method is for illustration only, and the steps of the method of the present disclosure are not limited to the order specifically described above unless otherwise specified. In addition, in some embodiments, the present disclosure may also be implemented as programs recorded in a recording medium, which include machine-readable instructions for implementing the methods according to the present disclosure. Thus, the present disclosure also covers recording media that store programs for executing the methods according to the present disclosure.

[0166] It should also be noted that in the apparatus, device, and method of the present disclosure, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present disclosure.

[0167] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the aspects shown herein, but rather to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0168] The above description has been provided for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present disclosure to the forms disclosed herein. Although a number of example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. A method for processing image data, comprising: Processing a first image and a second image using a first neural network to obtain a homography matrix, wherein the first image is taken at a first moment, the second image is taken at a second moment, and the first image and the second image have road surface elements in the same area, and the homography matrix is ​​determined based on a height of a camera relative to a road surface, road surface normal information, and a relative camera pose between the first image and the second image; determining, according to the homography matrix, a mapped image feature of a first image feature, wherein the first image feature is a feature extracted based on the first image; fusing the mapped image features and the second image features to obtain fused image features, wherein the second image features are features extracted based on the second image; The fused image features are processed using a second neural network to obtain a first pixel aspect ratio of the second image.

2. The method for processing image data according to claim 1, wherein: The method of processing the first image and the second image by using the first neural network to obtain a homography matrix includes: fusing the first image feature and the second image feature to obtain a third image feature; Processing the third image feature using a road surface subnetwork in the first neural network to determine road surface normal information; Processing the third image feature using a pose subnetwork in the first neural network to determine a relative camera pose between the first image and the second image; The homography matrix is ​​determined based on the pre-stored height of the camera relative to the road surface, the road surface normal information, and the relative posture of the camera.

3. The image data processing method according to claim 1, after processing the fused image features using a second neural network to obtain a first pixel aspect ratio, further comprising: determining a second pixel aspect ratio of the target object based on radar scanning data corresponding to the target object in the second image during an acquisition time of the second image; Based on the difference between the first pixel aspect ratio and the second pixel aspect ratio, parameters of the second neural network are adjusted.

4. The image data processing method according to claim 2, further comprising: after the first image and the second image are processed by the first neural network to obtain a homography matrix; Reconstructing the first image using the homography matrix to obtain a first reconstructed image; Adjusting matrix parameters of the homography matrix based on pixel displacements of road surface elements in the same area between the first reconstructed image and the second image; Based on the homography matrix after adjusting the matrix parameters, parameters of the road surface sub-network and the posture sub-network are adjusted.

5. The image data processing method according to claim 1, after processing the fused image features using a second neural network to obtain a first pixel aspect ratio, further comprising: determining an overall loss value based on radar scan data corresponding to the target object in the second image during an acquisition time of the second image, the first pixel aspect ratio, and a road surface mask of the second image; Based on the overall loss value, parameters of the first neural network and the second neural network are adjusted.

6. The method for processing image data according to claim 5, wherein: The determining of the overall loss value based on radar scan data corresponding to the target object in the second image during the acquisition time of the second image, the first pixel aspect ratio, and a road surface mask of the second image includes: determining a second pixel aspect ratio of the target object based on the radar scanning data; Determining a first loss value based on the first pixel aspect ratio and the second pixel aspect ratio; Reconstructing the first image using the homography matrix to obtain a first reconstructed image; determining a pixel displacement between a first image region and a second image region based on the first pixel aspect ratio, wherein the first image region is a remaining image region excluding a road surface image region in the first reconstructed image, and the second image region is a remaining image region excluding a road surface image region in the second image; adjusting pixel positions of the first reconstructed image based on the pixel displacement to obtain a second reconstructed image; determining a second loss value based on the second reconstructed image, the second image, and a road surface mask of the second image; The overall loss value is determined based on the first loss value and the second loss value.

7. The method for processing image data according to claim 6, wherein: The determining a second loss value based on the second reconstructed image, the second image, and a road surface mask of the second image includes: determining a full-image photometric error between the second reconstructed image and the second image; determining a photometric error between the second reconstructed image and the second image in a road surface image region based on the full image photometric error and a road surface mask of the second image; The second loss value is determined based on the full image photometric error and the photometric error between the second reconstructed image and the second image in the road surface image area.

8. An image data processing device, comprising: a homography matrix determination module, configured to process a first image and a second image using a first neural network to obtain a homography matrix, wherein the first image is taken at a first moment, the second image is taken at a second moment, and the first image and the second image have road surface elements in the same area, and the homography matrix is ​​determined based on a height of the camera relative to the road surface, road surface normal information, and a relative camera pose between the first image and the second image; a mapping image feature determination module, configured to determine a mapping image feature of a first image feature according to the homography matrix, wherein the first image feature is a feature extracted based on the first image; a fusion module, configured to fuse the mapped image features and the second image features to obtain fused image features, wherein the second image features are features extracted based on the second image; The first pixel height-to-depth ratio determination module is configured to process the fused image features using a second neural network to obtain a first pixel height-to-depth ratio of the second image.

9. A computer-readable storage medium storing a computer program, wherein the computer program is used to execute the image data processing method according to any one of claims 1 to 7.

10. An electronic device, comprising: processor; a memory for storing instructions executable by the processor; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the image data processing method described in any one of claims 1 to 7.

Images