Training method of neural network model, image processing method, device and equipment

Abstract

The method comprises the following steps: determining training image features corresponding to each view angle based on obtained training image data; adding random disturbance to original camera extrinsics corresponding to each view angle to obtain new camera extrinsics corresponding to each view angle after disturbance; converting the training image features corresponding to each view angle to a preset plane of a vehicle coordinate system based on the new camera extrinsics corresponding to each view angle to obtain first bird's-eye view features; and training a first neural network model obtained in advance based on the first bird's-eye view features and label data to obtain a target neural network model after training. The method can improve the adaptability of the model to distorted bird's-eye view features, thereby reducing the adverse effects of distortion caused by IPM transformation on image processing results when the model is applied to actual image processing, and improving the robustness and accuracy of the model.

Description

Technical Field

[0001] This disclosure relates to computer vision technology, and in particular to a method for training a neural network model, an image processing method, an apparatus, and a device. Background Technology

[0002] In the field of autonomous driving, subsequent decisions and planning based on camera perception results are all conducted in the 3D world in which the vehicle is located (such as the vehicle coordinate system). However, the images captured by the camera describe a 2D pixel world (image pixel coordinate system), making it extremely difficult to directly make autonomous driving decisions and plans based on camera perception results (image features). Therefore, it is usually necessary to transform the camera perception results into the 3D world in which the vehicle is located to obtain a bird's-eye view map (BEVMap). In related technologies, camera perception results are usually converted into BEVMaps based on IPM (Inverse Perspective Mapping). However, IPM transformation is based on a flat, uneven ground surface and fixed camera calibration parameters. Otherwise, the BEV features after IPM transformation will be distorted, resulting in inaccurate detection and segmentation results when used in applications such as object detection and semantic segmentation. Summary of the Invention

[0003] To address the technical problems such as inaccurate image processing results due to IPM transform distortion, this disclosure is proposed. Embodiments of this disclosure provide a method for training a neural network model, an image processing method, an apparatus, and a device.

[0004] According to one aspect of the present disclosure, a method for training a neural network model is provided, comprising: acquiring training image data and label data, wherein the training image data includes image data from at least one viewpoint; determining training image features corresponding to each viewpoint based on the training image data; adding random perturbations to the original camera extrinsic parameters corresponding to each viewpoint to obtain perturbed new camera extrinsic parameters corresponding to each viewpoint, wherein the original camera extrinsic parameters are calibration extrinsic parameters of the camera coordinate system relative to the vehicle coordinate system; transforming the training image features corresponding to each viewpoint to a preset plane of the vehicle coordinate system based on the new camera extrinsic parameters corresponding to each viewpoint to obtain a first bird's-eye view feature, wherein the preset plane is parallel to the ground plane; and training a pre-obtained first neural network model based on the first bird's-eye view feature and the label data to obtain a trained target neural network model.

[0005] According to another aspect of the present disclosure, an image processing method is provided, comprising: acquiring image data to be processed, the image data to be processed including images from at least one viewpoint; determining first image features corresponding to each of the viewpoints based on the image data to be processed; determining target bird's-eye view features based on the first image features corresponding to each of the viewpoints; and processing the target bird's-eye view features using a pre-trained target neural network model to obtain an image processing result, wherein the target neural network model is trained based on the training method of the neural network model described in any of the above embodiments.

[0006] According to another aspect of the present disclosure, a training apparatus for a neural network model is provided, comprising: a first acquisition module for acquiring training image data and label data, wherein the training image data includes image data from at least one viewpoint; a first processing module for determining training image features corresponding to each viewpoint based on the training image data; a second processing module for adding random perturbations to the original camera extrinsic parameters corresponding to each viewpoint to obtain perturbed new camera extrinsic parameters corresponding to each viewpoint, wherein the original camera extrinsic parameters are calibration extrinsic parameters of the camera coordinate system relative to the vehicle coordinate system; a third processing module for converting the training image features corresponding to each viewpoint to a preset plane of the vehicle coordinate system based on the new camera extrinsic parameters corresponding to each viewpoint to obtain a first bird's-eye view feature, wherein the preset plane is parallel to the ground plane; and a fourth processing module for training a pre-obtained first neural network model based on the first bird's-eye view feature and the label data to obtain a trained target neural network model.

[0007] According to another aspect of the present disclosure, an image processing apparatus is provided, comprising: a second acquisition module for acquiring image data to be processed, the image data to be processed including images from at least one viewpoint; a fifth processing module for determining first image features corresponding to each of the viewpoints based on the image data to be processed; a sixth processing module for determining target bird's-eye view features based on the first image features corresponding to each of the viewpoints; and a seventh processing module for processing the target bird's-eye view features using a pre-trained target neural network model to obtain an image processing result, wherein the target neural network model is trained based on the training method of the neural network model described in any of the above embodiments.

[0008] According to another aspect of the present disclosure, a computer-readable storage medium is provided, the storage medium storing a computer program for executing a training method for a neural network model according to any of the above embodiments of the present disclosure; or, the computer program for executing an image processing method according to any of the above embodiments of the present disclosure.

[0009] According to another aspect of the present disclosure, an electronic device is provided, the electronic device comprising: a processor; a memory for storing executable instructions of the processor; the processor being configured to read the executable instructions from the memory and execute the instructions to implement a training method for a neural network model according to any of the above embodiments of the present disclosure; or, to execute the instructions to implement an image processing method according to any of the above embodiments of the present disclosure.

[0010] Based on the neural network model training method, image processing method, apparatus, and device provided in the above embodiments of this disclosure, by applying perturbation to the extrinsic parameters of the calibration camera required for conversion during the process of converting multi-view training image data to bird's-eye view, noisy bird's-eye view features are obtained for training the neural network model, thereby improving the model's adaptability to distorted bird's-eye view features. Thus, when applied to actual image processing, the adverse effects of distortion caused by IPM transformation on image processing results can be reduced, and the robustness and accuracy of the model can be improved.

[0011] The technical solutions of this disclosure will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0012] The above and other objects, features, and advantages of this disclosure will become more apparent from the more detailed description of the embodiments thereof in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this disclosure and form part of the specification. They are used together with the embodiments of this disclosure to explain the disclosure and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0013] Figure 1 This is an exemplary application scenario of the training method for the neural network model provided in this disclosure;

[0014] Figure 2 This is a flowchart illustrating a training method for a neural network model provided in an exemplary embodiment of this disclosure;

[0015] Figure 3 This is a flowchart illustrating a method for training a neural network model provided in another exemplary embodiment of this disclosure;

[0016] Figure 4 This is a flowchart illustrating step 2052 provided in an exemplary embodiment of this disclosure;

[0017] Figure 5 This is a schematic flowchart of an image processing method provided in an exemplary embodiment of this disclosure;

[0018] Figure 6 This is a schematic diagram of the structure of a training apparatus for a neural network model provided in an exemplary embodiment of this disclosure;

[0019] Figure 7 This is a schematic diagram of the structure of a training apparatus for a neural network model provided in another exemplary embodiment of this disclosure;

[0020] Figure 8 This is a schematic diagram of the structure of an image processing apparatus provided in an exemplary embodiment of the present disclosure;

[0021] Figure 9 This is a schematic diagram of the structure of an application embodiment of the electronic device disclosed herein. Detailed Implementation

[0022] Hereinafter, exemplary embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present disclosure, and not all embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the exemplary embodiments described herein.

[0023] It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of this disclosure.

[0024] Those skilled in the art will understand that the terms "first," "second," etc., in the embodiments of this disclosure are only used to distinguish different steps, devices, or modules, and do not represent any specific technical meaning, nor do they indicate a necessary logical order between them.

[0025] It should also be understood that in the embodiments disclosed herein, "a plurality of" may refer to two or more, and "at least one" may refer to one, two or more.

[0026] It should also be understood that any component, data or structure mentioned in the embodiments of this disclosure can generally be understood as one or more unless expressly defined or given to the contrary in the context.

[0027] Furthermore, the term "and / or" in this disclosure is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this disclosure generally indicates that the preceding and following related objects have an "or" relationship.

[0028] It should also be understood that the description of the various embodiments in this disclosure emphasizes the differences between the various embodiments, and the similarities or similarities can be referred to each other. For the sake of brevity, they will not be described in detail.

[0029] At the same time, it should be understood that, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn according to actual scale.

[0030] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use.

[0031] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0032] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0033] The embodiments disclosed herein can be applied to electronic devices such as terminal devices, computer systems, and servers, and can operate together with a wide range of other general-purpose or special-purpose computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments, and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, and servers include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments including any of the above systems, etc.

[0034] Electronic devices such as terminal devices, computer systems, and servers can be described in the general context of computer system executable instructions (such as program modules) executed by a computer system. Typically, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in distributed cloud computing environments, where tasks are executed by remote processing devices linked through communication networks. In distributed cloud computing environments, program modules can reside on local or remote computing system storage media, including storage devices.

[0035] This disclosure outlines

[0036] In developing this disclosure, the inventors discovered that in the field of autonomous driving, subsequent decisions and planning based on camera perception results are all conducted in the 3D world of the vehicle (e.g., the vehicle coordinate system). However, the images captured by the camera describe a 2D pixel world (image pixel coordinate system), making it extremely difficult to directly perform autonomous driving decision-making and planning based on camera perception results (image features). Therefore, it is usually necessary to transform the camera perception results into the 3D world of the vehicle to obtain a Bird's Eye View Map (BEVMap). In related technologies, camera perception results are usually converted into BEVMaps based on IPM (Inverse Perspective Mapping). However, IPM transformation is based on a flat, unundulating ground surface and fixed camera calibration parameters. Otherwise, the BEV features after IPM transformation will be distorted, resulting in inaccurate detection and segmentation results when used in applications such as object detection and semantic segmentation.

[0037] Exemplary Overview

[0038] Figure 1 This is an exemplary application scenario of the training method for the neural network model provided in this disclosure.

[0039] In autonomous driving scenarios, multiple cameras can be configured on autonomous vehicles to form a vision system, such as a four-way surround-view system, for perceiving the surrounding environment. The four-way surround-view system can include cameras with four perspectives: front-view, rear-view, left-view, and right-view. It can collect image data from these four perspectives as training image data. Label data corresponding to the training image data is obtained through manual annotation, semi-automatic annotation, or automatic annotation. The label data includes the labels of each object included in the training image data in the vehicle coordinate system (or a preset plane of the vehicle coordinate system), such as bounding box labels on the Z=0 plane of the vehicle coordinate system, semantic segmentation labels, object type labels, etc. Z represents the coordinate axis perpendicular to the XOY plane of the vehicle coordinate system, X represents the vertical axis (length direction of the vehicle), and Y represents the horizontal axis (width direction of the vehicle). The specific settings can be configured according to the actual model requirements. Using the neural network model training method of this disclosure (executed in a neural network model training device), training can be performed... Training image data is used to determine the training image features corresponding to each viewpoint. Random perturbations are added to the original camera extrinsics corresponding to each viewpoint to obtain new perturbations for each viewpoint. Then, based on the new camera extrinsics for each viewpoint, the training image features corresponding to each viewpoint are transformed to a preset plane of the vehicle coordinate system to obtain the first bird's-eye view features. Based on the first bird's-eye view features and label data, a pre-obtained first neural network model is trained to obtain the trained target neural network model. The target neural network model is deployed in the image processing device on the autonomous vehicle to process the bird's-eye view features determined from the images of the four viewpoints collected during the vehicle's operation to obtain the image processing results. Since the target neural network model is trained based on noisy bird's-eye view features, the model has a strong adaptability to distorted bird's-eye view features. Therefore, when applied to actual image processing, it can reduce the adverse effects of distortion caused by IPM transformation on the image processing results and improve the robustness and accuracy of the perception results.

[0040] It should be noted that the training method of the neural network model disclosed herein is not limited to the autonomous driving scenario of the above-mentioned four-way surround view system. It can also be applied to any scenario that requires obtaining image processing results based on bird's-eye view features, such as a vision system with six RGB phases, without any specific limitation.

[0041] Exemplary methods

[0042] Figure 2 This is a flowchart illustrating a training method for a neural network model provided in an exemplary embodiment of this disclosure. This embodiment can be applied to electronic devices, specifically servers, terminals, and other electronic devices. Figure 2 As shown, it includes the following steps:

[0043] Step 201: Obtain training image data and label data. The training image data includes image data from at least one viewpoint.

[0044] The training image data can be acquired based on a vehicle equipped with at least one camera, with each camera corresponding to a viewpoint. The label data can include the labels of each object included in the training image data in the vehicle coordinate system (or a preset plane of the vehicle coordinate system). The label data can be obtained through manual annotation, semi-automatic annotation, automatic annotation, etc., and there are no specific limitations.

[0045] Step 202: Based on the training image data, determine the training image features corresponding to each viewpoint.

[0046] The training image features can be obtained by extracting features from the training image data using a specific feature extraction method. The specific feature extraction method can be set according to actual needs; for example, it can be based on a pre-trained feature extraction network. The image data for each viewpoint can include one or more frames, and each frame can be associated with its corresponding training image features.

[0047] Step 203: Add random perturbations to the original camera extrinsic parameters corresponding to each viewpoint to obtain the perturbed new camera extrinsic parameters corresponding to each viewpoint. The original camera extrinsic parameters are the calibration extrinsic parameters of the camera coordinate system relative to the vehicle coordinate system.

[0048] The original camera extrinsic parameters can be obtained by calibrating the camera beforehand. The original camera extrinsic parameters may include translation and rotation matrices, and random disturbances may include at least one of roll, pitch, and yaw disturbances, which can be set according to actual needs.

[0049] In an optional embodiment, a numerical range for the random perturbation can be set, thereby determining the random perturbation value within the numerical range. For example, the numerical range for the roll perturbation is (-2°, 2°), and the specific numerical range can be set according to actual needs.

[0050] Steps 203 and 202 are not in any particular order.

[0051] Step 204: Based on the new camera extrinsic parameters corresponding to each viewpoint, the training image features corresponding to each viewpoint are transformed to a preset plane of the vehicle coordinate system to obtain the first bird's-eye view features. The preset plane is parallel to the ground plane.

[0052] The preset plane can be set according to actual needs, such as the Z=0 plane of the vehicle coordinate system or other planes, without any specific limitation. The first bird's-eye view features can be obtained by transforming the features of each training image to the preset plane and then fusing them.

[0053] Step 205: Based on the features and label data of the first bird's-eye view, train the pre-obtained first neural network model to obtain the trained target neural network model.

[0054] The first neural network model can be an initial model or an intermediate model that has been trained to a certain extent, such as a model that has been iteratively optimized based on some training image data. No specific limitation is imposed.

[0055] In one optional embodiment, the first neural network model can be a model for at least one of the tasks, including object detection, semantic segmentation, and object type prediction, and can be specifically set according to actual needs.

[0056] The neural network model training method provided in this embodiment applies perturbation to the extrinsic parameters of the calibration camera required for the conversion during the process of converting multi-view training image data to a bird's-eye view. This obtains noisy bird's-eye view features for training the neural network model, improving the model's adaptability to distorted bird's-eye view features. As a result, when applied to actual image processing, it can reduce the adverse effects of distortion caused by IPM transformation on image processing results and improve the robustness and accuracy of the model.

[0057] Figure 3 This is a flowchart illustrating a method for training a neural network model provided in another exemplary embodiment of this disclosure.

[0058] In an optional embodiment, step 203 may specifically include the following steps:

[0059] Step 2031: For any viewpoint, generate random data based on preset generation rules. The random data includes at least one of roll angle disturbance value, pitch angle disturbance value, and yaw angle disturbance value.

[0060] The preset generation rules can include the data range for generating random data, such as the data range that the randomly generated roll angle disturbance value needs to meet, the data range that the pitch angle disturbance value needs to meet, and the data range that the yaw angle disturbance value needs to meet. The specific rules can be set according to actual needs.

[0061] For example, the roll angle disturbance value Δroll needs to meet the data range of -2° < Δroll < 2°, the pitch angle disturbance value Δpitch needs to meet the data range of -2° < Δpitch < 2°, and the yaw angle disturbance value needs to meet the data range of -180° < Δyaw < 180°.

[0062] Step 2032: Determine the perturbation rotation matrix based on the random data.

[0063] The perturbation rotation matrix has the same dimension as the rotation matrix of the original camera extrinsic parameters, both being 3x3 matrices. The perturbation rotation matrix can be obtained based on the transformation relationship from Euler angles to rotation matrices. For example, the rotation matrices corresponding to each Euler angle (pitch, yaw, roll) can be combined in a preset order. For instance, roll rotates around the camera's z-axis, pitch rotates around the x-axis, and yaw rotates around the y-axis. The rotation matrix corresponding to △roll is represented as Rz, the rotation matrix corresponding to △pitch is represented as Rx, and the rotation matrix corresponding to △yaw is represented as Ry. Then, the final determined perturbation rotation matrix can be represented as △R = Rz * Ry * Rx. The specific combination order is not limited to this order and can be set according to actual needs. This disclosure does not impose any restrictions on this.

[0064] Step 2033: Based on the perturbation rotation matrix and the original camera extrinsic parameters, determine the new camera extrinsic parameters corresponding to this viewpoint.

[0065] The new camera extrinsic parameters can be achieved by applying the perturbation rotation matrix to the rotation matrix in the original camera extrinsic parameters. For example, if the original camera extrinsic parameters are represented as [R,T], where R represents the rotation matrix and T represents the translation matrix, then the corresponding new camera extrinsic parameters are [R*ΔR,T].

[0066] This embodiment obtains new camera extrinsic parameters by adding random perturbations to the original camera extrinsic parameters, which are then used for IPM transformation. This can effectively expand the sample size of bird's-eye view features used for model training, simulate the distortion caused by IPM transformation, and enable the model to learn noisy and distorted bird's-eye view features, thereby improving the model's adaptability to distorted bird's-eye view features.

[0067] In an optional embodiment, step 205, which involves training a pre-obtained first neural network model based on the first bird's-eye view features and label data to obtain a trained target neural network model, includes:

[0068] Step 2051: Transform the label data based on the perturbation rotation matrix to obtain the transformed new label data.

[0069] In this process, since the camera extrinsic parameters are perturbed during the IPM transformation to obtain the first bird's-eye view features, the sample is expanded. To ensure the accuracy of the labels, the label data needs to be transformed in the same way based on the perturbation rotation matrix to obtain the new label data. For example, if the yaw angle is rotated by 90 degrees during the process of obtaining the first bird's-eye view features, the label data is also rotated by 90 degrees in the yaw angle to obtain the new label data. This new label data is used as the label data corresponding to the first bird's-eye view features obtained based on the new camera extrinsic parameter transformation and is used for training the model.

[0070] Step 2052: Based on the features of the first bird's-eye view and the new label data, train the first neural network model to obtain the target neural network model.

[0071] The target neural network model can be obtained through multiple iterations of optimization training. For each iteration, the first neural network model is the neural network model updated in the previous iteration. The first neural network model is used to process the features of the first bird's-eye view to obtain the processing result. The processing result is compared with the corresponding new label data to determine the network loss, which is used to update the network parameters to obtain the updated network parameters. The updated network parameters determine the optimized neural network model, which is used as the first neural network model. The above process is then repeated for iterative optimization until the network loss meets the preset conditions or the number of iterations reaches the maximum threshold, at which point the training ends and the target neural network model is obtained.

[0072] This embodiment effectively simulates the deviation between the actual camera rotation matrix and the calibration rotation matrix by rotating the label data in the same way as the random perturbation of the IPM transformation, thus ensuring the accuracy of the results of the target neural network model in processing the bird's-eye view features.

[0073] Figure 4 This is a flowchart illustrating step 2052 provided in an exemplary embodiment of this disclosure.

[0074] In an optional embodiment, step 2052, which involves training the first neural network model based on the first bird's-eye view features and the new label data to obtain the target neural network model, includes:

[0075] Step 20521: Process the features of the first bird's-eye view using the first neural network model to obtain the processing result.

[0076] The specific content of the processing result is determined based on the task type of the first neural network model. For example, if the first neural network model is a single-task object detection model, the processing result includes the predicted bounding box information of each detected target object in a preset plane of the vehicle coordinate system. The predicted bounding box information describes the position of the predicted bounding box, and the specific description method is not limited. For example, it can include the coordinates and size of the center point of the predicted bounding box, or it can include the coordinates of the four corner points of the predicted bounding box. If the first neural network model is a semantic segmentation model, the processing result can include the point set corresponding to each target object in the preset plane of the vehicle coordinate system. If the first neural network model is a multi-task model, the processing result can include the results corresponding to each task.

[0077] Step 20522: Based on the processing results and the new label data, determine the network loss.

[0078] The network loss can be determined based on a preset loss function. The specific loss function can be set according to actual needs, and this disclosure does not limit it. For example, for semantic segmentation tasks, similarity loss function, intersection-union loss function, cross-entropy loss function, etc. can be used.

[0079] Step 20523: Adjust the network parameters of the first neural network model based on the network loss to obtain the adjusted network parameters.

[0080] The adjustment of network parameters can be based on any feasible adjustment rule, such as stochastic gradient descent. The specific rules can be set according to actual needs, and this disclosure does not impose any restrictions.

[0081] Step 20524: Determine the second neural network model based on the adjusted network parameters.

[0082] Step 20525: Using the second neural network model as the first neural network model, repeatedly execute the step of adding random perturbations to the original camera extrinsic parameters corresponding to each viewpoint to obtain the perturbated new camera extrinsic parameters corresponding to each viewpoint.

[0083] The specific steps for this operation are described above and will not be repeated here.

[0084] Step 20526: In response to the satisfaction of the training termination condition, the target neural network model is obtained.

[0085] The training termination condition can be set according to actual needs, such as network loss convergence, or the number of iterations reaching the maximum threshold, etc., without any specific limitations.

[0086] In one alternative embodiment, the target neural network model may include an extraction network for further feature extraction from the first bird's-eye view features and a head network for at least one task. The extraction network may employ any implementable network structure, such as a Convolutional Neural Network (CNN), which can be configured according to actual needs. The head network may include at least one of a head network for object detection, a head network for semantic segmentation, or a head network for object classification.

[0087] This embodiment obtains more bird's-eye view feature samples through random perturbation for model training, which can effectively improve the robustness of the model.

[0088] In an optional embodiment, step 204, based on the new camera extrinsic parameters corresponding to each viewpoint, transforms the training image features corresponding to each viewpoint to a preset plane of the vehicle coordinate system to obtain the first bird's-eye view features, including:

[0089] Step 2041: For any viewpoint, based on the camera intrinsic parameters corresponding to that viewpoint, transform the training image features corresponding to that viewpoint to the camera coordinate system corresponding to that viewpoint to obtain the first feature in the camera coordinate system corresponding to that viewpoint.

[0090] The camera intrinsic parameters are pre-calibrated intrinsic parameters. These parameters can include the camera's focal length and the coordinates of the camera's optical center in the pixel coordinate system. The camera intrinsic parameters can be represented by an intrinsic parameter matrix, which is used to transform image features to the camera coordinate system, thus obtaining the first feature.

[0091] Step 2042: Based on the new camera extrinsic parameters corresponding to the viewpoint, transform the first feature corresponding to the viewpoint to the vehicle coordinate system to obtain the second feature in the vehicle coordinate system corresponding to the viewpoint.

[0092] The camera coordinate system and the vehicle coordinate system have a certain rotation and translation relationship. The camera extrinsic parameters are used to describe this rotation and translation relationship. The first feature in the camera coordinate system can be rotated and translated accordingly and transformed into the vehicle coordinate system through the camera extrinsic parameters.

[0093] Step 2043: Fuse the second features corresponding to each viewpoint to obtain the fused features in the vehicle coordinate system.

[0094] Since the second features of each perspective are features under a unified vehicle coordinate system, feature matching and splicing can be performed based on the corresponding positions of the second features of each perspective under the vehicle coordinate system to achieve feature fusion of multiple perspectives. The specific fusion method can be set according to actual needs, and this disclosure does not limit it.

[0095] Step 2044: Project the fused features onto a preset plane to obtain the first bird's-eye view features.

[0096] For example, if the preset plane is the Z=0 plane, then the Z coordinates of the coordinate points corresponding to the fused features are all considered to be 0, thereby realizing the projection of the fused features onto the Z=0 plane.

[0097] In an optional embodiment, the transformation matrix from the camera coordinate system to the vehicle coordinate system of the i-th camera is represented as: K i This represents the camera intrinsic parameter matrix. The rotation matrix represents the camera extrinsic parameters (which may be new camera extrinsic parameters in this disclosure), img represents the image coordinate system (also known as the pixel coordinate system), cam represents the camera coordinate system, and vcs represents the vehicle coordinate system. The translation matrix or translation vector representing the camera's extrinsic parameters can be obtained through... The projection onto the preset plane is achieved. vcs_gnd represents the ground plane of the vehicle coordinate system (i.e., the Z=0 plane), and (0,1,3) represents the vectors taken from the 0th, 1st, and 3rd columns.

[0098] In an optional embodiment, a mapping relationship between a preset plane and a bird's-eye view (BEVMap) can also be set. After the fused features are projected onto the preset plane, they are then mapped onto the bird's-eye view through the mapping relationship to obtain the first bird's-eye view features.

[0099] For example, the mapping relationship from the Z=0 plane to the bird's-eye view is represented as follows: The mapping relationship from the pixel coordinate system to the bird's-eye view is expressed as follows:

[0100] This embodiment uses IPM conversion based on the perturbed new camera extrinsic parameters to obtain bird's-eye view features, in order to simulate the changes in roll, pitch, and yaw angles of the vehicle relative to the ground, thereby increasing the noise of the bird's-eye view features and achieving feature distortion, so as to improve the model's adaptability to distorted features.

[0101] In an optional embodiment, step 202, which determines the training image features corresponding to each viewpoint based on the training image data, includes:

[0102] Step 2021: Use the pre-trained feature extraction network to extract features from the training image data corresponding to each viewpoint, and obtain the training image features corresponding to each viewpoint.

[0103] The feature extraction network can adopt any feasible network structure, such as a convolutional neural network (CNN), which can be set according to actual needs.

[0104] In an optional embodiment, the feature extraction network can be trained in conjunction with a task prediction network. For example, it can be trained in a supervised manner using prediction networks for tasks such as object detection, semantic segmentation, and object classification. The specific setup can be determined based on actual needs.

[0105] This disclosure optimizes the model's tolerance to IPM distortion and enhances the quality of BEV features extracted by the model by enabling the model to learn and adapt to perturbations, distortions and environmental noise in bird's-eye view features, thereby improving the model's robustness in multi-task expansion.

[0106] The embodiments described above can be implemented individually or in any combination without conflict. The specific implementation can be set according to actual needs, and this disclosure does not limit them.

[0107] The training method for any neural network model provided in this disclosure can be executed by any suitable device with data processing capabilities, including but not limited to terminal devices and servers. Alternatively, the training method for any neural network model provided in this disclosure can be executed by a processor, such as by a processor executing the training method for any neural network model mentioned in this disclosure by calling corresponding instructions stored in memory. Further details will not be elaborated below.

[0108] Figure 5 This is a schematic flowchart of an image processing method provided in an exemplary embodiment of this disclosure. This embodiment can be applied to electronic devices, specifically, for example, in-vehicle computing platforms. Figure 5 As shown, it includes the following steps:

[0109] Step 301: Obtain image data to be processed, which includes images from at least one viewpoint.

[0110] The image data to be processed can be image data collected in real time during vehicle operation.

[0111] Step 302: Based on the image data to be processed, determine the first image features corresponding to each viewpoint.

[0112] The determination of the first image feature is similar to that of the training image features mentioned above, and will not be repeated here.

[0113] Step 303: Determine the target bird's-eye view features based on the first image features corresponding to each viewpoint.

[0114] Among them, the target bird's-eye view features can be obtained by performing IPM transformation on each first image feature based on the camera parameters corresponding to each viewpoint.

[0115] Step 304: Using the pre-trained target neural network model, process the target bird's-eye view features to obtain image processing results. The target neural network model is trained based on the training method of the neural network model provided in any of the above embodiments.

[0116] The specific training process of the target neural network model is described in the aforementioned embodiments and will not be repeated here. The image processing result is determined based on the specific task function of the target neural network model.

[0117] Any of the image processing methods provided in this disclosure can be executed by any suitable device with data processing capabilities, including but not limited to: terminal devices and servers. Alternatively, any of the image processing methods provided in this disclosure can be executed by a processor, such as by a processor executing any of the image processing methods mentioned in this disclosure by calling corresponding instructions stored in memory. Further details will not be elaborated below.

[0118] Exemplary device

[0119] Figure 6 This is a schematic diagram of the structure of a training apparatus for a neural network model provided in an exemplary embodiment of this disclosure. The apparatus of this embodiment can be used to implement the corresponding neural network model training method embodiments of this disclosure, such as... Figure 6 The device shown includes: a first acquisition module 501, a first processing module 502, a second processing module 503, a third processing module 504, and a fourth processing module 505.

[0120] The first acquisition module 501 is used to acquire training image data and label data, wherein the training image data includes image data from at least one viewpoint.

[0121] The first processing module 502 is used to determine the training image features corresponding to each viewpoint based on the training image data.

[0122] The second processing module 503 is used to add random perturbations to the original camera extrinsic parameters corresponding to each viewpoint to obtain the new perturbated camera extrinsic parameters corresponding to each viewpoint. The original camera extrinsic parameters are the calibration extrinsic parameters of the camera coordinate system relative to the vehicle coordinate system.

[0123] The third processing module 504 is used to transform the training image features corresponding to each viewpoint to a preset plane of the vehicle coordinate system based on the new camera extrinsic parameters corresponding to each viewpoint, so as to obtain the first bird's-eye view features. The preset plane is parallel to the ground plane.

[0124] The fourth processing module 505 is used to train the pre-obtained first neural network model based on the features and label data of the first bird's-eye view, so as to obtain the trained target neural network model.

[0125] Figure 7 This is a schematic diagram of the structure of a training apparatus for a neural network model provided in another exemplary embodiment of this disclosure.

[0126] In an optional example, the second processing module 503 includes: a generation unit 5031, a first determination unit 5032, and a second determination unit 5033.

[0127] The generation unit 5031 is used to generate random data for any viewpoint based on a preset generation rule. The random data includes at least one of roll angle disturbance value, pitch angle disturbance value and yaw angle disturbance value.

[0128] The first determining unit 5032 is used to determine the perturbation rotation matrix based on random data.

[0129] The second determining unit 5033 is used to determine the new camera extrinsic parameters corresponding to the viewpoint based on the perturbation rotation matrix and the original camera extrinsic parameters.

[0130] In an optional embodiment, the fourth processing module 505 includes: a first processing unit 5051 and a second processing unit 5052.

[0131] The first processing unit 5051 is used to transform the label data based on the perturbation rotation matrix to obtain the transformed new label data.

[0132] The second processing unit 5052 is used to train the first neural network model based on the features of the first bird's-eye view and the new label data to obtain the target neural network model.

[0133] In an optional embodiment, the second processing unit 5052 is specifically used for:

[0134] The first neural network model is used to process the features of the first bird's-eye view to obtain the processing result; based on the processing result and the new label data, the network loss is determined; based on the network loss, the network parameters of the first neural network model are adjusted to obtain the adjusted network parameters; based on the adjusted network parameters, the second neural network model is determined; the second neural network model is used as the first neural network model, and the steps of adding random perturbations to the original camera extrinsic parameters corresponding to each viewpoint are repeated to obtain the perturbated new camera extrinsic parameters corresponding to each viewpoint; in response to the satisfaction of the training termination condition, the target neural network model is obtained.

[0135] In an optional embodiment, the third processing module 504 includes: a first conversion unit 5041, a second conversion unit 5042, and a feature fusion unit 5043.

[0136] The first transformation unit 5041 is used to transform the training image features corresponding to any viewpoint to the camera coordinate system corresponding to that viewpoint based on the camera intrinsic parameters corresponding to that viewpoint, so as to obtain the first feature in the camera coordinate system corresponding to that viewpoint.

[0137] The second conversion unit 5042 is used to convert the first feature corresponding to the viewpoint to the vehicle coordinate system based on the new camera extrinsic parameters corresponding to the viewpoint, so as to obtain the second feature in the vehicle coordinate system corresponding to the viewpoint.

[0138] The feature fusion unit 5043 is used to fuse the second features corresponding to each viewpoint to obtain the fused features in the vehicle coordinate system.

[0139] Projection unit 5044 is used to project the fused features onto a preset plane to obtain the first bird's-eye view features.

[0140] In an optional embodiment, the first processing module 502 includes:

[0141] The feature extraction unit 5021 is used to extract features from the training image data corresponding to each viewpoint using a pre-trained feature extraction network, so as to obtain the training image features corresponding to each viewpoint.

[0142] Figure 8 This is a schematic diagram of the structure of an image processing apparatus provided in an exemplary embodiment of the present disclosure. The apparatus of this embodiment can be used to implement the corresponding image processing method embodiments disclosed herein, such as… Figure 8 The device shown includes: a second acquisition module 601, a fifth processing module 602, a sixth processing module 603, and a seventh processing module 604.

[0143] The second acquisition module 601 is used to acquire image data to be processed, which includes images from at least one viewpoint.

[0144] The fifth processing module 602 is used to determine the first image features corresponding to each viewpoint based on the image data to be processed.

[0145] The sixth processing module 603 is used to determine the target bird's-eye view features based on the first image features corresponding to each viewpoint.

[0146] The seventh processing module 604 is used to process the features of the target bird's-eye view using a pre-trained target neural network model to obtain image processing results. The target neural network model is trained based on the training method of the neural network model provided in any of the above embodiments.

[0147] For the specific operation of each module and unit in the various device embodiments of this disclosure, please refer to the corresponding method embodiments described above.

[0148] In practical applications, each module and unit can be divided into more granular categories according to actual needs, or in other ways, without any specific restrictions.

[0149] Exemplary electronic devices

[0150] This disclosure also provides an electronic device, including: a memory for storing computer programs;

[0151] A processor is configured to execute a computer program stored in a memory, wherein when the computer program is executed, it implements a training method for a neural network model as described in any of the above embodiments of the present disclosure, or implements an image processing method as described in any of the above embodiments of the present disclosure.

[0152] Figure 9 This is a schematic diagram of an application embodiment of the electronic device disclosed herein. In this embodiment, the electronic device 10 includes one or more processors 11 and a memory 12.

[0153] The processor 11 may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 10 to perform desired functions.

[0154] The memory 12 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 11 may execute the program instructions to implement the methods of the various embodiments of this disclosure described above and / or other desired functions. Various contents such as input signals, signal components, and noise components may also be stored in the computer-readable storage medium.

[0155] In one example, the electronic device 10 may also include an input device 13 and an output device 14, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0156] For example, the input device 13 may be the microphone or microphone array described above, used to capture the input signal of the sound source.

[0157] In addition, the input device 13 may also include, for example, a keyboard, a mouse, etc.

[0158] The output device 14 can output various information to the outside, including determined distance information, direction information, etc. The output device 14 may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0159] Of course, for the sake of simplicity, Figure 9 Only some of the components of the electronic device 10 relevant to this disclosure are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device 10 may include any other suitable components depending on the specific application.

[0160] Exemplary computer program products and computer-readable storage media

[0161] In addition to the methods and apparatus described above, embodiments of this disclosure may also be computer program products comprising computer program instructions that, when executed by a processor, cause the processor to perform the steps of the methods according to various embodiments of this disclosure as described in the "Exemplary Methods" section of this specification.

[0162] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this disclosure. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's computing device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0163] Furthermore, embodiments of this disclosure may also be computer-readable storage media having computer program instructions stored thereon, which, when executed by a processor, cause the processor to perform the steps in the methods according to various embodiments of this disclosure described in the "Exemplary Methods" section above.

[0164] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0165] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0166] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For system embodiments, since they largely correspond to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0167] The block diagrams of devices, apparatuses, devices, and systems disclosed herein are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0168] The methods and apparatus of this disclosure may be implemented in many ways. For example, they may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above-described order of steps for the methods is for illustrative purposes only, and the steps of the methods of this disclosure are not limited to the order specifically described above unless otherwise specifically stated. Furthermore, in some embodiments, this disclosure may also be implemented as a program recorded on a recording medium, the program including machine-readable instructions for implementing the methods according to this disclosure. Thus, this disclosure also covers recording media storing programs for performing the methods according to this disclosure.

[0169] It should also be noted that in the apparatus, devices, and methods of this disclosure, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions to this disclosure.

[0170] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.

[0171] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. A method for training a neural network model, comprising: Acquire training image data and label data, wherein the training image data includes image data from at least one viewpoint; Based on the training image data, determine the training image features corresponding to each of the aforementioned viewpoints; Random perturbations are added to the original camera extrinsic parameters corresponding to each of the aforementioned viewpoints to obtain new perturbation camera extrinsic parameters corresponding to each of the aforementioned viewpoints. The original camera extrinsic parameters are the calibration extrinsic parameters of the camera coordinate system relative to the vehicle coordinate system. The random perturbations include at least one of roll angle perturbation, pitch angle perturbation, and yaw angle perturbation. Based on the new camera extrinsic parameters corresponding to each of the aforementioned viewpoints, the training image features corresponding to each of the aforementioned viewpoints are transformed to a preset plane of the vehicle coordinate system to obtain the first bird's-eye view features, wherein the preset plane is parallel to the ground plane; Based on the features of the first bird's-eye view and the label data, the pre-obtained first neural network model is trained to obtain the trained target neural network model. The step of adding random perturbations to the original camera extrinsic parameters corresponding to each of the aforementioned viewpoints to obtain new perturbed camera extrinsic parameters corresponding to each of the aforementioned viewpoints includes: For any of the aforementioned viewpoints, random data is generated based on a preset generation rule. The random data includes at least one of roll angle disturbance value, pitch angle disturbance value, and yaw angle disturbance value. Based on the random data, determine the perturbation rotation matrix; Based on the perturbation rotation matrix and the original camera extrinsic parameters, the new camera extrinsic parameters corresponding to the viewpoint are determined.

2. The method according to claim 1, wherein, The step of training a pre-obtained first neural network model based on the first bird's-eye view features and the label data to obtain a trained target neural network model includes: The label data is transformed based on the perturbation rotation matrix to obtain new label data after transformation; Based on the features of the first bird's-eye view and the new label data, the first neural network model is trained to obtain the target neural network model.

3. The method according to claim 2, wherein, The step of training the first neural network model based on the first bird's-eye view features and the new label data to obtain the target neural network model includes: The first neural network model is used to process the features of the first bird's-eye view to obtain the processing result; Based on the processing results and the new label data, determine the network loss; The network parameters of the first neural network model are adjusted based on the network loss to obtain the adjusted network parameters; Based on the adjusted network parameters, a second neural network model is determined; Using the second neural network model as the first neural network model, the step of adding random perturbations to the original camera extrinsic parameters corresponding to each of the aforementioned viewpoints to obtain the new perturbated camera extrinsic parameters corresponding to each of the aforementioned viewpoints is repeated. In response to the satisfaction of the training termination condition, the target neural network model is obtained.

4. The method according to claim 1, wherein, The step of transforming the training image features corresponding to each of the aforementioned viewpoints to a preset plane of the vehicle coordinate system based on the new camera extrinsic parameters corresponding to each viewpoint to obtain the first bird's-eye view features includes: For any of the aforementioned viewpoints, based on the camera intrinsic parameters corresponding to that viewpoint, the training image features corresponding to that viewpoint are transformed to the camera coordinate system corresponding to that viewpoint to obtain the first feature in the camera coordinate system corresponding to that viewpoint. Based on the new camera extrinsic parameters corresponding to the viewpoint, the first feature corresponding to the viewpoint is transformed into the vehicle coordinate system to obtain the second feature in the vehicle coordinate system corresponding to the viewpoint. The second features corresponding to each of the aforementioned viewpoints are fused to obtain the fused features in the vehicle coordinate system. The fused features are projected onto the preset plane to obtain the first bird's-eye view features.

5. The method according to claim 1, wherein, The step of determining the training image features corresponding to each of the aforementioned viewpoints based on the training image data includes: The pre-trained feature extraction network is used to extract features from the training image data corresponding to each of the aforementioned viewpoints, thereby obtaining the training image features corresponding to each of the aforementioned viewpoints.

6. An image processing method, comprising: Acquire image data to be processed, the image data to be processed including images from at least one viewpoint; Based on the image data to be processed, determine the first image features corresponding to each of the aforementioned viewpoints; Based on the first image features corresponding to each of the aforementioned viewpoints, the bird's-eye view features of the target are determined. The target bird's-eye view features are processed using a pre-trained target neural network model to obtain image processing results. The target neural network model is trained based on the training method of any one of the neural network models described in claims 1-5.

7. A training device for a neural network model, comprising: The first acquisition module is used to acquire training image data and label data, wherein the training image data includes image data from at least one viewpoint; The first processing module is used to determine the training image features corresponding to each of the aforementioned viewpoints based on the training image data. The second processing module is used to add random perturbations to the original camera extrinsic parameters corresponding to each of the aforementioned viewpoints to obtain new perturbated camera extrinsic parameters corresponding to each of the aforementioned viewpoints. The original camera extrinsic parameters are the calibration extrinsic parameters of the camera coordinate system relative to the vehicle coordinate system. The random perturbations include at least one of roll angle perturbation, pitch angle perturbation, and yaw angle perturbation. The third processing module is used to transform the training image features corresponding to each of the aforementioned viewpoints to a preset plane of the vehicle coordinate system based on the new camera extrinsic parameters corresponding to each of the aforementioned viewpoints, thereby obtaining the first bird's-eye view features. The preset plane is parallel to the ground plane. The fourth processing module is used to train the pre-obtained first neural network model based on the features of the first bird's-eye view and the label data to obtain the trained target neural network model. The second processing module includes: The generation unit is used to generate random data for any of the aforementioned viewpoints based on a preset generation rule. The random data includes at least one of roll angle disturbance value, pitch angle disturbance value, and yaw angle disturbance value. The first determining unit is used to determine the perturbation rotation matrix based on the random data; The second determining unit is used to determine the new camera extrinsic parameters corresponding to the viewpoint based on the perturbation rotation matrix and the original camera extrinsic parameters.

8. An image processing apparatus, comprising: The second acquisition module is used to acquire image data to be processed, the image data to be processed including images from at least one viewpoint; The fifth processing module is used to determine the first image features corresponding to each of the viewpoints based on the image data to be processed; The sixth processing module is used to determine the target bird's-eye view features based on the first image features corresponding to each of the aforementioned viewpoints; The seventh processing module is used to process the features of the target bird's-eye view using a pre-trained target neural network model to obtain image processing results. The target neural network model is trained based on the training method of any one of the neural network models described in claims 1-5.

9. A computer-readable storage medium storing a computer program for performing a training method for a neural network model according to any one of claims 1-5; or, the computer program for performing an image processing method according to claim 6.

10. An electronic device, the electronic device comprising: processor; Memory used to store the processor's executable instructions; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the training method of the neural network model according to any one of claims 1-5; or, execute the instructions to implement the image processing method according to claim 6.

Images