Model training methods, devices, vehicles, storage media, and computer program products

Abstract

This disclosure relates to a model training method, apparatus, vehicle, storage medium, and computer program product, belonging to the field of image processing technology. The method includes iteratively executing the following steps to obtain a target network model: inputting a collected image of a storage location into an initial network model to predict the predicted three-dimensional coordinates of a target point in the storage location; converting the predicted three-dimensional coordinates into predicted two-dimensional coordinates; updating the network parameters in the initial network model based on the loss value between the predicted two-dimensional coordinates and the actual two-dimensional coordinates; the actual two-dimensional coordinates are obtained by labeling the target point in the storage location image; if the loss value meets the convergence condition, the initial network model that meets the convergence condition is used as the target network model; the target network model is used to obtain the three-dimensional coordinates of the target point from the storage location image. Using the model training method proposed in this disclosure, the cost of manual labeling can be reduced while obtaining highly accurate predicted three-dimensional coordinates.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and more particularly to a model training method, apparatus, vehicle, storage medium, and computer program product. Background Technology

[0002] With rapid urbanization and increasing vehicle ownership, the demand for parking space is also rising. To efficiently utilize urban space, mechanical parking spaces are becoming increasingly common. A mechanical parking space is a mobile, non-independent space that uses a platform to lift or slide vehicles for parking. Mechanical parking spaces typically have two levels: the lower level allows direct vehicle access, while the upper level is mechanically lowered to the ground for user convenience. Once a user drives their vehicle into the mechanical parking space, the system automatically moves it to a suitable location.

[0003] In the automatic parking system, the vehicle simulates a 3D model of the mechanical parking space and performs automatic parking based on the 3D model of the mechanical parking space. However, the 3D model of the mechanical parking space is relatively complicated to annotate, and its annotation efficiency is low. Summary of the Invention

[0004] To overcome the problems existing in related technologies, this disclosure provides a model training method, apparatus, vehicle, storage medium, and computer program product.

[0005] According to a first aspect of the present disclosure, a model training method is provided, the method comprising iteratively performing the following steps to obtain a target network model: The collected images of the storage location are input into the initial network model to predict the predicted three-dimensional coordinates of the target point in the storage location. Convert the predicted three-dimensional coordinates into predicted two-dimensional coordinates; The network parameters in the initial network model are updated based on the loss value between the predicted two-dimensional coordinates and the actual two-dimensional coordinates; the actual two-dimensional coordinates are obtained by labeling the target points in the storage location image. If the loss value satisfies the convergence condition, the initial network model that satisfies the convergence condition is used as the target network model; the target network model is used to obtain the three-dimensional coordinates of the target point based on the storage location image.

[0006] Optionally, the initial network model includes a backbone network, a cross-attention mechanism, and a regression head; the step of inputting the collected reservoir location image into the initial network model to predict the predicted three-dimensional coordinates of the target point in the reservoir location includes: The images of the storage locations are processed through the backbone network to extract features, resulting in multi-scale features. The multi-scale features are input into the cross-attention mechanism to obtain the location features of the storage location; The reservoir location features are input into the regression head to obtain the predicted three-dimensional coordinates of the target point.

[0007] Optionally, the storage location image includes adjacent first storage location images and second storage location images; the step of inputting the multi-scale features into the cross-attention mechanism to obtain the storage location features of the storage location includes: The multi-scale features of the second storage location image, the predicted three-dimensional coordinates of the target point in the first storage location image, and the storage location features of the first storage location image are input into the cross-attention mechanism to obtain the storage location features corresponding to the second storage location image.

[0008] Optionally, the predicted three-dimensional coordinates of the target point in the first storage location image are input into the cross-attention mechanism, including: The predicted 3D coordinates of the target point in the first storage location image are updated according to the iteration step size; The updated predicted 3D coordinates are input into the cross-attention mechanism.

[0009] Optionally, the predicted three-dimensional coordinates of the target point in the first storage location image are input into the cross-attention mechanism, including: The predicted three-dimensional coordinates of the target point in the first storage location image are transformed from the first vehicle coordinate system to the second vehicle coordinate system to obtain the predicted three-dimensional coordinates of the target point located in the second vehicle coordinate system; the first vehicle coordinate system is the vehicle coordinate system in which the vehicle collects the first storage location image, and the second vehicle coordinate system is the vehicle coordinate system in which the vehicle collects the second storage location image.

[0010] Optionally, the actual two-dimensional coordinates of the target point are marked on the storage location image.

[0011] Optionally, the target location includes at least one of the following: a foot point in the storage location, any point on the line connecting the baffles on both sides of the storage location and the ground of the storage location.

[0012] Optionally, the storage location image includes a panoramic view and a circumferential view.

[0013] According to a second aspect of the present disclosure, a model training apparatus is provided, the model training apparatus comprising iteratively performing the following steps to obtain a target network model: The input module is configured to input the collected images of the storage location into the initial network model to predict the predicted three-dimensional coordinates of the target point in the storage location; The conversion module is configured to convert the predicted three-dimensional coordinates into predicted two-dimensional coordinates; The update module is configured to update the network parameters in the initial network model based on the loss value between the predicted two-dimensional coordinates and the actual two-dimensional coordinates; the actual two-dimensional coordinates are obtained by labeling the target points in the storage location image; The determination module is configured to, if the network parameters of the initial network model satisfy the convergence condition, use the initial network model that satisfies the convergence condition as the target network model; the target network model is used to obtain the three-dimensional coordinates of the target point based on the storage location image.

[0014] According to a third aspect of the present disclosure, a vehicle is provided, comprising: processor; Memory used to store processor-executable instructions; The processor is configured to perform the steps of the model training method provided in the first aspect of the present disclosure.

[0015] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, having stored thereon computer program instructions that, when executed by a processor, implement the steps of the model training method provided in the first aspect of the present disclosure.

[0016] According to a fifth aspect of the present disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of the model training method provided in the first aspect of the present disclosure.

[0017] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects: On the one hand, because the acquired warehouse location images themselves are accurate to a small margin of error, the accuracy of the 3D coordinates of the target points obtained based on the warehouse location images is also small. When automatically parking vehicles based on the 3D coordinates of the target points in the warehouse location with this small error accuracy, the vehicles can be automatically parked in the warehouse location accurately. On the other hand, because the annotation process only annotates the target points in the warehouse location images, the amount of annotation data is less compared to annotating a large amount of point cloud data. Moreover, it is a two-dimensional annotation on the warehouse location images, which only annotates the pixel coordinates of the target points. It only needs to annotate the rows and columns of the target points on the warehouse location images, without needing to rotate the target points for annotation. Therefore, the annotation is relatively simple, which can greatly reduce the annotation cost and improve the annotation efficiency.

[0018] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0019] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0020] Figure 1 This is a flowchart illustrating the steps of a model training method according to an exemplary embodiment.

[0021] Figure 2 This is a schematic diagram illustrating a mechanical storage location according to an exemplary embodiment.

[0022] Figure 3 This is a schematic diagram illustrating four foot points in a mechanical storage location according to an exemplary embodiment.

[0023] Figure 4 This is a schematic diagram illustrating four arbitrary points in a mechanical storage location according to an exemplary embodiment.

[0024] Figure 5 This is a schematic diagram illustrating a vehicle coordinate system according to an exemplary embodiment.

[0025] Figure 6 This is a schematic diagram illustrating a camera coordinate system according to an exemplary embodiment.

[0026] Figure 7 This is a schematic diagram illustrating an image coordinate system according to an exemplary embodiment.

[0027] Figure 8 This is a schematic diagram illustrating an initial network model according to an exemplary embodiment.

[0028] Figure 9 This is a flowchart illustrating the steps of a model training method according to an exemplary embodiment.

[0029] Figure 10 This is a block diagram illustrating a model training apparatus according to an exemplary embodiment.

[0030] Figure 11 This is a block diagram illustrating a vehicle according to an exemplary embodiment.

[0031] Figure 12 This is a block diagram illustrating a chip system according to an exemplary embodiment.

[0032] Figure 13 This is a block diagram illustrating a model training apparatus according to an exemplary embodiment. Detailed Implementation

[0033] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0034] It should be noted that all actions involving the acquisition of signals, information, or data in this disclosure are carried out in compliance with the relevant data protection laws and policies of the country where the location is situated, and with authorization from the owner of the relevant device.

[0035] Figure 1 This is a flowchart illustrating a model training method according to an exemplary embodiment, such as... Figure 1 As shown, iteratively execute steps S10 to S30 until the target network model that satisfies the convergence condition is obtained: In step S10, the collected images of the storage location are input into the initial network model to predict the predicted three-dimensional coordinates of the target point in the storage location.

[0036] Parking spaces include mechanical parking spaces, which can move vehicles parked in the space vertically and horizontally to a suitable position, thereby making full use of parking space and accommodating more vehicles. Please refer to [link / reference]. Figure 2 As shown, the mechanical storage space includes the storage space floor and the baffles mounted on both sides of the storage space floor. The storage space floor is used to support parked vehicles, and the baffles on both sides are used to restrict the vehicles within the area of ​​the storage space floor.

[0037] The storage location image is a mechanical storage location image captured by the vehicle. This image includes both circumferential and surround-view images. The circumferential image is captured by circumferential cameras mounted on the vehicle in all four directions (front, rear, left, and right), providing a comprehensive view of the vehicle's surroundings. The surround-view image is captured by surround-view cameras mounted on the vehicle in all four directions, as well as on other key locations such as the front and rear bumpers and rearview mirrors, enabling panoramic monitoring of the vehicle's surroundings. The difference between surround view images and perimeter view images is as follows: Surround view images are fisheye images. Because surround view cameras have a smaller field of view (FOV), they capture images of storage spaces that are relatively close to the vehicle, and are used to observe the storage spaces that are close to the vehicle. On the other hand, perimeter view cameras are generally pinhole cameras. Because perimeter view cameras have a larger field of view, they can capture images of storage spaces that are farther away, and are used to observe the storage spaces that are farther away from the vehicle.

[0038] The initial network model is used to predict the predicted three-dimensional coordinates (3D coordinates) of the target point in the storage location based on the collected storage location images.

[0039] The predicted 3D coordinates of the target location are used to indicate the 3D model of the entire storage site.

[0040] For example, please see Figure 3 As shown, Figure 3 The four black dots in the diagram represent four foot points. The target location includes the foot points in the storage area. These foot points are the beginning and end points on the connecting line between the baffles on both sides of the storage area and the storage area floor. There are at least four foot points. By using the three-dimensional coordinates of at least four foot points, the complete storage area floor can be simulated.

[0041] For example, please refer to Figure 4 As shown, the target points include any point on the line connecting the baffles on both sides of the storage location and the storage location ground. By using at least four arbitrary points, the complete storage location ground can also be simulated.

[0042] Understandably, if the vehicle can simulate the footholds in the storage space using the storage space image, then the complete storage space ground is simulated using the three-dimensional coordinates of the footholds; if the vehicle cannot simulate the footholds in the storage space using the storage space image, then the complete storage space ground is simulated by obtaining the three-dimensional coordinates of any point on the line connecting the baffles on both sides of the storage space and the storage space ground.

[0043] In step S20, the predicted three-dimensional coordinates are converted into predicted two-dimensional coordinates.

[0044] The predicted 3D coordinates can be projected into predicted 2D coordinates through 3D reprojection. For example, the predicted 3D coordinates in the vehicle coordinate system can be converted into predicted 3D coordinates in the camera coordinate system through the camera extrinsic parameters; then the predicted 3D coordinates in the camera coordinate system can be converted into predicted 2D coordinates in the image coordinate system through the camera's intrinsic parameter matrix.

[0045] Please refer to Figure 5 As shown, the vehicle coordinate system is a coordinate system constructed with the vehicle's center of gravity as the origin, the front of the vehicle as the x-axis, the right side of the vehicle as the y-axis, and the vertical direction of the vehicle as the z-axis. The vehicle's position changes dynamically as the vehicle moves, so the origin of the vehicle coordinate system also changes dynamically, which in turn causes the vehicle coordinate system to change dynamically as well.

[0046] Please refer to Figure 6 As shown, the camera coordinate system is constructed with the camera itself as the origin, the right side of the camera as the X-axis, the vertically downward side of the camera as the Y-axis, and the front of the camera as the Z-axis. Similarly, since the camera is located on the vehicle, when the vehicle's position changes, the camera's position also changes, causing the camera coordinate system to change dynamically.

[0047] Please refer to Figure 7 As shown, the image coordinate system defines the pixel coordinates of each pixel in the image. These pixel coordinates are predicted two-dimensional coordinates. The image coordinate system is usually constructed with the top left corner of the image as the origin, the horizontal direction to the right as the x-axis, and the vertical direction downward as the y-axis. The coordinates of a pixel in the image are usually represented by rows and columns.

[0048] Among them, camera extrinsic parameters are used to describe the position and orientation of the camera in the world coordinate system. They include the camera's position and rotation angle in the world coordinate system. Camera extrinsic parameters usually involve rotation matrix and translation vector. The rotation matrix determines the rotation angle of the camera in the world coordinate system, and the translation vector determines the position of the camera in the world coordinate system.

[0049] Camera intrinsic parameters refer to the inherent characteristics of the camera, including parameters such as focal length, principal point, image size, pixel size, and distortion coefficients. Camera intrinsic parameters are typically represented by a 3x3 camera matrix, which includes parameters such as focal length, principal point, image size, pixel size, and distortion coefficients.

[0050] In step S30, the network parameters in the initial network model are updated based on the loss value between the predicted two-dimensional coordinates and the actual two-dimensional coordinates.

[0051] The predicted two-dimensional coordinates are obtained by transforming the predicted three-dimensional coordinates obtained from the initial network model. These predicted two-dimensional coordinates are the pixel coordinates of the target point in the predicted storage location image.

[0052] The actual two-dimensional coordinates are obtained by labeling the target point in the storage location image. These actual two-dimensional coordinates are the actual pixel coordinates of the target point in the storage location image, and they can mark the row and column of the target point in the storage location image.

[0053] The loss value is the difference between the predicted two-dimensional coordinates and the actual two-dimensional coordinates. The larger the difference between the predicted two-dimensional coordinates and the actual two-dimensional coordinates, the less accurate the predicted two-dimensional coordinates obtained by the initial network model are.

[0054] The network parameters of the initial network model can be the weights, etc. This initial network model includes the backbone network, cross-attention mechanism, and regression head, etc. Therefore, the network parameters can include the network parameters of each network layer in the backbone network, cross-attention mechanism, and regression head.

[0055] During the update of the initial network model, if the loss value between the predicted 2D coordinates corresponding to the predicted 3D coordinates output by the initial network model and the actual 2D coordinates does not meet the convergence condition, the input storage location image and the network parameters of the initial network model are updated. The updated storage location image is then re-inputted into the updated initial network model to predict the predicted 3D coordinates of the target point in the storage location. The predicted 3D coordinates are then converted back to predicted 2D coordinates. The network parameters in the initial network model are then updated again using the loss value between the updated predicted 2D coordinates and the actual 2D coordinates, resulting in an updated initial network model. Through continuous updates, the predicted 2D coordinates corresponding to the predicted 3D coordinates output by the updated initial network model continuously approximate the actual 2D coordinates. When the loss value between the predicted 2D coordinates and the actual 2D coordinates meets the convergence condition, it indicates that the initial network model has been successfully trained. Since the predicted 2D coordinates output by the initial network model are relatively close to the actual 2D coordinates, the initial network model can be used as the target network model.

[0056] In step S40, if the network parameters of the initial network model satisfy the convergence condition, the initial network model that satisfies the convergence condition is taken as the target network model.

[0057] The target network model is used to obtain the accurate three-dimensional coordinates of target points in the input warehouse location image. The three-dimensional coordinates of the target points can be used to construct a three-dimensional model of the mechanical warehouse location, for example, by connecting two adjacent target points to obtain a three-dimensional model of the mechanical warehouse location.

[0058] Convergence conditions include any one of the following: The loss value between the predicted 2D coordinates and the actual 2D coordinates no longer decreases, the number of iterations of the initial training model is reached, and the loss value between the predicted 2D coordinates and the actual 2D coordinates is less than the preset value.

[0059] Currently, the following two methods can be used to detect warehouse location information: Option A involves using LiDAR to collect point cloud data of the storage location, and then labeling the point cloud data onto the vehicle coordinate system to obtain the three-dimensional coordinates of the storage location in the vehicle coordinate system.

[0060] On the one hand, although the point cloud data collected by LiDAR can represent the position of certain points in the parking space, there is an accuracy error of at least 10cm between the point cloud data collected by LiDAR and the actual position of the points. When automatically parking a vehicle into a mechanical parking space, the required parking accuracy is usually less than 3cm, that is, the error between the distance between the vehicle tires and the side panels of the parking space and the actual distance is less than 3cm. After the collected point cloud data is marked on the vehicle coordinate system to form the three-dimensional coordinates of the parking space, the 10cm error between the three-dimensional coordinates of the parking space and the actual coordinates of the parking space makes it impossible to identify the accurate position of the parking space. When parking, the vehicle is also prone to colliding with the side panels on the ground of the parking space, making it difficult to park in the parking space.

[0061] On the other hand, when labeling point cloud data into the vehicle coordinate system, each point cloud data needs to be rotated to three different angles (x, y, z) in the vehicle coordinate system to label the three coordinates separately, which is quite troublesome.

[0062] Thirdly, after obtaining the point cloud data, it is necessary to fuse it with the image data to obtain the storage location information. However, the point cloud data is collected by LiDAR, while the image data is collected by a camera. Since they are different sensor devices, the storage location images collected by the two are very likely not at the same time, and there will be errors in the fused storage location.

[0063] Option B is a 2D sensing method based on the assumption that the near-ground surface is flat. In sensing the storage location, it typically assumes the ground at the storage location is flat, i.e., assumes Z is 0, and then monitors the X and Y coordinates of the storage location.

[0064] However, the ground where vehicles are located is usually higher in the middle and lower on both sides, which will cause distortion in the collected images of the parking spaces, resulting in inaccurate X and Y coordinates. Similarly, when the parking space images are collected from the ground, there is usually a certain height difference between the parking space ground and the ground where the vehicle is located. The parking space ground of the mechanical parking space is higher than the ground where the vehicle is located, which will also cause distortion in the collected images of the parking spaces, resulting in inaccurate X and Y coordinates. The parking space position obtained based on inaccurate X, Y, and Z coordinates is also incorrect. Therefore, it is also difficult to automatically park the vehicle in the mechanical parking space based on the incorrect parking space position.

[0065] Compared to Solution A, the above technical solution offers several advantages. First, since the acquired warehouse location image has an accuracy of 1cm, the error of the three-dimensional coordinates of the target point obtained from the warehouse location image can also be accurate to 1cm. This error accuracy is less than 3cm. When automatically parking vehicles based on the three-dimensional coordinates of the target point in the warehouse location with an error accuracy of less than 3cm, the vehicle can be accurately parked in the warehouse location. Second, since the annotation process involves annotating the target point in the warehouse location image, the amount of annotation data is less than that required for annotating a large amount of point cloud data. Furthermore, the annotation is performed on the warehouse location image in two dimensions, annotating the pixel coordinates of the target point. It only requires annotating the rows and columns of the target point in the warehouse location image, without needing to rotate the target point for annotation, making the annotation process simpler. Third, it can also avoid errors caused by inconsistent external parameters of non-homogeneous sensors. Since this disclosure uses image data, specifically warehouse location images acquired by cameras, there will be no errors caused by non-homogeneous sensors.

[0066] Compared to Solution B, this solution inputs the storage location image into the initial network model to obtain predicted 3D coordinates, converts the predicted 3D coordinates into predicted 2D coordinates, and continuously updates the initial network model based on the loss value between the predicted 2D coordinates and the actual 2D coordinates. In this way, the error between the 2D coordinates converted from the 3D coordinates output by the target network model based on the storage location image and the actual 2D coordinates will be smaller. That is, this solution only uses the initial network model as the target network model to predict the 3D coordinates of subsequent target points after it is confirmed that the initial network model can output relatively accurate predicted 3D coordinates, resulting in more accurate 3D coordinates of the target points.

[0067] Furthermore, this disclosure employs a sparse Transformer (deep learning model) perception technique. In current Transformer perception techniques, after acquiring a 2D image, the image is converted to a BEV (bird's-eye view) space, which is then divided into multiple grids. The features in the 2D image are projected onto each grid in the BEV space, resulting in significant computational overhead. In contrast, the Transformer perception technique used in this disclosure, after acquiring a 2D image, simply inputs the multi-scale features of the storage location image into a cross-attention mechanism to obtain the storage location features. Based on these features, the predicted 3D coordinates of the target point are obtained. Essentially, this disclosure calculates the predicted 3D coordinates of the target point, without needing to convert the multi-scale features to BEV space to calculate the 3D coordinates of all points in the storage location image, thus greatly reducing computational overhead.

[0068] Figure 8 and 9 This is an exemplary embodiment illustrating how, in each iteration, an initial network model obtains the predicted 3D coordinates of a target point based on an input image of the reservoir location, comprising the following steps: In step S11, the location image is processed by the backbone network to extract features and obtain multi-scale features.

[0069] The backbone network is used to extract features at multiple scales from the image, obtaining multi-scale features at different resolutions. Multi-scale feature extraction includes downsampling feature extraction and upsampling feature extraction.

[0070] For example, taking a 10x10 resolution image of a storage location as an example, through downsampling feature extraction, the storage location image can be processed into images at different resolutions such as 8x8, 6x6, and 4x4, with the size and resolution of these images gradually decreasing. Through upsampling feature extraction, the storage location image can be processed into images at different resolutions such as 12x12, 14x14, and 16x16, with the size and resolution of these images gradually increasing. Thus, the multi-scale features corresponding to this storage location image at different resolutions include feature images of 8x8, 6x6, 4x4, 10x10, 12x12, 14x14, and 16x16.

[0071] By inputting the storage location image into the backbone network for feature extraction, the semantic representation of each pixel at different resolutions can be obtained, making the semantic expression of each pixel in the storage location image richer and more accurate, and the expression of local features more accurate.

[0072] The storage location image includes a panoramic view and a circumferential view. Therefore, the multi-scale features of the storage location image are obtained by extracting features from the backbone network. This includes: extracting features from the panoramic view through the backbone network to obtain the multi-scale features of the panoramic view; and extracting features from the circumferential view through the backbone network to obtain the multi-scale features of the circumferential view.

[0073] In step S12, the multi-scale features are input into the cross-attention mechanism to obtain the storage location features of the storage location.

[0074] The cross-attention mechanism is used to fuse multi-scale features of the panoramic and circumferential images to obtain the location features of the entire storage location.

[0075] For example, if the multi-scale features of both the panoramic view and the circumferential view are N frames of feature images at different resolutions, and the height H and width W of the N frames of feature images are the same, then 2N H*W*256 feature images will be input into the cross-attention mechanism. The 2 represents that the input is of two types: panoramic view and circumferential view, and 256 represents the feature dimension of the feature images. After the 2N H*W*256 feature images are fused by the cross-attention mechanism, a new 2N H*W*256 location feature will be obtained.

[0076] Alternatively, the multi-scale features of the panoramic image and the multi-scale features of the circumferential image can be input into an efficient defoemable cross attention mechanism to obtain the storage location features, which are used to represent the information of the entire storage location.

[0077] In step S13, the reservoir location features are input into the regression head to obtain the predicted three-dimensional coordinates of the target point.

[0078] The regression head (MLP) is used to obtain the predicted three-dimensional coordinates of the target point in the storage location image based on the storage location features in the input storage location image.

[0079] For example, after inputting 2N H*W*256 storage location features into the regression head, 2N 4*3 predicted 3D coordinates can be obtained. 2N represents the two types of images: panoramic and circumferential. 4 represents the number of target points. For example, if there are four foot points in the storage location image, then there are 4 target points. 3 represents the predicted 3D coordinates of the four target points, such as the XYZ coordinates.

[0080] In steps S11 to S13 above, the cross-attention mechanism can also combine adjacent first and second storage location images. It considers the temporal characteristics of image features and can input the predicted 3D coordinates (query anchor) of the target point in the first storage location image, the storage location feature (quer feature) of the first storage location image, and the multi-scale features of the second storage location image into the cross-attention mechanism to obtain the storage location feature corresponding to the second storage location image. Here, the first storage location image is the storage location image of the previous frame, and the second storage location image is the storage location image of the current frame.

[0081] For example, the cross-attention mechanism includes Q, K, and V, where Q, K, and V represent Query, Key, and Value, respectively. The Q value represents the information that needs to be focused on and processed in the initial network model. It can be the location features of the storage location in the first storage location image output by the cross-attention mechanism in the initial network model, or the predicted 3D coordinates of the target point in the first storage location image output by the subsequent regression head (MLP). The K and V values ​​represent the multi-scale features of the second storage location image.

[0082] Please see Figure 8 As shown, it can input the location features of the first location image and the predicted three-dimensional coordinates of the target point in the first location image as Q values ​​into the cross-attention mechanism, and input the multi-scale features of the second location image as K and V values ​​into the cross-attention mechanism. In this way, the cross-attention mechanism will obtain a new Q based on Q, K, and V, that is, obtain the location features of the second location image.

[0083] In steps S11 to S13 above, since the first and second storage location images are acquired at different times, the first and second storage location images input into the cross-attention mechanism belong to storage location images in different vehicle coordinate systems. To ensure that the first and second storage location images input into the cross-attention mechanism are in the same dimension, the following two solutions can be adopted: Option C involves transforming the predicted 3D coordinates of the target point in the first storage location image from the first vehicle coordinate system to the second vehicle coordinate system, thereby obtaining the first storage location image located in the second vehicle coordinate system. The first vehicle coordinate system is the vehicle coordinate system in which the vehicle acquires the first storage location image, and the second vehicle coordinate system is the vehicle coordinate system in which the vehicle acquires the second storage location image.

[0084] Specifically, the predicted three-dimensional coordinates of the target point located in the first vehicle coordinate system can be transformed through coordinate transformation relationship (anchor propagation) to obtain the predicted three-dimensional coordinates of the target point located in the second vehicle coordinate system.

[0085] For example, if the vehicle acquires the first storage location image at 10ms and the second storage location image at 11ms, then the vehicle is located in the first vehicle coordinate system at 10ms and in the second vehicle coordinate system at 11ms. The predicted three-dimensional coordinates of the target point in the first storage location image acquired by the vehicle at 10ms can be transformed from the first vehicle coordinate system to the second vehicle coordinate system, thereby obtaining the predicted three-dimensional coordinates of the target point located in the second vehicle coordinate system.

[0086] Option D: For the storage location features of the first storage location image, the storage location features of the first storage location image located in the first vehicle coordinate system can be converted into storage location features of the first storage location image located in the second vehicle coordinate system through a preset network (MOtionMLP).

[0087] After obtaining the storage location features of the first storage location image in the second vehicle coordinate system, the predicted three-dimensional coordinates of the target point in the first storage location image, and the multi-scale features of the second storage location image in the second vehicle coordinate system, these data are input into a cross-attention mechanism for fusion to obtain the storage location features of the second storage location image.

[0088] In steps S11 to S13 above, it is also necessary to incrementally update the predicted three-dimensional coordinates of the target point in the first storage location image and to incrementally update the storage location features of the first storage location image. These are described through the following two schemes: Scheme E, for the incremental update of the predicted three-dimensional coordinates of the target point in the first storage location image, can update the predicted three-dimensional coordinates of the target point in the first storage location image according to the iteration step size, and then input the updated predicted three-dimensional coordinates into the cross-attention mechanism.

[0089] An iterative step size can be superimposed on the predicted 3D coordinates of the target point in the first location image to obtain the updated predicted 3D coordinates; then the updated predicted 3D coordinates are transformed from the first vehicle coordinate system to the second vehicle coordinate system, and finally the updated predicted 3D coordinates located in the second vehicle coordinate system are input into the cross-attention mechanism.

[0090] Scheme F allows for the iterative update of the location features of the first location image. The location features of the first location image can be directly used as the location features input to the cross-attention mechanism next time, and the location features can be transformed from the first vehicle coordinate system to the second vehicle coordinate system and input into the cross-attention focus.

[0091] In some scenarios, please refer to Figure 8As shown, assuming the first storage location image is the previous frame storage location image, the second storage location image is the current frame storage location image, and the previous frame storage location image includes multiple frames of panoramic and circumferential images, and the current frame storage location image also includes multiple frames of panoramic and circumferential images, the initial network model obtains the predicted 3D coordinates of the target point in the storage location image based on the storage location image during each iteration of training. This includes: assuming the acquired current frame storage location image includes panoramic and circumferential images, after distortion correction of the panoramic and circumferential images, inputting the distorted panoramic and circumferential images into the backbone network to obtain the multi-scale features of the panoramic images; then updating the storage location features of the previous frame storage location image and the predicted 3D coordinates of the target point in the previous frame storage location image to obtain more... The updated storage location features of the previous storage location image and the predicted 3D coordinates of the target point in the previous storage location image are then transformed from the first vehicle coordinate system to the second vehicle coordinate system. The multi-scale features of the circumferential view image, the multi-scale features of the surround view image, the updated storage location features of the previous storage location image in the second vehicle coordinate system, and the predicted 3D coordinates of the target point in the updated storage location image in the second vehicle coordinate system are then input into a cross-attention mechanism to obtain the storage location features of the current storage location image. Finally, the storage location features of the current storage location image are input into a regression head to obtain the predicted 3D coordinates of the target point in the current storage location image.

[0092] In some scenarios, during the training of the initial network model, the following steps are iteratively executed: The acquired second storage location image is input into the initial network model to obtain the predicted 3D coordinates of the target point in the storage location; the predicted 3D coordinates are converted into predicted 2D coordinates; the loss value between the predicted 2D coordinates and the actual 2D coordinates is calculated; if the loss value does not meet the convergence condition, the network parameters in the initial network model are updated with the loss value, and the storage location features of the first storage location image and the predicted 3D coordinates of the target point in the first storage location image are updated; the updated storage location features of the first storage location image, the predicted 3D coordinates of the target point in the first storage location image, and the currently acquired second storage location image are input into the updated initial network model to obtain the updated predicted 3D coordinates; the updated predicted 3D coordinates are converted into updated predicted 2D coordinates, and the loss value between the updated predicted 2D coordinates and the actual 2D coordinates is calculated; the above steps are repeated until the loss value meets the convergence condition, thus obtaining the target network model.

[0093] The above technical solution provides a method that can accurately predict three-dimensional coordinates even with two-dimensional annotation. It utilizes a cross-attention mechanism and a regression head to obtain the predicted three-dimensional coordinates, and uses the loss value between the predicted two-dimensional coordinates after the three-dimensional coordinates are transformed and the actual two-dimensional coordinates to continuously update the network parameters of the initial network model, so that the initial network model after training can predict accurate three-dimensional coordinates.

[0094] On the one hand, by fusing the first storage location image of the previous frame with the second storage location image of the next frame through the cross-attention mechanism, the storage location features of the obtained second storage location image fully consider the temporal relationship of the vehicle when the storage location images are collected. When the image quality of the second storage location image is poor, the first storage location image of the previous frame with better image quality is used to supplement the second storage location image, so that the storage location features of the obtained second storage location image are more accurate. On the other hand, it constructs the loss value between the predicted two-dimensional coordinates and the actual two-dimensional coordinates, rather than the loss value between the predicted three-dimensional coordinates and the actual three-dimensional coordinates. Therefore, when the staff is annotating, they are annotating the actual two-dimensional coordinates of the target point in the storage location image, that is, they are annotating the pixel coordinates of the target point in the storage location image, rather than the three-dimensional coordinates in the vehicle coordinate system. The annotation is simpler and can greatly reduce the annotation cost.

[0095] Figure 10 This is a block diagram illustrating a model training apparatus according to an exemplary embodiment. (Refer to...) Figure 10 The model training device 1000 includes iteratively executing the following steps to obtain the target network model: The input module 1010 is configured to input the collected images of the storage location into the initial network model to predict the predicted three-dimensional coordinates of the target point in the storage location. The conversion module 1020 is configured to convert the predicted three-dimensional coordinates into predicted two-dimensional coordinates; The update module 1030 is configured to update the network parameters in the initial network model based on the loss value between the predicted two-dimensional coordinates and the actual two-dimensional coordinates; the actual two-dimensional coordinates are obtained by labeling the target points in the storage location image. The determination module 1040 is configured to, when the network parameters of the initial network model satisfy the convergence condition, use the initial network model that satisfies the convergence condition as the target network model; the target network model is used to obtain the three-dimensional coordinates of the target point based on the storage location image.

[0096] Optionally, the initial network model includes a backbone network, a cross-attention mechanism, and a regression head; the input module 1010 includes: The first input submodule is configured to extract features from the library image through the backbone network to obtain multi-scale features; The second input submodule is configured to input the multi-scale features into the cross-attention mechanism to obtain the location features of the storage location; The third input submodule is configured to input the reservoir location features into the regression head to obtain the predicted three-dimensional coordinates of the target point.

[0097] Optionally, the storage location image includes an adjacent first storage location image and a second storage location image; the second input submodule is further configured to input the multi-scale features of the second storage location image, the predicted three-dimensional coordinates of the target point in the first storage location image, and the storage location features of the first storage location image into the cross-attention mechanism to obtain the storage location features corresponding to the second storage location image.

[0098] Optionally, the second input submodule includes: The iterative submodule is configured to update the predicted 3D coordinates of the target point in the first storage location image according to the iteration step size; The update input submodule is configured to input the updated predicted 3D coordinates into the cross-attention mechanism.

[0099] Optionally, the second input submodule is further configured to transform the predicted three-dimensional coordinates of the target point in the first storage location image from the first vehicle coordinate system to the second vehicle coordinate system, thereby obtaining the predicted three-dimensional coordinates of the target point located in the second vehicle coordinate system; the first vehicle coordinate system is the vehicle coordinate system in which the vehicle acquires the first storage location image, and the second vehicle coordinate system is the vehicle coordinate system in which the vehicle acquires the second storage location image.

[0100] Optionally, the model training device 1000 includes: The annotation module is configured to annotate the actual two-dimensional coordinates of the target point on the storage location image.

[0101] Optionally, the target location includes at least one of the following: a foot point in the storage location, any point on the line connecting the baffles on both sides of the storage location and the ground of the storage location.

[0102] Optionally, the storage location image includes a panoramic view and a circumferential view.

[0103] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0104] This disclosure also provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the steps of the model training method provided in this disclosure.

[0105] Figure 11 This is a block diagram illustrating a vehicle 1100 according to an exemplary embodiment. For example, vehicle 1100 may be a hybrid vehicle, a non-hybrid vehicle, an electric vehicle, a fuel cell vehicle, or other types of vehicle. Vehicle 1100 may be an autonomous vehicle, a semi-autonomous vehicle, or a non-autonomous vehicle.

[0106] Reference Figure 11 The vehicle 1100 may include various subsystems, such as an infotainment system 1110, a perception system 1120, a decision control system 1130, a drive system 1140, and a computing platform 1150. The vehicle 1100 may also include more or fewer subsystems, and each subsystem may include multiple components. Furthermore, each subsystem and each component of the vehicle 1100 can be interconnected via wired or wireless means.

[0107] In some embodiments, the infotainment system 1110 may include a communication system, an entertainment system, and a navigation system, etc.

[0108] The perception system 1120 may include several types of sensors for sensing information about the environment surrounding the vehicle 1100. For example, the perception system 1120 may include a global positioning system (which may be a GPS system, a BeiDou system, or another positioning system), an inertial measurement unit (IMU), a lidar, a millimeter-wave radar, an ultrasonic radar, and a camera device.

[0109] The decision control system 1130 may include a computing system, a vehicle controller, a steering system, a throttle, and a braking system.

[0110] The drive system 1140 may include components that provide powered motion to the vehicle 1100. In one embodiment, the drive system 1140 may include an engine, an energy source, a transmission system, and wheels. The engine may be one or a combination of internal combustion engines, electric motors, and compressed air engines. The engine is capable of converting energy provided by the energy source into mechanical energy.

[0111] Some or all of the functions of vehicle 1100 are controlled by computing platform 1150. Computing platform 1150 may include at least one processor 1151 and memory 1152, and processor 1151 may execute instructions 1153 stored in memory 1152.

[0112] Processor 1151 can be any conventional processor, such as a commercially available CPU. Processors may also include graphics processing units (GPUs), field-programmable gate arrays (FPGAs), systems-on-chips (SoCs), application-specific integrated circuits (ASICs), or combinations thereof.

[0113] The memory 1152 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0114] In addition to instruction 1153, memory 1152 can also store data, such as road maps, route information, vehicle position, direction, speed, and other data. The data stored in memory 1152 can be used by computing platform 1150.

[0115] In this embodiment of the disclosure, the processor 1151 may execute instructions 1153 to complete all or part of the steps of the above-described model training method.

[0116] In another exemplary embodiment, a computer program product is also provided, the computer program product comprising a computer program executable by a programmable device, the computer program having a code portion for performing the model training method described above when executed by the programmable device.

[0117] Some embodiments of this disclosure also provide a chip system, such as Figure 12 As shown, the chip system includes at least one processor 1201 and at least one interface circuit 1202. The processor 1201 and the interface circuit 1202 are interconnected via lines. For example, the interface circuit 1202 can be used to receive signals from other devices (e.g., the memory of an electronic device). As another example, the interface circuit 1202 can be used to send signals to other devices (e.g., the processor 1201). Exemplarily, the interface circuit 1202 can read instructions stored in memory and send those instructions to the processor 1201. When the instructions are executed by the processor 1201, the model training device can perform the steps in the above embodiments. Of course, the chip system may also include other discrete components, and some embodiments of this disclosure do not specifically limit this.

[0118] In some embodiments of this disclosure, the interface circuit 1202 can acquire data, program instructions, and / or information from the internal storage area of ​​the chip system; it can also acquire data, program instructions, and / or information from outside the chip system.

[0119] Optionally, the chip system may also include a memory for storing necessary computer programs and data.

[0120] Figure 13 This is a block diagram illustrating an apparatus 1300 for model training according to an exemplary embodiment. For example, apparatus 1300 may be provided as a server. (Refer to...) Figure 13 The device 1300 includes a processing component 1322, which further includes one or more processors, and memory resources represented by memory 1332 for storing instructions, such as application programs, that can be executed by the processing component 1322. The application programs stored in memory 1332 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 1322 is configured to execute instructions to perform the aforementioned model training method.

[0121] Device 1300 may also include a power supply component 1326 configured to perform power management of device 1300, a wired or wireless network interface 1350 configured to connect device 1300 to a network, and an input / output interface 1358. Device 1300 can operate on an operating system stored in memory 1332.

[0122] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 1332 including instructions, which can be executed by the processor of device 1300 to complete the model training method described above. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0123] Those skilled in the art will also understand that the various illustrative logical blocks and steps listed in the embodiments of this application can be implemented by electronic hardware, computer software, or a combination of both. Whether such functionality is implemented through hardware or software depends on the specific application and the overall system design requirements. Those skilled in the art can implement the described functionality using various methods for each specific application, but such implementation should not be construed as exceeding the scope of protection of the embodiments of this application.

Claims

1. A model training method, characterized in that, The method includes iteratively performing the following steps to obtain the target network model: The collected images of the storage location are input into the initial network model to predict the predicted three-dimensional coordinates of the target point in the storage location. Convert the predicted three-dimensional coordinates into predicted two-dimensional coordinates; The network parameters in the initial network model are updated based on the loss value between the predicted two-dimensional coordinates and the actual two-dimensional coordinates; the actual two-dimensional coordinates are obtained by labeling the target points in the storage location image. If the loss value satisfies the convergence condition, the initial network model that satisfies the convergence condition is used as the target network model; the target network model is used to obtain the three-dimensional coordinates of the target point based on the storage location image.

2. The method according to claim 1, characterized in that, The initial network model includes a backbone network, a cross-attention mechanism, and a regression head; the step of inputting the collected reservoir location images into the initial network model to predict the predicted three-dimensional coordinates of the target points in the reservoir location includes: The images of the storage locations are processed through the backbone network to extract features, resulting in multi-scale features. The multi-scale features are input into the cross-attention mechanism to obtain the location features of the storage location; The reservoir location features are input into the regression head to obtain the predicted three-dimensional coordinates of the target point.

3. The method according to claim 2, characterized in that, The storage location image includes adjacent first storage location images and second storage location images; the step of inputting the multi-scale features into the cross-attention mechanism to obtain the storage location features of the storage location includes: The multi-scale features of the second storage location image, the predicted three-dimensional coordinates of the target point in the first storage location image, and the storage location features of the first storage location image are input into the cross-attention mechanism to obtain the storage location features corresponding to the second storage location image.

4. The method according to claim 3, characterized in that, Inputting the predicted 3D coordinates of the target point in the first storage location image into the cross-attention mechanism includes: The predicted 3D coordinates of the target point in the first storage location image are updated according to the iteration step size; The updated predicted 3D coordinates are input into the cross-attention mechanism.

5. The method according to claim 3, characterized in that, Inputting the predicted 3D coordinates of the target point in the first storage location image into the cross-attention mechanism includes: The predicted three-dimensional coordinates of the target point in the first storage location image are transformed from the first vehicle coordinate system to the second vehicle coordinate system to obtain the predicted three-dimensional coordinates of the target point located in the second vehicle coordinate system; the first vehicle coordinate system is the vehicle coordinate system in which the vehicle collects the first storage location image, and the second vehicle coordinate system is the vehicle coordinate system in which the vehicle collects the second storage location image.

6. The method according to claim 1, characterized in that, The method further includes: Mark the actual two-dimensional coordinates of the target point on the image of the storage location.

7. The method according to any one of claims 1 to 6, characterized in that, The target location includes at least one of the following: a foot point in the storage location, or any point on the line connecting the baffles on both sides of the storage location and the ground of the storage location.

8. The method according to any one of claims 1 to 6, characterized in that, The storage location image includes a panoramic view and a circumferential view.

9. A model training device, characterized in that, The model training device includes iteratively executing the following steps to obtain the target network model: The input module is configured to input the collected images of the storage location into the initial network model to predict the predicted three-dimensional coordinates of the target point in the storage location; The conversion module is configured to convert the predicted three-dimensional coordinates into predicted two-dimensional coordinates; The update module is configured to update the network parameters in the initial network model based on the loss value between the predicted two-dimensional coordinates and the actual two-dimensional coordinates; the actual two-dimensional coordinates are obtained by labeling the target points in the storage location image; The determination module is configured to, if the network parameters of the initial network model satisfy the convergence condition, use the initial network model that satisfies the convergence condition as the target network model; the target network model is used to obtain the three-dimensional coordinates of the target point based on the storage location image.

10. A vehicle, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured as follows: The steps of the method according to any one of claims 1 to 8.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the method described in any one of claims 1 to 8.

12. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the steps of the method according to any one of claims 1 to 8.

Images