Target image prediction method, storage medium, vehicle-mounted equipment and vehicle

Abstract

The invention discloses a target image prediction method, a storage medium, vehicle-mounted equipment and a vehicle, and the method comprises the steps: stacking at least one non-distorted image and at least one distorted image, inputting the stacked images into a feature extractor, obtaining a feature map of a preset number of layers, and enabling the non-distorted images and the distorted images to come from different types of cameras; obtaining a bird's-eye view query vector in a preset range with the own vehicle as the center; projecting the aerial view query vector back to the feature map to form an aerial view feature map; and inputting the aerial view feature map into a detector to predict a target image. In this way, image information can be effectively obtained, features of different cameras are fused, and the blind area sensing ability is enhanced.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to a target image prediction method, a storage medium, an on-board device and a vehicle. Background Art

[0002] In the existing vehicle visual environment processing methods, the collected distorted images need to be pre-processed, dedistorted, flattened, and cropped before being fed into the model. However, this processing method will reduce the acquired image information and narrow the field of view. When acquiring visual information around the vehicle, there are problems such as blind spots in forward perception and the inability to extract and fuse features collected by different types of cameras.

[0003] Therefore, a detection method is needed that can effectively obtain image information, integrate the features of different types of cameras and enhance the blind spot perception capability. Summary of the invention

[0004] The present application mainly provides a target image prediction method, a storage medium, a vehicle-mounted device and a vehicle to solve the problems of incomplete image information acquisition, the inability to fuse different camera features and weak forward visual perception capability.

[0005] To solve the above technical problems, a technical solution adopted in the present application is: to provide a target image prediction method, comprising: stacking at least one non-distorted image and at least one distorted image and inputting the stacked images into a feature extractor to obtain a feature map of a preset number of layers, wherein the non-distorted image and the distorted image are derived from different types of cameras; obtaining a bird's-eye view query vector within a preset range centered on the vehicle; projecting the bird's-eye view query vector back to the feature map to form a bird's-eye view feature map; and inputting the bird's-eye view feature map into a detector to predict a target image.

[0006] The information of the original image is maintained by directly extracting features from the distorted original image and the non-distorted image; the features are projected onto the image through the bird's-eye view query vector of the query feature to complete the feature fusion of the distorted image and the non-distorted image; the predicted target image is obtained based on the bird's-eye view feature map with fused features, the feature fusion of different types of camera images is realized, and a bird's-eye view feature map with close-range data annotation is formed.

[0007] In some embodiments, projecting the bird's-eye view query vector back to the feature map to form a bird's-eye view feature map includes: obtaining a normalized coordinate plane of the bird's-eye view query vector in a corresponding camera coordinate system; correcting the coordinate points of the normalized coordinate plane based on the corresponding camera type; and projecting the corrected coordinate points of the normalized coordinate plane back to the feature map to obtain the bird's-eye view feature map.

[0008] Through the above steps, the bird's-eye view query vector is projected onto the camera normalized coordinate plane, so that the camera parameters can be used to calculate the distortion of the bird's-eye view query vector on the camera normalized coordinate plane.

[0009] In some embodiments, the camera type includes a fisheye camera; the correction of the coordinate points of the normalized coordinate plane based on the corresponding camera type includes: calculating the coordinates of the distortion points for some coordinate points on the normalized coordinate plane corresponding to the fisheye camera, and using the coordinates of the distortion points as the updated new coordinate points; the distorted image originates from the fisheye camera.

[0010] Through the above steps, the distortion of the distorted image acquired by the fisheye camera is calculated on the camera normalization plane, so that the distorted coordinate points and the non-distorted coordinate points on the camera normalization plane can be projected onto the image at the same time.

[0011] In some embodiments, obtaining the normalized coordinate plane of the bird's-eye view query vector in the corresponding camera coordinate system includes: multiplying the coordinate points of the bird's-eye view query vector with the corresponding camera extrinsic parameters and performing normalization processing to obtain the normalized coordinate plane in the corresponding camera coordinate system; projecting the corrected coordinate points of the normalized coordinate plane back to the feature map includes: multiplying the corrected coordinate points of the normalized coordinate plane with the corresponding camera intrinsic parameters and projecting them back to the feature map.

[0012] Through the above steps, the coordinate points are projected onto the camera plane, and the coordinates of the coordinate points are calculated and normalized using the camera external parameters, so that the coordinate points are converted to the camera normalized plane, and the coordinate points can be mapped to the bird's-eye view image. Through the coordinate transformation of the coordinate points corresponding to the bird's-eye view query vector one by one, the process of projecting the bird's-eye view query vector back to the feature map is completed.

[0013] In some embodiments, before stacking at least one non-distorted image and at least one distorted image, the method further includes: enhancing forward blind area data of the distorted image.

[0014] Strengthening the forward blind area data of the distorted image through the above steps is beneficial to enhancing the model's forward perception ability and reducing the loss of image information.

[0015] In some embodiments, the enhancing the forward blind spot data of the distorted image includes: randomly selecting a sampling position coordinate within a preset forward sampling distance; forming a two-dimensional mask on the sampling position coordinate based on a randomly selected target in a mask library; adding distortion to the two-dimensional mask based on camera parameters, the camera parameters including camera intrinsic parameters and camera extrinsic parameters; pasting the two-dimensional mask with added distortion onto the real image to obtain a virtual target corresponding to the target; identifying the target in the blind spot of the distorted image based on the virtual target, and enhancing the forward blind spot data of the distorted image.

[0016] Through the above steps, a two-dimensional mask with added distortion is formed to form a virtual forward target. Through the conversion process between the real forward target and the virtual forward target, a virtual target corresponding to each target is formed for blind spot recognition of the distorted image, training the model's forward perception ability, and improving the ability to obtain forward image information.

[0017] In some embodiments, the forming of a two-dimensional mask at the sampling position coordinates based on a target randomly selected from a mask library includes: setting a three-dimensional graphic of the target randomly selected from the mask library at the sampling position coordinates; projecting the three-dimensional graphic back into the real image to obtain a two-dimensional graphic of the three-dimensional graphic in the real image; scaling the length and width of the two-dimensional mask corresponding to the target to be the same as the length and width of the two-dimensional graphic to obtain a two-dimensional mask at the sampling position coordinates.

[0018] Through the above steps, the three-dimensional graphics are projected onto the two-dimensional image on the real image to form a two-dimensional mask at the sampling position, which is used to limit the visible range on the node.

[0019] In some embodiments, adding distortion to the two-dimensional mask based on camera parameters includes: converting each pixel coordinate in the real image to a camera coordinate system, and calculating the radial distance of each pixel coordinate in the camera coordinate system; calculating each distorted pixel coordinate based on the camera parameters and the radial distance of each pixel coordinate; mapping each distorted pixel coordinate to the real image to obtain a distorted image; and cropping based on the boundary of a non-zero area of ​​the distorted image to obtain the two-dimensional mask with added distortion.

[0020] Through the above steps, the real image is distorted and then the image coordinate points are mapped, so that distortion is added to the two-dimensional figure on the real image, so that the distorted two-dimensional figure can be used for feature extraction training of the distorted image.

[0021] In some embodiments, obtaining a bird's-eye view query vector within a preset range centered on the vehicle includes: constructing a grid of reference points within the preset range centered on the vehicle; initializing a bird's-eye view query vector corresponding to each of the grid reference points using a word embedding module; and processing the bird's-eye view query vector using an attention mechanism.

[0022] Through the above steps, the model's ability to process bird's-eye view query vectors is trained, the model's global information of bird's-eye view query vectors in the space constructed by grid reference points is strengthened, and the model's ability to process spatial relationships and strengthen features is improved.

[0023] In some embodiments, the use of an attention mechanism to process the bird's-eye view query vector includes: inputting each of the bird's-eye view query vectors into a self-attention mechanism for processing; and inputting each of the bird's-eye view query vectors and the corresponding feature map into a deformable cross-attention mechanism for processing.

[0024] Through the deformable cross-attention mechanism, the model can flexibly adjust the receptive field when calculating the attention weight, better adapt to complex input structures, and improve the performance of the model when processing features of different scales and shapes.

[0025] In order to solve the above technical problems, another technical solution adopted by the present application is: providing a storage medium on which program data is stored, and when the program data is executed by a processor, the steps of the target image prediction method as described above are implemented.

[0026] The beneficial effects of the storage medium refer to the beneficial effects of the target image prediction method described above, which will not be described in detail here.

[0027] The present application also provides a vehicle-mounted device, including a processor and a memory connected to each other, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the target image prediction method as described above are implemented.

[0028] The vehicle-mounted device is a computer-type device. The beneficial effects of the vehicle-mounted device refer to the beneficial effects of the above-mentioned target image prediction method, which will not be repeated here.

[0029] The present application also provides a vehicle, including the vehicle-mounted device as described above. The vehicle can complete the process of the target image prediction method through the control of the vehicle-mounted device during driving.

[0030] The beneficial effects of the present application are as follows: Different from the prior art, the present application discloses a target image prediction method, a storage medium, a vehicle-mounted device and a vehicle, which completes feature extraction of different types of camera images by stacking a non-distorted image and a distorted image and inputting the stacked images into a feature extractor. The connection between the coordinates of each point in the space and the features is established through a bird's-eye view query vector, and the distorted points and the non-distorted points can be mapped to the image together, thereby avoiding the influence of distortion on the feature coordinate mapping of the bird's-eye view query vector, and also avoiding image incompleteness caused by clipping distortion, thereby achieving effective acquisition of image information and enhancing blind spot perception capabilities. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work, among which:

[0032] Figure 1 It is a flowchart of the first embodiment of the target image prediction method provided by the present application;

[0033] Figure 2 It is a flowchart of the second embodiment of the target image prediction method provided by the present application;

[0034] Figure 3 It is a flowchart of the target image prediction method embodiment 3 provided in the present application;

[0035] Figure 4 is a flowchart of a fourth embodiment of the target image prediction method provided by the present application;

[0036] Figure 5 is a flowchart of a fifth embodiment of the target image prediction method provided by the present application;

[0037] Figure 6 is a flowchart of a sixth embodiment of the target image prediction method provided by the present application;

[0038] Figure 7 is a flow chart of the seventh embodiment of the target image prediction method provided by the present application;

[0039] Figure 8 It is a structural schematic diagram of an embodiment of a storage medium provided by the present application;

[0040] Fig. 9 It is a structural schematic diagram of an embodiment of the vehicle-mounted device provided by the present application;

[0041] Fig.10It is a structural schematic diagram of a vehicle embodiment provided by the present application. DETAILED DESCRIPTION

[0042] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0043] The terms "first", "second", "third" in the embodiments of the present application are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Thus, the features defined as "first", "second", "third" can expressly or implicitly include at least one of the features. In the description of the present application, the meaning of "multiple" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but optionally also includes steps or units that are not listed, or optionally also includes other steps or units inherent to these processes, methods, products or devices.

[0044] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0045] When an intelligent vehicle performs visual perception, it collects images of the surroundings through on-board visual acquisition devices such as cameras. Depending on the type and location of the camera, the collected images also have differences in content and form. The more common form difference is the degree of image distortion. The present application provides a target image prediction method for feature extraction and fusion of images with different degrees of distortion.

[0046] See also Figure 1 , Figure 1 : is a flow chart of a target image prediction method embodiment 1 provided in the present application, the target image prediction method comprising:

[0047] Step 10: stack at least one non-distorted image and at least one distorted image and input them into a feature extractor to obtain a feature map of a preset number of layers.

[0048] The non-distorted images and distorted images captured by different cameras are stacked into image data with the same pixel size, that is, the same image size, and then the image data is input into the feature extractor to obtain a feature map of a preset number of layers.

[0049] Non-distorted image: The non-distorted image in this application refers to an image with zero distortion, or an image with a low distortion that does not affect feature extraction.

[0050] Distorted image: A distorted image in this application refers to an image that has a larger distortion than a non-distorted image and requires distortion calculation during the image feature extraction process.

[0051] Among them, the non-distorted image and the distorted image come from different types of cameras respectively.

[0052] Optionally, the non-distorted image can be acquired by a camera with less distortion, such as a vehicle-mounted pinhole camera.

[0053] Optionally, the distorted image can be acquired by a camera with a wide viewing angle but large distortion, such as a vehicle-mounted fisheye camera or a wide-angle camera.

[0054] Optionally, the stacking of images can be achieved by image processing software. The non-distorted images and distorted images acquired by different cameras are input into the image processing software. After setting the target pixel size, the image processing software adjusts the layer position and image size of each image, and stacks the input images into image data with the same pixel size.

[0055] For example, a fisheye camera for acquiring distorted images is set in front of the vehicle, and seven pinhole cameras for acquiring undistorted images are set on the sides and rear of the vehicle. The collected distorted image and seven undistorted images are stacked to form eight images with each image having a pixel size of 576×960. The stacked data is (8, 576, 960).

[0056] For another example, a fisheye camera is set at the front and rear of the vehicle, and four pinhole cameras are set at the side. Two distorted images and four non-distorted images are stacked to form six images with each image having a pixel size of 324×540. The stacked data is (6, 324, 540).

[0057] The feature extractor consists of a backbone network and a feature pyramid network (FPN). The backbone network is used to extract useful feature information from the input image, and the feature pyramid network is used to make the image contained in the feature pyramid have strong semantic representation and features at different observation levels, so as to better detect and identify the target.

[0058] The main network is a bottom-up deep convolutional neural network. The convolutional layers of the main network extract features layer by layer to form feature maps at different levels. The feature pyramid network is a top-down network structure. In the layer-by-layer calculation process, feature maps of different resolutions are generated. These feature maps have different semantic information. The shallow feature map has less semantic information, but the target position is relatively accurate; the deep feature map has rich semantic information, but the target position is relatively rough. The feature pyramid network fuses these feature maps at different levels through a top-down structure and lateral connections, thereby constructing feature maps with strong semantic information at all scales. The feature pyramid network extracts features from the main network, fuses features of different scales, that is, different observation levels, to form a feature pyramid, which contains feature representations of images at different scales. New pixel values ​​are inserted into the image through upsampling to generate a feature map with higher resolution. Then the upsampled feature map is horizontally connected and fused with the feature map of the corresponding resolution in the main network to obtain a new feature map. Repeat the above process until all feature maps of the required resolution are generated.

[0059] The main network is responsible for feature extraction, while the feature pyramid network is responsible for using these features to detect and identify multi-scale objects. The feature extractor combines the main network and the feature pyramid network to form an efficient object detection network, thereby more accurately extracting the feature information contained in the image.

[0060] A feature map is an image feature extracted from an input image through a convolution operation in a convolutional neural network. A feature map can be seen as an abstract representation of the original image, where each pixel represents some specific features.

[0061] Specifically, according to the needs of the actual image processing process, the number of layers of the feature map can be preset to 3 layers, 4 layers, 5 layers or 6 layers, etc.

[0062] For example, when the preset number of layers of the feature map is 4, the C2, C3, C4 and C5 layers in the main network are selected as the output layers of feature extraction, and the outputs of these layers have different spatial resolutions and semantic levels. The feature pyramid network starts from the highest layer C5 output by the main network, generates P5 with higher resolution by upsampling, and then connects and fuses P5 with C4 of the corresponding resolution in the main network horizontally to obtain a new feature map P4. Repeat the above steps until all feature maps of the required resolution are generated, including feature maps of the four layers P2, P3, P4 and P5 with the same number of channels, different spatial resolutions and different semantic levels.

[0063] By stacking the distorted image and the non-distorted image and then extracting features through a feature extractor, it is possible to jointly extract features from different types of images acquired by different cameras, thereby improving the efficiency of feature extraction and facilitating the image processing model to more accurately extract the feature information contained in the image, thereby better detecting and identifying the surrounding environment.

[0064] Step 20: Obtain a bird's-eye view query vector within a preset range centered on the vehicle.

[0065] A vehicle coordinate system centered on the vehicle is constructed, and multiple bird's-eye view query vectors (BEV queries) are formed within a preset range around the vehicle.

[0066] The bird's eye view query vector is a learnable parameter used to query the bird's eye view (BEV) features. The elements are randomly initialized, and you can choose to use random numbers generated by standard normal distribution (torch.randn) or generate random numbers between 0 and 1 (torch.rand) as needed. The bird's eye view query vector is usually organized into a grid structure, and each bird's eye view query vector corresponds to each grid in the grid structure, which can query and aggregate features of the area of ​​interest from multiple camera perspectives. The acquisition of the bird's eye view query vector can convert the neatly input sequence into a dense vector representation through the initialization of the functional module.

[0067] The vehicle coordinate system is a coordinate system that uses a preset point as the origin of the coordinate system and is developed according to the orientation of the vehicle as the direction of the coordinate axis. Specifically, the origin of the coordinate system can be the left front wheel, left rear wheel, right front wheel, right rear wheel, geometric center, forward camera position or radar sensor position of the vehicle, or the ground position where the left front wheel touches the ground, the ground position corresponding to the geometric center of the vehicle, etc. The extension direction of the coordinate axis can be from the origin of the coordinate system to the right side of the vehicle as the positive semi-axis of the x-axis, and from the origin of the coordinate system to the front of the vehicle as the positive semi-axis of the y-axis; it can also be from the origin of the coordinate system to the left side of the vehicle as the positive semi-axis of the x-axis, from the origin of the coordinate system to the front of the vehicle as the positive semi-axis of the y-axis, or other coordinate axis extension methods in the front, back, left and right directions. Among them, the horizontal angle between the x-axis and the y-axis is 90°. In the vehicle coordinate system, the z-axis can be extended from the origin of the coordinate system vertically upward or vertically downward from the ground as the positive semi-axis, and the extension direction of the z-axis is usually the observation direction when the positive semi-axis of the y-axis is at a position 90° counterclockwise from the positive semi-axis of the x-axis.

[0068] The preset range is a preset range for collecting features around the vehicle. The preset range can define a closed area with the distance of the vehicle in each direction as the dimension to form a bird's-eye view query vector within the area. The preset range can be a circle, square, rectangle or other polygon that includes the vehicle. The perception of distance in each direction can be obtained by devices such as ranging radar, ranging camera or infrared rangefinder, and this application does not limit this.

[0069] For example, a rectangle formed by a distance of 60 meters forward and 15 meters to the left, right and rear of the vehicle is set as the preset range, 75×30 evenly distributed 1×1 square meter squares are formed within the preset range, and 75×30 bird's-eye view query vectors are formed at the center point of each square.

[0070] For another example, a circle with a radius of 20m and a vehicle as the origin is used as the preset range. Within the preset range, a circle is formed every 1m from the vehicle to the boundary of the preset range, for a total of 20 circles. 30 points are randomly selected on each circle to form a total of 20×30 bird's-eye view query vectors.

[0071] By forming a bird's-eye view query vector within a preset area, features of the area of ​​interest from multiple camera perspectives can be queried and aggregated, which is conducive to accurately determining the spatial location of features and the query relationship between features.

[0072] Further, see Figure 2 , Figure 2 : is a flow chart of the second embodiment of the target image prediction method provided by the present application. Acquiring a bird's-eye view query vector within a preset range centered on the vehicle includes:

[0073] Step 21: Construct a grid of reference points within a preset range centered on the vehicle.

[0074] A vehicle coordinate system is constructed with the vehicle as the center, and a sampling range is set on the vehicle coordinate system to form grid reference points evenly distributed within the sampling range from a bird's-eye view.

[0075] The grid reference points are used to uniformly sample the spatial positions within a preset range. Each grid reference point corresponds to a bird's-eye view query vector, which is a position mapping of the bird's-eye view query vector on the bird's-eye view two-dimensional plane.

[0076] The position of the grid reference point in space can be represented by the coordinates (x, y, z), where x and y are the horizontal and vertical coordinates of the grid reference point in the current vehicle coordinate system, respectively, and z is a fixed constant. Specifically, the z value can be set to the height of the ranging radar, the height of the camera, the height of the wheel, or other positions that can be used as height references.

[0077] For example, a vehicle coordinate system is constructed with the geometric center of the vehicle chassis as the coordinate origin, the right direction of the vehicle as the positive direction of the x-axis, the front direction of the vehicle as the positive direction of the y-axis, and the vertical direction upward as the positive direction of the z-axis. In the vehicle coordinate system, the preset sampling range is set to be 60m forward of the coordinate origin and 15m to the left, right and rearward. 150×120 grid reference points are evenly distributed every 0.5m along the x-axis and every 0.25m along the y-axis. The x value range of the grid reference point is a multiple of 0.5 from -15 to 15, the y value range is a multiple of 0.25 from -15 to 60, and z is the height of the ranging device from the vehicle chassis, 0.4. The coordinate forms of the grid reference point are (2.5, 3, 0.4), (5, 8, 0.4), (15, 60, 0.4) and (-15, -15, 0.4).

[0078] Generally speaking, when a preset range is constructed with the vehicle as the center, due to the higher requirements for forward perception, a telephoto camera is often used as the forward camera, and the range set in the forward distance is greater than the range set in the lateral distance.

[0079] By constructing a grid of reference points centered on the vehicle, a uniform distribution of sampling positions in space is formed, making the spatial mapping of the bird's-eye view query vector more uniform, querying and aggregating features of the area of ​​interest from multiple camera perspectives. This is conducive to strengthening the spatial connection between features and strengthening features through uniformly distributed spatial relationships.

[0080] Step 22: Use the word embedding module to initialize the bird's-eye view query vector corresponding to each grid reference point.

[0081] Taking the number and position of grid reference points as reference, the word embedding module initializes the bird's-eye view query vector that corresponds one-to-one with the grid reference points.

[0082] For example, if there are 150×120 grid reference points in the preset range, the word embedding module initializes 150×120 bird's-eye view query vectors. Each bird's-eye view query vector can correspond to a grid reference point, and each grid reference point is used as a spatial position mapping of the corresponding bird's-eye view query vector.

[0083] The word embedding module (referring to the torch.nn.embedding module in this embodiment) is a module in the PyTorch deep learning framework, which is used to convert the input vocabulary index into a dense vector representation, that is, to initialize a set of bird's-eye view query vectors.

[0084] The bird's-eye view query vector corresponding to each grid reference point is initialized through the word embedding module, and the features indicated by each bird's-eye view query vector are mapped to the two-dimensional plane of the bird's-eye view. Each bird's-eye view query vector corresponds to each grid in the grid structure, and the features of the area of ​​interest from multiple camera perspectives can be queried and aggregated to the two-dimensional plane of the bird's-eye view.

[0085] Step 23: Use the attention mechanism to process the bird’s-eye view query vector.

[0086] By processing the bird's-eye view query vector through a variety of different attention mechanisms, the ability of the image processing model to extract image features and construct image spatial relationships can be enhanced, thereby improving the performance of the image processing model.

[0087] Optionally, the attention mechanism includes a self-attention mechanism or a deformable cross-attention mechanism.

[0088] Further, see Figure 3 , Figure 3 : is a flow chart of the target image prediction method embodiment 3 provided in the present application, and using the attention mechanism to process the bird's-eye view query vector includes:

[0089] Step 231: Input each bird's-eye view query vector into the self-attention mechanism for processing.

[0090] The bird's-eye view query vector is input into the self-attention mechanism, and self-attention training is performed between the bird's-eye view query vectors. By calculating the relationship between different positions in the input sequence, the global information is captured, thereby improving the performance of the image processing model.

[0091] Step 232: Input each bird's-eye view query vector and the corresponding feature map into the deformable cross attention for processing.

[0092] The bird's-eye view query vector and feature map are input into the deformable cross attention mechanism for deformable cross attention training, so that the model can flexibly adjust the receptive field when calculating the attention weight. Compared with the traditional fixed receptive field, the deformable cross attention can better adapt to complex input structures and improve the performance of image processing models when processing features of different scales and shapes.

[0093] Step 30: Project the bird's eye view query vector back to the feature map to form a bird's eye view feature map.

[0094] The bird's-eye view query vector is projected back to the feature map through the spatial position corresponding to the bird's-eye view query vector to form a bird's-eye view feature map on a two-dimensional plane under a bird's-eye view perspective.

[0095] The bird's-eye view query vector is a learnable parameter used to query the bird's-eye view features. The process of projecting each bird's-eye view query vector back to the image according to the corresponding position is also the process of accurately projecting the bird's-eye view features corresponding to each bird's-eye view query vector onto the image. This process realizes the formation of feature parameters on the plane image, strengthens the spatial relationship between feature parameters, and further refines the image processing model's feature processing capabilities for images.

[0096] Further, see Figure 4 , Figure 4 is a flow chart of the fourth embodiment of the target image prediction method provided by the present application, and step 30 also includes:

[0097] Step 31: Obtain the normalized coordinate plane of the bird's-eye view query vector in the corresponding camera coordinate system.

[0098] The bird's-eye view query vector is converted to the camera coordinate system through its corresponding spatial relationship, and the camera coordinate system is normalized to obtain the camera normalization plane and the coordinates of each point on the plane.

[0099] Furthermore, step 31 includes: multiplying the coordinate point of the bird's-eye view query vector by the corresponding camera extrinsic parameter and performing normalization processing to obtain a normalized coordinate plane in the corresponding camera coordinate system.

[0100] Each bird's-eye view query vector corresponds to each coordinate point. The coordinate point (x, y, z) is multiplied by the camera extrinsic parameter of the image source camera at its corresponding position to obtain the coordinate (xc, yc, zc) in the camera coordinate system; then each point is divided by the depth coordinate zc of the point itself, and each coordinate point in the camera coordinate system is projected to the depth-normalized camera coordinate system to obtain the coordinate of the coordinate point in the normalized coordinate plane (xc / zc, yc / zc, 1).

[0101] For example, the process of projecting the coordinate point (3, 5, 1.5) to the normalized coordinate plane is: multiply the coordinate point (3, 5, 1.5) by the external parameter function d of the image source camera at its corresponding position to obtain (3d, 5d, 1.5d), and then divide it by the depth coordinate of the point itself 1.5d to obtain the coordinates of the coordinate point (3, 5, 1.5) in the normalized coordinate plane (3d / 1.5d, 5d / 1.5d, 1).

[0102] Through the above steps, the coordinate points are projected onto the camera plane, and the coordinates of the coordinate points are calculated and normalized using the camera extrinsic parameters, so that the coordinate points are converted to the camera normalization plane, and the coordinate points can be mapped to the bird's-eye view.

[0103] Step 32: Correct the coordinate points of the normalized coordinate plane based on the corresponding camera type.

[0104] For distorted images, the camera parameters of the image source camera are obtained, and the image distortion is calculated using the mathematical model corresponding to this type of camera. The coordinates of the coordinate points within the image range (xc / zc, yc / zc, 1) are corrected to obtain the corrected coordinate points as the new position (xp, yp, 1) corresponding to the bird's-eye view query vector in space.

[0105] For a non-distorted image, since its distortion is small and does not affect image processing, the coordinates of the points in the corresponding area (xc / zc, yc / zc, 1) can be directly used.

[0106] For example, an image acquired by a fisheye camera contains a coordinate point (3d / 1.5d, 5d / 1.5d, 1). The distortion of the image is calculated using a mathematical model that describes the projection and back-projection process of the fisheye camera to obtain a corrected coordinate point (3.2, 5.9, 1).

[0107] The coordinate point in the corrected distorted image can represent the real spatial position of this point in the image. Therefore, by correcting the distorted image, the bird's-eye view query vector corresponding to the coordinate point can be reflected in the real spatial position, which is conducive to improving its position accuracy in the bird's-eye view feature map.

[0108] Specifically, the camera type includes a fisheye camera.

[0109] Furthermore, step 32 also includes: calculating the coordinates of the distorted points for some coordinate points on the normalized coordinate plane corresponding to the fisheye camera, and using the coordinates of the distorted points as updated new coordinate points.

[0110] When the distorted image comes from a fisheye camera, the coordinates of the distorted points contained in the image acquired by the fisheye camera are calculated on the normalized coordinate plane, and the distorted point coordinates are used as the updated new coordinate points reflecting the real spatial position of the bird's-eye view query vector.

[0111] For example, the image acquired by the forward fisheye camera contains a coordinate point (3d / 1.5d, 5d / 1.5d, 1). The mathematical model kb model adapted for the fisheye camera is used to calculate the distortion of the image. The nonlinear projection method of the fisheye camera is approximated by assuming that there is a polynomial proportional relationship between the distance and angle from the optical center of the image to the projection point, and the coordinates of the distortion point after adding the distortion are obtained (3.2, 5.9, 1).

[0112] The mathematical model is used to add distortion to the coordinate points in the fisheye image, so that the coordinates of the distorted points after adding the distortion can reflect the real spatial position of the bird's-eye view query vector.

[0113] Step 33: Project the coordinate points of the corrected normalized coordinate plane back to the feature map to obtain a bird's-eye view feature map.

[0114] The coordinate points of the corrected normalized coordinate plane are projected back to the corresponding positions on the feature map, and the bird's-eye view query vector is projected to the corresponding position on the feature map through the positions of the corresponding coordinate points on the feature map, thereby forming a bird's-eye view feature map with enhanced features.

[0115] The process of projecting the bird's-eye view query vector back to the feature map is completed by transforming the coordinates of the coordinate points corresponding to the bird's-eye view query vector one by one, which strengthens the spatial position corresponding to the feature information contained in the feature map, so that the feature map can more clearly represent the environmental characteristics around the vehicle.

[0116] Furthermore, step 33 includes: multiplying the coordinate point of the corrected normalized coordinate plane with the corresponding camera intrinsic parameter and projecting it back to the feature map.

[0117] Get the camera intrinsic parameters of the image source camera in the area corresponding to the coordinate point, multiply the coordinate point (xp, yp, 1) or (xc / zc, yc / zc, 1) of the corrected normalized coordinate plane by the corresponding camera intrinsic parameters to obtain a new projection point (u, v, 1). The bird's-eye view query vector is projected back to the corresponding position on the feature map through the projection point (u, v, 1) to form a bird's-eye view feature map.

[0118] By multiplying the coordinate point of the normalized coordinate plane with the camera intrinsic parameter, the camera parameter information contained in the camera intrinsic parameter and the corresponding image position information are improved to improve the accuracy of the position of the bird's-eye view query vector in the image.

[0119] Step 40: Input the bird's-eye view feature map into the detector to predict the target image.

[0120] The bird's-eye view feature map is input into the corresponding detection head, and the detection head outputs the predicted three-dimensional target image.

[0121] By classifying and regressing each position or area in the feature map to determine the existence, position and size of the target, the detection head can predict the three-dimensional bounding box and category of the target, so that the target can be accurately identified and represented in three-dimensional space.

[0122] The target image prediction method further includes enhancing the forward blind area data of the distorted image before step 10.

[0123] Enhancing forward blind spot data is beneficial to improving the vehicle's forward perception capabilities and improving the acquired forward image information.

[0124] Specifically, see Figure 5 , Figure 5 : is a flow chart of the fifth embodiment of the target image prediction method provided by the present application, which enhances the forward blind area data of the distorted image, including:

[0125] Step 50: Randomly select a sampling position coordinate within the forward preset sampling distance.

[0126] In the forward space at a preset sampling distance from the ego vehicle, a position (x, y, z) in the three-dimensional space of the ego vehicle coordinate system is randomly generated as the sampling position coordinate.

[0127] The preset sampling distance setting can be adjusted according to the actual camera type selected or the forward perception requirements of the vehicle, such as within 0-4m, 1-3m, 0-8m, 2.2m-6.1m or 3-4m in the forward direction of the vehicle.

[0128] For example, within the preset sampling distance range of 0-4 m in the forward direction of the vehicle, a sampling position coordinate (1, 2, 1.5) is randomly generated.

[0129] By randomly generating a sampling position, the distortion of the sampling position is calculated to reflect the distortion of the entire image, which is beneficial to strengthening the forward blind area data of the image.

[0130] Step 60: Based on the randomly selected target in the mask library, a two-dimensional mask on the sampling position coordinates is formed.

[0131] A mask library of foreground objects is constructed, and a target is randomly selected from the mask library, and the two-dimensional mask corresponding to the target is set at the sampling position coordinates (x, y, z).

[0132] The mask library contains multiple mask images such as cones, triangles, lane lines or parking lot signs. The mask image is an auxiliary image used to achieve transparent mapping. The transparent part is usually white and the non-transparent part is black. The mask image is used in conjunction with the target image to achieve a transparent pasting effect of the foreground target through specific image processing technology.

[0133] A two-dimensional mask is formed at the sampling position to calculate the distortion of the sampling position to reflect the distortion of the entire image, which is beneficial to strengthening the forward blind area data of the image.

[0134] Further, see Figure 6 , Figure 6 6 is a flow chart of the sixth embodiment of the target image prediction method provided by the present application, wherein step 60 includes:

[0135] Step 61: Setting the three-dimensional graphics of the randomly selected target in the mask library at the sampling position coordinates.

[0136] The information of the 3D frame is preset for each target in the mask library, and the 3D graphics corresponding to the randomly selected target are set at the sampling position coordinates to form a 3D graphics at the position.

[0137] Step 62: Project the three-dimensional graphic back into the real image to obtain a two-dimensional graphic of the three-dimensional graphic in the real image.

[0138] The three-dimensional image is projected back into the image by the method in step 30 to obtain the maximum shape range of the three-dimensional figure on the two-dimensional plane as the two-dimensional figure (x1, y1, x2, y2) of the target.

[0139] Step 63: Scale the length and width of the two-dimensional mask corresponding to the target to be the same as the length and width of the two-dimensional graphic, and obtain the two-dimensional mask at the sampling position coordinates.

[0140] The two-dimensional mask corresponding to the target is scaled to the same size as the two-dimensional graph of the target (x2-x1, y2-y1), and a new two-dimensional mask at the sampling position coordinates is obtained.

[0141] Step 70: Add distortion to the two-dimensional mask based on the camera parameters.

[0142] Adds distortion to the scaled 2D mask based on the camera parameters of the image source camera.

[0143] Specifically, camera parameters include camera intrinsic parameters and camera extrinsic parameters. Camera extrinsic parameters can identify the position and orientation of the camera in the world coordinate system, which is used to construct the camera coordinate system for image transformation; camera intrinsic parameters include focal length, principal point, pixel scale and distortion parameters, etc., which are used to process the image to restore the real image information.

[0144] Further, see Figure 7 , Figure 7 is a flow chart of the seventh embodiment of the target image prediction method provided by the present application, and step 70 includes:

[0145] Step 71: Convert each pixel coordinate in the real image to the camera coordinate system, and calculate the radial distance of each pixel coordinate in the camera coordinate system.

[0146] The coordinates of each pixel in the real image are converted from the original coordinate system to the camera coordinate system. In the camera coordinate system, the points in the image can be calculated and changed by referring to the camera parameters, and the radial distance of each pixel coordinate can be calculated in the camera coordinate system.

[0147] By calculating the radial distance, the distortion parameter at the point can be reflected, thereby calculating the overall distortion of the image.

[0148] Step 72: Calculate the distorted coordinates of each pixel based on the camera parameters and the radial distance of each pixel coordinate.

[0149] The radial distance of each pixel point is calculated based on the camera parameters to obtain the distortion parameters at that point, thereby obtaining the coordinates of each pixel after the transformation.

[0150] Step 73: Map the coordinates of each pixel after distortion to the real image to obtain a distorted image.

[0151] The coordinates of each distorted pixel are mapped to the real image, so that the positional relationship in the real space passes through the distorted pixel coordinates to form the distorted positional relationship, that is, the distorted image.

[0152] Step 74: Crop based on the boundary of the non-zero area of ​​the distorted image to obtain a two-dimensional mask with added distortion.

[0153] By finding the boundary of non-zero pixels in the distorted image, the boundary of the cropping area is determined for cropping, and the black area at the image boundary caused by the distortion is removed. After the result is returned, a two-dimensional mask with added distortion is obtained.

[0154] Step 80: Paste the distorted two-dimensional mask onto the real image to obtain a virtual target corresponding to the target.

[0155] The distorted two-dimensional mask is pasted onto the real image to form a distorted virtual target on the real image, which corresponds to the target of the two-dimensional mask before distortion.

[0156] Step 90: Identify a target in the blind spot of the distorted image based on the virtual target, and enhance forward blind spot data of the distorted image.

[0157] By forming a two-dimensional mask with added distortion, a virtual forward target corresponding to the real target is generated. Other targets in the real image can be analogized to the conversion process between the real forward target and the virtual forward target, forming virtual targets corresponding to each target for blind spot recognition of distorted images, training the model's forward perception ability, and improving the ability to obtain forward image information.

[0158] See also Figure 8 , Figure 8 It is a structural diagram of an embodiment of the storage medium provided by the present application.

[0159] The storage medium 300 stores program data 301. When the program data 301 is executed by the processor, the following is achieved: Figure 1 The steps of the target image prediction method are described.

[0160] The program data 301 is stored in a storage medium 300, and includes a number of instructions for enabling a network device (such as a router, a personal computer, a server, or other network device) or a processor to execute all or part of the steps of the method described in each embodiment of the present application.

[0161] Optionally, the storage medium 300 may be a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, an optical disk, or other medium that can store program data.

[0162] See also Fig. 9 , Fig. 9 It is a structural schematic diagram of an embodiment of the vehicle-mounted device provided by the present application.

[0163] The vehicle-mounted device 400 is installed in the vehicle and is a computer-type device. It works in conjunction with various sensor modules and control modules in the vehicle to implement image feature extraction and fusion functions. The vehicle-mounted computer device 400 includes a processor 420 and a memory 410 that are interconnected. The memory 410 stores a computer program. When the processor 420 executes the computer program, the steps of the target image prediction method as described above are implemented.

[0164] Based on this, see Fig.10 The present application also provides a vehicle 500, including the above-mentioned vehicle-mounted device 400. The vehicle 500 can complete the target image prediction process through the control of the vehicle-mounted device 400 during driving. The vehicle 500 can be a smart vehicle equipped with a visual perception device, such as a fuel vehicle, a new energy vehicle, a simulated driving vehicle, or a special type of vehicle.

[0165] Different from the prior art, the present application provides a target image prediction method, storage medium, vehicle-mounted equipment and vehicle. By stacking and extracting features from images acquired by different cameras, feature information in the same image from a bird's-eye view is obtained, and the distorted image is corrected so that the spatial position of its features is more consistent with the position in the bird's-eye view, so that the bird's-eye view feature map contains more accurate feature information, avoiding the influence of distortion on the feature coordinate mapping of the bird's-eye view query vector, and avoiding image incompleteness caused by clipping distortion, so as to effectively obtain image information and enhance blind spot perception capabilities.

[0166] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the storage medium embodiment and the vehicle-mounted device embodiment, since they are basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0167] The present application can be used in many general or special vehicle-mounted computing system environments or configurations, such as personal computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, network PCs, minicomputers, and distributed computing environments including any of the above systems or devices.

[0168] In the several embodiments provided in this application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device embodiments described above are only illustrative, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not performed.

[0169] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0170] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0171] The above descriptions are merely embodiments of the present application and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A target image prediction method, characterized in that: include: Stacking at least one non-distorted image and at least one distorted image and inputting the stacked images into a feature extractor to obtain a feature map of a preset number of layers, wherein the non-distorted image and the distorted image are from different types of cameras; Get a bird's-eye view query vector within a preset range centered on the vehicle; Projecting the bird's-eye view query vector back to the feature map to form a bird's-eye view feature map; The bird's-eye view feature map is input into a detector to predict a target image.

2. The target image prediction method according to claim 1, characterized in that: The projecting the bird's-eye view query vector back to the feature map to form a bird's-eye view feature map comprises: Obtaining a normalized coordinate plane of the bird's-eye view query vector in a corresponding camera coordinate system; Correcting the coordinate points of the normalized coordinate plane based on the corresponding camera type; The corrected coordinate points of the normalized coordinate plane are projected back to the feature map to obtain the bird's-eye view feature map.

3. The target image prediction method according to claim 2, characterized in that: The camera type includes a fisheye camera; The correcting the coordinate points of the normalized coordinate plane based on the corresponding camera type includes: The distorted point coordinates are calculated for some coordinate points on the normalized coordinate plane corresponding to the fisheye camera, and the distorted point coordinates are used as updated new coordinate points; the distorted image originates from the fisheye camera.

4. The target image prediction method according to claim 2, characterized in that: The obtaining of a normalized coordinate plane of the bird's-eye view query vector in a corresponding camera coordinate system includes: The coordinate point of the bird's-eye view query vector is multiplied by the corresponding camera extrinsic parameter and normalized to obtain a normalized coordinate plane in the corresponding camera coordinate system; The projecting the corrected coordinate points of the normalized coordinate plane back to the feature map comprises: The corrected coordinate point of the normalized coordinate plane is multiplied by the corresponding camera intrinsic parameter and projected back to the feature map.

5. The target image prediction method according to claim 1, characterized in that: Before stacking the at least one non-distorted image with the at least one distorted image, the method further includes: The forward blind area data of the distorted image is enhanced.

6. The target image prediction method according to claim 5, characterized in that: The step of enhancing the forward blind area data of the distorted image includes: Randomly select a sampling position coordinate within the preset forward sampling distance; Based on the randomly selected target in the mask library, forming a two-dimensional mask on the sampling position coordinates; Adding distortion to the two-dimensional mask based on camera parameters, wherein the camera parameters include camera intrinsic parameters and camera extrinsic parameters; Pasting the two-dimensional mask with added distortion onto the real image to obtain a virtual target corresponding to the target; The target in the blind area of ​​the distorted image is identified based on the virtual target, and the forward blind area data of the distorted image is enhanced.

7. The target image prediction method according to claim 6, characterized in that: The forming of a two-dimensional mask on the sampling position coordinates based on a randomly selected target in a mask library includes: Setting the three-dimensional figure of the target randomly selected from the mask library on the sampling position coordinates; Projecting the three-dimensional graphic back into the real image to obtain a two-dimensional graphic of the three-dimensional graphic in the real image; The length and width of the two-dimensional mask corresponding to the target are scaled to be the same as the length and width of the two-dimensional graphic, so as to obtain the two-dimensional mask at the sampling position coordinates.

8. The target image prediction method according to claim 6, characterized in that: The adding distortion to the two-dimensional mask based on camera parameters includes: Convert each pixel coordinate in the real image to a camera coordinate system, and calculate the radial distance of each pixel coordinate in the camera coordinate system; Calculate each pixel coordinate after distortion based on camera parameters and the radial distance of each pixel coordinate; Mapping each of the distorted pixel coordinates onto the real image to obtain a distorted image; Cropping is performed based on the boundary of the non-zero region of the distorted image to obtain the two-dimensional mask with increased distortion.

9. The target image prediction method according to claim 1, characterized in that: The step of obtaining a bird's-eye view query vector within a preset range centered on the vehicle includes: Construct a grid of reference points within a preset range centered on the vehicle; Initializing a bird's-eye view query vector corresponding to each of the grid reference points using a word embedding module; An attention mechanism is adopted to process the bird's-eye view query vector.

10. The target image prediction method according to claim 9, characterized in that: The adopting the attention mechanism to process the bird's-eye view query vector includes: Input each of the bird's-eye view query vectors into a self-attention mechanism for processing; Each of the bird's-eye view query vectors and the corresponding feature map are input into the deformable cross attention for processing.

11. A storage medium having program data stored thereon, characterized in that: When the program data is executed by a processor, the steps of the target image prediction method according to any one of claims 1 to 10 are implemented.

12. A vehicle-mounted device, characterized in that: The method comprises a processor and a memory connected to each other, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the target image prediction method according to any one of claims 1 to 10 are implemented.

13. A vehicle, characterized in that: The vehicle includes the vehicle-mounted device according to claim 12.

Images