View angle conversion method and device, electronic equipment and storage medium

Abstract

The invention relates to the field of computer vision, and provides a visual angle conversion method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining a first number of images under a plurality of visual angles, and determining mutually independent image groups in spatial distribution from the images; and converting the image group to the same aerial view angle BEV space at the same time to obtain a second number of initial BEV images, and aggregating the initial BEV images to obtain a BEV image, the second number being smaller than the first number. According to the visual angle conversion method and device, the electronic equipment and the storage medium provided by the invention, the mutually independent image groups in spatial distribution are determined from the images under the multiple visual angles, and the image groups are converted to the same aerial view visual angle BEV space at the same time, so that the calculation redundancy is greatly reduced, the calculation amount is reduced, and the calculation efficiency is improved. Therefore, the overall time consumption of the visual conversion task is reduced, and the conversion efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a perspective conversion method, device, electronic device and storage medium. Background Art

[0002] Bird's Eye View (BEV) space plays a vital role in the field of autonomous driving and computer vision. It provides a top-down perspective that unifies data from different sensors into a common coordinate system, thus simplifying the understanding and processing of the environment. More and more visual perception technologies use BEV solutions to achieve 3D perception modeling of the surrounding environment.

[0003] Converting surround-view multi-camera input to BEV space is a key step in implementing the BEV solution. Usually, this process requires processing a large amount of image data, which is time-consuming. Therefore, there is an urgent need for a method to reduce the overall time consumption of visual conversion tasks, which will help accelerate the evolution and implementation of tasks such as autonomous driving. Summary of the invention

[0004] The present invention provides a perspective conversion method, device, electronic device and storage medium, which are used to solve the defect in the prior art that the conversion of surround view multi-camera input to BEV space is time-consuming.

[0005] The present invention provides a viewing angle conversion method, comprising: Acquire a first number of images at multiple viewing angles, and determine image groups that are independent of each other in spatial distribution from the images; The image groups are simultaneously converted to the same bird's-eye view BEV space to obtain a second number of initial BEV images, and the initial BEV images are aggregated to obtain BEV images, where the second number is smaller than the first number.

[0006] According to the perspective conversion method provided by the present invention, the step of determining, from the images, image groups that are independent of each other in spatial distribution comprises: Based on the shooting angle of view of the image, image groups that are independent of each other in spatial distribution are determined from the image.

[0007] According to the perspective conversion method provided by the present invention, the image group is converted to the same BEV space at the same time to obtain the initial BEV image, including: extracting image features of each image in the image group; Based on the internal and external parameters of the camera, determine the mapping relationship between the image coordinate system and the BEV coordinate system; Based on the mapping relationship, the image features are simultaneously mapped to the same BEV space to obtain the initial BEV image.

[0008] According to the perspective conversion method provided by the present invention, the image groups are simultaneously converted to the same bird's-eye view perspective BEV space to obtain a second number of initial BEV images, and the initial BEV images are aggregated to obtain the BEV images, including: The operators in the perspective conversion process are merged to obtain a perspective conversion operator, wherein the perspective conversion operator is used to simultaneously convert the image group to the same BEV space to obtain a second number of initial BEV images, and the initial BEV images are aggregated to obtain a BEV image; Based on the perspective conversion operator, the images under the multiple perspectives are read, perspective conversion is performed and the BEV image is saved.

[0009] According to the perspective conversion method provided by the present invention, the operators in the perspective conversion process are fused to obtain the perspective conversion operator, including: The sampling operator, the multiplication operator, the summation operator and the division operator in the operators are integrated to obtain the perspective conversion operator; The sampling operator and the multiplication operator are used to convert the image group into the same BEV space to obtain an initial BEV image, and the summation operator and the division operator are used to aggregate the initial BEV image to obtain a BEV image.

[0010] According to the perspective conversion method provided by the present invention, the image groups are simultaneously converted to the same bird's-eye view perspective BEV space to obtain a second number of initial BEV images, and the initial BEV images are aggregated to obtain the BEV images, including: Based on a vectorized processor, the image group is simultaneously converted to the same bird's-eye view BEV space through vectorized instructions to obtain a second number of initial BEV images, and the initial BEV images are aggregated to obtain the BEV image.

[0011] According to the perspective conversion method provided by the present invention, the vectorization processor is deployed on the terminal side device.

[0012] The present invention also provides a viewing angle conversion device, comprising: An image acquisition unit, configured to acquire a first number of images at multiple viewing angles, and determine image groups that are independent of each other in spatial distribution from the images; A perspective conversion unit is used to simultaneously convert the image groups to the same bird's-eye view perspective BEV space to obtain a second number of initial BEV images, and aggregate the initial BEV images to obtain BEV images, wherein the second number is smaller than the first number.

[0013] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, any of the above-mentioned perspective conversion methods is implemented.

[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the perspective conversion method described in any one of the above is implemented.

[0015] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned perspective conversion methods.

[0016] The perspective conversion method, device, electronic device and storage medium provided by the present invention obtain a first number of images under multiple perspectives, and determine a group of images that are independent of each other in spatial distribution from the images; the image groups are simultaneously converted to the same bird's-eye view perspective BEV space to obtain a second number of initial BEV images, and the initial BEV images are aggregated to obtain BEV images, and the second number is less than the first number. Compared with the related art in which each input image needs to be mapped to the BEV space, resulting in a large computational redundancy, this embodiment simultaneously converts a group of images that are independent of each other in spatial distribution to the same bird's-eye view perspective BEV space. By converting in this way, the redundant points in the obtained initial BEV image are greatly reduced, that is, the computational redundancy is greatly reduced.

[0017] At the same time, since the number of initial BEV images is less than the number of single images, the computational complexity of the step of aggregating the initial BEV images is correspondingly reduced, thereby reducing the overall time consumption of the visual conversion task and improving the conversion efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 This is one of the flow charts of the perspective conversion method provided by the present invention.

[0020] Figure 2 It is a mapping relationship diagram from image to BEV space in the related technology.

[0021] Figure 3 It is a mapping relationship diagram from the image group to the BEV space provided by the present invention.

[0022] Figure 4 It is a schematic diagram of coordinate transformation provided by the present invention.

[0023] Figure 5 It is a structural schematic diagram of the viewing angle conversion device provided by the present invention.

[0024] Figure 6 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0026] With the development of visual perception technology, the traditional 2D visual perception method with single camera input has gradually become unsuitable for the requirements of tasks such as autonomous driving and robotics. More and more visual perception technologies have adopted the BEV solution to achieve 3D perception modeling of the surrounding environment. Taking the evolution of autonomous driving visual models as an example, models such as BEVDet and Maptr are visual perception solutions based on the BEV space. These solutions use the data fusion capabilities provided by the BEV space to significantly improve the effect of visual perception. In these solutions, the conversion of surround multi-camera input to the BEV space is a key step in implementing the BEV solution. Usually this process requires processing a large amount of image data, which is time-consuming, so there is an urgent need to accelerate this process.

[0027] In view of the above problems, an embodiment of the present invention proposes a perspective conversion method, in which a first number of images under multiple perspectives are obtained, and image groups that are independent of each other in spatial distribution are determined from the images; the image groups are simultaneously converted to the same bird's-eye view BEV space to obtain a second number of initial BEV images, and the initial BEV images are aggregated to obtain BEV images, and the second number is less than the first number. Compared with the related art in which each input image needs to be mapped to the BEV space, resulting in a large computational redundancy, this embodiment simultaneously converts image groups that are independent of each other in spatial distribution to the same bird's-eye view BEV space. By converting in this way, the redundant points in the obtained initial BEV image are greatly reduced, that is, the computational redundancy is greatly reduced.

[0028] At the same time, since the number of initial BEV images is less than the number of single images, the computational complexity of the step of aggregating the initial BEV images is correspondingly reduced, thereby reducing the overall time consumption of the visual conversion task and improving the conversion efficiency.

[0029] The embodiments of the present invention can be applied to scenes that require visual conversion, especially scenes that convert surround multi-camera input to bird's-eye view vision, such as autonomous driving, robot navigation, monitoring systems and other fields. The execution subject of the method can be an electronic device such as a terminal device, a computer, a server, a server cluster or a specially designed scene switching device, or a visual conversion device set in the electronic device, which can be implemented by software, hardware or a combination of both.

[0030] In the description of the embodiments of the present invention, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0031] Figure 1 is one of the flow charts of the perspective conversion method provided by the present invention, such as Figure 1 As shown, the method comprises the following steps: Step 110 , obtaining a first number of images at multiple viewing angles, and determining image groups that are independent of each other in spatial distribution from the images.

[0032] Specifically, images from multiple perspectives can be acquired through an image acquisition device. Multiple cameras can be arranged around the target area, and these cameras can capture images from different angles and positions. For example, in an autonomous vehicle, cameras may be installed on the front, rear, sides, and top of the vehicle.

[0033] These cameras are started to capture images simultaneously or sequentially, ensuring that each camera can capture environmental information within its field of view, and these images constitute a first number of image sets. Taking six surround view cameras as an example, the first number can be 6. Of course, the first number can be flexibly adjusted according to application scenarios and requirements, such as the first number can be 4.

[0034] In addition, the captured images can be pre-processed, such as denoising, contrast enhancement, etc., to improve the accuracy and efficiency of subsequent processing.

[0035] In the related art, usually each image needs to be mapped to the BEV space. Taking the input of six surround view cameras as an example, six initial BEV images are obtained after conversion. For each initial BEV image, the corresponding points must be taken from the camera image and filled into the corresponding position. However, this process has a lot of redundancy because the image of each camera cannot completely cover a BEV space.

[0036] Figure 2 It is a mapping relationship diagram from image to BEV space in related technology, such as Figure 2 As shown in the figure, a is the mapping from the front view image to the BEV space, b is the mapping from the rear view image to the BEV space, c is the mapping from the left front view image to the BEV space, d is the mapping from the left rear view image to the BEV space, e is the mapping from the right front view image to the BEV space, and f is the mapping from the right rear view image to the BEV space. When the grid sample operator is used to extract features from the camera image, the uncovered area must also be operated, resulting in about 50% redundant operations.

[0037] In addition, due to the limited field of view of each camera, each voxel feature is very sparse, for example, only about 17% of the positions are non-zero. Because of their large scale, the computational cost of aggregating these voxel features is very large, so these voxel features need to be fused to generate a dense voxel feature to avoid time-consuming voxel aggregation.

[0038] In this embodiment, after acquiring images from multiple viewing angles, a spatially independent image group is determined from the images, that is, any two images in the image group are spatially independent. If the overlap between the two images is low and they each cover different parts of the target area, they are considered as spatially independent image groups. Any image group may include images from at least two viewing angles.

[0039] The determination of image groups can be achieved through image processing algorithms. Use image processing algorithms (such as SIFT, SURF, ORB, etc.) to extract feature points in each image, calculate the descriptors of the feature points, and use the feature point descriptors to find matching feature point pairs between different images. Use the RANSAC algorithm or other robustness estimation methods to calculate the transformation matrix (such as the perspective transformation matrix) between images. According to the transformation matrix, map the feature points in the image to the world coordinate system. Analyze the distribution of feature points in the world coordinate system to determine which image groups are independent in spatial distribution. Independence can be determined by calculating the overlap degree and distance relationship of feature points between image groups. According to the results of spatial distribution analysis, filter out image groups that are independent in spatial distribution. Ensure that the images in each image group can cover different spatial areas to avoid duplication and redundancy.

[0040] In other embodiments, the image group can also be determined by the shooting angles of each image. Images from different shooting angles usually cover different parts of the target area, and images with completely opposite shooting angles are usually independent of each other in spatial distribution. For example, the front view image and the rear view image can be regarded as an image group, the left front view and the right rear view images can be regarded as an image group, and the left rear view and the right front view images can be regarded as an image group.

[0041] Step 120 , convert the image groups to the same bird's-eye view BEV space at the same time to obtain a second number of initial BEV images, and aggregate the initial BEV images to obtain BEV images, the second number being less than the first number.

[0042] Specifically, after obtaining a group of images that are independent of each other in spatial distribution, the image group can be simultaneously converted to the same bird's-eye view BEV space to obtain a second number of initial BEV images. Since the images in the image group are independent of each other in spatial distribution and can cover different spatial regions, the conversion is performed in the form of an image group, and the redundant points in the initial BEV image are greatly reduced, that is, the computational redundancy is greatly reduced.

[0043] At the same time, during the conversion process, each BEV space must perform a sampling operation, that is, the original sampling operation required a first number of times is reduced to only a second number of sampling operations. The second number is significantly smaller than the first number, which significantly reduces the computational complexity of the sampling operation.

[0044] Then, the initial BEV images are aggregated to obtain BEV images. Aggregation can be achieved by summing the eigenvalues ​​at the same position in the BEV space and then averaging them. As the number of initial BEV images decreases, the computational complexity of the aggregation operation is reduced accordingly.

[0045] The method provided by the embodiment of the present invention determines image groups that are independent in spatial distribution from images under multiple perspectives, and simultaneously converts the image groups to the same bird's-eye view BEV space, which greatly reduces computational redundancy and the amount of calculation, thereby reducing the overall time consumption of the visual conversion task and improving the conversion efficiency.

[0046] Based on any of the above embodiments, determining the image groups that are independent of each other in spatial distribution from the images in step 110 includes: Based on the shooting angle of the image, image groups that are independent of each other in spatial distribution are determined from the image.

[0047] Specifically, images from different shooting angles usually cover different parts of the target area, and images from completely opposite shooting angles are usually independent of each other in spatial distribution. For example, the front view image and the rear view image can be regarded as an image group, the left front view and the right rear view image can be regarded as an image group, and the left rear view and the right front view image can be regarded as an image group.

[0048] Taking the input of six surround view cameras as an example, by observing the mapping relationship diagram from image to BEV space, there are multiple pairs of images whose pixels are non-overlapping after being mapped to BEV space, that is, they are independent of each other in spatial distribution. Therefore, when mapping images to BEV space, a pair of non-overlapping images can be mapped simultaneously for one BEV image. Figure 3 is a mapping relationship diagram from an image group to a BEV space provided by the present invention, such as Figure 3 As shown, a is the mapping of the image group consisting of the front view image and the rear view image to the BEV space, b is the mapping of the image group consisting of the right front view image and the left rear view image to the BEV space, and c is the mapping of the image group consisting of the left front view image and the right rear view image to the BEV space.

[0049] contrast Figure 2 and Figure 3 It can be seen that the redundant points in the initial BEV image are greatly reduced, that is, the computational redundancy is greatly reduced. At the same time, the conversion process is reduced from sampling operations in each BEV space to sampling operations in only three BEV spaces, and the computational complexity of this step is reduced by 50%. The subsequent operation is to aggregate multiple BEV spaces. Since the number of BEV spaces is reduced by 50%, the computational complexity of subsequent aggregation operations will also be reduced by 50%, so the overall computational complexity of the algorithm is reduced by 50%.

[0050] The method provided by the embodiment of the present invention can further reduce the overall amount of calculation by determining image groups that are independent of each other in spatial distribution from the image based on the shooting angle of the image.

[0051] Based on any of the above embodiments, in step 120, the image groups are simultaneously converted to the same BEV space to obtain an initial BEV image, including: Step 121, extracting image features of each image in the image group; Step 122, determining a mapping relationship between the image coordinate system and the BEV coordinate system based on the internal and external parameters of the camera; Step 123 , based on the mapping relationship, the image features are simultaneously mapped to the same BEV space to obtain an initial BEV image.

[0052] Specifically, in order to simultaneously convert an image group into the same BEV space and obtain an initial BEV image, we can first extract the features of each image in the image group through a convolutional neural network (CNN) model; then use the mapping equations constructed using the internal and external parameters of the camera to map the features of all positions in the image group to the corresponding positions in the BEV to obtain the initial BEV image.

[0053] Step 122 mainly involves the conversion from the image coordinate system to the BEV coordinate system, and all pixels in the image need to be mapped to positions in the BEV space. Figure 4 is a schematic diagram of the coordinate conversion provided by the present invention, such as Figure 4As shown in the figure, this process requires: 1. Converting the pixel coordinate system to the image coordinate system through discretization; 2. Converting the image coordinate system to the camera coordinate system through perspective relationship; 3. Converting the camera coordinate system to the world coordinate system through rigid body transformation.

[0054] In order to perform the above process, it is necessary to obtain the internal parameters of the camera (such as focal length, principal point position, etc.) to convert the image coordinate system to the camera coordinate system, which is usually obtained through camera calibration. The internal parameter matrix K can be expressed as formula (1), and is the focal length, and are the principal point coordinates.

[0055] (1) It is also necessary to obtain the camera's external parameters to convert the camera coordinate system to the world coordinate system. The external parameters describe the position and posture of the camera relative to the world coordinate system. The external parameters include the rotation matrix R and the translation vector t. For a point (u, v) in the image, it can be mapped to a point (X, Y, Z) in three-dimensional space through coordinate transformation. Using the internal parameter matrix K and the external parameter matrix [R|t], the projection matrix P can be constructed as shown in formula (2).

[0056] (2) For each pixel point (u, v) in the image, assuming that its corresponding depth is Z (usually determined by depth estimation or known height), its three-dimensional coordinates (X, Y, Z) in the world coordinate system can be calculated by formula (3).

[0057] (3) Project the three-dimensional coordinates (X, Y, Z) onto the BEV plane. Since BEV is a top-down perspective, we usually only care about the X and Y coordinates and ignore the Z coordinate.

[0058] Assume that the resolution of BEV is , and the spatial range of BEV has been defined (for example, from -L / 2 to L / 2, from -W / 2 to W / 2). Then, the three-dimensional coordinates (X, Y) can be transformed into the BEV coordinate system ( , ).

[0059] (4) (5) In order to speed up this process, it is usually assumed that the depth distribution of pixels is uniform in the ray direction. In this case, after obtaining the intrinsic and extrinsic parameter matrices, the mapping from the image coordinate system to the BEV coordinate system is determined. The mapping relationship can be solidified using the lookup table method. When the image coordinate system needs to be converted to the BEV coordinate system, the mapping relationship can be found from the lookup table. According to this mapping relationship, the image features can be moved to the specified position in the BEV space to complete the conversion. This process is usually implemented using the CPU or GPU through the Grid Sample sampling operator.

[0060] Based on any of the above embodiments, the image groups are simultaneously converted to the same bird's-eye view BEV space to obtain a second number of initial BEV images, and the initial BEV images are aggregated to obtain BEV images, that is, step 120 specifically includes: Step 124, fusing the operators in the perspective conversion process to obtain a perspective conversion operator, the perspective conversion operator is used to convert the image group to the same BEV space at the same time to obtain a second number of initial BEV images, and aggregating the initial BEV images to obtain a BEV image; Step 125 , based on the perspective conversion operator, read images under multiple perspectives, perform perspective conversion and save the BEV image.

[0061] Specifically, considering that the operators involved in the calculation process of the existing algorithms are all memory-intensive operators, each time the corresponding data is obtained, only simple calculations are performed and the data is written back. This computing mode is very unfriendly to large-scale data, and there are problems such as poor data locality, insufficient data reuse, and insufficient hardware utilization. If the data locality of the program is not good, that is, frequent access to discontinuous memory addresses, it may cause a large number of cache misses, thereby increasing the latency of memory access. If the data in the program is not fully reused, each calculation needs to read the data from the memory again, which will cause unnecessary memory access and increase the pressure on the memory bandwidth. Modern processors usually have features such as multi-level cache systems and SIMD instruction sets to improve data processing capabilities. If the memory access ratio is too low, these hardware resources may not be fully utilized, thereby affecting performance.

[0062] The process of image feature perspective conversion requires multiple steps, first converting all image points to BEV space, and then aggregating these points. The operators involved in these processes include sampling operators, multiplication operators, summation operators, and division operators.

[0063] Among them, the sampling operator involves the Grid Sample operation, which converts the points into the BEV space; the multiplication operator involves the multiplication operation, which sets all the values ​​of the redundant points to zero, and retains the original values ​​of the normally mapped points to obtain the initial BEV image; the summation operator involves the summation operation, which is used to sum the eigenvalues ​​at the same position in the BEV space; the division operator involves the division operation, which is used to average the features, thereby aggregating the initial BEV image to obtain the BEV image.

[0064] Each of the above operations requires data access before and after, but the calculation of each operation is very simple, which will lead to a large number of redundant memory accesses. At the same time, the computing component can quickly process the acquired data, so the computing component will be in a state of waiting for data most of the time, and the computing power utilization of the hardware is very low.

[0065] Usually, the number of circuit cycles consumed by memory access is tens to hundreds of times that of computing operations. Therefore, multiple calculations are performed on each acquired data to balance the time of memory access and computing, thereby forming a pipeline, making both the memory access component and the computing component busy, and improving the utilization of hardware. In order to achieve this goal, in this embodiment, all operators in the image feature perspective conversion process are fused, and these all operators are fused to generate a single perspective conversion operator.

[0066] When executing the calculation process, only the data required by the perspective conversion operator needs to be read, and then the conversion calculation is performed. After all calculations are completed, the results are saved. The fused operator saves multiple memory access operations, reduces the most time-consuming part of the calculation process by 3 / 4, and reduces the calculation time.

[0067] The method provided by the embodiment of the present invention obtains a perspective conversion operator by fusing various operators in the perspective conversion process. The fused operator saves multiple memory access operations and reduces calculation time.

[0068] Based on any of the above embodiments, the image groups are simultaneously converted to the same bird's-eye view BEV space to obtain a second number of initial BEV images, and the initial BEV images are aggregated to obtain BEV images, that is, step 120 specifically includes: Step 126 , based on a vectorized processor, the image group is simultaneously converted to the same bird's-eye view BEV space through vectorized instructions to obtain a second number of initial BEV images, and the initial BEV images are aggregated to obtain the BEV images.

[0069] Specifically, the basic principle of parallel computing acceleration is to reduce the overall computing time by breaking down a large task into multiple small tasks and allowing these small tasks to be executed simultaneously on different processing units.

[0070] The means of accelerating parallel computing mainly include technologies at the hardware and software levels. Different hardware involves different acceleration methods and principles. There are different hardware solutions that can accelerate parallel computing, such as multi-core processors, graphics processing units (GPUs), field programmable gate arrays (FPGAs), vector processors, etc.

[0071] Multi-core processors utilize multiple processing cores in modern CPUs, each of which can perform tasks independently, thereby achieving parallel processing; GPUs have a large number of parallel processing units (CUDA cores or stream processors), which are particularly suitable for large-scale data parallel processing; FPGAs are semi-custom circuits that can be programmed according to specific application requirements to achieve high optimization and parallelization of specific algorithms; vector processors can perform operations on data vectors in a single instruction cycle. There are means such as parallel algorithm design at the software level. Different parallel algorithms often have different computational loads, which affects the time consumption. At the same time, parallel algorithms need to adapt to different hardware to maximize the computing power of the hardware and approach the theoretical time consumption of the algorithm.

[0072] In the end-side environment, there are several problems with the use of multi-core processors, GPUs, and FPGAs, which affect the practical application of parallel methods on these hardware. Multi-core processors have the problem of relatively small acceleration. The acceleration ratio of parallel computing depends on the number of multi-core processors. In the end-side environment, the number of multi-core processors is often small, so it is difficult to achieve significant acceleration. GPUs have the problem of high power consumption and cost. The end-side environment often has power constraints, which greatly restricts the application of GPUs in the end-side environment. FPGAs have the problem of high development complexity, long development cycle, and high cost. In addition, the existing algorithms are not optimized for end-side devices and are still implemented based on hardware characteristics such as GPUs and CPUs. This will result in the inference time being much longer than the inference time on the server side when the algorithm is deployed on the end-side device.

[0073] In this implementation, a vectorized processor is used to parallelize and accelerate the algorithm, which allows the processor to perform the same operation on multiple data items simultaneously, greatly reducing the time consumption of the algorithm execution.

[0074] The deployment of vectorized processors on the end side (i.e., the device side, such as smartphones, IoT devices, etc.) has many advantages, especially high performance and high energy efficiency. Vectorized processors use vector instructions for calculations. Vectorized instructions use special vector registers to store data. These registers can accommodate multiple data elements and support operations on all elements at once. Taking addition as an example, the vectorized instruction can be expressed as in formula (6). Multiple elements in vector A and vector B will be added separately and saved to the corresponding positions in vector C. This process is equivalent to executing n instructions at the same time.

[0075] C =A +B (6) In the process of image feature perspective conversion, Grid Sample sampling operation, multiplication operation, and division operation are all calculated separately for a single element, while the sum operation is an aggregated calculation process. These operations are very suitable for vectorized instructions to accelerate. Therefore, these operations can be vectorized and accelerated separately, and then the operations can be fused, thus completing the vectorization acceleration of the entire calculation process and reducing the algorithm running time by several times.

[0076] Based on any of the above embodiments, a perspective conversion method is provided, including: S1, acquiring a first number of images at multiple viewing angles; and determining image groups that are independent of each other in spatial distribution from the images based on the shooting viewing angles of the images.

[0077] S2, converting the image group to the same bird's-eye view BEV space at the same time, obtaining a second number of initial BEV images, and aggregating the initial BEV images to obtain BEV images, the second number is smaller than the first number, so as to reduce the amount of calculation of the algorithm. Specifically including: The image features of each image in the image group are extracted; based on the internal and external parameters of the camera, the mapping relationship between the image coordinate system and the BEV coordinate system is determined; based on the mapping relationship, the image features are simultaneously mapped to the same BEV space to obtain the initial BEV image.

[0078] S3, fuses the operators in the perspective conversion process to obtain a perspective conversion operator, reads images under multiple perspectives based on the perspective conversion operator, performs perspective conversion and saves the BEV image, so as to eliminate redundant data access operations.

[0079] S4, based on the vectorized processor, converts the image group to the same bird's-eye view BEV space at the same time through vectorized instructions to obtain a second number of initial BEV images, and aggregates the initial BEV images to obtain the BEV image. The vectorized processor is deployed on the terminal device to reduce the time consumption of the algorithm operation.

[0080] The method provided by the embodiment of the present invention, firstly, designs a more efficient parallel computing algorithm based on the analysis of the calculation process of converting the surround multi-camera input to the bird's-eye view perspective, thereby optimizing the amount of calculation. Secondly, multiple operators in the calculation process are integrated, and the principle of data locality is used to improve the data reuse rate in the calculation process, thereby improving the calculation-to-memory access ratio at the hardware level and reducing the time consumption caused by memory access. Finally, the vector processing unit of the vector processor parallelizes the calculation process, so that multiple subtasks of the algorithm can be executed simultaneously, reducing the running time of the algorithm. This solution improves the efficiency of algorithm reasoning on the end-side chip, reduces the reasoning time of the overall solution, and accelerates the evolution and implementation of the algorithm.

[0081] The viewing angle conversion device provided by the present invention is described below. The viewing angle conversion device described below and the viewing angle conversion method described above can be referenced to each other.

[0082] Based on any of the above embodiments, Figure 5 is a schematic diagram of the structure of the viewing angle conversion device provided by the present invention, such as Figure 5 As shown, the viewing angle conversion device includes: An image acquisition unit 510 is used to acquire a first number of images at multiple viewing angles, and determine image groups that are independent of each other in spatial distribution from the images; The perspective conversion unit 520 is used to simultaneously convert the image groups to the same bird's-eye view perspective BEV space to obtain a second number of initial BEV images, and aggregate the initial BEV images to obtain BEV images, where the second number is smaller than the first number.

[0083] The device provided by the embodiment of the present invention determines image groups that are independent in spatial distribution from images under multiple perspectives, and simultaneously converts the image groups to the same bird's-eye view BEV space, which greatly reduces computational redundancy and the amount of calculation, thereby reducing the overall time consumption of the visual conversion task and improving the conversion efficiency.

[0084] Based on any of the above embodiments, the image acquisition unit is specifically used for: Based on the shooting angle of view of the image, image groups that are independent of each other in spatial distribution are determined from the image.

[0085] Based on any of the above embodiments, the viewing angle conversion unit is specifically used for: extracting image features of each image in the image group; Based on the internal and external parameters of the camera, determine the mapping relationship between the image coordinate system and the BEV coordinate system; Based on the mapping relationship, the image features are simultaneously mapped to the same BEV space to obtain the initial BEV image.

[0086] Based on any of the above embodiments, the viewing angle conversion unit is specifically used for: The operators in the perspective conversion process are merged to obtain a perspective conversion operator, wherein the perspective conversion operator is used to simultaneously convert the image group to the same BEV space to obtain a second number of initial BEV images, and the initial BEV images are aggregated to obtain a BEV image; Based on the perspective conversion operator, the images under the multiple perspectives are read, perspective conversion is performed and the BEV image is saved.

[0087] Based on any of the above embodiments, the viewing angle conversion unit is specifically used for: The sampling operator, the multiplication operator, the summation operator and the division operator in the operators are integrated to obtain the perspective conversion operator; The sampling operator and the multiplication operator are used to convert the image group into the same BEV space to obtain an initial BEV image, and the summation operator and the division operator are used to aggregate the initial BEV image to obtain a BEV image.

[0088] Based on any of the above embodiments, the viewing angle conversion unit is specifically used for: Based on a vectorized processor, the image group is simultaneously converted to the same bird's-eye view BEV space through vectorized instructions to obtain a second number of initial BEV images, and the initial BEV images are aggregated to obtain the BEV image.

[0089] Based on any of the foregoing embodiments, the vectorization processor is deployed on a terminal-side device.

[0090] Figure 6 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 6 As shown, the electronic device may include: a processor 610, a communication interface 620, a memory 630 and a communication bus 640, wherein the processor 610, the communication interface 620 and the memory 630 communicate with each other through the communication bus 640. The processor 610 may call the logic instructions in the memory 630 to execute the perspective conversion method, which includes: acquiring a first number of images under multiple perspectives, and determining image groups that are independent of each other in spatial distribution from the images; converting the image groups to the same bird's-eye view perspective BEV space at the same time to obtain a second number of initial BEV images, and aggregating the initial BEV images to obtain BEV images, wherein the second number is less than the first number.

[0091] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0092] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the perspective conversion method provided by the above-mentioned methods, which includes: acquiring a first number of images under multiple perspectives, and determining image groups that are independent of each other in spatial distribution from the images; simultaneously converting the image groups to the same bird's-eye view perspective BEV space to obtain a second number of initial BEV images, and aggregating the initial BEV images to obtain BEV images, wherein the second number is less than the first number.

[0093] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the perspective conversion method provided by the above-mentioned methods, the method comprising: acquiring a first number of images under multiple perspectives, and determining image groups that are independent of each other in spatial distribution from the images; simultaneously converting the image groups to the same bird's-eye view perspective BEV space to obtain a second number of initial BEV images, and aggregating the initial BEV images to obtain BEV images, the second number being less than the first number.

[0094] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0095] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0096] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A perspective conversion method, characterized in that: include: Acquire a first number of images at multiple viewing angles, and determine image groups that are independent of each other in spatial distribution from the images; The image groups are simultaneously converted to the same bird's-eye view BEV space to obtain a second number of initial BEV images, and the initial BEV images are aggregated to obtain BEV images, where the second number is smaller than the first number.

2. The perspective conversion method according to claim 1, characterized in that: The step of determining, from the images, image groups that are independent of each other in spatial distribution comprises: Based on the shooting angle of view of the image, image groups that are independent of each other in spatial distribution are determined from the image.

3. The perspective conversion method according to claim 2, characterized in that: The step of simultaneously converting the image groups to the same bird's-eye view (BEV) space to obtain a second number of initial BEV images includes: extracting image features of each image in the image group; Based on the internal and external parameters of the camera, determine the mapping relationship between the image coordinate system and the BEV coordinate system; Based on the mapping relationship, the image features are simultaneously mapped to the same BEV space to obtain the initial BEV image.

4. The viewing angle conversion method according to any one of claims 1 to 3, characterized in that: The step of converting the image groups to the same bird's-eye view BEV space to obtain a second number of initial BEV images, and aggregating the initial BEV images to obtain the BEV images includes: The operators in the perspective conversion process are merged to obtain a perspective conversion operator, wherein the perspective conversion operator is used to simultaneously convert the image group to the same BEV space to obtain a second number of initial BEV images, and the initial BEV images are aggregated to obtain a BEV image; Based on the perspective conversion operator, the images under the multiple perspectives are read, perspective conversion is performed and the BEV image is saved.

5. The viewing angle conversion method according to claim 4, characterized in that: The operator in the perspective conversion process is fused to obtain a perspective conversion operator, including: The sampling operator, the multiplication operator, the summation operator and the division operator in the operators are integrated to obtain the perspective conversion operator; The sampling operator and the multiplication operator are used to convert the image group into the same BEV space to obtain an initial BEV image, and the summation operator and the division operator are used to aggregate the initial BEV image to obtain a BEV image.

6. The viewing angle conversion method according to any one of claims 1 to 3, characterized in that: The step of converting the image groups to the same bird's-eye view BEV space to obtain a second number of initial BEV images, and aggregating the initial BEV images to obtain the BEV images includes: Based on a vectorized processor, the image group is simultaneously converted to the same bird's-eye view BEV space through vectorized instructions to obtain a second number of initial BEV images, and the initial BEV images are aggregated to obtain the BEV image.

7. The viewing angle conversion method according to claim 6, characterized in that: The vectorization processor is deployed on the terminal device.

8. A viewing angle conversion device, characterized in that: include: An image acquisition unit, configured to acquire a first number of images at multiple viewing angles, and determine image groups that are independent of each other in spatial distribution from the images; A perspective conversion unit is used to simultaneously convert the image groups to the same bird's-eye view perspective BEV space to obtain a second number of initial BEV images, and aggregate the initial BEV images to obtain BEV images, wherein the second number is smaller than the first number.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the perspective conversion method according to any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the perspective conversion method as described in any one of claims 1 to 7 is implemented.

Images