Bird-eye view generation method and device, storage medium and electronic equipment

Abstract

The invention discloses an aerial view generation method and device, a storage medium and electronic equipment, and relates to the technical field of image processing. The method comprises the steps that a target parameter set, a surround view and an initial aerial view are acquired, and the target parameter set is used for representing the quantization process and the inverse quantization process of the surround view and the initial aerial view; dividing the surround view based on a preset thread to obtain an initial surround view feature block, and dividing the initial aerial view based on the preset thread to obtain an initial aerial view data block; adjusting the initial surround view feature block according to the target parameter group to obtain a target surround view feature block; mapping the target surround view feature block to the initial aerial view data block to obtain a target aerial view data block; and generating a target aerial view based on the target aerial view data block. According to the invention, the technical problems of large resource consumption and relatively large operation delay when the aerial view is generated in the prior art are solved.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more specifically, to a method, apparatus, storage medium, and electronic device for generating bird's-eye view images. Background Technology

[0002] A bird's-eye view (BEV) is a view of an area of ​​interest from a high vantage point, using a high-viewpoint perspective. Due to its intuitiveness, ease of understanding, and ease of integration with other sensors, BEV is widely used in autonomous driving and robotics. Optical cameras, due to their low cost, low noise, and high density, are the preferred sensing components for autonomous driving tasks. BEVs are typically generated from the front view captured by the optical camera (i.e., in the camera coordinate system), such as... Figure 1 As shown. However, converting the front view to BEV requires a large amount of memory operations or multiplication and accumulation operations, making it difficult for chips with low computing power and low bandwidth to run the BEV perception algorithm.

[0003] Currently, there are two methods for generating BEVs. One is the forward BEV generation method, such as Lift Shift and Short (LSS), which first explicitly predicts the depth, then samples spatial voxels based on the depth distribution, and finally compresses the voxels into BEV features. However, this method requires estimating multiple depths for each pixel, resulting in high computational and storage resource consumption. Furthermore, after predicting and generating depth information, it is necessary to sample the depth information to generate the BEV feature map, a step that is limited by bandwidth bottlenecks. The other BEV generation method first generates a BEV mesh, then projects the BEV mesh onto the forward view image, and finally samples the projected area. In this method, to improve accuracy, each BEV feature point corresponds to multiple sampling points in the toroidal view, thus being limited by the bandwidth of the edge chip, resulting in significant sampling computation latency.

[0004] There is currently no effective solution to the above problems. Summary of the Invention

[0005] This invention provides a method, apparatus, storage medium, and electronic device for generating bird's-eye view images, in order to at least solve the technical problems of high resource consumption and large computational latency when generating bird's-eye view images in related technologies.

[0006] According to one embodiment of the present invention, a method for generating a bird's-eye view is provided. The method may include: acquiring a target parameter set, a surround view, and an initial bird's-eye view, wherein the target parameter set is used to represent the quantization and dequantization processes of the surround view and the initial bird's-eye view; dividing the surround view based on a preset thread to obtain initial surround view feature blocks, and dividing the initial bird's-eye view based on a preset thread to obtain initial bird's-eye view data blocks; adjusting the initial surround view feature blocks according to the target parameter set to obtain target surround view feature blocks; mapping the target surround view feature blocks to the initial bird's-eye view data blocks to obtain target bird's-eye view data blocks; and generating a target bird's-eye view based on the target bird's-eye view data blocks.

[0007] Optionally, the following steps are taken: obtaining a target parameter set, a ring view, and an initial bird's-eye view, wherein the target parameter set represents the quantization and dequantization processes of the ring view and the initial bird's-eye view; dividing the ring view based on a preset thread to obtain initial ring view feature blocks, and dividing the initial bird's-eye view based on a preset thread to obtain initial bird's-eye view data blocks; adjusting the initial ring view feature blocks according to the target parameter set to obtain target ring view feature blocks; mapping the target ring view feature blocks to the initial bird's-eye view data blocks to obtain target bird's-eye view data blocks; and generating the target bird's-eye view based on the target bird's-eye view data blocks.

[0008] Optionally, obtaining the target parameter set includes: obtaining the first zoom value and the first offset value of the panorama view, and the second zoom value and the second offset value of the initial bird's-eye view; and performing a fusion process on the first zoom value, the first offset value, the second zoom value, and the second offset value to obtain the target parameter set.

[0009] Optionally, fusing the first scaling value, the first offset value, the second scaling value, and the second offset value to obtain the target parameter set includes: calculating the ratio of the first scaling value to the second scaling value to obtain a first ratio, and calculating the ratio of the second scaling value to the first scaling value to obtain a second ratio; performing a first product calculation on the first ratio to obtain a first product result, and performing a second product calculation on the second ratio to obtain a second product result; rounding the first product result to obtain a first rounded result, and rounding the second product result to obtain a second rounded result; vectorizing the first rounded result to obtain a first vector, and vectorizing the second rounded result to obtain a second vector; and determining the target parameter set based on the first vector and the second vector.

[0010] Optionally, performing a first product calculation on the first ratio to obtain the first product result includes: determining a first value; multiplying the first value by the first ratio to obtain the first product result.

[0011] Optionally, performing a second product calculation on the second ratio to obtain the second product result includes: determining a second value; multiplying the second ratio by a second offset value to obtain a first calculation result; subtracting the first calculation result from the first offset value to obtain a second calculation result; and multiplying the second value by the second calculation result to obtain the second product result.

[0012] Optionally, dividing the panorama view based on a preset thread to obtain initial panorama view feature blocks, and dividing the initial bird's-eye view based on a preset thread to obtain initial bird's-eye view data blocks, includes: dividing the panorama view in parallel based on a preset thread to obtain initial panorama view feature blocks, and dividing the initial bird's-eye view in parallel based on a preset thread to obtain initial bird's-eye view data blocks.

[0013] Optionally, adjusting the initial surround view feature block according to the target parameter set to obtain the target surround view feature block includes: converting the initial surround view feature block to a data type to obtain a first surround view feature block; performing a first shift operation on the first surround view feature block to obtain a second surround view feature block; adding the second surround view feature block to a second vector to obtain a third surround view feature block; multiplying the third surround view feature block to a first vector to obtain a fourth surround view feature block; and performing a second shift operation on the fourth surround view feature block to obtain the target surround view feature block.

[0014] Optionally, mapping the target panoramic view feature block to the initial bird's-eye view data block to obtain the target bird's-eye view data block includes: mapping the target panoramic view feature block to the corresponding initial bird's-eye view data block based on a preset thread to obtain the target bird's-eye view data block.

[0015] According to one embodiment of the present invention, a bird's-eye view generation apparatus is also provided. The apparatus may include: an acquisition module, configured to acquire a target parameter set, a surround view, and an initial bird's-eye view, wherein the target parameter set represents the quantization and dequantization processes of the surround view and the initial bird's-eye view; a partitioning module, configured to partition the surround view based on a preset thread to obtain initial surround view feature blocks, and to partition the initial bird's-eye view based on a preset thread to obtain initial bird's-eye view data blocks; an adjustment module, configured to adjust the initial surround view feature blocks according to the target parameter set to obtain target surround view feature blocks; a mapping module, configured to map the target surround view feature blocks to the initial bird's-eye view data blocks to obtain target bird's-eye view data blocks; and a generation module, configured to generate a target bird's-eye view based on the target bird's-eye view data blocks.

[0016] Optionally, the acquisition module is also used to acquire the first zoom value and the first offset value of the panoramic view, as well as the second zoom value and the second offset value of the initial bird's-eye view; and to perform fusion processing on the first zoom value, the first offset value, the second zoom value and the second offset value to obtain the target parameter group.

[0017] Optionally, the acquisition module is further configured to calculate the ratio of the first scaling value to the second scaling value to obtain a first ratio, and calculate the ratio of the second scaling value to the first scaling value to obtain a second ratio; perform a first product calculation on the first ratio to obtain a first product result, and perform a second product calculation on the second ratio to obtain a second product result; perform a rounding operation on the first product result to obtain a first rounded result, and perform a rounding operation on the second product result to obtain a second rounded result; perform vectorization operation on the first rounded result to obtain a first vector, and perform vectorization operation on the second rounded result to obtain a second vector; and determine the target parameter set based on the first vector and the second vector.

[0018] Optionally, the acquisition module is also used to determine a first value; multiply the first value by a first ratio to obtain a first product result.

[0019] Optionally, the acquisition module is further configured to determine a second value; multiply the second ratio by the second offset value to obtain a first calculation result; subtract the first calculation result from the first offset value to obtain a second calculation result; and multiply the second value by the second calculation result to obtain a second product result.

[0020] Optionally, the partitioning module is also used to partition the panorama view in parallel based on a preset thread to obtain an initial panorama view feature block, and to partition the initial bird's-eye view in parallel based on a preset thread to obtain an initial bird's-eye view data block.

[0021] Optionally, the adjustment module is further configured to perform data type conversion on the initial surround view feature block to obtain a first surround view feature block; perform a first shift operation on the first surround view feature block to obtain a second surround view feature block; add the second surround view feature block to a second vector to obtain a third surround view feature block; multiply the third surround view feature block to a first vector to obtain a fourth surround view feature block; and perform a second shift operation on the fourth surround view feature block to obtain a target surround view feature block.

[0022] Optionally, the mapping module is also used to map the target panoramic view feature block to the corresponding initial bird's-eye view data block based on a preset thread, so as to obtain the target bird's-eye view data block.

[0023] According to one embodiment of the present invention, a computer-readable storage medium is also provided, wherein a computer program is stored in the computer-readable storage medium, wherein the computer program is configured to execute the bird's-eye view generation method of the present invention when run on a computer or processor.

[0024] According to one embodiment of the present invention, a vehicle is also provided, which is used to perform the bird's-eye view generation method in the embodiments of the present invention.

[0025] According to one embodiment of the present invention, an electronic device is also provided, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to execute the bird's-eye view generation method in the embodiments of the present invention.

[0026] In this embodiment of the invention, a target parameter set, a surround view, and an initial bird's-eye view are obtained. The target parameter set represents the quantization and dequantization processes of the surround view and the initial bird's-eye view. The surround view is divided based on a preset thread to obtain initial surround view feature blocks, and the initial bird's-eye view is divided based on a preset thread to obtain initial bird's-eye view data blocks. The initial surround view feature blocks are adjusted according to the target parameter set to obtain target surround view feature blocks. The target surround view feature blocks are mapped to the initial bird's-eye view data blocks to obtain target bird's-eye view data blocks. The target bird's-eye view is then generated based on the target bird's-eye view data blocks. This achieves the goal of converting a surround view into a bird's-eye view with fewer resources and lower bandwidth, realizing a fast and efficient bird's-eye view generation technique. It can effectively alleviate bandwidth bottlenecks, achieving a trade-off between computation and bandwidth resource consumption, thereby improving overall operating speed and solving the technical problems of high resource consumption and large computational latency when generating bird's-eye views in related technologies. Attached Figure Description

[0027] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0028] Figure 1 This is a schematic diagram of converting a front view into a bird's-eye view;

[0029] Figure 2 This is a flowchart of a method for generating a bird's-eye view according to one embodiment of the present invention;

[0030] Figure 3 This is a schematic diagram of a 4-thread parallel optimization according to one embodiment of the present invention;

[0031] Figure 4 This is a flowchart of another method for generating a bird's-eye view according to one embodiment of the present invention;

[0032] Figure 5 This is a structural block diagram of a bird's-eye view generation device according to one embodiment of the present invention. Detailed Implementation

[0033] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0034] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0035] According to one embodiment of the present invention, an embodiment of a bird's-eye view generation method is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0036] This method embodiment can be executed in an electronic device, similar control device, or system that includes a memory and a processor, located in a vehicle. Taking an electronic device as an example, the electronic device may include one or more processors and a memory for storing data. Optionally, the electronic device may also include a communication device for communication functions. Those skilled in the art will understand that the above structural description is merely illustrative and does not limit the structure of the electronic device. For example, the electronic device may include more or fewer components than described above, or have a different configuration than described above.

[0037] A processor may include one or more processing units. For example, a processor may include a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processing (DSP) chip, a microcontroller unit (MCU), a field-programmable gate array (FPGA), a neural network processing unit (NPU), a tensor processing unit (TPU), or an artificial intelligence (AI) processor. Different processing units may be independent components or integrated into one or more processors. In some instances, electronic devices may also include one or more processors.

[0038] The memory can be used to store computer programs, such as the computer program corresponding to the bird's-eye view generation method in this embodiment of the invention. The processor implements the bird's-eye view generation method by running the computer program stored in the memory. The memory may include high-speed random access memory and non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to electronic devices via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0039] Communication equipment is used to receive or send data via a network. Specific examples of the network mentioned above may include a wireless network provided by the vehicle terminal's communication provider. In one example, the communication equipment includes a network interface controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the communication equipment may be a radio frequency (RF) module, used for wireless communication with the Internet.

[0040] This embodiment provides a method for generating bird's-eye view images running on an electronic device. Figure 2 This is a flowchart of a bird's-eye view generation method according to one embodiment of the present invention, such as... Figure 2 As shown, the process includes the following steps:

[0041] Step S20: Obtain the target parameter set, the ring view, and the initial bird's-eye view, wherein the target parameter set is used to represent the quantization process and the dequantization process of the ring view and the initial bird's-eye view.

[0042] The target parameter set is a parameter set obtained by fusing the quantization and dequantization parameters of the panorama view and the initial bird's-eye view. It is used to represent the process after fusing the quantization and dequantization processes of the panorama view and the initial bird's-eye view.

[0043] The panorama view can include a front view taken by at least one optical camera. That is, the panorama view can be a single image or multiple images. For example, when there is only one camera, the panorama view is a single front view. There is no limitation here.

[0044] The initial bird's-eye view is used to represent the BEV view coordinates in order to determine the size range of the final bird's-eye view.

[0045] For example, both the BEV view and the ring view can be stored in a storage device as an int8 data type, such as in memory or on a hard disk, without limitation.

[0046] Step S22: Divide the panorama view based on a preset thread to obtain an initial panorama view feature block, and divide the initial bird's-eye view based on a preset thread to obtain an initial bird's-eye view data block.

[0047] The preset number of threads can be customized and is not limited here.

[0048] The feature blocks of the ring view and the data blocks of the initial bird's-eye view can be divided equally according to a preset number of threads, without any restrictions here. For example, in order to improve the efficiency of memory operations, the ring view and the initial bird's-eye view can be divided in parallel according to a preset number of threads, but the data parallelism should not be too low. In order to improve memory access efficiency, it is best to ensure that the storage space of the ring view and the BEV view is contiguous.

[0049] It is understood that there is no fixed start and end order between threads. In order to improve the efficiency of multi-threaded computing, this embodiment of the invention may not use a fixed filling order. That is, if the pixels of multiple ring views are mapped to the pixels of the same BEV view, then only the last one or a few pixels can be saved among these multiple ring view pixels to reduce inefficient thread synchronization operations.

[0050] For example, the initial ring view feature blocks and initial bird's-eye view data blocks obtained from the above division can be stored in the corresponding registers, which is not limited here.

[0051] Figure 3 This is a schematic diagram of a 4-thread parallel optimization according to one embodiment of the present invention, such as... Figure 3 As shown, Figure 3 This demonstrates a four-thread approach to partitioning the toroidal view, where each row represents a single image within the toroidal view, and each column represents a BEV feature point. It can be seen that... Figure 3 There are a total of 6 ring views and 10,000 BEV feature points. Each color represents an element processed by a thread.

[0052] Step S24: Adjust the initial ring view feature block according to the target parameter group to obtain the target ring view feature block.

[0053] Adjusting the pixels of the initial toroidal view feature block based on the obtained target parameter set yields the adjusted target toroidal view feature block, which can be understood as having undergone quantization and dequantization operations.

[0054] Step S26: Map the target panoramic view feature block to the initial bird's-eye view data block to obtain the target bird's-eye view data block.

[0055] The target panoramic view feature block is mapped to the initial bird's-eye view data block to obtain the target bird's-eye view data block.

[0056] For example, the mapping relationship between the BEV view and the loop view can be constructed using methods such as the inverse projection matrix (IPM) and transformer, and there are no restrictions here.

[0057] Step S28: Generate a target bird's-eye view based on the target bird's-eye view data blocks.

[0058] The target bird's-eye view is generated based on the obtained target bird's-eye view data blocks, thereby enabling the conversion of a panoramic view into a bird's-eye view. This method is more suitable for tasks with limited bandwidth resources and surplus computing resources, achieving a trade-off between memory access and computation, which is beneficial to improving overall operating efficiency.

[0059] Through the above steps, the target parameter set, the toroidal view, and the initial bird's-eye view are obtained. The target parameter set represents the quantization and dequantization processes of the toroidal view and the initial bird's-eye view. The toroidal view is divided based on a preset thread to obtain initial toroidal view feature blocks, and the initial bird's-eye view is divided based on the preset thread to obtain initial bird's-eye view data blocks. The initial toroidal view feature blocks are adjusted according to the target parameter set to obtain target toroidal view feature blocks. These target toroidal view feature blocks are mapped to the initial bird's-eye view data blocks to obtain target bird's-eye view data blocks. Finally, the target bird's-eye view is generated based on these data blocks. This achieves the goal of converting a toroidal view into a bird's-eye view with fewer resources and lower bandwidth, realizing a fast and efficient bird's-eye view generation technique. It effectively alleviates bandwidth bottlenecks, achieving a trade-off between computational and bandwidth resource consumption, thereby improving overall operating speed and solving the technical problems of high resource consumption and large computational latency in related technologies when generating bird's-eye views.

[0060] Optionally, obtaining the target parameter set in step S20 may include the following execution steps:

[0061] Step S200: Obtain the first zoom value and the first offset value of the panorama view, and the second zoom value and the second offset value of the initial bird's-eye view;

[0062] Step S201: The first scaling value, the first offset value, the second scaling value, and the second offset value are fused to obtain the target parameter group.

[0063] You can use a per-tensor linear quantization mode, where each tensor uses a quantization scaling factor and an offset factor.

[0064] The first zoom and first offset values ​​of the toroidal view can be understood as the scale and offset of the toroidal view, denoted as f respectively. s and f o .

[0065] The second zoom and second offset values ​​of the initial bird's-eye view can be understood as the scale and offset of the BEV view, respectively, v s and v o .

[0066] It is understandable that f s and v s It is a floating-point number, f o and v o It is an integer.

[0067] By analyzing f s f o v s and v oBy performing fusion processing, the target parameter set can be obtained.

[0068] Optionally, in step S201, fusing the first scaling value, the first offset value, the second scaling value, and the second offset value to obtain the target parameter set may include the following steps:

[0069] Step S2010: Calculate the ratio of the first scaling value to the second scaling value to obtain the first ratio; and calculate the ratio of the second scaling value to the first scaling value to obtain the second ratio.

[0070] Step S2011: Perform a first product calculation on the first ratio to obtain the first product result, and perform a second product calculation on the second ratio to obtain the second product result;

[0071] Step S2012: Round the first product result to obtain a first rounded result, and round the second product result to obtain a second rounded result;

[0072] Step S2013: Vectorize the first rounding result to obtain the first vector, and vectorize the second rounding result to obtain the second vector;

[0073] Step S2014: Determine the target parameter set based on the first vector and the second vector.

[0074] When performing the fusion process, the ratio of the scale of the loop view to the scale of the BEV view is first calculated to obtain the first ratio. And calculate its reciprocal to obtain the second ratio. It is understandable that the first and second ratios obtained are both floating-point numbers, which are scalar operations.

[0075] Then, by performing a first product calculation on the first ratio, which expands it to a larger floating-point number, we obtain the first product result. Similarly, by performing a second product calculation on the second ratio, which expands it to a larger floating-point number, we obtain the second product result. For example, the first and second ratios can each be multiplied by a large power of 2 integer, such as 2^28. p and 2 q The first product result is obtained. Second product result

[0076] It is important to note that the result of the first product and the result of the second product must both be much greater than 1 and much less than 2. 31 This step is also a scalar operation.

[0077] Then, the first and second product results are rounded to obtain the first and second rounded results. For example, rounding can be done directly, or rounding can be performed using operations such as round, floor, and ceil to obtain the first rounded result. Second round result This step is also a scalar operation.

[0078] Then, the first and second rounding results are vectorized to obtain the first vector and the second vector, respectively. For example, this can be extended by data parallelization (i.e., vectorization) of the first and second rounding results. and Perform vectorization expansion on each vector to obtain the first vector vec. fv Second vector vec vf And store it in a register.

[0079] Finally, based on the first vector vec fv Second vector vec vf Obtain the target parameter set.

[0080] For example, the above quantification parameter steps can be represented by formula (1).

[0081]

[0082] Where x represents a feature point of the input ring view, that is, a pixel of the ring view.

[0083] Optionally, in step S2011, calculating the first product of the first ratio to obtain the first product result may include the following steps:

[0084] Step S20110: Determine the first value;

[0085] Step S20111: Multiply the first value by the first ratio to obtain the first product result.

[0086] For example, the first value being an integer that is a power of 2 can be denoted as 2. p The first product result is obtained by multiplying the first value by the first ratio.

[0087] Optionally, in step S2011, calculating the second product of the second ratio to obtain the second product result may include the following steps:

[0088] Step S20112, determine the second value;

[0089] Step S20113: Multiply the second ratio by the second offset value to obtain the first calculation result;

[0090] Step S20114: Subtract the first calculation result from the first offset value to obtain the second calculation result;

[0091] Step S20115: Multiply the second value by the second calculation result to obtain the second product result.

[0092] For example, the second value being an integer that is a power of 2 can be denoted as 2. q The first calculation result is obtained by multiplying the second ratio by the second offset value. Then, subtract the first calculation result from the first offset value to obtain the second calculation result. Then multiply the second value by the second calculation result to obtain the second product result.

[0093] Optionally, in step S22, dividing the panorama view based on a preset thread to obtain initial panorama view feature blocks, and dividing the initial bird's-eye view based on a preset thread to obtain initial bird's-eye view data blocks may include the following execution steps:

[0094] Step S220: The panorama view is divided in parallel based on a preset thread to obtain the initial panorama view feature block, and the initial bird's-eye view is divided in parallel based on a preset thread to obtain the initial bird's-eye view data block.

[0095] This means that the loop view and the initial bird's-eye view can be divided in parallel based on a preset thread to improve the efficiency of memory operations.

[0096] Optionally, in step S24, adjusting the initial ring view feature block according to the target parameter set to obtain the target ring view feature block may include the following execution steps:

[0097] Step S240: Convert the data type of the initial ring view feature block to obtain the first ring view feature block;

[0098] Step S241: Perform a first shifting process on the first annular view feature block to obtain the second annular view feature block;

[0099] Step S242: Add the second ring view feature block to the second vector to obtain the third ring view feature block;

[0100] Step S243: Multiply the third ring view feature block with the first vector to obtain the fourth ring view feature block;

[0101] Step S244: Perform a second shifting process on the fourth ring view feature block to obtain the target ring view feature block.

[0102] For example, when adjusting the initial toroidal view feature block according to the target parameter set to obtain the target toroidal view feature block, the initial toroidal view feature block can first be read from the corresponding storage unit. Then, the data type of the initial toroidal view feature block can be converted from int8 to int32 to obtain the first toroidal view feature block. Next, the first toroidal view feature block is shifted left by q bits to obtain the second toroidal view feature block. Finally, the second toroidal view feature block is compared with the second vector vec. vf Add them together to obtain the third ring view feature block. Then multiply the third ring view feature block by the first vector vec. fv The fourth ring view feature block is obtained. Finally, the fourth ring view feature block is shifted to the right by (q+p) bits to obtain the target ring view feature block. The target ring view feature block, after being adjusted based on the target parameter set, is then written into the corresponding storage unit.

[0103] It is understandable that the operations in steps S240 to S244 above are all performed using vectors.

[0104] For example, the steps described above for adjusting the initial ring view feature blocks can be represented by formula (2).

[0105] y = [vec fv ·(x< <q+vec vf Formula (2) >> (q+p)

[0106] In this context, >> represents a right shift operation for integers, << represents a left shift operation, and · represents the Hadamard operation.

[0107] It is understandable that the initial toroidal view feature blocks are adjusted in parallel, and the calculation results are written to the storage unit in a data-parallel manner. If the data processing is complete, the operation ends; otherwise, the remaining initial toroidal view feature blocks are adjusted.

[0108] Optionally, in step S26, mapping the target panoramic view feature block to the initial bird's-eye view data block to obtain the target bird's-eye view data block may include the following execution steps:

[0109] Step S260: Based on a preset thread, the target panoramic view feature block is mapped to the corresponding initial bird's-eye view data block to obtain the target bird's-eye view data block.

[0110] For example, when mapping the target panoramic view feature block to the initial bird's-eye view data block, the target panoramic view feature block stored in the storage unit can be mapped to the storage unit corresponding to the thread according to the divided thread, that is, mapped to the corresponding initial bird's-eye view data block, so as to obtain the target bird's-eye view data block.

[0111] The target bird's-eye view data block can be of the int8 data type; there are no restrictions here.

[0112] Figure 4 This is a flowchart of another bird's-eye view generation method according to one embodiment of the present invention, such as... Figure 4 As shown, both the BEV view and the ring view are stored in int8 format on a storage device (which can be memory, hard disk, etc.). First, the BEV view and ring view are divided according to a preset number of threads. Each thread reads a data block into a register in a data-parallel manner. Then, the quantization and dequantization parameters of the BEV view and ring view are read, and the quantization parameter A of the ring view and the quantization parameter B of the BEV view are merged into a single parameter set (of type fp32, int8). Subsequently, the merged parameter is used to adjust the aforementioned data block stored in the register, calculating a new quantization value. Finally, the adjusted data block is saved to the storage unit corresponding to that thread, and the saved data is still of int8 data type, thus obtaining the BEV view.

[0113] Since each int8 data type occupies only one byte, while each float32 data type occupies four bytes, using int8 to store the BEV view and ring view saves storage space and reduces the amount of data accessed, thereby reducing bandwidth pressure. The quantization parameters of the BEV view and the ring view are generally different, requiring data adjustment based on both sets of quantization parameters during computation. However, on data-parallel computing chips, the conversion between floating-point and fixed-point numbers is time-consuming, so some chips do not provide data-parallel instructions for cross-data type conversion between floating-point and fixed-point numbers, resulting in low efficiency in quantization and dequantization. The solution in this embodiment of the invention utilizes quantization parameter fusion technology to solve this problem.

[0114] As can be seen, the quantization operation in this embodiment of the invention can significantly reduce the amount of data accessed to memory. For example, quantizing from FP32 to int8 can compress the bandwidth requirement for reading and writing memory to 1 / 4 of the original.

[0115] Meanwhile, the data parallel operation in this embodiment of the invention can improve memory access efficiency and the computational efficiency of quantization and dequantization. By pre-calculating the scale ratio, this invention can reduce the operations of the scalar arithmetic unit and the number of synchronizations between the scalar and vector arithmetic units. By pre-saving fixed intermediate results of vector operations to registers, read / write latency can be reduced. Furthermore, by using integer shift operations, division operations can be avoided, further reducing computational latency.

[0116] Furthermore, compared to scalar computation, the speedup effect of data parallelism is related to the degree of data parallelism. For example, on Qualcomm chips that support HVX, data parallelism can achieve a speedup of 32 times, using 4-thread parallelism can achieve a speedup of nearly 4 times, and combining the two can achieve a speedup of over 100 times.

[0117] Therefore, the present invention proposes a fast and efficient BEV view generation method for data-parallel chips. By using numerical quantization technology to reduce memory access bandwidth and using multi-threading optimization and data parallel technology to improve the efficiency of BEV feature generation, a higher speedup ratio is achieved.

[0118] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause the air pressure adjustment system of a vehicle or air conditioner to execute the methods described in the various embodiments of the present invention.

[0119] This embodiment also provides a bird's-eye view generation device, which is used to implement the above embodiments and preferred embodiments, and will not be repeated as already described. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0120] Figure 5 This is a structural block diagram of a bird's-eye view generation device according to one embodiment of the present invention, such as... Figure 5 As shown, taking a bird's-eye view generation device 500 as an example, the device includes: an acquisition module 501, used to acquire a target parameter set, a surround view, and an initial bird's-eye view, wherein the target parameter set is used to represent the quantization process and the dequantization process of the surround view and the initial bird's-eye view; a partitioning module 502, used to partition the surround view based on a preset thread to obtain an initial surround view feature block, and to partition the initial bird's-eye view based on a preset thread to obtain an initial bird's-eye view data block; an adjustment module 503, used to adjust the initial surround view feature block according to the target parameter set to obtain a target surround view feature block; a mapping module 504, used to map the target surround view feature block to the initial bird's-eye view data block to obtain a target bird's-eye view data block; and a generation module 505, used to generate a target bird's-eye view based on the target bird's-eye view data block.

[0121] Optionally, the acquisition module 501 is further configured to acquire the first zoom value and the first offset value of the panoramic view, as well as the second zoom value and the second offset value of the initial bird's-eye view; and to perform fusion processing on the first zoom value, the first offset value, the second zoom value and the second offset value to obtain the target parameter group.

[0122] Optionally, the acquisition module 501 is further configured to calculate the ratio of the first scaling value to the second scaling value to obtain a first ratio, and calculate the ratio of the second scaling value to the first scaling value to obtain a second ratio; perform a first product calculation on the first ratio to obtain a first product result, and perform a second product calculation on the second ratio to obtain a second product result; perform a rounding operation on the first product result to obtain a first rounded result, and perform a rounding operation on the second product result to obtain a second rounded result; perform vectorization operation on the first rounded result to obtain a first vector, and perform vectorization operation on the second rounded result to obtain a second vector; and determine the target parameter set based on the first vector and the second vector.

[0123] Optionally, the acquisition module 501 is further configured to determine a first value; multiply the first value by a first ratio to obtain a first product result.

[0124] Optionally, the acquisition module 501 is further configured to determine a second value; multiply the second ratio by the second offset value to obtain a first calculation result; subtract the first calculation result from the first offset value to obtain a second calculation result; and multiply the second value by the second calculation result to obtain a second product result.

[0125] Optionally, the partitioning module 502 is also used to partition the panorama view in parallel based on a preset thread to obtain an initial panorama view feature block, and to partition the initial bird's-eye view in parallel based on a preset thread to obtain an initial bird's-eye view data block.

[0126] Optionally, the adjustment module 503 is further configured to perform data type conversion on the initial surround view feature block to obtain a first surround view feature block; perform a first shift operation on the first surround view feature block to obtain a second surround view feature block; add the second surround view feature block to a second vector to obtain a third surround view feature block; multiply the third surround view feature block to a first vector to obtain a fourth surround view feature block; and perform a second shift operation on the fourth surround view feature block to obtain a target surround view feature block.

[0127] Optionally, the mapping module 504 is also used to map the target panoramic view feature block to the corresponding initial bird's-eye view data block based on a preset thread, so as to obtain the target bird's-eye view data block.

[0128] It should be noted that the above modules can be implemented by software or hardware. For the latter, they can be implemented in the following ways, but are not limited to: all the above modules are located in the same processor; or, the above modules are located in different processors in any combination.

[0129] Embodiments of the present invention also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to perform the steps in any of the above method embodiments when run on a computer or processor.

[0130] Optionally, in this embodiment, the computer-readable storage medium may be configured to store a computer program for performing the following steps:

[0131] Step S1: Obtain the target parameter set, the ring view, and the initial bird's-eye view, wherein the target parameter set is used to represent the quantization and dequantization processes of the ring view and the initial bird's-eye view;

[0132] Step S2: Divide the ring view based on a preset thread to obtain the initial ring view feature block, and divide the initial bird's-eye view based on a preset thread to obtain the initial bird's-eye view data block;

[0133] Step S3: Adjust the initial ring view feature block according to the target parameter group to obtain the target ring view feature block;

[0134] Step S4: Map the target panoramic view feature block to the initial bird's-eye view data block to obtain the target bird's-eye view data block;

[0135] Step S5: Generate a target bird's-eye view based on the target bird's-eye view data blocks.

[0136] Optionally, in this embodiment, the computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0137] Embodiments of the present invention also provide a vehicle for performing the bird's-eye view generation method described in any of the preceding claims.

[0138] Optionally, in this embodiment, the vehicle may perform the following steps:

[0139] Step S1: Obtain the target parameter set, the ring view, and the initial bird's-eye view, wherein the target parameter set is used to represent the quantization and dequantization processes of the ring view and the initial bird's-eye view;

[0140] Step S2: Divide the ring view based on a preset thread to obtain the initial ring view feature block, and divide the initial bird's-eye view based on a preset thread to obtain the initial bird's-eye view data block;

[0141] Step S3: Adjust the initial ring view feature block according to the target parameter group to obtain the target ring view feature block;

[0142] Step S4: Map the target panoramic view feature block to the initial bird's-eye view data block to obtain the target bird's-eye view data block;

[0143] Step S5: Generate a target bird's-eye view based on the target bird's-eye view data blocks.

[0144] Embodiments of the present invention also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0145] Optionally, in this embodiment, the processor in the above-described electronic device may be configured to run a computer program to perform the following steps:

[0146] Step S1: Obtain the target parameter set, the ring view, and the initial bird's-eye view, wherein the target parameter set is used to represent the quantization and dequantization processes of the ring view and the initial bird's-eye view;

[0147] Step S2: Divide the ring view based on a preset thread to obtain the initial ring view feature block, and divide the initial bird's-eye view based on a preset thread to obtain the initial bird's-eye view data block;

[0148] Step S3: Adjust the initial ring view feature block according to the target parameter group to obtain the target ring view feature block;

[0149] Step S4: Map the target panoramic view feature block to the initial bird's-eye view data block to obtain the target bird's-eye view data block;

[0150] Step S5: Generate a target bird's-eye view based on the target bird's-eye view data blocks.

[0151] Optionally, specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated here.

[0152] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0153] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0154] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0155] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0156] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0157] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0158] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for generating a bird's-eye view, characterized in that, include: Obtain a target parameter set, a ring view, and an initial bird's-eye view, wherein the target parameter set is used to represent the quantization and dequantization processes of the ring view and the initial bird's-eye view; The panorama view is divided based on a preset thread to obtain an initial panorama view feature block, and the initial bird's-eye view is divided based on the preset thread to obtain an initial bird's-eye view data block. Adjust the initial ring view feature block according to the target parameter set to obtain the target ring view feature block; The target panoramic view feature block is mapped to the initial bird's-eye view data block to obtain the target bird's-eye view data block; A target bird's-eye view is generated based on the target bird's-eye view data blocks.

2. The method according to claim 1, characterized in that, Obtaining the target parameter set includes: Obtain the first zoom value and the first offset value of the panorama view, and the second zoom value and the second offset value of the initial bird's-eye view; The first scaling value, the first offset value, the second scaling value, and the second offset value are fused to obtain the target parameter group.

3. The method according to claim 2, characterized in that, The process of fusing the first scaling value, the first offset value, the second scaling value, and the second offset value to obtain the target parameter set includes: Calculate the ratio of the first scaling value to the second scaling value to obtain a first ratio, and calculate the ratio of the second scaling value to the first scaling value to obtain a second ratio; The first ratio is multiplied by a first product to obtain a first product result, and the second ratio is multiplied by a second product to obtain a second product result; The first product result is rounded down to obtain a first rounded result, and the second product result is rounded down to obtain a second rounded result; The first rounding result is vectorized to obtain a first vector, and the second rounding result is vectorized to obtain a second vector; The target parameter set is determined based on the first vector and the second vector.

4. The method according to claim 3, characterized in that, The calculation of the first product on the first ratio to obtain the first product result includes: Determine the first value; Multiply the first value by the first ratio to obtain the first product result.

5. The method according to claim 3, characterized in that, The calculation of the second product on the second ratio to obtain the second product result includes: Determine the second value; Multiply the second ratio by the second offset value to obtain the first calculation result; Subtract the first calculation result from the first offset value to obtain the second calculation result; Multiply the second value by the second calculation result to obtain the second product result.

6. The method according to claim 1, characterized in that, The process of dividing the panorama view based on a preset thread to obtain initial panorama view feature blocks, and dividing the initial bird's-eye view based on the preset thread to obtain initial bird's-eye view data blocks, includes: The panorama view is divided in parallel based on a preset thread to obtain the initial panorama view feature block, and the initial bird's-eye view is divided in parallel based on the preset thread to obtain the initial bird's-eye view data block.

7. The method according to claim 3, characterized in that, The step of adjusting the initial ring view feature block according to the target parameter set to obtain the target ring view feature block includes: The initial ring view feature block is converted to a new data type to obtain the first ring view feature block. The first annular view feature block is subjected to a first shifting process to obtain the second annular view feature block; Add the second ring view feature block to the second vector to obtain the third ring view feature block; Multiply the third ring view feature block by the first vector to obtain the fourth ring view feature block; The fourth ring view feature block is subjected to a second shifting process to obtain the target ring view feature block.

8. A bird's-eye view generation device, characterized in that, include: The acquisition module is used to acquire a target parameter set, a ring view, and an initial bird's-eye view, wherein the target parameter set is used to represent the quantization process and the dequantization process of the ring view and the initial bird's-eye view; The segmentation module is used to segment the panorama based on a preset thread to obtain an initial panorama feature block, and to segment the initial bird's-eye view based on the preset thread to obtain an initial bird's-eye view data block. The adjustment module is used to adjust the initial ring view feature block according to the target parameter set to obtain the target ring view feature block; The mapping module maps the target panoramic view feature block to the initial bird's-eye view data block to obtain the target bird's-eye view data block; The generation module is used to generate a target bird's-eye view based on the target bird's-eye view data block.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein the computer program is configured to execute the bird's-eye view generation method as described in any one of claims 1 to 7 when run on a computer or processor.

10. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to run the computer program to perform the bird's-eye view generation method as described in any one of claims 1 to 7.

Images