Point cloud projection transformation method and device based on matrix splitting, equipment and medium

Abstract

The embodiment of the invention discloses a point cloud projection transformation method and device based on matrix splitting, equipment and a medium, and the method comprises the steps: carrying out the multi-view image collection of a target scene through a plurality of image collection devices disposed on a movable carrier, and obtaining a plurality of scene images; splitting the projection matrix corresponding to the image acquisition equipment to obtain a plurality of split matrixes; wherein the size of each split matrix is the same as that of the projection matrix; determining image coordinates of spatial points in a three-dimensional point cloud in the scene image based on the plurality of split matrixes; wherein the three-dimensional point cloud corresponds to the target scene; based on the image coordinates, determining point cloud features corresponding to spatial points in the three-dimensional point cloud, and obtaining a target three-dimensional point cloud with the point cloud features; according to the embodiment of the invention, the calculation overhead is effectively reduced, the calculation efficiency is improved, and the calculation precision is ensured.

Description

Technical Field

[0001] The present disclosure relates to the field of machine vision perception technology, and in particular to a point cloud projection transformation method, device, equipment and medium based on matrix splitting. Background Art

[0002] The rapid development of autonomous driving technology has made the perception system of autonomous driving vehicles increasingly important. The role of the perception system is to provide accurate and comprehensive external environment information for autonomous driving vehicles, including roads, vehicles, pedestrians, obstacles, etc., so that the autonomous driving system can make effective decisions and actions to ensure the safety and stability of the vehicle. The perception system collects spatial information through a variety of sensors, such as lidar, millimeter wave radar, ultrasonic sensors, etc. At the same time, it is also a common method to extract three-dimensional information from multi-view two-dimensional cameras using multi-view consistency. These 3D information from different sources provide the system with more comprehensive perception capabilities. This 3D information is usually represented in the form of point clouds.

[0003] Autonomous driving perception algorithms need to be deployed on edge computing devices with limited computing power, such as vehicles. In order to increase the running speed of the algorithm, neural network quantization technology is usually used to replace floating-point data in the neural network with integer data and floating-point operations with integer operations, which can reduce the storage and computing overhead of running the neural network. However, due to the limitation of the number of register bits of the inference chip, converting floating-point data into integer data and subsequent calculations will bring large cutoff errors and rounding errors to the projection process. Summary of the invention

[0004] In order to solve the above technical problems, the present disclosure is proposed. The embodiments of the present disclosure provide a point cloud projection transformation method, device, equipment and medium based on matrix decomposition.

[0005] According to one aspect of an embodiment of the present disclosure, a point cloud projection transformation method based on matrix decomposition is provided, comprising:

[0006] Capturing multi-view images of a target scene by using a plurality of image acquisition devices arranged on a movable carrier to obtain a plurality of scene images;

[0007] Splitting the projection matrix corresponding to the image acquisition device to obtain a plurality of split matrices; wherein each split matrix has the same size as the projection matrix;

[0008] Based on the multiple split matrices, determining the image coordinates of the spatial points in the three-dimensional point cloud in the scene image; wherein the three-dimensional point cloud corresponds to the target scene;

[0009] Based on the image coordinates, point cloud features corresponding to the spatial points in the three-dimensional point cloud are determined to obtain a target three-dimensional point cloud with point cloud features.

[0010] Optionally, the splitting process of the projection matrix corresponding to the image acquisition device to obtain a plurality of split matrices includes:

[0011] Splitting each element in the projection matrix into a plurality of sub-elements with the same position;

[0012] The plurality of sub-elements corresponding to the same bit position are determined as a split matrix to obtain a plurality of split matrices.

[0013] Optionally, splitting each element in the projection matrix into a plurality of sub-elements with the same position includes:

[0014] Determining a sign bit of each element in the projection matrix;

[0015] Splitting each element in the projection matrix according to bits to obtain multiple partial elements;

[0016] The sign bit corresponding to each of the partial elements is determined based on the sign bit to obtain the multiple sub-elements.

[0017] Optionally, determining the image coordinates of the spatial points in the three-dimensional point cloud in the scene image based on the multiple split matrices includes:

[0018] Processing the spatial point by each splitting matrix of the multiple splitting matrices to obtain the three-dimensional coordinates of the spatial point in the device coordinate system corresponding to the image acquisition device;

[0019] Based on the three-dimensional coordinates, the image coordinates corresponding to the spatial point in the scene image are determined.

[0020] Optionally, the processing of the spatial point by each splitting matrix in the multiple splitting matrices to obtain the three-dimensional coordinates of the spatial point in the device coordinate system corresponding to the image acquisition device includes:

[0021] Perform matrix multiplication on each of the multiple split matrices and the spatial point to obtain multiple intermediate results; each of the intermediate results corresponds to one split matrix;

[0022] Based on the multiple intermediate results, the three-dimensional coordinates of the spatial point in the device coordinate system are determined.

[0023] Optionally, determining the three-dimensional coordinates of the spatial point in the device coordinate system based on the multiple intermediate results includes:

[0024] determining a first intermediate result and at least one second intermediate result from the plurality of intermediate results;

[0025] Reducing the at least one second intermediate result by a preset multiple to obtain at least one processed third intermediate result; wherein the preset multiples corresponding to each of the second intermediate results are different;

[0026] Based on the first intermediate result and the at least one third intermediate result, a three-dimensional coordinate of the spatial point in the device coordinate system is determined.

[0027] Optionally, determining the image coordinates corresponding to the spatial point in the scene image based on the three-dimensional coordinates includes:

[0028] The three-dimensional coordinates are converted into an image coordinate system corresponding to the scene image through coordinate system conversion to obtain the image coordinates corresponding to the spatial point in the scene image.

[0029] Optionally, determining point cloud features corresponding to spatial points in the three-dimensional point cloud based on the image coordinates to obtain a target three-dimensional point cloud having point cloud features includes:

[0030] Determining whether the image coordinates meet preset conditions;

[0031] In response to the image coordinates meeting the preset condition, obtaining a corresponding feature vector from a feature map corresponding to the scene image based on the image coordinates;

[0032] The feature vector is determined as a point cloud feature of a spatial point in the three-dimensional point cloud to obtain the target three-dimensional point cloud.

[0033] Optionally, before the splitting process of the projection matrix corresponding to the image acquisition device to obtain a plurality of split matrices, the process further includes:

[0034] Performing a first magnification process on the floating-point projection matrix corresponding to the image acquisition device;

[0035] The enlarged floating-point projection matrix is ​​integerized to obtain the projection matrix.

[0036] According to another aspect of an embodiment of the present disclosure, a point cloud projection transformation device based on matrix decomposition is provided, comprising:

[0037] An image acquisition module is used to acquire multiple scene images by performing multi-viewing image acquisition on a target scene through multiple image acquisition devices arranged on a movable carrier;

[0038] A matrix splitting module, used for splitting the projection matrix corresponding to the image acquisition device to obtain a plurality of split matrices; wherein each split matrix has the same size as the projection matrix;

[0039] An image mapping module, configured to determine image coordinates of spatial points in a three-dimensional point cloud in the scene image based on the multiple split matrices; wherein the three-dimensional point cloud corresponds to the target scene;

[0040] The point cloud projection module is used to determine the point cloud features corresponding to the spatial points in the three-dimensional point cloud based on the image coordinates, and obtain a target three-dimensional point cloud with point cloud features.

[0041] Optionally, the matrix splitting module includes:

[0042] An element splitting unit, used for splitting each element in the projection matrix into a plurality of sub-elements with the same position;

[0043] The matrix unit is used to determine the multiple sub-elements corresponding to the same bit position as a split matrix to obtain multiple split matrices.

[0044] Optionally, the element splitting unit is specifically used to determine the sign bit of each element in the projection matrix; split each element in the projection matrix according to the bit position to obtain multiple partial elements; determine the sign bit corresponding to each of the partial elements based on the sign bit to obtain the multiple sub-elements.

[0045] Optionally, the image mapping module includes:

[0046] A device projection unit, used for processing the spatial point through each splitting matrix of the multiple splitting matrices to obtain the three-dimensional coordinates of the spatial point in the device coordinate system corresponding to the image acquisition device;

[0047] The image projection unit is used to determine the image coordinates corresponding to the spatial point in the scene image based on the three-dimensional coordinates.

[0048] Optionally, the device projection unit is specifically used to perform matrix multiplication on each of the multiple split matrices with the spatial point to obtain multiple intermediate results; each intermediate result corresponds to one split matrix; based on the multiple intermediate results, determine the three-dimensional coordinates of the spatial point in the device coordinate system.

[0049] Optionally, when the device projection unit determines the three-dimensional coordinates of the spatial point in the device coordinate system based on the multiple intermediate results, it is specifically used to determine a first intermediate result and at least one second intermediate result from the multiple intermediate results; reduce the at least one second intermediate result by a preset multiple to obtain at least one processed third intermediate result; wherein the preset multiples corresponding to each second intermediate result are different; and determine the three-dimensional coordinates of the spatial point in the device coordinate system based on the first intermediate result and the at least one third intermediate result.

[0050] Optionally, the image projection unit is specifically configured to transform the three-dimensional coordinates into an image coordinate system corresponding to the scene image through coordinate system transformation, so as to obtain image coordinates corresponding to the spatial point in the scene image.

[0051] Optionally, the point cloud projection module is specifically used to determine whether the image coordinates meet preset conditions; in response to the image coordinates meeting the preset conditions, obtain a corresponding feature vector from a feature map corresponding to the scene image based on the image coordinates; determine the feature vector as a point cloud feature of a spatial point in the three-dimensional point cloud, and obtain the target three-dimensional point cloud.

[0052] Optionally, the device further comprises:

[0053] The matrix preprocessing module is used to magnify the floating-point projection matrix corresponding to the image acquisition device by a first multiple; and to integerize the amplified floating-point projection matrix to obtain the projection matrix.

[0054] According to another aspect of the embodiments of the present disclosure, there is provided an electronic device, including:

[0055] A memory for storing a computer program product;

[0056] The processor is used to execute the computer program product stored in the memory, and when the computer program product is executed, the method described in any one of the above embodiments is implemented.

[0057] According to another aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the method described in any of the above embodiments is implemented.

[0058] Based on the above-mentioned embodiments of the present disclosure, a point cloud projection transformation method, device, equipment and medium based on matrix splitting are provided. A plurality of image acquisition devices arranged on a movable carrier are used to perform multi-view image acquisition on a target scene to obtain a plurality of scene images; the projection matrix corresponding to the image acquisition device is split to obtain a plurality of split matrices; wherein each split matrix is ​​the same size as the projection matrix; based on the plurality of split matrices, the image coordinates of the spatial points in the three-dimensional point cloud in the scene image are determined; wherein the three-dimensional point cloud corresponds to the target scene; based on the image coordinates, the point cloud features corresponding to the spatial points in the three-dimensional point cloud are determined to obtain a target three-dimensional point cloud with point cloud features; the embodiments of the present disclosure split the projection matrix when projecting the point cloud, and realize point cloud projection by splitting the matrix. While improving the calculation accuracy, the projection calculation process reduces the computing power requirements of the computing chip, and can be applied to computing chips with smaller computing power, thereby expanding the application scenarios that point cloud projection can adapt to.

[0059] The technical solution of the present disclosure is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] The accompanying drawings, which constitute a part of the specification, illustrate embodiments of the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0061] The present disclosure may be more clearly understood from the following detailed description with reference to the accompanying drawings, in which:

[0062] Figure 1 It is a flowchart of a point cloud projection transformation method based on matrix decomposition provided by an exemplary embodiment of the present disclosure;

[0063] Figure 2 This disclosure Figure 1 A schematic flow chart of step 104 in the embodiment shown;

[0064] Figure 3 This disclosure Figure 1 A schematic flow chart of step 106 in the embodiment shown;

[0065] Figure 4 This disclosure Figure 1 A schematic flow chart of step 108 in the embodiment shown;

[0066] Figure 5 It is a structural schematic diagram of a point cloud projection transformation device based on matrix decomposition provided by an exemplary embodiment of the present disclosure;

[0067] Figure 6 A block diagram of an electronic device according to an embodiment of the present disclosure is illustrated. DETAILED DESCRIPTION

[0068] Below, the exemplary embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all the embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the exemplary embodiments described here.

[0069] It should be noted that the relative arrangement of components and steps, the numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure unless specifically stated otherwise.

[0070] Those skilled in the art can understand that the terms "first" and "second" in the embodiments of the present disclosure are only used to distinguish different steps, devices or modules, etc., and neither represent any specific technical meaning nor indicate the necessary logical order between them.

[0071] It should also be understood that in the embodiments of the present disclosure, “plurality” may refer to two or more than two, and “at least one” may refer to one, two, or more than two.

[0072] It should also be understood that any component, data or structure mentioned in the embodiments of the present disclosure can generally be understood as one or more, unless explicitly limited or otherwise indicated in the context.

[0073] In addition, the term "and / or" in this disclosure is only a description of the association relationship of associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this disclosure generally indicates that the previous and next associated objects are in an "or" relationship. The data referred to in this disclosure may include unstructured data such as text, images, and videos, and may also be structured data.

[0074] It should also be understood that the description of the various embodiments in the present disclosure focuses on the differences between the various embodiments, and the same or similar aspects thereof can be referenced to each other, and for the sake of brevity, they will not be described one by one.

[0075] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0076] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present disclosure, its application, or uses.

[0077] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered as part of the specification.

[0078] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0079] The disclosed embodiments can be applied to electronic devices such as terminal devices, computer systems, servers, etc., which can operate with many other general or special computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, servers, etc. include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network personal computers, small computer systems, large computer systems, and distributed cloud computing technology environments including any of the above systems, etc.

[0080] Electronic devices such as terminal devices, computer systems, servers, etc. can be described in the general context of computer system executable instructions (such as program modules) executed by computer systems. Generally, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in a distributed cloud computing environment, where tasks are performed by remote processing devices linked through a communication network. In a distributed cloud computing environment, program modules can be located on local or remote computing system storage media including storage devices.

[0081] Application Overview

[0082] In the process of implementing the present disclosure, the inventors found that due to the different positions and angles of different sensors, the coordinate systems of the acquired point cloud information are also inconsistent, and the coordinate system needs to be converted to project all point cloud information into a unified coordinate system. In order to eliminate occlusion, reduce the computational complexity and memory usage of the perception algorithm, and perform efficient calculations under limited computing resources, the 3D point cloud is usually projected onto the plane directly above the vehicle to form a top view. Therefore, projecting the 3D point cloud into the BEV space is one of the common steps in the autonomous driving perception algorithm, and real-time calculation is required. There are at least the following problems: the projection process is very sensitive to these errors, and the existing common algorithms do not perform integer quantization on the projection process, but calculate it in floating point form. However, the floating-point computing power of the inference chip used in the vehicle is much smaller than the integer computing power, so it will greatly reduce the computing speed of the entire perception algorithm and become a bottleneck for the execution of the entire neural network.

[0083] The point cloud projection operator requires real-time calculation in the autonomous driving perception algorithm, so its computational efficiency is crucial. However, the common inference chips on the market have integer computing capabilities that are far greater than floating-point computing capabilities. If more floating-point units are designed, the cost of the inference chip will be greatly increased. Therefore, the floating-point point cloud projection operator is not computationally efficient on the inference chip, which becomes a bottleneck for the execution of the entire autonomous driving perception algorithm. To solve the above problems, a full-integer point cloud projection algorithm with low computational overhead and high computational accuracy is needed to give full play to the advantages of integer computing of the inference chip and reduce the latency of the autonomous driving perception algorithm. Based on the above problems, the present disclosure proposes the following point cloud projection method.

[0084] Exemplary Methods

[0085] Figure 1 is a flow chart of a point cloud projection transformation method based on matrix decomposition provided by an exemplary embodiment of the present disclosure. This embodiment can be applied to electronic devices, such as Figure 1 As shown, the following steps are included:

[0086] Step 102 , performing multi-view image acquisition on a target scene by using a plurality of image acquisition devices disposed on a movable carrier to obtain a plurality of scene images.

[0087] Among them, the movable carrier can be equipment such as vehicles, airplanes, aircrafts, drones, ships, or other types of carriers; multiple image acquisition devices are arranged on the movable carrier to perform multi-perspective image acquisition on the target scene where the movable carrier is currently located, and each image acquisition device obtains a scene image of a perspective. A panoramic view of the target scene can be obtained through multiple scene images, and then the three-dimensional point cloud corresponding to the target scene can be mapped; for example, 6 image acquisition devices are arranged on the movable carrier (respectively arranged at the left front, right front, left side, right side, left rear, and right rear), and images of the target scene are acquired from 6 directions to obtain a target scene image that looks around the movable carrier.

[0088] Step 104, splitting the projection matrix corresponding to the image acquisition device to obtain a plurality of split matrices.

[0089] Each split matrix has the same size as the projection matrix.

[0090] In one embodiment, since the position of the image acquisition device on the movable carrier is fixed (the external parameter matrix is ​​known), the projection matrix of the image acquisition device relative to the device coordinate system (the coordinate system with the center of the movable carrier as the origin of the coordinate system) can be known; optionally, when the movable carrier is moving, the point cloud projection at the current moment can be realized based on the scene image obtained at the current moment. Optionally, the splitting process in this embodiment can be to split each element in the projection matrix according to the bit position, for example, split the 32-bit element according to the bit position into a high-order element of the upper 16 bits and a low-order element of the lower 16 bits. Of course, the 32-bit element can also be split into 4 sub-elements according to the bit position, corresponding to 0 to 7 bits, 8 to 15 bits, 16 to 23 bits, and 24 to 31 bits respectively; in this embodiment, since the projection matrix is ​​split to adapt to the calculation bit width of the inference chip, and since in the prior art, the general inference chip usually mainly supports int16 (16-bit operation) or int8 (8-bit operation) Therefore, the projection matrix can be split according to the bits supported by the inference chip, so that the split matrix obtained after the split can achieve high-performance calculations in the inference chip; in addition, due to the splitting step, when the projection matrix is ​​integerized (since the inference chip supports full-integer calculations, the projection matrix is ​​converted into a full integer to improve calculation efficiency), the elements in the projection matrix can be enlarged more to reduce the loss during integerization. For example, when the projection matrix is ​​enlarged to 64 bits, the prior art cannot perform calculations on such data. This embodiment splits the projection matrix to achieve point cloud projection without changing the hardware device.

[0091] Step 106: determining the image coordinates of the spatial points in the three-dimensional point cloud in the scene image based on the multiple split matrices.

[0092] Among them, the three-dimensional point cloud corresponds to the target scene.

[0093] In the prior art, matrix multiplication is directly performed on the projection matrix and the spatial point to determine the image coordinates of the spatial point in the scene image. In the present embodiment, since the projection matrix is ​​split into multiple split matrices, the spatial point coordinates are respectively matrix multiplied with each split matrix to obtain multiple multiplication results, and then the image coordinates of the spatial point in the scene image are determined based on the multiple multiplication results. In this process, the spatial point coordinates only need to perform matrix multiplication with the split matrix with fewer bits, which has low requirements on the bit support of the inference chip, and can achieve higher computing efficiency and accuracy based on the existing inference chip.

[0094] Step 108 , based on the image coordinates, determine the point cloud features corresponding to the spatial points in the three-dimensional point cloud, and obtain a target three-dimensional point cloud with the point cloud features.

[0095] Optionally, feature extraction of the scene image can be performed through any deep neural network to obtain image features corresponding to the scene image, and the collected image can be expressed through the image features. In this embodiment, after determining the image coordinates of the spatial point in the scene image, the feature vector corresponding to the image coordinates can be obtained from the image features based on the image coordinates, and the point cloud features corresponding to the spatial point are represented by the feature vector. By determining the point cloud features corresponding to each spatial point in the three-dimensional point cloud, the target three-dimensional point cloud is obtained.

[0096] The above-mentioned embodiment of the present disclosure provides a point cloud projection transformation method based on matrix splitting, including: performing multi-perspective image acquisition of a target scene by multiple image acquisition devices arranged on a movable carrier to obtain multiple scene images; splitting the projection matrix corresponding to the image acquisition device to obtain multiple split matrices; wherein each split matrix has the same size as the projection matrix; based on the multiple split matrices, determining the image coordinates of the spatial points in the three-dimensional point cloud in the scene image; wherein the three-dimensional point cloud corresponds to the target scene; based on the image coordinates, determining the point cloud features corresponding to the spatial points in the three-dimensional point cloud to obtain a target three-dimensional point cloud with point cloud features; the embodiment of the present disclosure splits the projection matrix when projecting the point cloud, and realizes point cloud projection by splitting the matrix, thereby improving the calculation accuracy while reducing the computing power requirements of the projection calculation process on the computing chip, and can be applied to computing chips with smaller computing power, thereby expanding the application scenarios that point cloud projection can adapt to.

[0097] In order to achieve high-precision point cloud projection on an inference chip that only supports int16 integer calculations, this embodiment proposes a full-integer calculation method, which splits the projection matrix into high and low bits, performs matrix operations separately, and then splices the results together; effectively reducing the calculation overhead, improving the calculation efficiency, and ensuring the calculation accuracy. With the characteristics of low calculation overhead and high calculation accuracy, it can be effectively applied to the acceleration and optimization of autonomous driving perception algorithms.

[0098] like Figure 2 As shown in the above Figure 1 Based on the illustrated embodiment, step 104 may include the following steps:

[0099] Step 1041, split each element in the projection matrix into multiple sub-elements with the same position.

[0100] In this embodiment, each element in the projection matrix is ​​split into multiple sub-elements with the same bit. For example, in an optional example, M_{int32} (a 32-bit projection matrix) is split into two int16 split matrices, which respectively represent the high bit and the low bit of M_{int32}; wherein each 32-bit element is split into two 16-bit sub-elements, for example, [1000010010101110 1000011111000101] is split into [1000010010101110] and [1000011111000101].

[0101] Step 1042: multiple sub-elements corresponding to the same bit positions are determined as a split matrix to obtain multiple split matrices.

[0102] Optionally, since the embodiment of the present disclosure aims to solve the problem of limited bit width format supported by the inference chip, in order to not change the hardware chip, the bit width of the bit corresponding to the sub-element of this embodiment is less than or equal to the bit width of the bit corresponding to the inference chip, so that the existing inference chip can perform matrix multiplication on the split matrix, thereby improving computing efficiency.

[0103] In addition, since the data in each element has positive and negative signs, the positive and negative signs of the data are represented by the sign bit (highest bit). In the process of splitting the data, it is hoped that the positive and negative signs of each sub-element obtained by splitting are the same as those of the original element to avoid errors when the calculation results are subsequently merged. Therefore, this embodiment includes the following steps when performing element splitting:

[0104] Determine the sign bit of each element in the projection matrix; split each element in the projection matrix according to the bit position to obtain a plurality of partial elements; determine the sign bit corresponding to each partial element based on the sign bit to obtain a plurality of sub-elements.

[0105] Optionally, the positive or negative value of the element is determined by the value of the sign bit (highest bit) of each element. For example, a sign bit of 1 indicates that the element is negative, and a sign bit of 0 indicates that the element is positive. After the elements are split according to the bits, some elements that do not include the highest bit will not have a sign bit. At this time, the sign bit of the element is brought into the highest bit of some elements to obtain sub-elements with signs, thereby avoiding calculation errors caused by splitting.

[0106] like Figure 3 As shown in the above Figure 1 Based on the illustrated embodiment, step 106 may include the following steps:

[0107] Step 1061, processing the spatial point through each splitting matrix of the multiple splitting matrices to obtain the three-dimensional coordinates of the spatial point in the device coordinate system corresponding to the image acquisition device.

[0108] Optionally, matrix multiplication is performed on each of the multiple split matrices and the spatial point to obtain multiple intermediate results; each intermediate result corresponds to a split matrix.

[0109] Based on the multiple intermediate results, determine the three-dimensional coordinates of the spatial point in the device coordinate system.

[0110] In the prior art, the matrix multiplication of the projection matrix and the three-dimensional coordinates of the spatial point is directly implemented based on the inference chip. Since the projection matrix is ​​split in this embodiment, the matrix multiplication of each split matrix and the spatial point can be implemented based on the inference chip to obtain multiple intermediate results, and then the multiple intermediate results are merged to obtain the three-dimensional coordinates of the spatial point in the device coordinate system corresponding to the image acquisition device.

[0111] Step 1062: Based on the three-dimensional coordinates, determine the image coordinates corresponding to the spatial point in the scene image.

[0112] Optionally, the three-dimensional coordinates are converted into an image coordinate system corresponding to the scene image through coordinate system conversion to obtain the image coordinates corresponding to the spatial point in the scene image.

[0113] In this embodiment, the scene image is obtained based on the image acquisition device. Therefore, there is a corresponding conversion relationship between the device coordinate system and the image coordinate system (the conversion relationship can be determined based on the existing technology). Based on the conversion relationship, the three-dimensional coordinates can be projected into the scene image to obtain the image coordinates of the spatial point. This embodiment splits the projection matrix to achieve the calculation of the projection matrix that is enlarged to a higher precision but has a bit width exceeding the inference chip without changing the hardware device, thereby improving the accuracy of point cloud projection.

[0114] Optionally, based on the multiple intermediate results, a specific process of determining the three-dimensional coordinates of the spatial point in the device coordinate system may include:

[0115] A first intermediate result and at least one second intermediate result are determined from the plurality of intermediate results.

[0116] After the matrix multiplication process, the bit width of the intermediate result is expanded. For example, after two 16-bit data are multiplied, the intermediate result obtained is 32 bits. Therefore, when merging multiple intermediate results, the intermediate results need to be processed; wherein, the first intermediate result is calculated based on the split matrix composed of the sub-elements corresponding to the highest bit position in each element, and the intermediate results corresponding to the other split matrices are used as the second intermediate results;

[0117] The at least one second intermediate result is reduced by a preset multiple to obtain at least one processed third intermediate result.

[0118] Among them, each second intermediate result corresponds to a different preset multiple; optionally, the second intermediate result can be reduced by right shifting the bits, for example, right shifting 15 bits to reduce it by 2 15 times the effect; since the elements in the split matrix corresponding to each second intermediate result do not have the same bits as the elements in the original projection matrix, the corresponding preset multiples are different. For example, M_{int32} (32-bit projection matrix) is split into four int8 split matrices, which are the first split matrix (where the corresponding elements correspond to bit31-bit24), the second split matrix (where the corresponding elements correspond to bit23-bit16), the third split matrix (where the corresponding elements correspond to bit15-bit8) and the fourth split matrix (where the corresponding elements correspond to bit7-bit0) from high to low in terms of bit position; four intermediate results are obtained based on the four split matrices, at which time the intermediate result corresponding to the first split matrix is ​​the first intermediate result, and the other intermediate results are the second intermediate results. The preset multiple of the intermediate result corresponding to the second split matrix is ​​smaller than the preset multiple of the intermediate result corresponding to the third split matrix, and the preset multiple of the intermediate result corresponding to the third split matrix is ​​smaller than the preset multiple of the intermediate result corresponding to the fourth split matrix. For example, the preset multiple of the intermediate result corresponding to the second split matrix is ​​2 7 (Because the sign bit is included, the right shift is 8-1), the preset multiple of the reduction of the intermediate result corresponding to the third split matrix is ​​2 15 , the intermediate result corresponding to the fourth split matrix is ​​reduced by a preset multiple of 2 23 ; By merging all intermediate results, the three-dimensional coordinates of int32 can be obtained.

[0119] Based on the first intermediate result and the at least one third intermediate result, a three-dimensional coordinate of the spatial point in the device coordinate system is determined.

[0120] Optionally, the first intermediate result and at least one third intermediate result may be merged by adding. For example, M_{int32} (32-bit projection matrix) is split into two int16 split matrices, which are: the split matrix M_low corresponding to the low bit int16 The split matrix M_high corresponding to the high bit int16 ;p int16 represents a spatial point. At this time, the three-dimensional coordinates can be determined based on the following formula (1):

[0121] C int32 =(x int32 ,y int32 , z int32 )=(M_low int16 *p int16 >>15)+M_highint16 *p int16 Formula (1)

[0122] Among them, C int32 Represents 32-bit three-dimensional coordinates, including 32-bit coordinate results corresponding to the x-axis, y-axis, and z-axis respectively. int32 ,y int32 and z int32 .

[0123] like Figure 4 As shown in the above Figure 1 Based on the illustrated embodiment, step 108 may include the following steps:

[0124] Step 1081, determining whether the image coordinates meet preset conditions.

[0125] Optionally, the preset condition may be that the values ​​of the three coordinate axes representing the image coordinates are all positive, that is, indicating that the spatial point has a projection point within the image. When at least one of the values ​​of the three coordinate axes is negative, it means that the projection point of the spatial point is outside the image. At this time, there is no need to continue to determine the feature vector for the spatial point based on the image.

[0126] Step 1082, in response to the image coordinates meeting a preset condition, obtaining a corresponding feature vector from a feature map corresponding to the scene image based on the image coordinates.

[0127] It may also include, in response to the image coordinates not meeting the preset conditions, not determining the corresponding point cloud features for the spatial point based on the image coordinates; in this embodiment, after the scene image is acquired based on the image acquisition device, a feature map (for example, with dimensions of H×W×C) corresponding to the scene image can be determined based on any deep learning method (for example, a deep neural network, etc.), and each pixel point in the scene image corresponds to a feature vector (with dimensions of 1×C) in the feature map. Therefore, after determining that the image coordinates are in the scene image, the feature vector corresponding to the spatial point can be determined.

[0128] Step 1083, determining the feature vector as the point cloud feature of the spatial point in the three-dimensional point cloud, and obtaining the target three-dimensional point cloud.

[0129] In this embodiment, the spatial coordinates of each spatial point in the three-dimensional point cloud are known. After the point cloud features corresponding to each spatial point are determined, the point cloud features are brought into the position corresponding to the spatial point in the three-dimensional point cloud. When the point cloud features of all spatial points in the three-dimensional point cloud are determined (determined by multiple images, each image corresponds to the point cloud features of some spatial points in the three-dimensional point cloud), the target three-dimensional point cloud can be obtained. The full-integer point cloud projection acceleration method provided in this embodiment can effectively improve the real-time performance of the autonomous driving perception algorithm and ensure the accuracy of the algorithm; compared with other algorithms, it is more efficient and has better practicality and application prospects.

[0130] In the field of autonomous driving, point cloud projection is a critical task. At present, mainstream inference chips mainly support high-performance computing in the format of int16 or int8 when performing integer calculations. Therefore, in order to represent the projection matrix M, at most int16 is used. However, the value range of the elements in M ​​is very large, mainly because the movable carrier will produce a large displacement during driving. Since the translation vector in the projection matrix M will have a much larger value range than the rotation matrix, directly mapping the floating-point projection matrix into an int16 integer matrix will produce a large rounding error. The integerization of floating-point numbers is usually implemented by rounding (round), resulting in a large number of offsets in the pixels obtained after point cloud projection, especially when the 3D point is very close to the camera plane. Therefore, how to achieve high-precision full-integer point cloud projection under limited bit width is an important issue.

[0131] In order to reduce the rounding error, this embodiment may further include the following steps before splitting the projection matrix:

[0132] A first magnification process is performed on the floating-point projection matrix corresponding to the image acquisition device; and an integer process is performed on the amplified floating-point projection matrix to obtain a projection matrix.

[0133] In this embodiment, in order to provide accuracy, each element in the floating-point projection matrix is ​​enlarged, and then the enlarged floating-point projection matrix is ​​integerized, which can greatly reduce rounding errors; optionally, the first multiple is determined according to the number of integer bits supported by the inference chip, for example, each element in the floating-point projection matrix with each element being 16 bits is enlarged to 32 bits.

[0134] In some optional examples, in order to ensure the accuracy of the projection matrix, the present disclosure may use 21 bits to represent its decimal part. If M_{i nt32} is directly sent to the projection calculation unit, the multiplication and addition operation will overflow. In this embodiment, M_{i nt32} is split into two i nt16 matrices, which represent the high and low bits of M_{i nt32} respectively. The high and low parts are calculated separately, and the results are merged. While saving the calculation overhead, the high accuracy of the calculation results is guaranteed, thereby improving the perception ability of the automatic driving system to the external environment.

[0135] This embodiment can embed the point cloud projection operator into a full-integer hardware accelerator, thereby greatly improving the computing speed and efficiency. At the same time, the above-mentioned embodiment of the present disclosure adopts a full-integer algorithm, which can not only save computing overhead and energy consumption overhead, but also obtain high-precision calculation results. Compared with the floating-point point cloud projection algorithm in the prior art, the full-integer point cloud projection acceleration solution proposed in the present disclosure has faster computing efficiency, can effectively improve the real-time performance of the autonomous driving perception algorithm, and does not affect the accuracy of the algorithm.

[0136] Any of the point cloud projection transformation methods based on matrix splitting provided in the embodiments of the present disclosure can be executed by any appropriate device with data processing capabilities, including but not limited to: terminal devices and servers, etc. Alternatively, any of the point cloud projection transformation methods based on matrix splitting provided in the embodiments of the present disclosure can be executed by a processor, such as the processor executing any of the point cloud projection transformation methods based on matrix splitting mentioned in the embodiments of the present disclosure by calling corresponding instructions stored in a memory. This will not be repeated below.

[0137] Exemplary Devices

[0138] Figure 5 Schematic diagram of a point cloud projection transformation device based on matrix decomposition provided by an exemplary embodiment of the present disclosure. Figure 5 As shown, the device provided in this embodiment includes:

[0139] The image acquisition module 51 is used to acquire multiple scene images by performing multi-viewing angle image acquisition on a target scene through multiple image acquisition devices arranged on a movable carrier.

[0140] The matrix splitting module 52 is used to split the projection matrix corresponding to the image acquisition device to obtain multiple split matrices.

[0141] Each split matrix has the same size as the projection matrix.

[0142] The image mapping module 53 is used to determine the image coordinates of the spatial points in the three-dimensional point cloud in the scene image based on the multiple split matrices.

[0143] Among them, the three-dimensional point cloud corresponds to the target scene.

[0144] The point cloud projection module 54 is used to determine the point cloud features corresponding to the spatial points in the three-dimensional point cloud based on the image coordinates, and obtain a target three-dimensional point cloud with the point cloud features.

[0145] The above-mentioned embodiment of the present disclosure provides a point cloud projection transformation device based on matrix splitting, comprising: performing multi-perspective image acquisition of a target scene by multiple image acquisition devices arranged on a movable carrier to obtain multiple scene images; splitting the projection matrix corresponding to the image acquisition device to obtain multiple split matrices; wherein each split matrix has the same size as the projection matrix; based on the multiple split matrices, determining the image coordinates of the spatial points in the three-dimensional point cloud in the scene image; wherein the three-dimensional point cloud corresponds to the target scene; based on the image coordinates, determining the point cloud features corresponding to the spatial points in the three-dimensional point cloud to obtain a target three-dimensional point cloud with point cloud features; the embodiment of the present disclosure splits the projection matrix when projecting the point cloud, and realizes point cloud projection by splitting the matrix, thereby improving the calculation accuracy while reducing the computing power requirements of the projection calculation process on the computing chip, and can be applied to computing chips with smaller computing power, thereby expanding the application scenarios that point cloud projection can adapt to.

[0146] In some optional embodiments, the matrix splitting module 52 includes:

[0147] An element splitting unit is used to split each element in the projection matrix into multiple sub-elements with the same position;

[0148] The matrix unit is used to determine the multiple sub-elements corresponding to the same bit position as a split matrix to obtain multiple split matrices.

[0149] Optionally, the element splitting unit is specifically used to determine the sign bit of each element in the projection matrix; split each element in the projection matrix according to the bit position to obtain multiple partial elements; determine the sign bit corresponding to each partial element based on the sign bit to obtain multiple sub-elements.

[0150] In some optional embodiments, the image mapping module 53 includes:

[0151] A device projection unit, used for processing a spatial point through each splitting matrix of a plurality of splitting matrices to obtain a three-dimensional coordinate of the spatial point in a device coordinate system corresponding to the image acquisition device;

[0152] The image projection unit is used to determine the image coordinates corresponding to the spatial point in the scene image based on the three-dimensional coordinates.

[0153] Optionally, the device projection unit is specifically used to perform matrix multiplication on each split matrix of multiple split matrices and the spatial point to obtain multiple intermediate results; each intermediate result corresponds to a split matrix; based on the multiple intermediate results, determine the three-dimensional coordinates of the spatial point in the device coordinate system.

[0154] Optionally, when the device projection unit determines the three-dimensional coordinates of a spatial point in a device coordinate system based on multiple intermediate results, it is specifically used to determine a first intermediate result and at least one second intermediate result from the multiple intermediate results; reduce the at least one second intermediate result by a preset multiple to obtain at least one processed third intermediate result; wherein the preset multiples corresponding to each second intermediate result are different; and determine the three-dimensional coordinates of the spatial point in the device coordinate system based on the first intermediate result and the at least one third intermediate result.

[0155] Optionally, the image projection unit is specifically configured to transform the three-dimensional coordinates into an image coordinate system corresponding to the scene image through coordinate system transformation, so as to obtain image coordinates corresponding to the spatial point in the scene image.

[0156] In some optional embodiments, the point cloud projection module 54 is specifically used to determine whether the image coordinates meet the preset conditions; in response to the image coordinates meeting the preset conditions, obtain a corresponding feature vector from a feature map corresponding to the scene image based on the image coordinates; determine the feature vector as a point cloud feature of a spatial point in a three-dimensional point cloud, and obtain a target three-dimensional point cloud.

[0157] In some optional embodiments, the device provided in this embodiment may further include:

[0158] The matrix preprocessing module is used to magnify the floating-point projection matrix corresponding to the image acquisition device by a first multiple; and to integerize the floating-point projection matrix after the magnification process to obtain a projection matrix.

[0159] Exemplary Electronic Devices

[0160] Below, reference Figure 6 The electronic device according to the embodiment of the present disclosure is described. The electronic device provided by the present disclosure may be a full-integer hardware accelerator. The electronic device may be any one or both of the first device and the second device, or a stand-alone device independent of them, and the stand-alone device may communicate with the first device and the second device to receive the collected input signal from them.

[0161] Figure 6 A block diagram of an electronic device according to an embodiment of the present disclosure is illustrated.

[0162] like Figure 6 As shown, the electronic device includes one or more processors and memory.

[0163] The processor may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device to perform desired functions.

[0164] The memory may store one or more computer program products, and the memory may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, a random access memory (RAM) and / or a cache memory (cache), etc. The non-volatile memory may include, for example, a read-only memory (ROM), a hard disk, a flash memory, etc. One or more computer program products may be stored on the computer-readable storage medium, and the processor may run the computer program product to implement the point cloud projection transformation method based on matrix decomposition of the various embodiments of the present disclosure described above and / or other desired functions.

[0165] In one example, the electronic device may further include: an input device and an output device, and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0166] In addition, the input device may also include, for example, a keyboard, a mouse, and the like.

[0167] The output device can output various information to the outside, including determined distance information, direction information, etc. The output device can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and the like.

[0168] Of course, to simplify, Figure 6 Only some of the components related to the present disclosure in the electronic device are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application situations, the electronic device may further include any other appropriate components.

[0169] In addition to the above-mentioned methods and devices, an embodiment of the present disclosure may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the point cloud projection transformation method based on matrix splitting according to various embodiments of the present disclosure described in the above part of this specification.

[0170] The computer program product may be written in any combination of one or more programming languages ​​to write program code for performing the operations of the disclosed embodiments, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0171] In addition, an embodiment of the present disclosure may also be a computer-readable storage medium having computer program instructions stored thereon, and when the computer program instructions are executed by a processor, the processor executes the steps of the point cloud projection transformation method based on matrix splitting according to various embodiments of the present disclosure described in the above part of this specification.

[0172] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0173] The basic principles of the present disclosure are described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, effects, etc. mentioned in the present disclosure are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. are required by each embodiment of the present disclosure. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, and are not limitations. The above details do not limit the present disclosure to the necessity of adopting the above specific details to be implemented.

[0174] Each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system embodiment, since it basically corresponds to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0175] The block diagrams of the devices, apparatuses, equipment, and systems involved in this disclosure are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these devices, apparatuses, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including," "comprising," "having," and the like are open words, referring to "including but not limited to," and can be used interchangeably therewith. The words "or" and "and" used herein refer to the words "and / or," and can be used interchangeably therewith, unless the context clearly indicates otherwise. The word "such as" used herein refers to the phrase "such as but not limited to," and can be used interchangeably therewith.

[0176] The method and apparatus of the present disclosure may be implemented in many ways. For example, the method and apparatus of the present disclosure may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above order of steps for the method is for illustration only, and the steps of the method of the present disclosure are not limited to the order specifically described above, unless otherwise specifically stated. In addition, in some embodiments, the present disclosure may also be implemented as a program recorded in a recording medium, which includes machine-readable instructions for implementing the method according to the present disclosure. Therefore, the present disclosure also covers a recording medium storing a program for executing the method according to the present disclosure.

[0177] It should also be noted that in the apparatus, device and method of the present disclosure, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present disclosure.

[0178] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the aspects shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

[0179] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present disclosure to the forms disclosed herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

1. A point cloud projection transformation method based on matrix decomposition, characterized in that: include: Capturing multi-view images of a target scene by using a plurality of image acquisition devices arranged on a movable carrier to obtain a plurality of scene images; Splitting the projection matrix corresponding to the image acquisition device to obtain a plurality of split matrices; wherein each split matrix has the same size as the projection matrix; Based on the multiple split matrices, determining the image coordinates of the spatial points in the three-dimensional point cloud in the scene image; wherein the three-dimensional point cloud corresponds to the target scene; Based on the image coordinates, point cloud features corresponding to the spatial points in the three-dimensional point cloud are determined to obtain a target three-dimensional point cloud with point cloud features.

2. The method according to claim 1, characterized in that The projection matrix corresponding to the image acquisition device is split to obtain a plurality of split matrices, including: Splitting each element in the projection matrix into a plurality of sub-elements with the same position; The plurality of sub-elements corresponding to the same bit position are determined as a split matrix to obtain a plurality of split matrices.

3. The method according to claim 2, characterized in that The step of splitting each element in the projection matrix into a plurality of sub-elements with the same position comprises: Determining a sign bit of each element in the projection matrix; Splitting each element in the projection matrix according to bits to obtain multiple partial elements; The sign bit corresponding to each of the partial elements is determined based on the sign bit to obtain the multiple sub-elements.

4. The method according to any one of claims 1 to 3, characterized in that: The determining, based on the multiple split matrices, the image coordinates of the spatial points in the three-dimensional point cloud in the scene image comprises: Processing the spatial point by each splitting matrix of the multiple splitting matrices to obtain the three-dimensional coordinates of the spatial point in the device coordinate system corresponding to the image acquisition device; Based on the three-dimensional coordinates, the image coordinates corresponding to the spatial point in the scene image are determined.

5. The method according to claim 4, characterized in that The step of processing the spatial point by each splitting matrix of the multiple splitting matrices to obtain the three-dimensional coordinates of the spatial point in the device coordinate system corresponding to the image acquisition device includes: Perform matrix multiplication on each of the multiple split matrices and the spatial point to obtain multiple intermediate results; each of the intermediate results corresponds to one split matrix; Based on the multiple intermediate results, the three-dimensional coordinates of the spatial point in the device coordinate system are determined.

6. The method according to claim 5, characterized in that The determining, based on the plurality of intermediate results, the three-dimensional coordinates of the spatial point in the device coordinate system comprises: determining a first intermediate result and at least one second intermediate result from the plurality of intermediate results; Reducing the at least one second intermediate result by a preset multiple to obtain at least one processed third intermediate result; wherein the preset multiples corresponding to each of the second intermediate results are different; Based on the first intermediate result and the at least one third intermediate result, a three-dimensional coordinate of the spatial point in the device coordinate system is determined.

7. The method according to any one of claims 4 to 6, characterized in that: Determining the image coordinates corresponding to the spatial point in the scene image based on the three-dimensional coordinates includes: The three-dimensional coordinates are converted into an image coordinate system corresponding to the scene image through coordinate system conversion to obtain the image coordinates corresponding to the spatial point in the scene image.

8. The method according to any one of claims 1 to 7, characterized in that: The step of determining point cloud features corresponding to the spatial points in the three-dimensional point cloud based on the image coordinates to obtain a target three-dimensional point cloud having point cloud features includes: Determining whether the image coordinates meet preset conditions; In response to the image coordinates meeting the preset condition, obtaining a corresponding feature vector from a feature map corresponding to the scene image based on the image coordinates; The feature vector is determined as a point cloud feature of a spatial point in the three-dimensional point cloud to obtain the target three-dimensional point cloud.

9. The method according to any one of claims 1 to 8, characterized in that: Before the projection matrix corresponding to the image acquisition device is split to obtain a plurality of split matrices, the method further includes: Performing a first magnification process on the floating-point projection matrix corresponding to the image acquisition device; The enlarged floating-point projection matrix is ​​integerized to obtain the projection matrix.

10. A point cloud projection transformation device based on matrix decomposition, characterized in that: include: An image acquisition module is used to acquire multiple scene images by performing multi-viewing image acquisition on a target scene through multiple image acquisition devices arranged on a movable carrier; A matrix splitting module, used for splitting the projection matrix corresponding to the image acquisition device to obtain a plurality of split matrices; wherein each split matrix has the same size as the projection matrix; An image mapping module, configured to determine image coordinates of spatial points in a three-dimensional point cloud in the scene image based on the multiple split matrices; wherein the three-dimensional point cloud corresponds to the target scene; The point cloud projection module is used to determine the point cloud features corresponding to the spatial points in the three-dimensional point cloud based on the image coordinates, and obtain a target three-dimensional point cloud with point cloud features.

11. An electronic device, characterized in that: include: a memory for storing a computer program product; A processor is used to execute the computer program product stored in the memory, and when the computer program product is executed, it implements the method described in any one of claims 1 to 9.

12. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method described in any one of claims 1 to 9 is implemented.

Images