Subgraph decomposition and model acceleration method for fewest pixel 3D area coverage

Abstract

The invention provides a method for decomposing a perspective image into a plurality of sub-images and application of the method in the aspects of model acceleration and the like, and aims to cover potential obstacles in a designated 3D area of a ground plane by using fewest pixels through the sub-image decomposition, so that the input size of a model algorithm is reduced to achieve the purpose of acceleration. Comprising the following steps of: obtaining optimal sub-graph decomposition through optimization methods such as dynamic planning and the like by utilizing priori and predefined potential obstacle 3D information of a 3D ground plane region; performing sub-graph deep neural network modeling based on middle layer interaction to obtain a feature layer corresponding to each sub-graph; based on algorithm adaptation of a sub-graph feature layer, existing 2D and 3D model algorithms are seamlessly supported. According to the subgraph decomposition method provided by the invention, the number of pixels can be remarkably reduced under the condition of meeting obstacles in a specified 3D area covering the ground plane, so that the training and reasoning speed of the model is improved, and in addition, the data transmission overhead and the storage cost can be correspondingly reduced by adopting the subgraph decomposition system.

Description

Technical Field

[0001] The present invention relates to the field of perspective image processing technology, including but not limited to the field of autonomous driving vision, and specifically to a sub-image decomposition method with minimum pixel 3D area coverage, and its application in model acceleration, reducing data transmission overhead and storage costs, etc. Background Art

[0002] Camera images are the most common perspective images and are widely used in autonomous driving, smart transportation, home security, and other aspects. Data from other sensors such as lidar can also be converted into perspective images through range-view for processing.

[0003] In most scenarios, including autonomous driving, the main focus is on obstacles in a specific area on the ground plane. Since the imaging plane of the perspective image is approximately perpendicular to the ground plane, the obstacles on the ground plane appear larger when they are closer and smaller when they are farther away, and sparse when they are closer and dense when they are farther away. The present invention only considers such scenarios.

[0004] Current deep learning models use perspective images as a whole after scaling appropriately as input. In order to characterize distant obstacles, a sufficiently large input resolution is required. At this time, most of the computing and network capacity are used for relatively sparse large objects nearby. This imbalance is essentially an unnecessary and huge waste. In addition, many scenarios such as autonomous driving only care about the ground plane area near the road surface, while the road surface area only occupies a small part of the perspective image's 3D field of view. The current modeling method does not consider the potential value of this factor in model acceleration.

[0005] As the requirements for perception effects and perception distances in fields such as autonomous driving continue to increase, more and more cameras with 8M or even higher resolution are used. At this time, data transmission has gradually become a bottleneck restricting system performance. In addition, a large number of high-resolution images also brings considerable storage costs. Summary of the invention

[0006] The present invention provides a sub-image decomposition method with minimum pixel 3D area coverage, which can effectively solve the problems of time-consuming model, transmission bottleneck, storage cost, etc. of perspective images in the background technology.

[0007] The present invention is implemented by the following technical solution, which is described in three parts:

[0008] 1. A sub-image decomposition method for minimum pixel 3D region coverage, comprising the following steps:

[0009] Step 1: Get the original image , according to the calibration information, obtain the internal parameter matrix , the camera height and angle relative to the ground plane, and the lateral field of view .

[0010]

[0011]

[0012]

[0013] Step 2: Predefine the 3D obstacles to be considered:

[0014]

[0015] in , , Indicates the true height, width, and length of the 3D obstacle; , , Indicates the orientation angle of the 3D obstacle. Usually, the algorithm only requires that the projection height is greater than MinPixelHeight and the obstacle is perpendicular to the ground plane, so it can be equivalent to , at this time, the 3D obstacle is determined only by its true height, which can be simply written as:

[0016]

[0017] Indicates the number of 3D obstacles that need to be covered. For example, the common categories that need to be considered for dynamic obstacle detection in autonomous driving are pedestrians, cyclists, and vehicles, whose actual heights are all around 1.7m. For some large vehicles, it can be assumed that the 1.7m part close to the ground is sufficient to meet the detection algorithm requirements.

[0018] Step 3: Get the ground plane 3D area in the camera coordinate system. Different obstacles can use different ground plane 3D areas, but for simplicity, all obstacles share the same ground plane 3D area by default. The following two common types are considered:

[0019] Type 1, standard rectangular area within FOV:

[0020]

[0021] Where a, b, c are the ground plane equation parameters determined by the camera height and angle, and Respectively represent the left and right and front and back ranges, in general application scenarios (such as autonomous driving) Significantly greater than , This is to eliminate the possibility of abnormal obstacle projection in a certain area ahead. Indicates that ground points are calibrated based on mature online-calibration methods such as image vanishing points.

[0022] Type 2, road area within FOV:

[0023]

[0024] The map can be an offline high-precision map or an online map. Indicates that ground points are calibrated based on mature online-calibration methods such as image vanishing points.

[0025] Step 4: By Performing multiple different Crop-Resize operations will result in different sets of sub-images. , through the following optimization problem, the minimum pixel area coverage sub-image decomposition is obtained:

[0026]

[0027] when When it represents a set of sub-images obtained by performing Crop-Resize operations on multiple different images at the same time, the optimal multi-image joint coverage sub-image decomposition can also be obtained through this optimization problem.

[0028] Step 5: Solve the optimization problem in step 4 to obtain the optimal sub-image decomposition. In general multi-view scenarios (such as autonomous driving), the overlapping area of ​​FOV of different images is very small, so each image can be processed separately. The following method is only for a single image:

[0029] Consider first For a standard rectangular area within the regular FOV:

[0030] The ground plane 3D area is uniformly discretized into N rectangular or trapezoidal slices along the z direction. They represent the left and right endpoints of the N+1 horizontal dividing lines from near to far, and Corresponding to the image For the projection point on , and Indicates that the obstacle is located at the endpoint. Another vertex in the image The projection point on Indicates that the obstacles on the segmentation line are Upper projection height:

[0031]

[0032] make The minimum number of pixels that represents the optimal solution to a sub-problem of the original problem, that is, , whose coverage area is numbered The middle area of ​​the two horizontal dividing lines N+1 is composed of the following inequality:

[0033]

[0034] The equal sign is in Established when It may not necessarily be true. Indicates that , whose coverage area is numbered and The middle area of ​​the two horizontal dividing lines is composed of , and only one sub-image is used to cover it. At this time, the minimum pixels of the sub-image can be calculated as follows:

[0035]

[0036] Represents the crop area of ​​the sub-image, represents the scaling factor, and there are also initial conditions:

[0037]

[0038] Therefore, based on the above inequality, the optimal solution upper bound is obtained through dynamic programming ,when , since the equality holds, this upper bound is the optimal solution. When , the equality does not hold and the upper bound is a suboptimal solution.

[0039] Reconsider For the general road area within the FOV:

[0040] when For a relatively simple longitudinal single road, the z direction is evenly distributed along the central axis of the road. Discretize into N quadrilateral slices, and the rest are processed in the same way as the standard rectangular area within the FOV mentioned above.

[0041] when For more complex bifurcated roads, decompose them into longitudinal main roads , and Several other non-connected horizontal areas separated . The subgraph decomposition is obtained by processing the longitudinal single road in the same way as above; Each region corresponds to a sub-graph independently. Indicates the crop area of ​​the sub-image. Indicates the scaling factor as follows:

[0042]

[0043] when For other more complex road conditions, they are handled in a similar way to forked roads. Depending on the different road topologies, different heuristic strategies can be adopted mainly based on the dynamic programming algorithm.

[0044] 2. The subgraph deep neural network modeling method based on the interaction of the intermediate layers includes the following steps:

[0045] Step 1: The optimal subgraph set is ,Will Input the deep neural network (in addition to the sub-image itself, the input can also optionally add 2D position encoding, scale position encoding, 3D position encoding, etc.) to obtain the feature layers of each stage , s and S represent the stage number and total number. The deep neural network can select the commonly used CNN or VIT, and the network parameters of each subgraph can be shared or not.

[0046] Step 2: Insert the following feature layer interaction operators after several high-level stages of the deep neural network: ,in You can choose The feature layers corresponding to the sub-images with more overlapped crop areas and similar scaling ratios on the original image. You can choose interpolation alignment or Deformable Attention Feature mapping and fusion , Deformable Attention refers to the Deformable DETR algorithm. In this way, the features of each sub-graph neural network can be reused.

[0047] 3. Algorithm adaptation method based on sub-graph feature layer, including the following parts:

[0048] Algorithm adaptation method based on original image feature layer: The feature layer of the sth stage of the original image can be aggregated by the following sub-image feature layer aggregation operator Obtain, among which , and Op can choose interpolation alignment or Deformable Attention to aggregate the sub-image feature layer into the original image feature layer. In this way, algorithms based on the original image feature layer, such as the common Faster-rcnn, Yolo and other algorithms, can be seamlessly adapted.

[0049] Algorithm adaptation method based on 2d-object-query: Perform global attention on the 2d-object-query and sub-image feature layers, or Perform sparse attention on the 2d-object-query and sub-image feature layers, and They correspond to the corresponding operators of DETR and Deformable DETR algorithms respectively. Represents the features of all 2d-object-queries, You can select the high-level feature layers of each sub-graph. This means that the spatial dimensions of the feature layers of each sub-graph are flattened and concat together. In this way, 2d-object-query-based algorithms such as DETR and Deformable DETR can be seamlessly adapted.

[0050] Algorithm adaptation method based on BEV feature layer: using operators Indicates that multiple images are viewed from the The feature layer of BEV is constructed, and express The same operator is used for subgraph decomposition of Construct a BEV feature layer based on the sub-image feature layer; You can choose to aggregate 2D features to 3D positions based on LSS or 3D Deformable Attention. LSS and 3D Deformable Attention refer to BevDet and BevFormer algorithms respectively. In this way, algorithms based on the BEV feature layer, such as BevDet, BevFormer, etc., can be seamlessly adapted.

[0051] Algorithm adaptation method based on 3d-object-query: using operators Indicates that the 3d-object-query looks at multiple images The feature layer query aggregate features, Represents the features of all 3d-object-queries, let express The same operator is used for subgraph decomposition of Used for 3d-object-query to query aggregate features from the sub-graph feature layer; You can choose to aggregate 2D features to 3d-object-query based on Global Cross Attention or 3D position projection interpolation, where Global Cross Attention and 3D position projection interpolation refer to Petr and Detr3D algorithms respectively. In this way, algorithms based on 3d-object-query, such as Petr, Detr3D, etc., can be seamlessly adapted.

[0052] Compared with the modeling method based on the original perspective image, the above-mentioned modeling and adaptation method based on sub-graph decomposition can significantly reduce the input resolution, thereby achieving the purpose of model acceleration.

[0053] In addition, before the perspective image is stored and transmitted, the sub-image is decomposed first, and then only the sub-image is stored and transmitted. In this way, the storage and transmission overhead can be greatly reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention.

[0055] In the attached picture:

[0056] Figure 1 It is the overall framework diagram in the example of the present invention.

[0057] Figure 2 It is a schematic diagram of a 3D region in an example of the present invention.

[0058] Figure 3 It is a schematic diagram of 3D region discretization in an example of the present invention.

[0059] Figure 4 It is a schematic diagram of 3D region division corresponding to the optimal subgraph decomposition in the example of the present invention.

[0060] Figure 5 It is a schematic diagram of the Crop area of ​​the original image corresponding to the optimal sub-image decomposition in the example of the present invention.

[0061] Figure 6 It is a schematic diagram of the optimal subgraph set in the example of the present invention.

[0062] Figure 7 It is a schematic diagram of the intermediate layer interactive deep residual network in the example of the present invention.

[0063] Figure 8 It is a schematic diagram of the middle layer interaction operator in the example of the present invention.

[0064] Fig. 9 It is a schematic diagram of 2D algorithm adaptation in an example of the present invention.

[0065] Fig.10 It is a schematic diagram of 3D algorithm adaptation in an example of the present invention. DETAILED DESCRIPTION

[0066] In order to more clearly illustrate the purpose, technical solutions and advantages of the present invention, the following is a further detailed description in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0067] Example: Figure 1 As shown, the present invention provides a sub-graph decomposition method with minimum pixel 3D area coverage and its application in model acceleration, etc., including three parts: first, a sub-graph decomposition method with minimum pixel 3D area coverage; second, a sub-graph deep neural network modeling method based on intermediate layer interaction; third, an algorithm adaptation method based on a sub-graph feature layer, which are introduced in the following three parts:

[0068] 1. A sub-image decomposition method for minimum pixel 3D region coverage, comprising the following steps:

[0069] Step 1: Get the original image , according to the calibration information, obtain the internal parameter matrix , the camera height and angle relative to the ground plane, and the lateral field of view .

[0070]

[0071]

[0072]

[0073] In this embodiment, if Figure 2 As shown in the figure, the image is a front view image in an autonomous driving scenario, with a width and height of 2633 and 795 pixels respectively; the camera coordinate system is the standard Camera coordinate system, with x pointing to the right, y pointing to the bottom, and z pointing to the front; the offline calibrated camera height and pitch angle are 1.45 meters and 0° respectively; the internal reference , , , They are 1529.66, 1292.27, 1529.66, 366.45 respectively; , They are -40.19° and 41.23° respectively.

[0074] Step 2: Predefine the 3D obstacles to be considered:

[0075]

[0076] in , , Indicates the true height, width, and length of the 3D obstacle; , , Indicates the orientation angle of the 3D obstacle. Usually, the algorithm only requires that the projection height is greater than MinPixelHeight and the obstacle is perpendicular to the ground plane, so it can be equivalent to , at this time, the 3D obstacle is determined only by its true height, which can be simply written as:

[0077]

[0078] In this example, we only consider common obstacle types in autonomous driving scenarios: pedestrians, cyclists, and vehicles. Their actual heights are all around 1.7m. For some large vehicles, we can assume that the 1.7m part close to the ground is sufficient to meet the detection algorithm requirements. Therefore, let: .

[0079] Step 3: Get the 3D area of ​​the ground plane in the camera coordinate system, such as Figure 2 As shown in the figure, the road area in this scene is approximately a standard rectangular area, so this example uses the standard rectangular area within the FOV:

[0080]

[0081] Where a, b, c are the ground plane equation parameters determined by the camera height and angle, and Respectively represent the left and right and front and back ranges, in general application scenarios (such as autonomous driving) Significantly greater than , This is to eliminate the possibility of abnormal obstacle projection in a certain area ahead. Indicates that ground points are calibrated based on mature online-calibration methods such as image vanishing points.

[0082] In this example, a, b, and c are calculated to be 0, 0, and 1.45 respectively; let , are -7.65 and 3.25 respectively, which is equivalent to expanding the width of one lane on both sides of the road; is 0.5 (in Figure 2 The center intersects with the FOV boundary line at points O1 and O2). is 162.5 (in Figure 2The CD line corresponds to the position of the 1.7m obstacle in the image, and the 1.7m obstacle corresponds to 16 pixels in the image); the online calibration method based on the vanishing point only corrects the pitch angle to 0.2°. At this time, the ground point calibration function is ;like Figure 2 As shown, at this time, the ground plane 3D area is a polygon AO1O2BCD. When the left and right boundary extension lines AD and BC are projected onto the image, they are consistent with the road boundary direction and intersect at the image vanishing point in the distance. Therefore, the ground plane 3D area is basically accurate for the image.

[0083] Step 4: By Performing multiple different Crop-Resize operations will result in different sets of sub-images. , through the following optimization problem, the minimum pixel area coverage sub-image decomposition is obtained. In this example, MinPixelHeight is 32 pixels:

[0084]

[0085] Step 5: Solve the optimization problem in step 4 to obtain the optimal subgraph decomposition. In this example, It is a standard rectangular area within the regular FOV, so the following dynamic programming algorithm is used to solve it:

[0086] like Figure 3 As shown, the ground plane 3D area is uniformly discretized into N rectangular or trapezoidal slices along the z direction. They represent the left and right endpoints of the N+1 horizontal dividing lines from near to far, and Corresponding to the image For the projection point on , and Indicates that the obstacle is located at the endpoint. Another vertex in the image The projection point on Indicates that the obstacles on the segmentation line are Upper projection height:

[0087]

[0088] make The minimum number of pixels that represents the optimal solution to a sub-problem of the original problem, that is, , whose coverage area is numbered The middle area of ​​the two horizontal dividing lines N+1 is composed of the following inequality:

[0089]

[0090] The equal sign is in Established when It may not necessarily be true. Indicates that , whose coverage area is numbered and The middle area of ​​the two horizontal dividing lines is composed of , and only one sub-image is used to cover it. At this time, the minimum pixels of the sub-image can be calculated as follows:

[0091]

[0092] Represents the crop area of ​​the sub-image, represents the scaling factor, and there are also initial conditions:

[0093]

[0094] Therefore, based on the above inequality, the optimal solution upper bound is obtained through dynamic programming ,when , since the equality holds, this upper bound is the optimal solution. When , the equality does not hold and the upper bound is a suboptimal solution.

[0095] In this example, N is set to 200, because , so according to the above dynamic programming, the minimum pixels of the optimal sub-graph decomposition can be obtained ; According to the conventional idea of ​​dynamic programming, the array By backtracking, we can get Figure 4 The 3D region division corresponding to the optimal subgraph decomposition shown in FIG. 7 is 7 sub-regions numbered 1 to 7 from near to far. Figure 5 The figure shows the Crop region of the original graph corresponding to the optimal subgraph decomposition determined by Q during the backtracking process. , numbered 1-7 accordingly; Figure 6 Shown is the Crop area according to The optimal subgraph set after scaling is numbered 1-7 accordingly.

[0096] Second, sub-graph deep neural network modeling method based on middle layer interaction, such as Figure 7-8 As shown, the following steps are included:

[0097] Step 1: The optimal subgraph set is ,Will Input the deep neural network (in addition to the sub-image itself, the input can also optionally add 2D position encoding, scale position encoding, 3D position encoding, etc.) to obtain the feature layers of each stage , s and S represent the stage number and total number. The deep neural network can select the commonly used CNN or VIT, and the network parameters of each subgraph can be shared or not. In this example, Figure 7 As shown, the deep neural network selects ResNet50, and the parameters of each subgraph are shared.

[0098] Step 2: Insert the following feature layer interaction operators after several high-level stages of the deep neural network: ,in You can choose The feature layers corresponding to the sub-images with more overlapped crop areas and similar scaling ratios on the original image. You can choose interpolation alignment or Deformable Attention Feature mapping and fusion , Deformable Attention refers to the Deformable DETR algorithm. In this way, each subgraph neural network can reuse features. In this example, Figure 7 As shown, a feature layer interaction operator is inserted after each ResBlock, such as Figure 8 As shown, Choose The feature layers corresponding to the two sub-images with the closest scaling ratio, Select the interpolation alignment method, that is, use bilinear interpolation to resize the two sub-image feature layers to the same The feature layer has the same scale, and then it is aligned with The feature layers are concat together and then feature fusion is performed through 1x1 convolution.

[0099] 3. Algorithm adaptation method based on sub-graph feature layer, including the following parts:

[0100] Algorithm adaptation method based on original image feature layer: The feature layer of the sth stage of the original image can be aggregated by the following sub-image feature layer aggregation operator Obtain, among which , and Op can choose interpolation alignment or Deformable Attention to aggregate the sub-image feature layer into the original image feature layer. In this way, algorithms based on the original image feature layer, such as the common Faster-rcnn, Yolo and other algorithms, can be seamlessly adapted. In this example, Fig. 9 As shown, take , Op takes Deformable Attention, through Get the feature layer of the last stage of the original image, that is, the feature layer after ResBlock 4.

[0101] Algorithm adaptation method based on 2d-object-query: Perform global attention on the 2d-object-query and sub-image feature layers, or Perform sparse attention on the 2d-object-query and sub-image feature layers, and They correspond to the corresponding operators of DETR and Deformable DETR algorithms respectively. Represents the features of all 2d-object-queries, You can select the high-level feature layers of each sub-graph. Indicates that the spatial dimensions of the feature layers of each sub-graph are flattened and concat together. In this way, algorithms based on 2d-object-query, such as DETR and Deformable DETR, can be seamlessly adapted. In this example, Fig. 9 As shown, take ,pass Use the sub-image feature layer to update the features of the 2d-object-query through sparse attention.

[0102] Algorithm adaptation method based on BEV feature layer: using operators Indicates that multiple images are viewed from the The feature layer of BEV is constructed, and express The same operator is used for subgraph decomposition of Construct a BEV feature layer based on the sub-image feature layer; You can choose to aggregate 2D features to 3D positions based on LSS or 3D Deformable Attention. LSS and 3D Deformable Attention refer to BevDet and BevFormer algorithms respectively. In this way, algorithms based on the BEV feature layer, such as BevDet, BevFormer, etc., can be seamlessly adapted. In this example, Fig.10 As shown in Figure 1, the right view image, rear view image and other surround camera images are used to obtain the feature layer corresponding to the optimal sub-image set in the same way as the front view image. Then, all sub-image feature layers of all camera images are merged into one set. Take 3D Deformable Attention and aggregate these sub-image feature layers to obtain the BEV feature layer.

[0103] Algorithm adaptation method based on 3d-object-query: using operators Indicates that the 3d-object-query looks at multiple images The feature layer query aggregate features, Represents the features of all 3d-object-queries, let express The same operator is used for subgraph decomposition of Used for 3d-object-query to query aggregate features from the sub-graph feature layer; You can choose to aggregate 2D features to 3d-object-query based on Global Cross Attention or 3D position projection interpolation, where Global Cross Attention and 3D position projection interpolation refer to Petr and Detr3D algorithms respectively. In this way, algorithms based on 3d-object-query, such as Petr, Detr3D, etc., can be seamlessly adapted. In this example, Fig.10 As shown in Figure 1, the right view image, rear view image and other surround camera images are used to obtain the feature layer corresponding to the optimal sub-image set in the same way as the front view image. Then, all sub-image feature layers of all camera images are merged into one set. Take the 3D position projection interpolation method and use the sub-image feature layer to update the features of the 3d-object-query.

[0104] Compared with the modeling method based on the original perspective image, the above-mentioned modeling and adaptation method based on sub-image decomposition can significantly reduce the input resolution, thereby achieving the purpose of model acceleration. In this example, the number of pixels of the original image is 2093235 (equivalent to the 2M image resolution commonly used in autonomous driving), and the total number of pixels of the optimal sub-image set is 124425, thus achieving a theoretical acceleration of 16.82 times. In addition, in order to detect obstacles at H=162.5, the original image needs to detect small targets of 16 pixels, and the MinPixelHeight=32 taken by the sub-image decomposition in this example is relatively large, so there is still room for further acceleration. If the MinPixelHeight is set to 16 (consistent with the original image detection MinSize), a theoretical acceleration of 67.29 times can be achieved.

[0105] In addition, before storing and transmitting the perspective image, the sub-image is first decomposed, and then only the sub-image is stored and transmitted, which can greatly reduce the storage and transmission overhead. In this example, similarly, the storage and transmission overhead can be reduced by 16.82 or even 67.29 times.

[0106] Finally, it should be noted that the above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A sub-image decomposition method with minimum pixel 3D area coverage, characterized in that: It includes the following steps: Step 1: Get the original image , according to the calibration information, obtain the internal parameter matrix , the camera height and angle relative to the ground plane, and the lateral field of view . Step 2: Predefine the 3D obstacles to be considered: in , , Indicates the true height, width, and length of the 3D obstacle; , , Indicates the orientation angle of the 3D obstacle. Usually, the algorithm only requires that the projection height is greater than MinPixelHeight and the obstacle is perpendicular to the ground plane, so it can be equivalent to , at this time, the 3D obstacle is determined only by its true height, which can be simply written as: Indicates the number of 3D obstacles that need to be covered. For example, the common categories that need to be considered for dynamic obstacle detection in autonomous driving are pedestrians, cyclists, and vehicles, whose actual heights are all around 1.7m. For some large vehicles, it can be assumed that the 1.7m part close to the ground is sufficient to meet the detection algorithm requirements. Step 3: Get the ground plane 3D area in the camera coordinate system. Different obstacles can use different ground plane 3D areas, but for simplicity, all obstacles share the same ground plane 3D area by default. The following two common types are considered: Type 1, standard rectangular area within FOV: Where a, b, c are the ground plane equation parameters determined by the camera height and angle, and Respectively represent the left and right and front and back ranges, in general application scenarios (such as autonomous driving) Significantly greater than , This is to eliminate the possibility of abnormal obstacle projection in a certain area ahead. Indicates that ground points are calibrated based on mature online-calibration methods such as image vanishing points. Type 2, road area within FOV: The map can be an offline high-precision map or an online map. Indicates that ground points are calibrated based on mature online-calibration methods such as image vanishing points. Step 4: By Performing multiple different Crop-Resize operations will result in different sets of sub-images. , through the following optimization problem, the minimum pixel area coverage sub-image decomposition is obtained: when When it represents a set of sub-images obtained by performing Crop-Resize operations on multiple different images at the same time, the optimal multi-image joint coverage sub-image decomposition can also be obtained through this optimization problem. Step 5: Solve the optimization problem in step 4 to obtain the optimal sub-image decomposition. In general multi-view scenarios (such as autonomous driving), the overlapping area of ​​FOV of different images is very small, so each image can be processed separately. The following claims are only for a single image unless otherwise specified.

2. The method for sub-image decomposition with minimum pixel 3D area coverage as claimed in claim 1, characterized in that: In the step of solving the optimization problem, when When it is a standard rectangular area within the FOV, the following dynamic programming algorithm is used to solve it: The ground plane 3D area is uniformly discretized into N rectangular or trapezoidal slices along the z direction. They represent the left and right endpoints of the N+1 horizontal dividing lines from near to far, and Corresponding to the image For the projection point on , and Indicates that the obstacle is located at the endpoint. Another vertex in the image The projection point on Indicates that the obstacles on the segmentation line are Upper projection height: make The minimum number of pixels that represents the optimal solution to a sub-problem of the original problem, that is, , whose coverage area is numbered The middle area of ​​the two horizontal dividing lines N+1 is composed of the following inequality: The equal sign is in Established when It may not necessarily be true. Indicates that , whose coverage area is numbered and The middle area of ​​the two horizontal dividing lines is composed of and only one sub-image is used to cover it. At this time, the minimum pixels of the sub-image can be calculated as follows: Represents the crop area of ​​the sub-image, represents the scaling factor, and there are also initial conditions: Therefore, based on the above inequality, the optimal solution upper bound is obtained through dynamic programming ,when , since the equality holds, this upper bound is the optimal solution. When , the equality does not hold and the upper bound is a suboptimal solution.

3. The method for sub-image decomposition with minimum pixel 3D area coverage as claimed in claim 1, characterized in that: In the step of solving the optimization problem, when For a road area within a general FOV, the following method is used to solve it: when For a relatively simple longitudinal single road, the z direction is evenly distributed along the central axis of the road. Discretized into N quadrilateral slices, and the rest are processed in the same way as the standard rectangular area within the FOV in claim 2. when For more complex bifurcated roads, decompose them into longitudinal main roads , and Several other non-connected horizontal areas separated . The subgraph decomposition is obtained by processing the longitudinal single road in the same way as above; Each region corresponds to a sub-graph independently. Indicates the crop area of ​​the sub-image. Indicates the scaling factor as follows: when For other more complex road conditions, they are handled in a similar way to forked roads. Depending on the different road topologies, different heuristic strategies can be adopted mainly based on the dynamic programming algorithm.

4. Subgraph deep neural network modeling based on middle layer interaction, characterized by: It includes the following steps: Step 1: The optimal subgraph set is ,Will Input the deep neural network (in addition to the sub-image itself, the input can also optionally add 2D position encoding, scale position encoding, 3D position encoding, etc.) to obtain the feature layers of each stage , s and S represent the stage number and total number. The deep neural network can select the commonly used CNN or VIT, and the network parameters of each subgraph can be shared or not. Step 2: Insert the following feature layer interaction operators after several high-level stages of the deep neural network: ,in You can choose The feature layers corresponding to the sub-images with more overlapped crop areas and similar scaling ratios on the original image. You can choose interpolation alignment or Deformable Attention Feature mapping and fusion , Deformable Attention refers to the Deformable DETR algorithm. In this way, the features of each sub-graph neural network can be reused.

5. The algorithm adaptation method based on the sub-image feature layer is characterized by: It includes the following parts: Algorithm adaptation method based on original image feature layer: The feature layer of the sth stage of the original image can be aggregated by the following sub-image feature layer aggregation operator Obtain, among which , and Op can choose interpolation alignment or Deformable Attention to aggregate the sub-image feature layer into the original image feature layer. In this way, algorithms based on the original image feature layer, such as the common Faster-rcnn, Yolo and other algorithms, can be seamlessly adapted. Algorithm adaptation method based on 2d-object-query: Perform global attention on the 2d-object-query and sub-image feature layers, or Perform sparse attention on the 2d-object-query and sub-image feature layers, and They correspond to the corresponding operators of DETR and Deformable DETR algorithms respectively. Represents the features of all 2d-object-queries, You can select the high-level feature layers of each sub-graph. This means that the spatial dimensions of the feature layers of each sub-graph are flattened and concat together. In this way, 2d-object-query-based algorithms such as DETR and Deformable DETR can be seamlessly adapted. Algorithm adaptation method based on BEV feature layer: using operators Indicates that multiple images are viewed from the The feature layer of BEV is constructed, and express The same operator is used for subgraph decomposition of Construct a BEV feature layer based on the sub-image feature layer; You can choose to aggregate 2D features to 3D positions based on LSS or 3D Deformable Attention. LSS and 3D Deformable Attention refer to BevDet and BevFormer algorithms respectively. In this way, algorithms based on the BEV feature layer, such as BevDet, BevFormer, etc., can be seamlessly adapted. Algorithm adaptation method based on 3d-object-query: using operators Indicates that the 3d-object-query looks at multiple images The feature layer query aggregate features, Represents the features of all 3d-object-queries, let express The same operator is used for subgraph decomposition of Used for 3d-object-query to query aggregate features from the sub-graph feature layer; You can choose to aggregate 2D features to 3d-object-query based on Global Cross Attention or 3D position projection interpolation, where Global Cross Attention and 3D position projection interpolation refer to Petr and Detr3D algorithms respectively. In this way, algorithms based on 3d-object-query, such as Petr, Detr3D, etc., can be seamlessly adapted.

6. Model acceleration method based on subgraph decomposition, characterized in that: It includes the following: The use of the subgraph decomposition-based modeling and adaptation method as claimed in claims 1 to 5 significantly reduces the input resolution, thereby achieving the purpose of model acceleration.

7. A method for reducing storage and transmission overhead based on subgraph decomposition, characterized in that: It includes the following: Before the perspective image is stored and transmitted, sub-image decomposition is first performed according to claims 1 to 3, and then only the sub-images are stored and transmitted, thereby reducing storage and transmission costs.

Images