Visual conversion method and device, electronic equipment, storage medium and program product

Abstract

The present application relates to the technical field of computer vision, and discloses a visual conversion method and device, electronic equipment, storage medium and program product.The method comprises the following steps: acquiring multiple images; performing feature extraction on the multiple images through a visual conversion model; compressing the image features in the height dimension and the width dimension; compressing the depth features in the height dimension and the width dimension; the visual conversion model comprises an image auxiliary branch and a depth auxiliary branch in the training stage, and is removed in the inference stage; the image auxiliary branch is used for supervising the image feature compression process; the depth auxiliary branch is used for supervising the depth feature compression process; the height image features and the height depth features are fused and projected to the longitudinal area of the bird's eye view; the width image features and the width depth features are fused and projected to the transverse area of the bird's eye view; the height intermediate features and the width intermediate features are fused and compressed to obtain the bird's eye view joint features.The above scheme improves the visual conversion efficiency.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and more specifically to visual conversion methods, devices, electronic devices, storage media, and program products. Background Technology

[0002] In the fields of autonomous driving and computer vision, the Bird's Eye View (BEV) space provides a top-down perspective, unifying data from different sensors into a common coordinate system. This simplifies the understanding and processing of the environment, enabling 3D perception modeling of the surroundings. Transforming the input from multiple surround-view cameras into the BEV space is a crucial prerequisite for subsequent modeling. This process typically requires processing large amounts of image data, involving numerous matrix operations and feature extraction.

[0003] In related technologies, bottom-up methods (such as LSS and BEVDet) or top-down methods (such as constructing BEV through Transformer) are used to transform high-dimensional view frustum features into BEV space. However, these methods involve huge computational loads or require high hardware computing power, often resulting in significant time consumption, making it difficult to achieve real-time operation, and exhibiting low efficiency and poor real-time performance, which is insufficient to meet the needs of autonomous driving scenarios. Summary of the Invention

[0004] This invention provides a visual conversion method, apparatus, electronic device, storage medium, and program product to solve the problems of low efficiency and poor real-time performance in related visual conversion methods.

[0005] In a first aspect, the present invention provides a visual transformation method, the method comprising:

[0006] Acquire multi-channel images from multiple cameras;

[0007] Feature extraction is performed on the multi-path images using a visual conversion model to obtain image features and depth features;

[0008] The image features are compressed in both height and width dimensions to obtain height image features and width image features.

[0009] The depth features are compressed in both height and width dimensions to obtain height-depth features and width-depth features. The visual transformation model includes an image-assisted branch and a depth-assisted branch during the training phase. The image-assisted branch predicts pixel-level semantic segmentation maps based on the height and width image features to supervise the image feature compression process. The depth-assisted branch predicts pixel-level depth maps based on the height and width depth features to supervise the depth feature compression process. The image-assisted branch and depth-assisted branch are removed during the inference phase.

[0010] The height image features and height depth features are fused and projected onto the vertical region of the bird's-eye view to obtain the intermediate height features;

[0011] The width image features and width and depth features are fused and projected onto the horizontal region of the bird's-eye view to obtain the width midpoint feature;

[0012] The height intermediate feature and the width intermediate feature are fused and channel compressed to obtain the bird's-eye view joint feature.

[0013] In one optional implementation, the compression of the image features in both height and width dimensions to obtain height image features and width image features includes:

[0014] Spatial compression of image features in the height dimension yields height image features;

[0015] Spatial compression of image features along the width dimension yields the width image features;

[0016] The compression of the depth features in both height and width dimensions to obtain height-depth features and width-depth features includes:

[0017] Spatial compression of depth features along the height dimension yields high-depth features;

[0018] Spatial compression of depth features along the width dimension yields width-depth features.

[0019] In one optional implementation, the step of fusing the height image features and height depth features and projecting them onto the vertical region of the bird's-eye view to obtain intermediate height features includes:

[0020] The height image features and height depth features are multiplied and fused to obtain the height fused intermediate features;

[0021] Based on the corresponding camera extrinsic parameters, the highly fused intermediate features are projected onto the vertical region of the bird's-eye view space to obtain the highly fused intermediate features;

[0022] The step of fusing the width image features and width-depth features and projecting them onto the horizontal region of the bird's-eye view to obtain the width midpoint feature includes:

[0023] The width image features and width-depth features are multiplied and fused to obtain the width-fused intermediate features;

[0024] The width fusion intermediate feature is projected onto the horizontal region of the bird's-eye view space based on the corresponding camera extrinsic parameters to obtain the width intermediate feature.

[0025] In one alternative implementation, the vertical region of the bird's-eye view includes a grid column in the width direction; the horizontal region of the bird's-eye view includes a grid row in the height direction.

[0026] In one optional implementation, the image auxiliary branch includes a lightweight decoding head; the lightweight decoding head restores the fused height and width image features to the original image resolution through an upsampling operation and outputs a pixel-level semantic segmentation map.

[0027] In an optional implementation, the method further includes:

[0028] During the training phase, a multi-task loss function is constructed; the multi-task loss function includes at least a main task sub-loss term, a semantic segmentation sub-loss term of the image auxiliary branch, and a deep regression sub-loss term of the depth auxiliary branch; wherein, each sub-loss term is dynamically weighted by learnable parameters;

[0029] The visual conversion model is updated via backpropagation based on the multi-task loss function.

[0030] In a second aspect, the present invention provides a visual conversion device, the device comprising:

[0031] The acquisition module is used to acquire multi-channel images from multiple cameras;

[0032] The feature extraction module is used to extract features from the multi-channel images through a visual transformation model to obtain image features and depth features;

[0033] The first compression module is used to compress the image features in the height and width dimensions respectively, to obtain the height image features and the width image features;

[0034] The second compression module is used to compress the depth features in both height and width dimensions to obtain height-depth features and width-depth features. The visual transformation model includes an image-assisted branch and a depth-assisted branch during the training phase. The image-assisted branch predicts pixel-level semantic segmentation maps based on the height and width image features to supervise the image feature compression process. The depth-assisted branch predicts pixel-level depth maps based on the height and width depth features to supervise the depth feature compression process. The image-assisted branch and depth-assisted branch are removed during the inference phase.

[0035] The first fusion module is used to fuse the height image features and height depth features, and project them onto the vertical region of the bird's-eye view to obtain the intermediate height features;

[0036] The second fusion module is used to fuse the width image features and the width and depth features, and project them onto the horizontal area of ​​the bird's-eye view to obtain the width midpoint feature.

[0037] The final fusion module is used to perform feature fusion and channel compression on the height intermediate feature and the width intermediate feature to obtain the joint feature of the bird's-eye view.

[0038] Thirdly, the present invention provides an electronic device, comprising: a memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the computer instructions to perform the visual conversion method of the first aspect or any corresponding embodiment described above.

[0039] Fourthly, the present invention provides a computer-readable storage medium storing computer instructions for causing a computer to perform the visual conversion method of the first aspect or any corresponding embodiment thereof.

[0040] Fifthly, the present invention provides a computer program product, including computer instructions for causing a computer to execute the visual conversion method of the first aspect or any corresponding embodiment thereof.

[0041] The technical solution provided by this invention may include the following beneficial effects:

[0042] The visual transformation method provided by this invention compresses the image features in both height and width dimensions, and also compresses the depth features in both height and width dimensions, decomposing the high-dimensional space into two orthogonal one-dimensional path processing methods. This avoids the computational redundancy caused by constructing complete five-dimensional view frustum features and performing global projection in related technologies, significantly reducing computational complexity and improving inference efficiency. Furthermore, by fusing height image features and height / depth features and projecting them onto the vertical region of the bird's-eye view, the method obtains the intermediate height feature of the structural information before and after encoding. Conversely, by fusing width image features and width / depth features and projecting them onto the horizontal region of the bird's-eye view, the method obtains the intermediate height feature of the structural information before and after encoding. The region is used to obtain the width of the middle feature reflecting the left and right layout. Finally, through dual-path feature fusion and channel compression, a complete and semantically rich two-dimensional bird's-eye view joint feature is generated. This not only ensures the spatial positioning accuracy, but also improves the model's ability to model the distribution of objects at near and far distances and sides, achieving efficient and accurate spatial mapping. By introducing an image auxiliary branch to supervise the quality of image features during the training phase, and introducing a depth auxiliary branch to optimize deep representation learning, the information retention ability during the image and depth feature compression process is effectively improved through backpropagation. The auxiliary branch is removed during the inference phase, which significantly improves the perceptual performance of the visual conversion model while ensuring real-time performance. Attached Figure Description

[0043] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0044] Figure 1 This is a schematic diagram of the first type of visual conversion method according to an embodiment of the present invention;

[0045] Figure 2 This is a schematic diagram of a second process of the visual conversion method according to an embodiment of the present invention;

[0046] Figure 3 This is a schematic diagram of the third process of the visual conversion method according to an embodiment of the present invention;

[0047] Figure 4 This is a structural block diagram of a visual conversion device according to an embodiment of the present invention;

[0048] Figure 5 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0050] It is understood that before using the technical solutions disclosed in the various embodiments of the present invention, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in the present invention and their authorization should be obtained in accordance with relevant laws and regulations through appropriate means.

[0051] The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0052] In the fields of autonomous driving and computer vision, the Bird's Eye View (BEV) space plays a crucial role. It provides a top-down perspective, unifying data from different sensors into a common coordinate system, thereby simplifying the understanding and processing of the environment and enabling 3D perception modeling of the surroundings. Transforming the input from multiple surround-view cameras into the BEV space is a critical prerequisite for subsequent modeling. This process typically requires processing large amounts of image data, involving numerous matrix operations and feature extraction.

[0053] In related technologies, two main categories of methods are used to convert surround-view multi-camera input to BEV space:

[0054] 1. Bottom-up methods, such as LSS (Lift, Splat, Shoot) (a BEV perception algorithm) and BEVDet (a BEV-based 3D object detection algorithm), require the transformation of high-dimensional frustum features into the BEV space, which involves a huge amount of computation.

[0055] 2. Top-down approach, such as building BEV through Transformer (a deep learning model architecture), requires high hardware computing power and may even require specially customized chips for acceleration, making deployment difficult.

[0056] Therefore, the methods in related technologies often consume a lot of time, making it difficult to achieve real-time operation. They are inefficient, have poor real-time performance, and are difficult to meet the needs of autonomous driving scenarios.

[0057] According to an embodiment of the present invention, a visual conversion method embodiment is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0058] This embodiment provides a visual conversion method that can be used in vehicle-mounted computers. Figure 1 This is a flowchart of a visual conversion method according to an embodiment of the present invention, such as... Figure 1 As shown, the process includes the following steps:

[0059] Step S101: Acquire multi-channel images from multiple cameras.

[0060] The onboard computer can be a vehicle-mounted domain controller or an autonomous driving computing platform. Multiple cameras are connected to the onboard computer to send acquired images. The onboard computer receives multiple images from the cameras, with each camera corresponding to one image stream. Each camera has image acquisition and depth estimation capabilities, and its intrinsic and extrinsic parameters are known for subsequent feature projection onto a bird's-eye view space. Each image contains both image information and depth information.

[0061] The specific number of cameras can be selected according to the actual application scenario. For example, in driving scenarios, a forward-looking three-eye or multi-eye camera module can be used, including a telephoto lens (narrow field of view, long detection distance), a standard medium telephoto lens (medium-range coverage, used for general target detection), and a wide-angle lens (wide field of view, large near-field field of view, suitable for turning and lane-cutting recognition). In parking scenarios, a four-way fisheye camera can be used, which can be installed in the front bumper, rear bumper, below the left rearview mirror, and below the right rearview mirror, with a very large field of view (usually ≥190°), capturing circular images with severe distortion. In fusion perception scenarios, it is necessary to cover all key areas around the vehicle. Three cameras (telephoto, medium telephoto, and wide-angle) can be used in the forward direction for long-distance detection and traffic light recognition. One wide-angle or standard lens can be used in the rear direction for rear obstacle detection and reversing assistance. One or two narrow-view or standard lenses can be used on the left and right sides for lane change assistance, door opening warning, and intersection collision avoidance. Four fisheye lenses can be used at the four corners for parking perception, low-speed obstacle avoidance, etc.

[0062] Step S102: The multi-channel images are used to extract features through a visual conversion model to obtain image features and depth features.

[0063] Image features are extracted from each image in the multi-channel image processing to obtain image features, which are used to express semantic content, including visual information such as color, edge, texture, etc. Depth features are extracted from each image in the multi-channel image processing to obtain depth features, which are used to express spatial geometric information, including distance, nearness, three-dimensional position, etc.

[0064] It should be noted that the visual transformation model includes an image-assisted branch and a depth-assisted branch during the training phase, which can be performed on a dedicated model training server. The image-assisted branch is used to predict pixel-level semantic segmentation maps based on height and width image features, serving as a first auxiliary supervision signal to supervise the image feature compression process and improve the expressive power of height and width image features. The depth-assisted branch is used to predict pixel-level depth maps based on height and width depth features, serving as a second auxiliary supervision signal to supervise the depth feature compression process and enhance the learning quality of height and width depth features.

[0065] After training, the model is deployed in the onboard computer and enters the inference phase to perform model inference for the autonomous driving task. Steps S101 to S107 are executed once whenever multiple images from multiple cameras are acquired. The image-assisted branch and the depth-assisted branch are removed during the inference phase. That is, the only difference in network structure between the visual translation model in the training phase and the visual translation model in the inference phase is that the visual translation model in the training phase still includes the image-assisted branch and the depth-assisted branch. In other words, apart from executing the steps related to the image-assisted branch and the depth-assisted branch, the visual translation model in the training phase performs steps similar to steps S101 to S107 in the inference phase.

[0066] Step S103: Compress the image features in both height and width dimensions to obtain height image features and width image features.

[0067] Image features are compressed in the height dimension (vertical direction of the image) to obtain height image features, preserving the contextual semantics along the height direction and reflecting the distribution of objects at different distances. Image features are also compressed in the width dimension (horizontal direction of the image) to obtain width image features, encoding lateral layout information along the width direction. This step decouples and extracts structured semantic representations along two orthogonal directions (height and width) from the two-dimensional image feature map, achieving efficient dimensionality reduction and orientation-specific modeling of spatial information.

[0068] Step S104: Compress the depth features in both height and width dimensions to obtain height depth features and width depth features.

[0069] Depth features are compressed in the height dimension to obtain height-depth features; depth features are compressed in the width dimension to obtain width-depth features. This step captures the vertical depth trend and lateral ground extension characteristics at different distances, achieving dimensionality reduction and directional decoupling of depth information, providing geometric guidance for subsequent directional fusion with image features.

[0070] In related technologies, when converting an image to a bird's-eye view space, it is necessary to construct a complete five-dimensional view frustum feature (N,D,H,W,C) and perform global projection. However, in this embodiment, before feature conversion, the image features and depth features are dimensionally compressed in the height and width directions, respectively, to extract lightweight "height prime features" and "width prime features". By decomposing the high-dimensional space into two orthogonal one-dimensional paths, redundant full-view feature modeling is avoided, significantly reducing matrix operations and memory usage, thereby effectively reducing overall computational overhead and improving visual conversion efficiency.

[0071] Step S105: The height image features and height depth features are fused and projected onto the vertical region of the bird's-eye view to obtain the intermediate height features.

[0072] By fusing height image features and height-depth features in the height dimension, it is possible to achieve structured modeling of the space ahead along the vehicle's direction of travel by combining the semantic information of image features in the vertical direction with the depth distribution of the corresponding positions. This allows for the identification of which height rows correspond to nearby objects and which belong to the distant background. Based on this, the effective features are accurately projected into the vertical grid of the bird's-eye view space, forming a vertical region of the bird's-eye view that contains both distance and semantic information. This provides a key basis for the forward and backward layout of the subsequent construction of a complete two-dimensional bird's-eye view, while significantly reducing the overhead of full-dimensional view frustum calculation.

[0073] Step S106: The width image features and width-depth features are fused and projected onto the horizontal region of the bird's-eye view to obtain the width intermediate features.

[0074] By fusing the width and depth features of the image, the spatial layout of objects on both sides of the road can be modeled by combining the semantic information of the image features in the horizontal direction with the corresponding depth distribution. This enables efficient encoding of the lateral structure in the bird's-eye view space, identifies which drivable areas or obstacles exist in the lateral positions, and projects them into the horizontal grid of the bird's-eye view space based on the depth information, forming a horizontal region of the bird's-eye view that reflects the left and right distribution. This provides orthogonal support for subsequent fusion with vertical features to generate a complete two-dimensional bird's-eye view, improving spatial positioning accuracy while reducing computational overhead.

[0075] Step S107: Perform feature fusion and channel compression on the height intermediate feature and the width intermediate feature to obtain the joint feature of the bird's-eye view.

[0076] By fusing two orthogonal intermediate features (height intermediate feature and width intermediate feature) that encode the vertical structure (front and back) and the horizontal structure (left and right) respectively in the bird's-eye view space, efficient cross-modeling of semantic and geometric information is achieved, generating a high-dimensional joint representation covering the entire two-dimensional ground region. Subsequently, redundancy is removed and feature dimensions are integrated through channel compression, reducing computational overhead while preserving key spatial semantics. The final output is a compact and expressive bird's-eye view joint feature, providing a high-quality, low-latency unified perception foundation for subsequent downstream tasks such as object detection and segmentation.

[0077] The visual transformation method provided in this embodiment compresses the image features in both height and width dimensions, and also compresses the depth features in both height and width dimensions. This decomposes the high-dimensional space into two orthogonal one-dimensional path processing methods, avoiding the computational redundancy caused by constructing complete five-dimensional view frustum features and performing global projection in related technologies. This significantly reduces computational complexity and improves inference efficiency. By fusing the height image features and the height and depth features and projecting them onto the vertical region of the bird's-eye view, the intermediate height feature of the structural information before and after encoding is obtained. Similarly, the width image features and the width and depth features are fused and projected onto the horizontal region of the bird's-eye view. The model obtains the width midpoint feature reflecting the left-right layout from the region. Finally, through dual-path feature fusion and channel compression, it generates a complete and semantically rich 2D bird's-eye view joint feature, which not only ensures spatial positioning accuracy but also improves the model's ability to model the distribution of objects at near and far distances and sides, achieving efficient and accurate spatial mapping. By introducing an image auxiliary branch to supervise the quality of image features during the training phase and introducing a depth auxiliary branch to optimize deep representation learning, the model effectively improves the information retention capability during image and depth feature compression through backpropagation. Finally, the auxiliary branch is removed during the inference phase, which significantly improves the perceptual performance of the visual conversion model while ensuring real-time performance.

[0078] This embodiment provides a visual conversion method that can be used in vehicle-mounted computers. Figure 2 This is a flowchart of a visual conversion method according to an embodiment of the present invention, such as... Figure 2 As shown, the process includes the following steps:

[0079] Step S201: Obtain the visual conversion model.

[0080] Receive a visual translation model pre-trained by a cloud / server. The cloud / server will train the visual translation model using sample images with labeled information.

[0081] The visual transformation model includes an image-assisted branch and a depth-assisted branch during the training phase. The image-assisted branch is used to predict pixel-level semantic segmentation maps based on height and width image features to supervise the image feature compression process. The depth-assisted branch is used to predict pixel-level depth maps based on height and width depth features to supervise the depth feature compression process. The image-assisted branch and the depth-assisted branch are removed during the inference phase.

[0082] The image-assisted branch includes a lightweight decoding head; this lightweight decoding head restores the fused height and width image features to the original image resolution through upsampling operations and outputs a pixel-level semantic segmentation map.

[0083] Optionally, during the training phase, a multi-task loss function is constructed; this multi-task loss function includes at least a main task sub-loss term, a semantic segmentation sub-loss term for the image auxiliary branch, and a deep regression sub-loss term for the deep auxiliary branch; wherein each sub-loss term is dynamically weighted by learnable parameters; then, the visual conversion model is updated by backpropagation based on this multi-task loss function.

[0084] Optionally, the semantic segmentation sub-loss term in the image-assisted branch adopts the cross-entropy loss function to supervise the model in learning more discriminative image orientation compression features; the depth regression sub-loss term in the depth-assisted branch adopts L1 loss or Berhu loss to supervise the structural consistency of depth features.

[0085] By setting an image-assisted branch, backpropagation can optimize image compression features, forcing the model to retain sufficient semantic information when compressing image features in the width / height directions, avoiding the loss of key features due to pooling or dimensionality reduction; it enhances the semantic expressive power of height and width image features, making them not only contain location information but also class discrimination (e.g., whether a region is a car or a road); it alleviates the gradient sparsity problem of the main task, providing an additional supervision path so that shallow features can also receive effective gradient update signals, accelerating convergence. By setting a depth-assisted branch, the spatial consistency of depth features can be strengthened, ensuring that the compressed directional depth features still accurately reflect the 3D geometric structure; it improves the accuracy of depth estimation, and explicit depth modeling helps to correctly project image features into the bird's-eye view space, especially for the localization of objects at different distances; it improves the accuracy of bird's-eye view projection, effectively avoiding incorrect depth causing features to be projected to the wrong location.

[0086] Since the inference goal of the visual conversion model in the inference stage is to efficiently generate bird's-eye view features for detection or planning, without the need for additional output of segmentation maps or depth maps, and the main path is sufficient to support downstream tasks, the image auxiliary branch and depth auxiliary branch are removed in the inference stage to simplify the structure of the visual conversion model, avoid redundant calculations, reduce the number of parameters and memory usage, make it more suitable for automotive chip deployment, and maintain end-to-end efficiency.

[0087] Step S202: Acquire multi-channel images from multiple cameras.

[0088] Please see details Figure 1 Step S101 of the illustrated embodiment will not be described again here.

[0089] Step S203: The multi-channel images are used to extract features through a visual conversion model to obtain image features and depth features.

[0090] Optionally, image feature extraction is performed by the backbone feature extraction network of the visual translation model, such as ResNet-18 / 34 / 50; depth feature extraction is performed by the depth estimation head of the visual translation model to predict the depth distribution from the image features and generate depth features.

[0091] Optionally, a cross-camera attention mechanism can be used to spatially align image features from multiple cameras to compensate for geometric and semantic differences between adjacent views in overlapping regions. This cross-camera attention mechanism constructs attention weights based on the extrinsic parameters of each camera. Specifically, it determines the spatial correspondence between the query and the key based on the intersection of the field of view angles between two cameras and applies a geometrically constrained attention mask. It should be noted that this cross-camera attention mechanism can also be enabled only during the training phase and removed during the inference phase to reduce latency.

[0092] Step S204: Compress the image features in both height and width dimensions to obtain height image features and width image features.

[0093] Specifically, spatial compression is performed on image features in the height dimension to obtain height image features; spatial compression is performed on image features in the width dimension to obtain width image features.

[0094] Optionally, global average pooling or global max pooling operations can be used to reduce the dimensionality along the width and height dimensions of the image features, respectively, to obtain the height image features and the width image features.

[0095] Optionally, the visual transformation model can perform different perception tasks in different application scenarios. When compressing the image features in a high dimension, it can also compress the image features in a high dimension according to a preset attention weight map along the height direction. The preset attention weight map is used to indicate the importance of different vertical positions to the current perception task.

[0096] Step S205: Compress the depth features in both height and width dimensions to obtain height depth features and width depth features.

[0097] Specifically, spatial compression is performed on the depth features in the height dimension to obtain the height depth features; spatial compression is performed on the depth features in the width dimension to obtain the width depth features.

[0098] Optionally, pooling operations can be performed along the height and width dimensions respectively to obtain height-depth features and width-depth features.

[0099] Optionally, steps S204 and S205 are performed by the directional feature compression module of the visual conversion model.

[0100] Step S206: The height image features and height depth features are fused and projected onto the vertical region of the bird's-eye view to obtain the intermediate height features.

[0101] Specifically, the height image feature and the height depth feature are multiplied and fused to obtain the height fusion intermediate feature; the height fusion intermediate feature is then projected onto the vertical region of the bird's-eye view space according to the corresponding camera extrinsic parameters to obtain the height intermediate feature; the vertical region of the bird's-eye view contains only one grid column in the width direction.

[0102] Step S207: The width image feature and the width and depth features are fused and projected onto the horizontal region of the bird's-eye view to obtain the width intermediate feature.

[0103] Specifically, the width image feature and the width and depth features are multiplied and fused to obtain the width fused intermediate feature; the width fused intermediate feature is then projected onto the horizontal region of the bird's-eye view space according to the corresponding camera extrinsic parameters to obtain the width intermediate feature; the horizontal region of the bird's-eye view contains a grid row in the height direction.

[0104] Step S208: Perform feature fusion and channel compression on the height intermediate feature and the width intermediate feature to obtain the joint feature of the bird's-eye view.

[0105] The height and width intermediate features are fused by outer product, and then channel compression is performed to adapt to the input requirements of downstream detection or segmentation heads, resulting in the joint features of the bird's-eye view.

[0106] Optionally, steps S206 to S208 are performed by the bird's-eye view projection and fusion module of the visual transformation model.

[0107] After obtaining the joint features of the bird's-eye view, the joint features of the bird's-eye view are further processed by the downstream task detection / segmentation head of the visual transformation model. Based on the task type executed by the visual transformation model, the target detection box or semantic segmentation result under the bird's-eye view is output. This embodiment mainly focuses on the visual transformation process, and the subsequent detection / segmentation process will not be described in detail.

[0108] As one or more specific application embodiments of the present invention, the optimal implementation scheme or the scheme that the inventors most want to embody is described in combination with the specific application scenario.

[0109] Figure 3 This is a flowchart of a visual conversion method according to an embodiment of the present invention, such as... Figure 3 As shown, the process includes the following steps:

[0110] a. Obtain multi-view visual images, extract the Iw dimension feature nch (width image feature, I represents image, w represents width) and the Ih dimension feature ncw (height image feature, h represents height) respectively, and perform dot product on the Iw dimension feature and Ih dimension feature of each image. If the dimensions dot product is mismatched, add 1 to match, for example, ncw×nch=nc1w×nch1=nchw. Use this feature to segment the image of that image as auxiliary branch 1 (image auxiliary branch), which only participates in training and does not participate in subsequent model inference.

[0111] b. Extract the Dw-dimensional feature ndh (width depth feature, D represents depth) and the Dh-dimensional feature ndw (height depth feature) of the depth features respectively, and perform a dot product of the Dw-dimensional feature and Dh-dimensional feature of each image. Use this feature to predict the depth of the image for that image as auxiliary branch 2 (depth auxiliary branch), which only participates in training and does not participate in subsequent model inference.

[0112] c. Assuming there are four cameras (front, back, left, and right), extract the Ih-dimensional image feature ncw and the Dh-dimensional depth feature ndw from each camera. Multiply the Ih-dimensional image feature and the Dh-dimensional depth feature by a dot product to obtain a new feature ncdw, and project it onto bev1(1×w0) (the vertical region of the bird's-eye view) to obtain a new feature fb1(ncdw0).

[0113] d. Extract the Iw-dimensional image features nch and the Dw-dimensional depth features ndh from each camera. Multiply the Iw-dimensional image features and the Dw-dimensional depth features by a dot product to obtain a new feature ncdh, and project it onto bev2(h0×1) (the horizontal region of the bird's-eye view) to obtain a new feature fb2(ncdh0).

[0114] e. Perform a dot product on fb1 and fb2 to obtain the feature (ncdh0w0) on the new bev(h0×w0).

[0115] f. Compress the new feature (ncdh0w0) in the dimension to obtain a new (nch0w0), and upsample it to obtain (n×c×(2h0)×(2w0)) for subsequent target segmentation or detection tasks.

[0116] This embodiment also provides a visual conversion device for implementing the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0117] This embodiment provides a visual conversion device, such as Figure 4 As shown, it includes:

[0118] The acquisition module 401 is used to acquire multi-channel images from multiple cameras;

[0119] Feature extraction module 402 is used to extract features from the multi-channel images through a visual conversion model to obtain image features and depth features;

[0120] The first compression module 403 is used to compress the image features in the height and width dimensions respectively to obtain height image features and width image features;

[0121] The second compression module 404 is used to compress the depth features in both the height and width dimensions to obtain height-depth features and width-depth features. During the training phase, the visual transformation model includes an image-assisted branch and a depth-assisted branch. The image-assisted branch predicts pixel-level semantic segmentation maps based on the height and width image features to supervise the image feature compression process. The depth-assisted branch predicts pixel-level depth maps based on the height and width depth features to supervise the depth feature compression process. Both the image-assisted branch and the depth-assisted branch are removed during the inference phase.

[0122] The first fusion module 405 is used to fuse the height image features and the height depth features, and project them onto the vertical region of the bird's-eye view to obtain the intermediate height features.

[0123] The second fusion module 406 is used to fuse the width image features and the width and depth features, and project them onto the horizontal area of ​​the bird's-eye view to obtain the width mid-width features.

[0124] The final fusion module 407 is used to perform feature fusion and channel compression on the intermediate height feature and the intermediate width feature to obtain the joint features of the bird's-eye view.

[0125] In an optional implementation, the first compression module is further configured to:

[0126] Spatial compression of image features in the height dimension yields height image features;

[0127] Spatial compression of image features along the width dimension yields the width image features;

[0128] The second compression module is also used for:

[0129] Spatial compression of depth features along the height dimension yields high-depth features;

[0130] Spatial compression of depth features along the width dimension yields width-depth features.

[0131] In an optional implementation, the first fusion module is further configured to:

[0132] The height image features and height depth features are multiplied and fused to obtain the height fused intermediate features;

[0133] Based on the corresponding camera extrinsic parameters, the height-fused intermediate feature is projected onto the vertical region of the bird's-eye view space to obtain the height intermediate feature;

[0134] The second fusion module is also used for:

[0135] The width image features and width and depth features are multiplied and fused to obtain the width fused intermediate features;

[0136] Based on the corresponding camera extrinsic parameters, the width fusion intermediate feature is projected onto the horizontal region of the bird's-eye view space to obtain the width intermediate feature.

[0137] In one alternative implementation, the vertical region of the bird's-eye view includes a grid column in the width direction; the horizontal region of the bird's-eye view includes a grid row in the height direction.

[0138] In one alternative implementation, the image-assisted branch includes a lightweight decoding head; the lightweight decoding head restores the fused height and width image features to the original image resolution through an upsampling operation and outputs a pixel-level semantic segmentation map.

[0139] In an optional implementation, the method further includes a model optimization module for:

[0140] During the training phase, a multi-task loss function is constructed; this multi-task loss function includes at least a main task sub-loss term, a semantic segmentation sub-loss term of the image auxiliary branch, and a deep regression sub-loss term of the deep auxiliary branch; wherein, each sub-loss term is dynamically weighted by learnable parameters;

[0141] The visual conversion model is updated via backpropagation based on this multi-task loss function.

[0142] The visual conversion device provided in this embodiment of the invention can execute the visual conversion method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects for executing the method. Further functional descriptions of the various modules and units described above are the same as in the corresponding embodiments described above, and will not be repeated here.

[0143] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention.

[0144] The following is a detailed reference. Figure 5This diagram illustrates a suitable structural schematic for implementing an electronic device according to embodiments of the present invention. The electronic device may include a processor (e.g., a central processing unit, graphics processor, etc.) 501, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 502 or a program loaded from memory 508 into random access memory (RAM) 503. The RAM 503 also stores various programs and data required for the operation of the electronic device. The processor 501, ROM 502, and RAM 503 are interconnected via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0145] Typically, the following devices can be connected to I / O interface 505: input devices 506 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 507 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; memory devices 508 including, for example, magnetic tapes, hard disks, etc.; and communication devices 509. Communication device 509 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 5 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown, and more or fewer devices may be implemented or have instead.

[0146] In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 509, or installed from a memory 508, or installed from a ROM 502. When the computer program is executed by the processor 501, it performs the functions defined in the visual conversion method of the embodiments of the present invention.

[0147] Figure 5 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of the present invention.

[0148] This invention also provides a computer-readable storage medium. The methods described above according to embodiments of the invention can be implemented in hardware or firmware, or implemented as recordable on a storage medium, or implemented as computer code originally stored on a remote storage medium or a non-transitory machine-readable storage medium and subsequently stored on a local storage medium after being downloaded via a network. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium may also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code. When the software or computer code is accessed and executed by the computer, processor, or hardware, the visual conversion method shown in the above embodiments is implemented.

[0149] A portion of this invention can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide the methods and / or technical solutions according to the invention through the operation of the computer. Those skilled in the art will understand that the forms in which computer program instructions exist in a computer-readable medium include, but are not limited to, source files, executable files, installation package files, etc. Correspondingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executing the instructions, or the computer compiling the instructions and then executing the corresponding compiled program, or the computer reading and executing the instructions, or the computer reading and installing the instructions and then executing the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to a computer.

[0150] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations all fall within the scope defined by the invention.

Claims

1. A visual transformation method, characterized in that, The method includes: Acquire multi-channel images from multiple cameras; Feature extraction is performed on the multi-path images using a visual conversion model to obtain image features and depth features; The image features are compressed in both height and width dimensions to obtain height image features and width image features. The depth features are compressed in both height and width dimensions to obtain height-depth features and width-depth features. The visual transformation model includes an image-assisted branch and a depth-assisted branch during the training phase. The image-assisted branch predicts pixel-level semantic segmentation maps based on the height and width image features to supervise the image feature compression process. The depth-assisted branch predicts pixel-level depth maps based on the height and width depth features to supervise the depth feature compression process. The image-assisted branch and depth-assisted branch are removed during the inference phase. The height image features and height depth features are fused and projected onto the vertical region of the bird's-eye view to obtain the intermediate height features; The width image features and width and depth features are fused and projected onto the horizontal region of the bird's-eye view to obtain the width midpoint feature; The height intermediate feature and the width intermediate feature are fused and channel compressed to obtain the joint feature of the bird's-eye view; The step of fusing the height image features and height depth features and projecting them onto the vertical region of the bird's-eye view to obtain intermediate height features includes: The height image features and height depth features are multiplied and fused to obtain the height fused intermediate features; Based on the corresponding camera extrinsic parameters, the highly fused intermediate features are projected onto the vertical region of the bird's-eye view space to obtain the highly fused intermediate features; The step of fusing the width image features and width-depth features and projecting them onto the horizontal region of the bird's-eye view to obtain the width midpoint feature includes: The width image features and width-depth features are multiplied and fused to obtain the width-fused intermediate features; Based on the corresponding camera extrinsic parameters, the width fusion intermediate feature is projected onto the horizontal region of the bird's-eye view space to obtain the width intermediate feature; The method further includes: During the training phase, a multi-task loss function is constructed; the multi-task loss function includes at least a main task sub-loss term, a semantic segmentation sub-loss term of the image auxiliary branch, and a deep regression sub-loss term of the depth auxiliary branch; wherein, each sub-loss term is dynamically weighted by learnable parameters; The visual conversion model is updated via backpropagation based on the multi-task loss function.

2. The method according to claim 1, characterized in that, The process of compressing the image features in both height and width dimensions to obtain height image features and width image features includes: Spatial compression of image features in the height dimension yields height image features; Spatial compression of image features along the width dimension yields the width image features; The compression of the depth features in both height and width dimensions to obtain height-depth features and width-depth features includes: Spatial compression of depth features along the height dimension yields high-depth features; Spatial compression of depth features along the width dimension yields width-depth features.

3. The method according to claim 1, characterized in that, The bird's-eye view's vertical region contains a grid column in the width direction; the bird's-eye view's horizontal region contains a grid row in the height direction.

4. The method according to claim 1, characterized in that, The image-assisted branch includes a lightweight decoding head; the lightweight decoding head restores the fused height and width image features to the original image resolution through upsampling operations and outputs a pixel-level semantic segmentation map.

5. A visual conversion device, characterized in that, The device includes: The acquisition module is used to acquire multi-channel images from multiple cameras; The feature extraction module is used to extract features from the multi-channel images through a visual transformation model to obtain image features and depth features; The first compression module is used to compress the image features in the height and width dimensions respectively, to obtain the height image features and the width image features; The second compression module is used to compress the depth features in both height and width dimensions to obtain height-depth features and width-depth features. The visual transformation model includes an image-assisted branch and a depth-assisted branch during the training phase. The image-assisted branch predicts pixel-level semantic segmentation maps based on the height and width image features to supervise the image feature compression process. The depth-assisted branch predicts pixel-level depth maps based on the height and width depth features to supervise the depth feature compression process. The image-assisted branch and depth-assisted branch are removed during the inference phase. The first fusion module is used to fuse the height image features and height depth features, and project them onto the vertical region of the bird's-eye view to obtain the intermediate height features; The second fusion module is used to fuse the width image features and the width and depth features, and project them onto the horizontal area of ​​the bird's-eye view to obtain the width midpoint feature. The final fusion module is used to perform feature fusion and channel compression on the height intermediate feature and the width intermediate feature to obtain the joint features of the bird's-eye view; The first fusion module is also used for: The height image features and height depth features are multiplied and fused to obtain the height fused intermediate features; Based on the corresponding camera extrinsic parameters, the highly fused intermediate features are projected onto the vertical region of the bird's-eye view space to obtain the highly fused intermediate features; The second fusion module is also used for: The width image features and width-depth features are multiplied and fused to obtain the width-fused intermediate features; Based on the corresponding camera extrinsic parameters, the width fusion intermediate feature is projected onto the horizontal region of the bird's-eye view space to obtain the width intermediate feature; The device further includes a training module for: During the training phase, a multi-task loss function is constructed; the multi-task loss function includes at least a main task sub-loss term, a semantic segmentation sub-loss term of the image auxiliary branch, and a deep regression sub-loss term of the depth auxiliary branch; wherein, each sub-loss term is dynamically weighted by learnable parameters; The visual conversion model is updated via backpropagation based on the multi-task loss function.

6. An electronic device, characterized in that, include: A memory and a processor are communicatively connected, the memory storing computer instructions, and the processor executing the computer instructions to perform the visual conversion method of any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing a computer to perform the visual conversion method according to any one of claims 1 to 4.

8. A computer program product, characterized in that, Includes computer instructions for causing a computer to perform the visual conversion method according to any one of claims 1 to 4.

Images