Data processing methods and apparatus, electronic devices, storage media, and software products

Abstract

This disclosure provides a data processing method and apparatus, electronic device, storage medium, and program product, belonging to the field of computer technology. The method includes: acquiring first image information from a first perspective of a first vehicle; identifying obstacles contained in the first image information and generating a mask containing occlusion regions obscured by the obstacles; obtaining a text embedding vector of a prompt text, the prompt text describing an object located in the occlusion region; generating a target semantic completion image of the first image information based on the first image information, the mask, and the text embedding vector, the target semantic completion image containing completions of objects in the occlusion region; and generating second image information from a second perspective of the first vehicle based on the first image information and the target semantic completion image.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and more specifically, to a data processing method and apparatus, electronic equipment, computer-readable storage medium, and computer program product. Background Technology

[0002] Vision-based autonomous driving methods typically follow an end-to-end design architecture, directly generating a Bird's Eye View (BEV) map from RGB (Red, Green, Blue) image information obtained from a front-facing monocular camera. However, occlusion from surrounding obstacles, such as other vehicles, can lead to the loss of some road background semantic information, such as obscured road edges or the direct line of sight. This poses a significant obstacle to the semantic integrity of the BEV and subsequent planning. Therefore, further understanding and reasoning of this occluded information is necessary.

[0003] Related technologies attempt to supplement information by using multi-view images or depth maps, and combine them with feature information corresponding to the occluded area to complete the BEV generation effect.

[0004] However, in perspective projection, when objects occlude each other, the occluded parts appear as partial absences or distortions in the perspective drawing. This occlusion becomes even more complex when projecting a given perspective view into a top-down view, as the top view is a vertical view from above, significantly different from the perspective view. Therefore, during the conversion process, occluded areas exhibit divergence / convergence due to changes in the projection angle, appearing as radial or irregular stripes on the BEV map. This increases the complexity of semantic understanding of the regions and further impacts the reliability of driving perception. Accurate identification of elements such as roads, vehicles, and pedestrians is crucial, especially in applications like autonomous driving. If elements on the BEV map become difficult to identify due to occlusion and projection issues, the decisions of the autonomous driving system will be affected. Summary of the Invention

[0005] This disclosure provides a data processing method, which includes: acquiring first image information from a first perspective of a first vehicle; identifying obstacles contained in the first image information and generating a mask containing occlusion areas obscured by the obstacles; obtaining a text embedding vector of a prompt text, the prompt text being used to describe an object located in the occlusion area; generating a target semantic completion image of the first image information based on the first image information, the mask, and the text embedding vector, the target semantic completion image containing completions of objects in the occlusion area; and generating second image information from a second perspective of the first vehicle based on the first image information and the target semantic completion image.

[0006] This disclosure provides a data processing apparatus, comprising: a data acquisition module for acquiring first image information from a first perspective of a first vehicle; a perception and reasoning module for identifying obstacles contained in the first image information and generating a mask containing occlusion areas obscured by the obstacles; obtaining a text embedding vector of a prompt text, the prompt text being used to describe an object located in the occlusion area; generating a target semantic completion image of the first image information based on the first image information, the mask, and the text embedding vector, the target semantic completion image containing completions of objects in the occlusion area; and an image generation module for generating second image information from a second perspective of the first vehicle based on the first image information and the target semantic completion image.

[0007] This disclosure provides an electronic device, including: one or more processors; and a memory configured to store one or more programs, which, when executed by the one or more processors, cause the electronic device to implement the method described in any embodiment of this disclosure.

[0008] This disclosure provides a computer-readable storage medium having a computer program stored thereon, which, when run on a computer, causes the computer to execute the methods described in any embodiment of this disclosure.

[0009] This disclosure provides a computer program product, including a computer program that, when executed by a computer, implements the methods described in any embodiment of this disclosure. Attached Figure Description

[0010] Figure 1 This is a flowchart of a data processing method according to an exemplary embodiment of the present disclosure.

[0011] Figure 2 This is a schematic diagram of the framework of an end-to-end autonomous driving system according to an exemplary embodiment of the present disclosure.

[0012] Figure 3 This is a schematic diagram of a semantic completion network processing procedure according to an exemplary embodiment of this disclosure.

[0013] Figure 4 This is a schematic diagram of the structure of a generative network according to an exemplary embodiment of this disclosure.

[0014] Figure 5 This is a schematic diagram of the feature matching and fusion module of an exemplary embodiment of this disclosure.

[0015] Figure 6 This is a schematic diagram of the structure of a data processing apparatus according to an exemplary embodiment of the present disclosure.

[0016] Figure 7 A schematic structural diagram of an electronic device according to an embodiment of the present disclosure is shown. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this disclosure more apparent, exemplary embodiments according to this disclosure will now be described in detail with reference to the accompanying drawings. In the drawings, the same reference numerals denote the same elements throughout. It should be understood that the embodiments described herein are merely illustrative and should not be construed as limiting the scope of this disclosure.

[0018] Figure 1 This is a flowchart of a data processing method according to an exemplary embodiment of the present disclosure. Figure 1 The methods provided in the embodiments can be executed by any electronic device, such as a terminal (e.g., including vehicle-mounted terminals) and / or a server, and this disclosure does not limit this. Figure 1 As shown, the method provided in this embodiment includes the following steps.

[0019] In S110, the first image information from the first perspective of the first vehicle is acquired.

[0020] The first vehicle in this disclosure embodiment can be any means of transportation. In the following embodiments, the first vehicle is illustrated as any autonomous vehicle, but this disclosure is not limited thereto. An autonomous vehicle refers to an intelligent car that enables the vehicle to drive autonomously without human intervention or with only partial human intervention through artificial intelligence, sensors, and other technologies.

[0021] In this embodiment of the disclosure, an on-board sensor is installed at an appropriate location on the first vehicle, which can acquire first image information of the first vehicle from a first perspective. Exemplarily, the on-board sensor includes a camera (hereinafter referred to as an external camera) or image sensor installed at an appropriate location on the exterior of the first vehicle. In some embodiments, the external camera can be a monocular camera and / or a surround-view camera. In some embodiments, the monocular camera can be installed at any appropriate location on the first vehicle, for example, capturing images or videos facing the front and / or rear and / or sides of the first vehicle. The surround-view camera can be used to capture images or videos of the environment surrounding the first vehicle. Exemplarily, the on-board sensor includes any one or more of lidar, millimeter-wave radar, ultrasonic radar, etc., installed on the first vehicle.

[0022] In this embodiment of the disclosure, the first perspective is relative to the subsequent second perspective, and the first perspective and the second perspective are different. For example, the first image information of the first perspective includes images captured by the vehicle's onboard sensors relative to the normal driving direction of the first vehicle (e.g., the direction the front of the vehicle is facing), i.e., image information of the front of the first vehicle. For example, the first image information of the first perspective includes images captured by the vehicle's onboard sensors in the opposite direction to the normal driving direction of the first vehicle (e.g., the direction the rear of the vehicle is facing), i.e., image information of the rear of the first vehicle, such as during reversing. For example, the first image information of the first perspective includes images or videos of the surrounding environment captured by the vehicle's onboard sensors.

[0023] In the following embodiments, the first perspective is the front of the first vehicle, and the second perspective is a downward view relative to the ground, and it is assumed that the ground is flat, but this disclosure is not limited to this.

[0024] In an exemplary embodiment, the first image information is a perspective view. A perspective view, based on the principle of perspective, projects objects in three-dimensional space onto a two-dimensional plane to show their three-dimensionality and spatial relationships. A perspective view is a graphic obtained by projecting objects in three-dimensional space onto a two-dimensional plane in a manner that simulates human eye observation. That is, the first image information is two-dimensional.

[0025] In S120, obstacles contained in the first image information are identified, and a mask containing the occlusion area obscured by the obstacles is generated.

[0026] In this embodiment of the disclosure, image detection can identify the targets, their categories, and locations contained in the first image information. Based on the category of each target, obstacles can be identified, and based on the location of the obstacles, a corresponding mask m is generated. This disclosure does not limit the specific image detection method.

[0027] In this embodiment of the disclosure, an obstacle refers to any target in the first image information that may affect the autonomous driving decision of the first vehicle when the first vehicle is in autonomous driving mode and needs to rely on the second image information of the converted second perspective to make autonomous driving decisions. For example, the first vehicle is driving normally on the road, and there may be other vehicles on the road (hereinafter referred to as second vehicles, that is, vehicles other than the first vehicle that exist in the first image information, which may be one or more vehicles). Other vehicles may obstruct road signs, pedestrians, and buildings along the road. Therefore, in the following embodiments, other vehicles identified in the first image information are used as examples of obstacles, but this disclosure is not limited to this.

[0028] In this embodiment, the mask m represents the occluded and unoccluded areas in the first image information using different pixel values. The specific pixel values ​​are not limited, as long as they can distinguish between the occluded and unoccluded areas. The unoccluded area refers to the area in the first image information other than the occluded area.

[0029] In S130, the text embedding vector of the prompt text is obtained, the prompt text being used to describe the object in the occluded area.

[0030] In this embodiment of the disclosure, a prompt text is obtained, which is used to guide the prediction of objects contained in the occluded area in the first image information, so as to complete the objects such as roads or buildings that are obscured by obstacles in the occluded area.

[0031] In this embodiment, the prompt text can be converted to obtain its text embedding vector. This disclosure does not limit the method used to convert the prompt text into a text embedding vector.

[0032] In S140, a target semantic completion image of the first image information is generated based on the first image information, the mask, and the text embedding vector. The target semantic completion image includes the completion of objects in the occluded region.

[0033] In an exemplary embodiment, generating a target semantic completion image of the first image information based on the first image information, the mask, and the text embedding vector includes: performing noise processing t times on the first image information to obtain a first noisy image at a first time step, where t is a positive integer greater than 1; performing noise reduction processing on the first noisy image at the first time step based on the text embedding vector to obtain a first noisy image at the first time step; performing noise processing t-1 times on the first image information to obtain a second noisy image at the first time step; obtaining a semantic completion image at the first time step based on the first noisy image at the first time step, the second noisy image at the first time step, and the mask; and obtaining the target semantic completion image based on the semantic completion image at the first time step.

[0034] In an exemplary embodiment, obtaining the target semantically complete image based on the semantically complete image at the first time step includes: performing t noise addition processing on the semantically complete image at the first time step to obtain a first noisy image at the second time step, where t is a positive integer greater than 1; performing noise reduction processing on the first noisy image at the second time step based on the text embedding vector to obtain a first noisy image at the second time step; performing t-1 noise addition processing on the first image information to obtain a second noisy image at the second time step; obtaining the semantically complete image at the second time step based on the first noisy image at the second time step, the second noisy image at the second time step, and the mask; and obtaining the target semantically complete image based on the semantically complete image at the second time step.

[0035] For a specific implementation of S140, please refer to the following example. Figure 3 Examples of implementation methods are provided.

[0036] In S150, second image information of the second perspective of the first vehicle is generated based on the first image information and the target semantic completion image.

[0037] In this embodiment of the disclosure, when the second perspective is a top-down view relative to the ground, the second image information is referred to as a BEV image. A BEV image is a view observed from a high vantage point or from a bird's-eye view. This mode of observation provides a broad and comprehensive perspective, allowing people to clearly see various details and layouts on the ground. In the field of autonomous driving, BEV perception technology uses data collected by onboard sensors (including the aforementioned first image information) to convert this data into a top-down perspective, providing more comprehensive and accurate environmental perception information. This helps vehicles more accurately identify obstacles such as roads, pedestrians, and other vehicles, thereby making safer driving decisions.

[0038] Semantic perception and segmentation in bird's-eye view (BEV) images play a crucial role in autonomous driving technology, directly impacting the effectiveness of subsequent vehicle planning and control. The core objective of BEV perception mapping is to convert RGB image information (including the aforementioned first image information) acquired from onboard sensors such as a monocular camera into a scene representation from a bird's-eye view perspective, i.e., converting it into second image information. Therefore, in this embodiment, before projecting the first image information onto the BEV, heuristic reasoning and semantic supplementation of occluded areas are performed using the spatial characteristics of the perspective view itself, which is beneficial to the completeness and coherence of BEV semantics.

[0039] In an exemplary embodiment, generating second image information of the second perspective of the first vehicle based on the first image information and the target semantic completion image includes: extracting features from the first image information and the target semantic completion image through a first branch and a second branch respectively, to obtain a first feature map and a second feature map of the first image information and the target semantic completion image at layer i, wherein the first branch and the second branch each include n layers, where n is a positive integer greater than 1 and i is a positive integer greater than or equal to 1 and less than or equal to n; obtaining a fused feature map of layer i based on the first feature map and the second feature map of layer i; merging the second feature map of layer i with the fused feature map of layer i to obtain a merged feature map of layer i; and generating the second image information based on the merged feature map of layer i.

[0040] In an exemplary embodiment, obtaining a fused feature map of the i-th layer based on a first feature map and a second feature map of the i-th layer includes: concatenating the first feature map and the second feature map of the i-th layer to obtain a concatenated feature map of the i-th layer; obtaining a first attention weight of the first feature map of the i-th layer and a second attention weight of the second feature map of the i-th layer based on the concatenated feature map of the i-th layer; and obtaining a fused feature map of the i-th layer based on the first feature map of the i-th layer and its first attention weight, and the second feature map of the i-th layer and its second attention weight.

[0041] For a specific implementation of S150, please refer to the following example. Figure 4 and Figure 5 Examples of implementation methods are provided.

[0042] In an exemplary embodiment, the method provided by this disclosure further includes: obtaining navigation instructions; generating control instructions based on the navigation instructions and the second image information; and controlling the first vehicle to drive autonomously based on the control instructions. The following examples illustrate the application of the method provided by this disclosure to an autonomous vehicle.

[0043] The data processing method provided in this embodiment performs semantic completion on the occluded areas in the first image information of the first vehicle obtained from a first viewpoint using a mask and prompt text to obtain a target semantically completed image of the first image information from the first viewpoint. Then, based on the first image information and the target semantically completed image, a second image information of the first vehicle from a second viewpoint is generated. On the one hand, this can improve the accuracy of the conversion from the first image information to the second image information; on the other hand, it does not need to consider multi-view and multi-temporal characteristic information, nor does it need to use depth estimation for multimodal fusion, thus reducing the complexity of network processing.

[0044] For example, this disclosure relates to a semantic perception method in the field of autonomous driving, and more particularly to a BEV inference framework that combines region semantic completion and feature matching mechanisms to improve the environmental perception capability of autonomous driving systems. Unlike BEV perception methods in related technologies, the method provided in this disclosure aims to address the problem of insufficient region perception caused by occlusion from surrounding vehicles or obstacles in BEV generation technologies in related technologies. In the inference framework, a generative model can be used to predict the latent semantics of the occlusion space (i.e., the occluded region) and generate a background image (i.e., the target semantic completion image) to assist in guiding the BEV generation process.

[0045] The framework provided in this disclosure mainly includes two core modules: a perception and reasoning module and a BEV generation module that incorporates a feature matching mechanism. In autonomous driving applications, cameras or external cameras, due to the impenetrability of their imaging features, cannot capture the occluded surrounding environment, even though these areas often contain important road structure information and semantic features. Therefore, in this disclosure, the perception and reasoning module first preprocesses the input image (here referring to the first image information), detecting the main occluded areas and combining surrounding spatial features and relevant semantic information to perform semantic completion operations on specific areas (i.e., occluded areas). This process can effectively compensate for the limitations of fixed cameras in spatial perception, ensuring that the semantic information of occluded areas is recovered, thereby improving the overall environmental perception capability. Based on this, the BEV generation module optimizes the generation process of the BEV map.

[0046] like Figure 2 As shown in the illustration, this disclosure also provides an end-to-end autonomous driving system 100, which includes a data acquisition module 110, a perception and reasoning module 120, a BEV generation module 130, a planning and control module 140, and an execution module 150. The data acquisition module 110 may include an external camera 111 and / or other sensors 112, such as one or more of lidar, millimeter-wave radar, and ultrasonic radar. The data acquisition module 110 acquires raw road condition data, which includes images, including the aforementioned first image information.

[0047] For example, the raw road condition data collected by the data acquisition module 110 is preprocessed and then input into the perception reasoning module 120 and the BEV generation module 130, respectively.

[0048] The perception and reasoning module 120 processes the preprocessed raw road condition data to obtain semantic supplementary information (i.e., the target semantic completion image), and inputs the semantic supplementary information to the BEV generation module 130. The BEV generation module 130 includes an encoder and a decoder, which process the preprocessed raw road condition data and its semantic supplementary information to generate a BEV map. The generated BEV map is then input to the planning and control module 140. The planning and control module 140 includes a decision submodule 141, a planning submodule 142, and a PID controller 143. The planning and control module 140 acquires navigation instructions and the BEV map, generates control instructions, and inputs the control instructions to the execution module 150. The execution module 150 receives the control instructions.

[0049] This disclosure provides a BEV perception system and method for end-to-end autonomous driving. To overcome the aforementioned shortcomings, this disclosure proposes a BEV perception method. First, a generative model is used to segment and semantically predict occluded areas caused by obstacles, such as other vehicles, thereby enhancing the integrity of background information in the first image. Second, the designed method uses the inferred global background information (i.e., the target semantically complete image) as a supplement to the original information (i.e., the original road condition data) collected by the onboard sensors, and establishes a correspondence by combining a feature attention mechanism to achieve accurate BEV perception. This method does not require consideration of multi-view, multi-temporal characteristic information, nor does it require the use of depth estimation for multimodal fusion, reducing the complexity of network processing.

[0050] For example, the system of this disclosure acquires visual data of the road scene ahead of the autonomous vehicle through an external camera 111, such as an RGB camera, and its associated storage device installed at the front of the autonomous vehicle, as images in the raw road condition data. The RGB camera is responsible for capturing and recording continuous video images of the environment ahead of the autonomous vehicle in real time during its operation. This visual data provides basic data for subsequent perception and processing modules (including the perception inference module and the BEV generation module), and can reflect the real-time visual characteristics and traffic conditions of the road currently being traveled by the autonomous vehicle, providing effective support for subsequent perception decisions in autonomous driving.

[0051] In some embodiments, this disclosure allows for compatibility with other sensor information acquired by other sensors 112, such as LiDAR data acquired by LiDAR. For example, the corresponding LiDAR data is cropped and converted into a feature space consistent with the visual image (i.e., the first image information). This integration of multi-source data can effectively enrich the relevant input information.

[0052] In some embodiments, the data generated by the external camera 111 is a 2D image, which has a high accuracy in perceiving the shape and category of objects, while the lidar, due to its working principle, makes up for the lack of depth information in the 2D image to a certain extent.

[0053] In some embodiments, decision-level fusion can be employed. For example, the sensor information (including 2D images acquired by an external camera and LiDAR data acquired by a lidar) is processed separately, and the resulting detection or segmentation results are then unified to the same coordinate system and merged. The BEV map is obtained based on the merged information. That is, the merged information is then input to the perception inference module and the BEV generation module.

[0054] In other embodiments, a feature-layer fusion approach can be used to project point cloud information from LiDAR data and image information acquired by an external camera (including the first image information mentioned above) onto the same BEV viewpoint. For example, in a monocular camera scenario, LiDAR point cloud information within a certain angle can be selected based on the camera's field of view; in a surround-view scenario, point clouds that are too high or too low can be removed. For a pixel (grid) on the BEV, its neighboring points are first found in the point cloud data / information. The spatial features of these points in 3D space are encoded and mapped to the image space as additional feature channels. Finally, the additional channel with depth information is merged with the RGB channels of the image to obtain the fused feature information. The BEV map is then obtained based on the fused feature information. That is, the fused feature information is then input to the perception inference module and the BEV generation module. This strategy can fully utilize the complementarity between sensor data / information.

[0055] In autonomous driving, point cloud information (3D spatial data obtained from LiDAR scanning) and image information (2D images captured by cameras) have different perspectives. To make them work together, they must be "adjusted" to the same perspective, which is the BEV perspective.

[0056] If the first vehicle uses a monocular camera (with only one lens), its field of view is limited, with a specific size of field of view. When processing point cloud information, point cloud information within a certain angular range is selected from the entire point cloud data based on the size of this field of view; that is, point cloud information within the field of view of the monocular camera.

[0057] If the vehicle is equipped with a surround-view camera (multiple lenses that can see the area around the vehicle), then remove point clouds that are too high or too low. For example, point cloud information generated by drones flying overhead or very low pebbles on the ground is not relevant to the movement of the vehicle and should be removed to make the data more useful.

[0058] After processing the point cloud information, from the BEV perspective, for a single pixel (the BEV image can be imagined as a large chessboard composed of many small squares, each square being a pixel or grid), we need to find its neighboring points in the point cloud data. These points, in real 3D space, each have their own spatial characteristics (such as position and distance). Then, the features of these 3D spatial points are transformed into image space, forming a new channel containing depth information, adding a new dimension to the original image that represents distance.

[0059] This new channel, containing depth information, is merged with the original RGB channels of the image. With this fused feature information, the autonomous driving system can better "understand" its surroundings and make more accurate decisions.

[0060] For example, the raw road condition data is transmitted to the scene perception module (including the perception inference module and the BEV generation module) via the CAN (Controller Area Network) bus. The main task of the scene perception module is to analyze and process the visual information (including images) in the raw road condition data to generate a road condition feature map (i.e., a BEV map).

[0061] In some embodiments, before sending the raw road condition data to the scene perception module, the input raw road condition data is first preprocessed, including steps such as image denoising, color correction, and feature extraction. That is, the images in the raw road condition data are preprocessed and used as the first image information.

[0062] In some embodiments, if LiDAR sensor input information (i.e., LiDAR data) is detected in the raw road condition data, the system can process heterogeneous modal information using feature layer fusion to unify it into a common BEV space. For monocular camera scenarios, all point clouds within the image space angle can be cropped according to the camera's field of view size. For a pixel (grid) on the BEV, its neighboring points are found in the point cloud data, and the spatial features of these points are encoded and mapped to the image space as additional feature channels. The additional channels with depth information are merged with the image's RGB channels to obtain the fused feature information.

[0063] The main function of the perception and reasoning module 120 is to identify the occluded area corresponding to the second vehicle and to provide semantic supplementation to the occluded area based on relevant prompt information (i.e. prompt text). Figure 3 The embodiment describes the process by which the perceptual reasoning module uses a semantic completion network to semantically complete the occluded area and generate a target semantically completed image.

[0064] like Figure 3 As shown, the prompt text d is assumed to be "roads / buildings obscured by vehicles," where "vehicles" refers to the second vehicle, and "roads / buildings" are assumed to be the objects that need to be completed. The input image is represented as x0 = x org x org It can be the first image information, which can be an image from the original road condition data or an image after preprocessing.

[0065] For example, semantic completion networks include Fast R-CNN (Fast Region-Convolutional Neural Networks, also referred to as Fast R-CNN) and CLIP diffusion models. Taking a single frame of input image x... org For example, the perception reasoning module first uses the pre-trained Fast R-CNN and CLIP diffusion model to complete the vehicle (referring to the second vehicle) target detection and semantic guidance tasks, respectively.

[0066] Figure 3 In the embodiment, the input image x0 = x org The input is fed into Fast R-CNN for processing to obtain the mask m. Fast R-CNN is used to detect scene maps (input image x0 = x...). org Other vehicles in the image are identified, and corresponding masks m for the occluded areas are generated. Specifically, Fast R-CNN is used to process the input image x0 = x org The process involves identifying the category and location of each target within the mask, thereby locating the occluded area where the second vehicle is located. The occluded area is represented by a first pixel value, and the unoccluded area by a second pixel value, thus generating a mask m. For example, Figure 3 In this diagram, white pixels represent occluded areas and black pixels represent unoccluded areas, but this disclosure is not limited to this. Furthermore, this disclosure is not limited to generating the mask m using Fast R-CNN.

[0067] refer to Figure 3 For the input image x0=x org Perform q(x) t The process involves processing x0, i.e., performing t noise addition operations to obtain the first noisy image x at the first time step. t .

[0068] For the first noisy image x at the first time step t The prompt text d is processed by the CLIP diffusion model to obtain x. t-1,fgFor example, the CLIP Diffusion model includes a CLIP (Contrastive Language-Image Pre-Training) model and a diffusion model. The CLIP Diffusion model is a technique that combines the CLIP model and the Diffusion model to generate high-quality target semantic completion images from prompt text.

[0069] CLIP is a text-image pre-trained model that understands the relationship between text and images. It is pre-trained using a large number of unlabeled image-text pairs to learn a cross-modal embedding space that minimizes the distance between the image and its corresponding text embedding vector. The CLIP model encodes text and images separately and then calculates their similarity. During training, the CLIP model continuously optimizes the encoding method, bringing similar text and images closer together in the embedding space. The CLIP model further includes a text encoder and an image encoder.

[0070] Diffusion models are a method for gradually eliminating random noise to recover the original signal. In image generation, they can produce highly realistic images. Starting with Gaussian noise, a Diffusion model iteratively reduces the noise and generates an image that matches the input prompt text through a series of iterations. In each iteration, the Diffusion model predicts the next noise reduction step based on the current noisy image and the input prompt text. Diffusion models are a type of generative model that gradually "diffused" (or destroyed) information from the input image and then reversed to generate a new image structure. This type of model is used for text-to-image generation tasks.

[0071] CLIP Diffusion incorporates the understanding capabilities of the CLIP model into the Diffusion model. Leveraging the text-image relationship information provided by the CLIP model, the Diffusion model can more accurately generate images that match the input prompt text.

[0072] For example, the CLIP Diffusion model in this embodiment employs Blended Diffusion. Blended Diffusion combines the CLIP model and Denoising Diffusion Probabilistic Models (DDPM). In Blended Diffusion, the CLIP model provides text embedding vectors of the cue text to guide the generation process of the target semantic image. DDPM is a type of diffusion model that leverages pre-training to achieve image denoising and generation. In Blended Diffusion, DDPM is used to denoise noisy images and generate target semantic images that conform to the description of the text embedding vectors.

[0073] In this embodiment of the disclosure, an input image x0 = x is provided for the Blended Diffusion model. org The model takes a mask *m* and a cue text *d*. The mask *m* specifies the occluded region in the input image that needs editing, while the cue text *d* describes the desired result after editing. The CLIP model converts the cue text *d* into a text embedding vector, which serves as conditional information to guide the denoising process of DDPM. During the denoising process of DDPM, the model progressively reduces the noise in the first noisy image *x*. t To mitigate noise in the input image, Blended Diffusion employs a background-preserving loss function to maintain background content. In each denoising step, the model mixes the CLIP-guided diffusion latent code with the corresponding noise image from the background of the input image, then inputs both together into the next denoising process. After multiple iterations (e.g., t iterations), DDPM generates a target semantic completion image that matches the text embedding vector description. This target semantic completion matches the cue text within the occluded region, while preserving the content of the input image outside the occluded region.

[0074] In this embodiment, the CLIP model performs embedding representation based on the provided cue text d, generating a region content supervision signal (i.e., a text embedding vector) to guide the subsequent visual generation task. Specifically, the cue text d is input into the text encoder of the CLIP model, which outputs the text embedding vector of the cue text d. Then, the text embedding vector is input into the diffusion model to guide it to generate an image that is as consistent as possible with the text embedding vector.

[0075] In the completion task of this embodiment, it is desired to replace the occluded region indicated by the mask m with the scene described by the cue text d. Therefore, in the subsequent loss function design, the cosine distance between the embedding vector of the generated image and the text embedding vector of the cue text d is calculated. Thus, the method proposed in this embodiment utilizes CLIP to generate the text embedding vector of the cue text d and constrains the generation loss to limit the content of the inference completion. In this way, the perceptual inference module can identify the occluded region in the input image and supplement the occluded region with relevant semantic information.

[0076] For example, in semantic completion, a variant of the diffusion model, BlendedDiffusion, is introduced as the generative backbone network. Its role is to process the first noisy image x under the guidance of the cue text d and the mask m. t Perform t-step denoising to restore the target output image. That is, to complete the target semantic image.

[0077] In this embodiment, the diffusion model controls the noise (Gaussian noise) using two hyperparameters: noise variance and time step. The noise variance can be set by adjusting the scale. The maximum and minimum values ​​are controlled by linear increase or cosine scheduling; the time step t needs to be adjusted according to the generation quality or experience, such as 1000 steps or 800 steps, or it can be randomly obtained within a certain range (such as between 5000 steps and 1000 steps) to train the diversity of downsampling at different time steps.

[0078] Figure 3 In, q(x) t |x0) represents t noise addition processes. ε represents noise (Gaussian noise). β i The noise control level parameter is set between (0, 1). For the first noisy image x at the first time step... t First, denoising is performed based on CLIP Guidance (i.e., the text embedding vector of the prompt text d). This involves adding noise t times to the input image x0 and then subtracting it once, resulting in the image x. t-1,fg This refers to the first denoised image at the first time step. Additionally, the input image x0 is subjected to t-1 denoising operations q(x) according to the Blended Diffusion formula. t-1 Let x be the result of adding noise to the input image x0 t-1 times. t-1,bg This refers to the second noisy image at the first time step. For the two noisy images x obtained in different ways... t-1,fg and x t-1,bgMixing is performed using a given mask m, for example, according to formula x. t-1 =x t-1,fg ⊙m+x t-1,bg ⊙(1-m) is mixed to obtain the semantically completed image x at the first time step. t-1 , can also be expressed as The same processing method is used in each subsequent step. This process is repeated t times to obtain the target output image (i.e., the target semantically complete image). It can also be expressed as

[0079] In this embodiment of the disclosure, the objective of the entire diffusion process is to minimize the cosine distance between the image generated by the occluded area in the mask and the prompt text. This objective can be expressed by the following formula:

[0080]

[0081] In the formula above, ⊙ represents element-wise multiplication. This represents the semantically completed image at time step i. This represents the image corresponding to the occluded region in the semantically completed image at time step i. This represents the embedding vector generated by inputting the image corresponding to the occluded region in the semantically completed image at time step i into the image encoder of the CLIP model. txt (d) represents the text embedding vector generated by the text encoder of the CLIP model when the prompt text d is input into it. Let represent the cosine distance between the embedding vector of the image corresponding to the occluded region in the semantic completion image at time step i and the text embedding vector of the prompt text d.

[0082] In this embodiment of the disclosure, the loss is calculated using the following formula at the i-th time step to update the gradient of the CLIP diffusion model:

[0083]

[0084] In the above formula, the value of i is a positive integer greater than or equal to 1 and less than or equal to t. This indicates that, in the current case, the input image x0 has undergone i rounds of noise addition, and the image before the i rounds of noise addition is calculated in reverse. The subscript is (0, i). After t steps of denoising, the semantically complete image of the occluded part or occluded region is obtained. This loss value is used to update the network's gradient, allowing the network to continuously adjust its parameters during training to better achieve the goal of semantic completion.

[0085] In the semantic completion process disclosed herein, Blended Diffusion serves as the generative network, using cue text d and a mask m to perform t-step denoising to obtain the target output image. The specific implementation process is as follows:

[0086] Step 1: Determine the input image x0 = x org The text prompt d and the mask m, as well as the time step t.

[0087] Step 2, Noise Addition Processing: For the input image x0 = x org Perform t noise addition processes, specifically calculated as q(x) t -x0). Different noise images x can be obtained at different time steps t. t .

[0088] Step 3, First Denoising: For the noisy image x t A denoising process is performed based on CLIP Guidance, and the result is denoted as x. t-1,fg This step utilizes the semantic supervision information provided by the CLIP network to guide the denoising process in a direction consistent with the prompt text d.

[0089] Step 4, Second noise addition: For the input image x0 = x org Perform t-1 noise addition processes, specifically calculated as q(x) t-1 -x0), the result is denoted as x t-1,bg .

[0090] Step 5, Image Blending: Using the given mask m, blend the noisy images x obtained from the two different methods described above. t-1,fg and x t-1,bg To mix, the formula is x t-1 =x t-1,fg ⊙m+x t-1,bg ⊙(1-m). Through mask m, retain x t-1,fg The denoising result for the middle mask region (i.e., the occluded region) while preserving x t-1,bg Information from the unobstructed area (i.e., the non-obstructed area) is used to achieve information fusion.

[0091] Step 6, Iteration Process: Repeat steps 3-5 above for a total of t times. Each iteration is based on the result of the previous iteration, continuously optimizing the noise reduction effect to make the image closer to the target output image.

[0092] After t-step denoising, the final target semantic supplement image of the occluded part is obtained. This target semantic supplement image restores the semantic information of the occluded area, providing more complete and accurate image data for subsequent BEV generation, which helps to improve the autonomous driving system's ability to perceive the environment.

[0093] In the second iteration described above, the first denoising step uses the blended image x obtained at the end of the first iteration. t-1 Based on this, processing is performed. In the first iteration, x... t-1 By x t-1,fg and x t-1,bg It is obtained by mixing through a mask m. In the second iteration, this x is... t-1 Treating this as the new "original image", we add noise to it t times and then subtract noise once, thus obtaining x in the second iteration. t-1,fg This is because each iteration relies on CLIP Guidance for denoising, utilizing the semantic supervision information provided by the CLIP network to continuously optimize the denoising direction, improving the image in a way that aligns with the prompt text d. The second denoising step uses the initial input image x0 = x org Based on this, perform t-1 noise additions to obtain x in the second iteration. t-1,bg Throughout the semantic completion process, following the Blended Diffusion formula, each iteration starts from the initial input image x0 = x org We begin by performing t-1 iterations of noise addition. This ensures that the information source for the unoccluded regions remains consistent and stable in each iteration. Then, we compare this result with the image portion x after CLIP Guidance denoising. t-1,fg The noise reduction effect is gradually optimized by using a mask m for blending, making the final image closer to the target output image. Other iterations are performed in the same manner.

[0094] This disclosure utilizes Blended Diffusion to semantically complete occluded regions in an input image while preserving the rest of the input image, resulting in a highly coherent generated image. Guided by the CLIP model, Blended Diffusion can generate a suitable image based on the prompt text. By combining the advantages of both the diffusion model and the CLIP model, Blended Diffusion can generate high-quality, detailed images.

[0095] Another component in this embodiment is a BEV generation module. The image from the preprocessed original road condition data (as the first image information) and the target semantic completion image are input into the BEV generation module. The main function of the BEV generation module is to generate a top-down view BEV image based on the input forward-looking plane information (including the first image information and the target semantic completion image).

[0096] Specifically, the BEV generation module uses both the original input image x (referring to the first image information) and its corresponding target semantically complete image y as input. It employs an attention-based feature matching fusion module to perform deep information fusion on corresponding pixels in the two input images x and y. The feature matching mechanism utilizes the information from the inference completion to effectively address the problem of insufficient spatial features in occluded regions, thereby improving the semantic completeness and spatial accuracy of the generated BEV image.

[0097] For example, the BEV generation module includes a generation network as its encoder and decoder. Figure 4 This is a schematic diagram of the structure of a generative network according to an exemplary embodiment of this disclosure. Figure 4 As shown, the input is the original image x and the semantically completed image y (here referring to the aforementioned target semantically completed image). The generative network includes a first branch and a second branch connected in parallel. Both the first and second branches consist of n layers connected in series. The first branch processes the input original image x, obtaining the first feature map 21 at layer 1, the first feature map 22 at layer 2, and so on up to the nth layer of the first branch. Figure 2 n. The second branch is used to process the semantically completed image y. The second feature map 31 of the semantically completed image y at layer 1, the second feature map 32 at layer 2, and so on up to layer n are obtained through the first layer, second layer, up to layer n, respectively. Figure 3 n.

[0098] The first feature map 21 of the original input image x at layer 1 and the second feature map 31 of the semantically completed image y at layer 1 are processed by AFF to obtain the fused feature map 41 of layer 1. The first feature map 22 of the original input image x at layer 2 and the second feature map 32 of the semantically completed image y at layer 2 are processed by AFF to obtain the fused feature map 42 of layer 2. And so on, the first feature map 21 of the original input image x at layer n is processed by AFF to obtain the fused feature map 42 of layer 2. Figure 2 n and the second feature of semantic completion image y at layer n Figure 3 After AFF processing, the fusion feature of layer n is obtained. Figure 4 n.

[0099] For example, the input to the BEV generation module includes the original input image x and the semantically completed image generated by the perceptual reasoning module. The overall network structure of the BEV generation module adopts a 5-layer U-shaped encoder-decoder structure, i.e., assuming n=5, but this disclosure is not limited to this. The input end contains two different processing branches, referred to as the first branch and the second branch, used to process feature extraction for x and y respectively. Each layer consists of two residual blocks cascaded together, extracting multi-scale feature information during the encoding process. The residual blocks solve the gradient vanishing and gradient exploding problems in deep neural networks by introducing "skip connections," thus enabling deeper networks to be trained effectively. It allows gradients to propagate by skipping a certain number of layers, thereby maintaining the gradient flow of the network and making deep networks easier to train. For example, a residual block contains two paths: a main path (main branch) and skip connections (bypasses). The main path extracts features through multiple convolutional layers, batch normalization layers, and non-linear activation functions. Skip connections directly connect the input to the output, thereby achieving residual learning. It allows the input signal to skip one or more layers directly and be added to the output of the main path. This connection method helps maintain the gradient flow of the network and prevents gradient vanishing. In the residual module, the outputs of the main path and the skip connections are added together, and then the output is obtained by passing it through an activation function (such as ReLU).

[0100] In this embodiment of the disclosure, for the same layer in the encoding path, the network additionally uses an Attentional Feature Fusion (AFF) module to fuse features from the two branches. The AFF module employs a cross-attention mechanism, which calculates the attention weights between the original input image and the semantically completed image, and then weights and fuses the features of the two, further improving the completeness of the BEV map.

[0101] Feature fusion methods in related technologies are implemented through simple operations such as summation or concatenation. However, these methods are not optimal because they merely provide a fixed linear aggregation of feature maps without considering whether such a combination is suitable for a specific object. AFF introduces an attention mechanism to dynamically and selectively fuse features, achieving dynamic selection and weighting of features, thereby improving the flexibility and accuracy of feature fusion. It can overcome semantic inconsistencies and effectively integrate feature structures of different scales, improving the quality of fused features.

[0102] Specifically, AFF connects each pixel of the original input image x with the pixels of the semantically complete image y through a multi-scale channel attention module (MS-CAM), enabling local features in the image to be better combined with global background information, thereby improving the accuracy and completeness of the generated BEV map.

[0103] An exemplary AFF module structure for feature matching fusion is as follows: Figure 5 As shown. The AFF input receives two feature vectors fx and fy of size (C×H×W), where C, H, and W are all positive integers greater than or equal to 1. For the input features fx and fy (which can be the above...), ... Figure 4 Given the first and second feature maps output from any identical layer, the AFF processing procedure can be expressed as the following formula:

[0104]

[0105] In the above formula, fz represents the fused feature map of the corresponding layer. This indicates that the first and second feature maps of the same layer will be merged. This represents the first attention weight of the first feature map of the corresponding layer obtained by MS-CAM after pixel addition of the first and second feature maps of the same layer. This represents the second attention weight of the corresponding layer's second feature map obtained through MS-CAM after pixel-wise summation of the first and second feature maps of the same layer. MS-CAM is a fundamental submodule in AFF, which extracts channel attention parameters—that is, extracts the first and second attention weights—through two branch convolutional paths with different scales. This indicates a multiplication operation.

[0106] MS-CAM is a component of AFF (Automatic Feature Fusion). Its design enables the attention module to aggregate contextual information from different receptive domains for objects of different scales, thereby improving the quality of feature fusion. MS-CAM consists of two branches: a global branch and a local branch. The global branch extracts global contextual information through operations such as global average pooling. The local branch directly performs pointwise convolutions on the original input to extract local contextual information. The outputs of the two branches are combined through addition or other methods and then used to generate attention weights (including first and second attention weights) using the sigmoid activation function.

[0107] In this embodiment, the decoding network introduces a feature skip connection mechanism, whereby the second feature map of the semantically completed image y at layer i is concatenated with the fused feature map of layer i obtained by AFF fusion. This design preserves shallow semantic information and enhances the overall features of objects and roads, thereby improving the spatial representation capability of the BEV map. Specifically, the fused feature map 41 of layer i and the second feature map 31 of layer i are concatenated; the fused feature map 42 of layer i and the second feature map 32 of layer i are concatenated; and so on, until the fused feature map of layer n is concatenated. Figure 4 The second feature of n and the nth layer Figure 3 The features are merged using n. Based on these merged feature maps, a BEV map is output. It's important to note that in the encoding path, each layer doubles the number of channels in the current layer's features by adjusting the number of convolutional kernels in the residual module. Then, downsampling (max pooling) halves the size of the feature map. In the decoding path, the number of output channels in each layer is half the number of input channels. This compresses the feature channels after the skip connections merge, and the feature map size is restored through upsampling (interpolation).

[0108] For example, the merged feature maps obtained from different layers through AFF and skip connections have different scales and semantic information. Lower-level feature maps contain more detailed information, while higher-level feature maps contain more abstract and global semantic information. For example, cross-scale feature fusion methods, such as Feature Pyramid Networks (FPN), can be used to fuse merged feature maps from different scales. For instance, for high-level feature maps, their size can be adjusted to be the same as that of low-level feature maps through upsampling operations, and then they can be fused together by element-wise addition or concatenation to obtain a feature map with integrated features, thus comprehensively utilizing feature information from different scales. For example, to further enhance the expressive power of features, channel attention mechanisms, such as the Squeeze-and-Excitation (SE) module, can be introduced during feature integration. The SE module can adaptively adjust the feature response of each channel, highlighting important feature channels and suppressing unimportant channels. For each merged feature map, the feature information of each channel is compressed into a scalar by global average pooling. Then, the weights of each channel are obtained through a fully connected layer and an activation function. Finally, these weights are multiplied with the original feature map to obtain the feature map after feature integration, thus achieving channel-level feature enhancement.

[0109] During the decoding phase, the feature map after feature integration is progressively upsampled and convolutionally processed. Each decoding layer can incorporate feature information from the corresponding layer passed from the encoding phase via skip connections to further enrich the feature representation. For example, during the decoding of a certain layer, the feature map obtained from the previous layer is upsampled and then concatenated with the stitched feature map obtained by AFF processing of that layer in the encoding phase. This is then passed through convolutional layers for feature fusion and extraction, gradually recovering a high-resolution feature map. The feature map obtained after the decoding process can be used to generate the final BEV image through the output layer (classification or segmentation head). The segmentation head consists of one or two fully connected layers with three output channels, resulting in the final output BEV image.

[0110] For example, based on the optimized BEV map, the planning control module 140, in conjunction with relevant navigation instructions, will output control commands or instructions recognizable by the first vehicle, providing decision support for the autonomous driving system. The planning control module 140 includes a decision submodule 141, a planning submodule 142, and a PID controller 143, which are responsible for converting perceived information (including the BEV map and navigation instructions) into specific vehicle operation instructions (i.e., control instructions).

[0111] Based on the BEV diagram, the decision submodule 141 analyzes the current traffic conditions and generates driving decisions. These decisions include how to respond to traffic signals, how to avoid obstacles, and how to choose a driving path. The decision submodule 141 makes reasonable judgments and decisions by combining real-time traffic information and preset driving rules to ensure the vehicle (referring to the first vehicle) can drive safely. The planning submodule 142 generates the vehicle's driving trajectory based on the driving decisions provided by the decision submodule 141. This process includes trajectory optimization and path planning operations to ensure the vehicle can efficiently complete the predetermined driving task while following traffic rules. The planning submodule 142 considers factors such as road curves, traffic signals, and obstacle positions to calculate the optimal driving trajectory and performs path adjustments and optimizations. The trajectory information from the planning submodule 142 is transmitted to the PID controller 143, which converts the planned trajectory into actual control commands. These control commands include braking, accelerator, and steering wheel adjustments to control the vehicle's movement. The PID controller 143 ensures the vehicle drives according to the planned trajectory and maintains stable operating performance through real-time feedback and adjustments.

[0112] The generated control commands are sent to the execution module 150 to complete a series of operations for the autonomous vehicle. The execution module 150 is responsible for adjusting the vehicle's movement according to the control commands, including acceleration, deceleration, and turn signals. Through precise execution control, the vehicle can drive according to the planned trajectory and decisions, achieving autonomous driving functionality. The execution module 150 also includes a monitoring and feedback mechanism for the vehicle's status, ensuring that the vehicle can respond promptly to various emergencies during execution. Through real-time monitoring and adjustment, the execution module 150 can guarantee the stability and safety of the vehicle under various driving conditions.

[0113] The system and method proposed in the embodiments of this disclosure are described below through specific examples.

[0114] Step 1: The data acquisition module acquires raw road condition data through external cameras and / or other sensors installed on the autonomous vehicle, including but not limited to color images and LiDRA point cloud data (3D data projection) on the frontal plane. This data acquisition module captures visual information of the surrounding environment in real time through cameras, providing basic data for subsequent perception and processing modules (including the perception reasoning module and the BEV generation module).

[0115] The second step involves processing the raw road condition data in the perception and inference module. This module detects target objects around the autonomous vehicle and performs semantic completion on occluded areas. First, the perception and inference module analyzes the input visual data using deep learning algorithms. A pre-trained Fast R-CNN model is used to extract obstacle regions (i.e., occluded regions) in the scene and generate corresponding masks. Based on these masks, combined with the original image and prompt text, a diffusion-based semantic completion network is used to complete the occluded areas, generating a complete background image, i.e., the target semantically completed image. These processed images provide rich and complete semantic information for subsequent BEV generation.

[0116] The third step involves feeding the road condition image (i.e., the first image information) and the target semantically complete image into the BEV generation module, which then generates the corresponding BEV image. The BEV generation module employs a dual-branch input encoder-decoder network architecture, and the network performs feature fusion processing through a cross-attention mechanism. This module utilizes the supplementary semantic information to transform the input image from a camera viewpoint into a bird's-eye view BEV. This top-down view accurately displays comprehensive information about the vehicle's surroundings, providing crucial spatial positioning data for the autonomous driving system.

[0117] Step 4: The generated BEV map and related navigation information (including navigation commands) are input into the planning and control module to generate control commands. The decision submodule generates driving decisions based on the BEV map and extracted feature maps, including vehicle acceleration, braking, and steering. The planning submodule calculates the optimal driving trajectory based on current road conditions and target location, and generates specific control commands through a PID controller for actual operation. This module ensures that the vehicle can be precisely controlled according to the environment and target.

[0118] Step 5: The control commands generated by the planning and control module are sent to the execution module, which is responsible for adjusting the vehicle's acceleration, braking, and steering actions according to the control commands. The execution module ensures that the vehicle travels along the planned trajectory, thereby achieving the autonomous driving function and ensuring the reliability and stability of the autonomous driving system.

[0119] In related technologies, BEV (bird's-eye view) generation methods mainly rely on input image data and known spatial features, but they often lack effective processing for occluded areas, resulting in perception blind spots in certain scenarios. This disclosure introduces a language-guided generative model to perform semantic completion on areas occluded by surrounding vehicles or obstacles, generating a complete road space background map. Compared to related technologies, this disclosure effectively avoids spatial distortion and structural information loss problems during the completion process from a top-down perspective. By performing semantic completion on the front view, it effectively utilizes structural features in perspective space, enabling more accurate recovery of occluded spatial information and reducing errors caused by perspective transitions.

[0120] In some embodiments, this disclosure also improves the accuracy of BEV generation. Embodiments of this disclosure introduce a network structure with a cross-attention mechanism into the BEV generation module. This structure can efficiently fuse semantic completion information generated by the perception inference module and known environmental information at the feature information level, effectively solving the feature loss problem caused by occlusion by a vehicle ahead. The cross-attention mechanism establishes an attention matching relationship between the input image and the semantic completion image, promoting the network's ability to comprehensively consider the contextual information of the occluded region.

[0121] Furthermore, the embodiments of this disclosure also make full use of the shallow features in the decoding path and effectively fuse these shallow features with deep features, thereby further alleviating the problem of decreased accuracy caused by insufficient spatial structure information in the BEV diagram.

[0122] The above measures significantly improve the spatial representation capability of BEV maps, ensuring the stability and accuracy of the environmental perception system in complex scenarios.

[0123] Furthermore, the embodiments of this disclosure enhance the intelligence of decision-making and control. The semantic completion method introduced in these embodiments predicts occluded areas that sensors cannot detect, establishing a more complete BEV map. This facilitates rational planning using global information in subsequent path planning and obstacle avoidance scenarios. At the control and execution end, by monitoring road conditions provided by the perception end in real time, the system can more accurately generate and adjust driving decisions and trajectories, ensuring the stability and safety of the vehicle under various driving conditions. This vision-based intelligent control method not only improves the overall performance of the system but also enhances the intelligence level of autonomous driving, optimizing the vehicle's driving performance in complex and ever-changing traffic environments.

[0124] Figure 6 This is a schematic diagram of the structure of a data processing apparatus according to an exemplary embodiment of this disclosure. Figure 6 As shown, the data processing device 600 provided in this embodiment includes a data acquisition module 610, a perception and reasoning module 620, and an image generation module 630.

[0125] The data acquisition module 610 is used to acquire first image information from the first perspective of the first vehicle.

[0126] The perception reasoning module 620 is used to identify obstacles contained in the first image information, generate a mask containing occluded areas blocked by the obstacles, obtain a text embedding vector of a prompt text, the prompt text being used to describe an object in the occluded area, and generate a target semantic completion image of the first image information based on the first image information, the mask and the text embedding vector, the target semantic completion image containing the completion of the object in the occluded area.

[0127] The image generation module 630 is used to generate second image information of the second perspective of the first vehicle based on the first image information and the target semantic completion image.

[0128] The data acquisition module 610 may be, for example, the data acquisition module 610 described above; the perception and reasoning module 620 may be, for example, the perception and reasoning module 120 described above; and the image generation module 630 may include, for example, the BEV generation module 130 described above.

[0129] In an exemplary embodiment, the data processing device 600 further includes: a planning and control module for obtaining navigation instructions; generating control instructions based on the navigation instructions and the second image information; and an execution module for controlling the first vehicle to drive autonomously based on the control instructions.

[0130] Figure 6 Other aspects of the embodiments can be found in the other embodiments described above.

[0131] In this disclosure, the terms "module" or "submodule" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or submodules. Furthermore, each module or submodule can be part of an overall module or submodule that includes the functionality of that module or unit.

[0132] Figure 7 A schematic structural diagram of an electronic device 700 according to an embodiment of the present disclosure is shown. Figure 7 The illustrated electronic device 700 includes a processor 710, which can call and run computer programs from memory to implement the methods in the embodiments of this disclosure.

[0133] Optionally, such as Figure 7 As shown, the electronic device 700 may further include a memory 720. The processor 710 can retrieve and run computer programs from the memory 720 to implement the methods described in this embodiment.

[0134] The memory 720 can be a separate device independent of the processor 710, or it can be integrated into the processor 710.

[0135] Optionally, such as Figure 7 As shown, the electronic device 700 may also include a transceiver 730, which the processor 710 can control to communicate with other devices. Specifically, it can send information or data to other devices or receive information or data sent by other devices.

[0136] Optionally, the processor 710, memory 720, and transceiver 730 can communicate bidirectionally with each other via the communication bus 740.

[0137] It should be understood that the processor in this embodiment of the disclosure may be an integrated circuit chip with signal processing capabilities. In implementation, the steps of the above method embodiments can be completed by integrated logic circuits in the processor's hardware or by software instructions.

[0138] The aforementioned processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this disclosure. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this disclosure can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory; the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above methods.

[0139] It is understood that the memory in the embodiments of this disclosure can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchlink DRAM (SLDRAM), and Direct Rambus RAM (DR RAM). It should be noted that the memory used in the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory. It should be understood that the above-described memory is exemplary and not limiting.

[0140] This disclosure also provides a computer-readable storage medium for storing a computer program. This computer program causes a computer to perform corresponding processes in the various methods of the embodiments of this disclosure; for brevity, these will not be elaborated upon here.

[0141] This disclosure also provides a computer program product, including computer program instructions. These computer program instructions cause a computer to execute corresponding processes in the various methods of this disclosure; for brevity, they will not be described in detail here.

[0142] This disclosure also provides a computer program. When the computer program is run on a computer, it causes the computer to execute the corresponding processes in the various methods of this disclosure embodiment, which will not be described in detail here for the sake of brevity.

[0143] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this disclosure.

Claims

1. A data processing method, characterized in that, include: Acquire the first image information from the first perspective of the first vehicle; Identify obstacles contained in the first image information and generate a mask containing the occlusion areas blocked by the obstacles; Obtain the text embedding vector of the prompt text, which describes the object in the occluded area; Based on the first image information, the mask, and the text embedding vector, a target semantic completion image of the first image information is generated, wherein the target semantic completion image includes the completion of objects in the occluded region; Based on the first image information and the target semantic completion image, second image information from a second perspective of the first vehicle is generated.

2. The method as described in claim 1, characterized in that, Based on the first image information, the mask, and the text embedding vector, a target semantic completion image of the first image information is generated, including: The first image information is subjected to noise addition processing t times to obtain the first noisy image at the first time step, where t is a positive integer greater than 1; The first noisy image at the first time step is denoised based on the text embedding vector to obtain the first denoised image at the first time step; The first image information is subjected to noise processing t-1 times to obtain the second noisy image at the first time step; Based on the first noise-reduced image of the first time step, the second noise-added image of the first time step, and the mask, the semantic completion image of the first time step is obtained; The target semantic completion image is obtained based on the semantic completion image of the first time step.

3. The method as described in claim 2, characterized in that, Obtaining the target semantically complete image based on the semantically complete image at the first time step includes: The semantic completion image at the first time step is subjected to t noise addition processes to obtain the first noisy image at the second time step, where t is a positive integer greater than 1; The first noisy image at the second time step is denoised based on the text embedding vector to obtain the first denoised image at the second time step; The first image information is subjected to noise processing t-1 times to obtain the second noisy image at the second time step; Based on the first noise-reduced image of the second time step, the second noise-added image of the second time step, and the mask, the semantic completion image of the second time step is obtained; The target semantic completion image is obtained based on the semantic completion image at the second time step.

4. The method as described in claim 1, characterized in that, Based on the first image information and the target semantic completion image, second image information of the first vehicle from a second perspective is generated, including: Feature extraction is performed on the first image information and the target semantic completion image through the first branch and the second branch respectively to obtain the first feature map and the second feature map of the first image information and the target semantic completion image at the i-th layer. The first branch and the second branch both include n layers, where n is a positive integer greater than 1 and i is a positive integer greater than or equal to 1 and less than or equal to n. Based on the first and second feature maps of the i-th layer, the fused feature map of the i-th layer is obtained; The second feature map of the i-th layer is merged with the fused feature map of the i-th layer to obtain the merged feature map of the i-th layer; The second image information is generated based on the merged feature map of the i-th layer.

5. The method as described in claim 4, characterized in that, Based on the first and second feature maps of the i-th layer, the fused feature map of the i-th layer is obtained, including: The first feature map and the second feature map of the i-th layer are concatenated to obtain the concatenated feature map of the i-th layer. The first attention weight of the first feature map of the i-th layer and the second attention weight of the second feature map of the i-th layer are obtained based on the spliced ​​feature map of the i-th layer. The fused feature map of layer i is obtained based on the first feature map of layer i and its first attention weight, and the second feature map of layer i and its second attention weight.

6. The method as described in claim 1, characterized in that, Also includes: Obtain navigation instructions; Based on the navigation instructions and the second image information, control instructions are generated; The first vehicle is controlled to drive automatically according to the control command.

7. A data processing apparatus, characterized in that, include: The data acquisition module is used to acquire first image information from the first perspective of the first vehicle; A perception and reasoning module is used to identify obstacles contained in the first image information, generate a mask containing occluded areas blocked by the obstacles, obtain a text embedding vector of a prompt text, the prompt text being used to describe an object in the occluded area, and generate a target semantic completion image of the first image information based on the first image information, the mask and the text embedding vector, the target semantic completion image containing the completion of the object in the occluded area. The image generation module is used to generate second image information of the first vehicle from a second perspective based on the first image information and the target semantic completion image.

8. The apparatus as claimed in claim 7, characterized in that, Also includes: A planning and control module is used to obtain navigation instructions and generate control instructions based on the navigation instructions and the second image information. An execution module is used to control the first vehicle to drive automatically according to the control instructions.

9. An electronic device, characterized in that, include: One or more processors; A memory configured to store one or more programs that, when executed by one or more processors, cause the electronic device to perform the method of any one of claims 1 to 6.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is run on a computer, it causes the computer to perform the method of any one of claims 1 to 6.

11. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 6.

Images