Map construction method for camera fault and electronic equipment

Abstract

The invention discloses a map construction method for camera faults and electronic equipment, and belongs to the technical field of image processing, the method comprises the following steps: obtaining a group of target perspective views, the target perspective views being multi-view camera images; determining a complete multi-view PV feature based on a 2D feature extractor, and converting the complete multi-view PV feature into a complete BEV feature; the method comprises the following steps: determining an incomplete multi-view PV feature, and initializing the incomplete multi-view PV feature; constructing a panoramic perspective view based on a G-PVR module; determining reconstruction loss according to the panoramic perspective view; a distillation-based BEV feature correction module determines a correction loss; and determining a training target according to map loss, the reconstruction loss and the correction loss. According to the method and the device, the map construction accuracy can be kept even if the camera breaks down.

Description

Technical Field

[0001] The present application belongs to the field of image processing technology, and in particular relates to a map construction method and electronic device for camera failure. Background Art

[0002] Online high-definition (HD) map construction is a critical and challenging task in autonomous driving, which provides accurate and rich static environment information for vehicle planning and navigation. These maps enable vehicles to accurately localize themselves on the road and predict upcoming features. HD maps can capture and provide important semantics such as road boundaries, lane dividers, and road markings. Depending on the input sensor modality, HD map construction models are generally divided into three categories: camera sensor-based, lidar sensor-based, and camera-lidar sensor fusion models.

[0003] Existing methods based on multi-view cameras usually prioritize improving model performance without addressing the challenges brought about by sensor failure or damage. There is no better solution to the problem of incomplete multi-view images caused by sensor failure or damage. As a result, once the sensor fails or is damaged, the accuracy of high-definition map construction based on incomplete multi-view cameras is low, and map construction cannot be achieved efficiently and accurately.

[0004] In view of the above problems, a map construction method and electronic device for camera failure are proposed in the present application. Summary of the invention

[0005] In order to address the deficiencies of the prior art, the present application provides a map construction method and electronic device for camera failure, which can solve the problem of low map construction accuracy and difficulty caused by the loss of multi-perspective images due to sensor failure or damage in the prior art.

[0006] The technical effects to be achieved by this application are achieved through the following solutions: In a first aspect, the present application provides a method for constructing a map for camera failure, the method comprising: Acquire a set of target perspective images, wherein the target perspective images are multi-view camera images; Determine a complete multi-view PV feature based on the 2D feature extractor, and convert the complete multi-view PV feature into a complete BEV feature; Determining an incomplete multi-view PV feature and initializing the incomplete multi-view PV feature; Construct panoramic perspective images based on G-PVR module; determining a reconstruction loss based on the panoramic perspective image; The distillation-based BEV feature correction module determines the correction loss; A training target is determined according to the map loss, the reconstruction loss and the correction loss.

[0007] In some embodiments, the set of target perspectives is ,in N Indicates the number of camera views, N is a positive integer, Indicates N A target perspective.

[0008] In some embodiments, the determining of the complete multi-view PV features based on the 2D feature extractor comprises: Using a 2D feature extractor, from each set of target perspectives Extract multi-scale 2D features and determine the complete multi-view PV features ,in Representing the target perspective The corresponding complete multi-view PV features.

[0009] In some embodiments, converting the complete multi-view PV signature into a complete BEV signature comprises: The complete multi-view PV features are converted using the 2D to BEV feature conversion module Convert to full BEV features .

[0010] In some embodiments, determining the incomplete multi-view PV features includes: Randomly mask the Mth 2D image feature and determine the incomplete multi-view PV feature as , where M is a positive integer, mask is an integer and 1 <mask< N .

[0011] In some embodiments, the panoramic perspective view constructed based on the G-PVR module satisfies: , in, Indicates that the features of all available views are connected. Indicates the number of available views, H , W , C Respectively represent the spatial height, spatial width and number of channels of the feature map, concat Indicates a connection operation. , Represents the characteristics of all available views, Represents the incomplete multi-view PV features Target perspective image with random masking removed The corresponding complete multi-view PV features , represents the set of real numbers; And generate a set of Gaussian-based reference points to overlay the panoramic perspective: , , in, Indicates reference point coordinate, Indicates reference point coordinate, and are Gaussian distribution and uniform distribution respectively. is the variance.

[0012] In some embodiments, determining the reconstruction loss according to the panoramic perspective image comprises: , in, , Indicates a learnable query. represents reconstruction losses; The distillation-based BEV feature correction module determines a correction loss, comprising: The incomplete multi-view PV features are converted into BEV features using a PV to BEV feature conversion module. Converted to incomplete BEV features , and calculate the correction loss : .

[0013] In some embodiments, determining the training target according to the map loss, the reconstruction loss, and the correction loss includes: Calculate the map loss based on classification loss, point-to-point loss, and edge direction loss ; Determine the training target according to the map loss, the reconstruction loss and the correction loss ,in satisfy: , in, and is a hyperparameter.

[0014] In a second aspect, the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described in the first aspect when executing the computer program.

[0015] In a third aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the method described in the first aspect.

[0016] The embodiments of the present application provide a map construction method and electronic device for camera failure. The method proposes a robust high-precision map construction framework (SafeMap), extracts features from complete multi-view camera images, and uses view transformation to effectively convert them into a unified BEV space. In order to simulate the emergency situation of camera failure, random view masking (RVM) and recovery schemes are used to reconstruct the missing view information, and then a high-definition map is constructed, achieving the technical effect of maintaining map construction accuracy even in the event of a camera failure. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present application or the existing technical solutions, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0018] Figure 1 This is a flow chart of a method for constructing a map for camera failure in one embodiment of the present application; Figure 2 A Gaussian-based reconstruction schematic diagram of a map construction method for camera failure in an embodiment of the present application; Figure 3 It is a schematic block diagram of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0020] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in one or more embodiments of the present application should be understood by people with ordinary skills in the field to which the present application belongs. The "first", "second" and similar words used in one or more embodiments of the present application do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0021] In the related art, due to the significant progress in bird's-eye-view (BEV) perception, HD map construction methods based on multi-view cameras have attracted increasing attention. Compared with LiDAR-based methods and fusion-based methods, multi-view camera-based methods are easier to deploy and more cost-effective. However, these methods are usually designed under the assumption that complete multi-view camera images can be accessed. When one or more camera views fail or are blocked, the model may fail catastrophically due to the lack of critical visual information. The lack of images from different perspectives can severely degrade the overall map construction quality. Therefore, it is crucial to improve the accuracy of online vectorized HD map construction in the presence of incomplete observations.

[0022] Existing multi-view camera-based methods typically prioritize improving model performance without addressing the challenges posed by sensor failure or damage. MapBench evaluates the reliability of HD map models against a variety of natural real-world damage, showing that sensor failure can significantly degrade performance and compromise traffic safety. Methods such as MetaBEV and UniBEV have begun to address camera failure issues in 3D object detection tasks, maintaining impressive performance even when camera views or LiDAR signals are lost. However, these methods still rely on complete multi-view camera images and do not address the challenges of incomplete observations due to camera damage or occlusions. Therefore, HD map construction methods based on incomplete multi-view cameras remain underexplored.

[0023] Therefore, it is necessary to adopt the map construction method for camera failure provided by this application to solve the problem of poor performance caused by camera failure. This application proposes a novel robust high-precision map construction framework (SafeMap) that can maintain accuracy even when a camera fails.

[0024] The core of SafeMap consists of two components: a Gaussian-based global view reconstruction (G-GVR) module and a distillation-based bird's-eye view correction (D-BEVC) module.

[0025] G-GVR adopts Gaussian-based sparse sampling to generate core reference points according to view importance as spatial priors for the missing views. These reference points are then used for deformable attention to dynamically focus on the most informative regions in the available views, thereby effectively reconstructing the missing data.

[0026] Meanwhile, D-BEVC uses the complete panoramic BEV features to further correct the BEV features extracted from incomplete observations. Importantly, these core components are simple yet effective plug-and-play techniques that are compatible with existing pipelines. Experimental results show that SafeMap outperforms previous methods by a large margin on both complete and incomplete observations, demonstrating superior performance and robustness.

[0027] Various non-limiting implementations of the present application are described in detail below in conjunction with the accompanying drawings.

[0028] The core idea of ​​this application is to first extract features from the complete multi-view camera images, and then use view transformation to effectively transform them into a unified bird's-eye view (BEV) space. In order to simulate the emergency situation of camera failure, this application adopts random view masking (RVM) and recovery scheme, Gaussian-based perspective reconstruction (G-PVR) module and distillation-based bird's-eye view correction (D-BEVC) module to reconstruct the missing view information. Finally, the reconstructed BEV features are input into the map decoder and prediction head to build a high-definition map.

[0029] Below, refer to Figure 1 , the map construction method for camera failure of this application is described in detail: In a first aspect, the present application provides a method for constructing a map for camera failure, the method comprising: S1: Acquire a set of target perspective images, wherein the target perspective images are multi-view camera images; S2: determining a complete multi-view PV feature based on a 2D feature extractor, and converting the complete multi-view PV feature into a complete BEV feature; S3: Determine an incomplete multi-view PV feature and initialize the incomplete multi-view PV feature; S4: Construct panoramic perspective image based on G-PVR module; S5: determining a reconstruction loss according to the panoramic perspective image; S6: Determine the correction loss based on the distillation-based BEV feature correction module; S7: Determine a training target according to the map loss, the reconstruction loss and the correction loss.

[0030] The map construction method for camera failure provided in the embodiment of the present application proposes a robust high-precision map construction framework (SafeMap), extracts features from the complete multi-view camera image, and uses view transformation to effectively convert it into a unified BEV space. In order to simulate the emergency situation of camera failure, random view masking (RVM) and recovery schemes are used to reconstruct the missing view information, and then build a high-definition map, achieving the technical effect of maintaining map construction accuracy even in the event of a camera failure.

[0031] Specifically, the present application takes complete or incomplete multi-view camera images as input and predicts vectorized map elements in the BEV space as well as the types of map elements (supported types include road boundaries, lane dividers, and pedestrian crossings).

[0032] In some embodiments, the set of target perspectives is ,in N Indicates the number of camera views, N is a positive integer, Indicates N a target perspective view; illustratively, ,in , Represents the image height and image width respectively.

[0033] This application builds a map encoder based on the high-definition map construction method MapTR, which consists of a 2D feature extractor and a perspective view (PV) to BEV conversion module.

[0034] In some embodiments, the determining of the complete multi-view PV features based on the 2D feature extractor comprises: Using a 2D feature extractor, from each set of target perspectives Extract multi-scale 2D features and determine the complete multi-view PV features ,in Representing the target perspective The corresponding complete multi-view PV features.

[0035] For example, I 1 , …,I N , represents the id of the image, for example I 1 It can be represented as the first target perspective image, I N Can indicate the N target perspectives, for example, there are 6 target perspectives in total. N The value of is 6.

[0036] Specifically, the 2D feature extractor includes, for example, CNN and Swin. This is for illustrative purposes only, and other 2D feature extractors may also be applied here without limitation.

[0037] In some embodiments, converting the complete multi-view PV signature into a complete BEV signature comprises: The complete multi-view PV features are converted using the 2D to BEV feature conversion module Convert to full BEV features To simulate the emergency situation of camera failure in the testing phase, a view masking and restoration scheme is adopted in the training phase.

[0038] In some embodiments, determining the incomplete multi-view PV features includes: Randomly mask the Mth 2D image feature and determine the incomplete multi-view PV feature as , where M is a positive integer, mask is an integer and 1 <mask< N , and then initialized using the spatial features of its surroundings .

[0039] For example, from arrive The Mth complete multi-view PV feature is randomly selected for masking, and the masked multi-view PV feature is , that is, Imask means I 1 ~ I N A certain image feature randomly masked between I mask Indicates that the I mask The 2D image features of the image, the mask is random and its value is greater than 1 and less than N An integer.

[0040] The challenge of reconstructing missing views from multiple perspectives is crucial to ensure comprehensive environmental perception. Related methods rely on cropping some pixels from the left and right neighboring views and processing them through a Transformer-based model to reconstruct the missing views. While this approach is effective, it is limited by the requirement of predefined cropping ratios, cannot fully utilize the information of all available views, and may ignore valuable contextual clues.

[0041] To address these limitations, this paper introduces a novel Gaussian-based perspective view reconstruction (G-PVR) module, which consists of a learnable query vector that serves as a flexible representation of the missing view and combines multi-view values ​​derived from all available viewpoints. Specifically, G-PVR samples features at the location of the reference point p to construct the keys and values ​​of the multi-head attention module: , in, = ; in, V It is a learnable query. and is the transformation matrix, and is key and value embedding, is a lightweight network used to generate offsets, is the sampling function from deformable attention; are features of all available views. However, the original reference points are generated from a uniform grid, ignoring the important fact that different views contribute unequally to the reconstruction of the missing view. Neighboring views usually contain the most relevant information and should be given higher importance, while farther views (e.g., opposite to the missing view) may contain less information and should be weighted accordingly.

[0042] The G-PVR module incorporates prior knowledge about view importance to dynamically focus on the most informative regions, such as Figure 2 Specifically, using the left and right views of the missing perspective image as a starting point, the present application tiles all available frames according to the spatial distance from the missing frame to construct a panoramic perspective image.

[0043] In some embodiments, the panoramic perspective view constructed based on the G-PVR module satisfies: , in, Indicates that the features of all available views are connected. Indicates the number of available views, H, W , C Respectively represent the spatial height, spatial width and number of channels of the feature map, concat Indicates a connection operation. , Represents the characteristics of all available views, Represents the incomplete multi-view PV features Target perspective image with random masking removed The corresponding complete multi-view PV features , represents the set of real numbers; And generate a set of Gaussian-based reference points to overlay the panoramic perspective: , ,

[0044] in, and are Gaussian distribution and uniform distribution respectively. is the variance; in this field, it is usually used N represents a Gaussian distribution, which is used here to represent the number of camera views in the previous text. To distinguish, so use represents Gaussian distribution, letter It does not have any other limiting meaning and is only used to facilitate formula expression.

[0045] The G-PVR module effectively guides the module to focus on the most informative regions in the input views, and can easily adapt to different numbers of input views and different spatial relationships between views. Through a deformable attention module with Gaussian-based reference points and panoramic perspectives, the learnable query $V$ aggregates information from available views, where "V" here represents a vector used to store the reconstructed feature vector.

[0046] In some embodiments, determining the reconstruction loss according to the panoramic perspective image comprises: , in, , V Indicates a learnable query, represents reconstruction losses; In addition to reconstructing the perspective features of partially missing views, it is also necessary to establish a global BEV feature learning space to further correct the BEV features extracted from incomplete observations.

[0047] Specifically, this application utilizes the full multi-view panoramic BEV feature As a supervisory signal, the incomplete observed BEV features are corrected by the correction loss corresponding to the MSE. .

[0048] The distillation-based BEV feature correction module determines a correction loss, comprising: The incomplete multi-view PV features are converted into BEV features using a PV to BEV feature conversion module. Converted to incomplete BEV features , and calculate the correction loss : .

[0049] The map decoder module of this application follows the MapTR model and adopts its BEV feature decoder, which consists of multiple decoding layers and a prediction head. Each decoding layer learns through self-attention and cross-attention. Self-attention is used to promote the interaction between hierarchical map queries in a decoupled manner. The cross-attention in the decoder is specifically designed to enable the interaction between map queries and input BEV features. Specifically, the input to the BEV decoder is from incomplete observations. The MapHead then uses the classification and point branches to produce the final predictions of map element categories and point locations.

[0050] In order to improve the accuracy of online construction of vectorized HD maps in the case of incomplete observations, this application will map loss Combined with the above reconstruction corresponding loss, including perspective reconstruction loss ( ) and BEV feature correction loss ( ).

[0051] In some embodiments, determining the training target according to the map loss, the reconstruction loss, and the correction loss includes: Calculate the map loss based on classification loss, point-to-point loss, and edge direction loss , map loss Calculated according to MapTR; Determine the training target according to the map loss, the reconstruction loss and the correction loss ,in satisfy: ,in, and is a hyperparameter; illustratively, It can be set to 0.05. Can be set to 5.

[0052] The following is a performance comparison of this application. The performance comparison between this application model and the MapTR model when 6 perspectives are missing is shown in the following table: Standard Configuration method <![CDATA[AP ped. ]]> <![CDATA[AP div. ]]> <![CDATA[AP bou. ]]> mAP All Views MapTR Application 46.348.1 51.554.3 53.155.3 50.352.5 Lost perspective method <![CDATA[AP ped. ]]> <![CDATA[AP div. ]]> <![CDATA[AP bou. ]]> mAP Front view (center) MapTR Application 25.736.6 34.545.0 33.645.8 31.342.4 Front left view (left) MapTR Application 37.944.3 47.852.0 45.652.4 43.749.5 Front right view (right) MapTR Application 38.845.3 46.652.3 47.151.7 44.249.4 Rear view (center) MapTR Application 33.439.6 25.240.8 27.041.2 28.540.5 Rear left view (left) MapTR Application 41.345.5 48.353.1 47.852.9 45.850.5 Rear right view (right) MapTR Application 41.246.4 49.553.1 47.853.2 46.150.9

[0053] The performance comparison between the application model and the HIMap model when 6 perspectives are missing is shown in the following table: Standard Configuration method <![CDATA[AP ped. ]]> <![CDATA[AP div. ]]> <![CDATA[AP bou. ]]> mAP All Views HIMap This application 62.262.6 66.566.7 67.968.7 65.566.0 Lost perspective method <![CDATA[AP ped. ]]> <![CDATA[AP div. ]]> <![CDATA[AP bou. ]]> mAP Front view (center) HIMap Application 39.250.7 41.655.7 33.156.3 38.054.3 Front left view (left) HIMap Application 51.559.2 59.664.2 60.065.7 57.063.0 Front right view (right) HIMap Application 57.060.0 62.164.2 62.666.1 60.663.4 Rear view (center) HIMap Application 46.351.4 31.450.8 21.751.6 33.151.3 Rear left view (left) HIMap Application 58.160.7 63.865.4 64.267.0 62.064.4 Rear right view (right) HIMap Application 58.560.8 62.865.1 63.466.5 61.664.2

[0054] As can be seen, this application proposes a novel robust HD map framework (SafeMap) that aims to maintain accuracy even in the absence of camera views. The core of SafeMap is the Gaussian-based global view reconstruction (G-GVR) module, which exploits the relationship between available camera views and fills in missing information using a Gaussian distribution. In addition, SafeMap also incorporates a distillation-based bird's-eye view correction (D-BEVC) module, which uses the complete panoramic BEV features to further correct the BEV features extracted from incomplete observations.

[0055] Experimental results show that SafeMap achieves significant performance improvements over baseline models with fewer parameters in both complete and incomplete observation scenarios, while maintaining comparable inference speed and memory efficiency, improving the accuracy and efficiency of map construction.

[0056] It should be noted that the method of one or more embodiments of the present application can be performed by a single device, such as a computer or server. The method of this embodiment can also be applied to a distributed scenario and completed by multiple devices cooperating with each other. In the case of such a distributed scenario, one of the multiple devices can only perform one or more steps in the method of one or more embodiments of the present application, and the multiple devices will interact with each other to complete the described method.

[0057] It should be noted that the above describes specific embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0058] Based on the same inventive concept, corresponding to any of the above-mentioned embodiments and methods, the present application also discloses an electronic device; Specifically, Figure 3The hardware structure diagram of an electronic device for a method for building a map for camera failure provided in this embodiment is shown, and the device may include: a processor 410, a memory 420, an input / output interface 430, a communication interface 440, and a bus 450. The processor 410, the memory 420, the input / output interface 430, and the communication interface 440 are connected to each other in communication within the device through the bus 450.

[0059] The processor 410 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application.

[0060] The memory 420 may be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 420 may store an operating system and other application programs. When the technical solution provided in the embodiment of the present application is implemented by software or firmware, the relevant program code is stored in the memory 420 and is called and executed by the processor 410.

[0061] The input / output interface 430 is used to connect the input / output module to realize information input and output. The input / output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.

[0062] The communication interface 440 is used to connect a communication module (not shown in the figure) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired mode (for example, USB, network cable, etc.) or a wireless mode (for example, mobile network, WIFI, Bluetooth, etc.).

[0063] The bus 450 includes a path that transmits information between the various components of the device (eg, the processor 410 , the memory 420 , the input / output interface 430 , and the communication interface 440 ).

[0064] It should be noted that, although the above device only shows the processor 410, the memory 420, the input / output interface 430, the communication interface 440 and the bus 450, in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include the components necessary for implementing the embodiment of the present application, and does not necessarily include all the components shown in the figure.

[0065] The electronic device of the above-mentioned embodiment is used to implement the corresponding map construction method for camera failure in any of the above-mentioned embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0066] Based on the same inventive concept, corresponding to any of the above-mentioned embodiments and methods, one or more embodiments of the present application also provide a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the map construction method for camera failure as described in any of the above embodiments.

[0067] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0068] The computer instructions stored in the storage medium of the above embodiment are used to enable the computer to execute the map construction method for camera failure as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0069] A person skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present application (including the claims) is limited to these examples. In line with the concept of the present application, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of different aspects of one or more embodiments of the present application as described above, which are not provided in detail for the sake of simplicity.

[0070] In addition, to simplify the description and discussion, and in order not to make one or more embodiments of the present application difficult to understand, the well-known power / ground connections to the integrated circuit (IC) chip and other components may or may not be shown in the provided figures. In addition, the device may be shown in the form of a block diagram to avoid making one or more embodiments of the present application difficult to understand, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform on which one or more embodiments of the present application will be implemented (that is, these details should be fully within the scope of understanding of those skilled in the art). Where specific details (e.g., circuits) are set forth to describe exemplary embodiments of the present application, it is obvious to those skilled in the art that one or more embodiments of the present application can be implemented without these specific details or with changes in these specific details. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0071] Although the present application has been described in conjunction with specific embodiments of the present application, many replacements, modifications and variations of these embodiments will be apparent to those skilled in the art based on the foregoing description. For example, other memory architectures (e.g., DRAM, i.e., dynamic RAM) may use the embodiments discussed.

[0072] One or more embodiments of the present application are intended to cover all such substitutions, modifications and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of the present application should be included in the scope of protection of the present application.

Claims

1. A method for constructing a map for camera failures, characterized in that: The method comprises: Acquire a set of target perspective images, wherein the target perspective images are multi-view camera images; Determine a complete multi-view PV feature based on the 2D feature extractor, and convert the complete multi-view PV feature into a complete BEV feature; Determining an incomplete multi-view PV feature and initializing the incomplete multi-view PV feature; Construct panoramic perspective images based on G-PVR module; determining a reconstruction loss based on the panoramic perspective image; The distillation-based BEV feature correction module determines the correction loss; A training target is determined according to the map loss, the reconstruction loss and the correction loss.

2. The method for constructing a map for camera failure according to claim 1, characterized in that: The set of target perspectives is ,in N Indicates the number of camera views, N is a positive integer, Indicates N A target perspective.

3. The map construction method for camera failure according to claim 1 or 2, characterized in that: The method of determining the complete multi-view PV features based on the 2D feature extractor includes: Using a 2D feature extractor, from each set of target perspectives Extract multi-scale 2D features and determine the complete multi-view PV features ,in Representing the target perspective The corresponding complete multi-view PV features.

4. The method for constructing a map for camera failure according to claim 3, characterized in that: The converting the complete multi-view PV features into a complete BEV features comprises: The complete multi-view PV features are converted using the 2D to BEV feature conversion module Convert to full BEV features .

5. The method for constructing a map for camera failure according to claim 3, characterized in that: The determining of incomplete multi-view PV features comprises: Randomly mask the Mth 2D image feature and determine the incomplete multi-view PV feature as , where M is a positive integer, mask is an integer and 1 <mask< N .

6. The method for constructing a map for camera failure according to claim 5, characterized in that: The panoramic perspective view constructed based on the G-PVR module satisfies: ， in, Indicates that the features of all available views are connected. Indicates the number of available views, H , W , C Respectively represent the spatial height, spatial width and number of channels of the feature map, concat Indicates a connection operation. , Represents the characteristics of all available views, Represents incomplete multi-view PV features Target perspective image with random masking removed The corresponding complete multi-view PV features , represents the set of real numbers; And generate a set of Gaussian-based reference points to overlay the panoramic perspective: ， ， in, Indicates reference point coordinate, Indicates reference point coordinate, and are Gaussian distribution and uniform distribution respectively. is the variance.

7. The method for constructing a map for camera failure according to claim 6, characterized in that: The determining of the reconstruction loss according to the panoramic perspective image comprises: ， in, , V Indicates a learnable query, represents reconstruction losses; The distillation-based BEV feature correction module determines a correction loss, comprising: The incomplete multi-view PV features are converted into BEV features using a PV to BEV feature conversion module. Converted to incomplete BEV features , and calculate the correction loss : 。 8. The method for constructing a map for camera failure according to claim 7, characterized in that: The determining of the training target according to the map loss, the reconstruction loss and the correction loss comprises: Calculate the map loss based on classification loss, point-to-point loss, and edge direction loss ; Determine the training target according to the map loss, the reconstruction loss and the correction loss ,in satisfy: ， in, and is a hyperparameter.

9. An electronic device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the map construction method for camera failure as described in any one of claims 1 to 8 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the map construction method for camera failure as described in any one of claims 1 to 8.

Images