Multi-camera generated local map template understanding enhanced object detection method and system

Abstract

The application relates to a multi-camera generated local map template understanding reinforced target detection method and system, which comprises the following steps: taking images of multiple vehicle-mounted cameras in the same frame as information sources to construct a generative local map within a self-vehicle perception range; based on the generative local map, interactive learning and cross attention reinforcement of a static environment template generative map and dynamic target detection are carried out, whether a static element is blocked is distinguished by target detection obstacle information, a blocked area is completed, the position and state of a dynamic obstacle are constrained by the static element, and target detection is improved; after target detection post-processing is constrained by the generative local map, a semantic layer of the static element and a three-dimensional bounding box of the dynamic element are output, and the integrity of a local map generation process is enhanced by target detection. The application uses a cross attention mechanism to process constraint of a dynamic target on a static environment template and blocking of the static template by the dynamic target, and jointly enhances the performance of the local map and the target detection.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving, and in particular to a method and system for target detection enhanced by multi-camera generation of local map templates and understanding. Background Technology

[0002] In autonomous driving, vision-based multi-camera environmental perception solutions are a widely studied research area. Environmental perception includes the perception of dynamic targets and the perception of static road features. There are generally three methods for target detection using surround-view cameras. The first is monocular camera detection methods, including direct regression of 3D bounding boxes from feature pyramids and pseudo-LiDAR methods pre-trained with monocular depth. For multi-camera configurations, global non-maximum suppression is used. The second is native multi-camera detection methods, including point cloud fusion methods using multiple pseudo-LiDARs and implicit ensemble prediction methods using transformers. None of these target detection methods consider the constraints of objects in the environment. For example, normally operating vehicles generally face parallel directions to the road, pedestrians generally walk on sidewalks, and vehicles generally travel between two lanes—positional constraints that are typically implemented using high-precision maps.

[0003] The practice of using high-precision maps to form "map sensors" to enhance environmental perception has been widely studied. In 2018, Uber's advanced technology team attempted to represent geometric and semantic prior information from maps in a bird's-eye view and use it as a data channel, feeding it into a deep learning network along with point cloud data. After fusing the geometric and semantic prior information from the map, the accuracy of target detection and localization was significantly improved. In 2019, Manato Hirabayashi et al. improved traffic light recognition by cross-validating features from high-precision maps and camera features. Also in 2019, Ming-Fang Chang et al. provided the rich map perception dataset Argoverse and pointed out three gains of maps for multi-target tracking tasks: drivable area recognition, ground removal, and road orientation priors. These methods provide static environmental templates for target perception, but the requirements for high-precision positioning are very stringent. Inaccurate positioning will cause the input of the map sensor to be completely inaccurate and ineffective. Centimeter-level positioning devices are expensive, and the coverage of high-precision maps is currently limited and involves geographic information security, so it is not currently possible to cover mainstream structured urban roads on a large scale.

[0004] Local map generation is a novel method for perceiving map features using multi-view cameras on vehicles. The main artificial intelligence technology is semantic segmentation. It uses manually collected and labeled high-precision semantic maps as labels, inputting multiple images from the same frame into a neural network. The output is a bird's-eye view with multiple semantic category segments. Finally, through raster clustering of the same category and edge sharpening, the boundaries of each semantic layer are formed, namely, the vector map feature information of road boundaries and lane boundaries. Accurate localization and correct maps are only required when training the network to generate the map. Furthermore, this method can be decoupled from localization and share the same set of sensors for dynamic object perception. However, there is currently no constraint method for tightly coupling dynamic objects with static features. Also, dynamic objects inevitably occlude static map features in traffic scenes, necessitating the design of local map completion methods to address this occlusion. Summary of the Invention

[0005] To address the aforementioned problems, the purpose of this invention is to provide a multi-camera-generated local map template understanding-enhanced target detection method and system. This method and system can use a cross-attention mechanism to handle the constraints of dynamic targets on static environment templates and the occlusion of static templates by dynamic targets, thereby jointly enhancing the performance of local maps and target detection.

[0006] To achieve the above objectives, in a first aspect, the technical solution adopted by the present invention is: a method for enhancing target detection by multi-camera generation of local map templates, characterized in that it includes:

[0007] Using images from multiple vehicle-mounted cameras in the same frame as information sources, a generative local map is constructed within the vehicle's perception range;

[0008] Based on generative local maps, interactive learning and cross-attention enhancement between static environment template generative maps and dynamic object detection are used to identify whether static elements are occluded by object detection obstacle information, complete the occluded areas, and constrain the position and state of dynamic obstacles through static elements to improve object detection.

[0009] Generative local map constraints are used to constrain target detection post-processing, outputting semantic layers of static features and 3D bounding boxes of dynamic features.

[0010] Furthermore, the construction of the generative local map includes a semantic surface layer and a separator layer, enabling an understanding of the static environment template, including:

[0011] Semantic segmentation is performed on the bird's-eye view to obtain a dense semantic grid;

[0012] In monocular vision, for area map features, raster elements that are close in location and have the same semantic attribute are clustered into one class, and their outer envelope is defined by polygons; for line map features, a class of semantic attributes is fitted by polyline segments.

[0013] When using multiple cameras, accurate intrinsic and extrinsic parameter calibration is performed, vector data correlation is established for the observed edges, and then smoothing is applied to obtain a panoramic 360° vector map.

[0014] Furthermore, the semantic segmentation includes:

[0015] A semantic segmentation backbone network is applied to the image plane, and then a two-dimensional to three-dimensional viewpoint transformation is performed to make the image form a light-view cone.

[0016] By transferring the camera's intrinsic and extrinsic parameters to a 3D ground plane, stacked convolutional networks are applied to the bird's-eye view for final semantic segmentation.

[0017] Furthermore, the implementation of the interactive learning and cross-attention includes:

[0018] Setting up a cross-attention mechanism enables the target detection framework to take into account information about the static geographic environment in which the target is located, and implements a transformer structure in the prediction head;

[0019] The output of the transformer structure is an object-level query, and the dimensions of the query are the number of hidden layers and the box encoding length. Global self-attention is formed in the transformer structure by changing the interaction matrix of the query, key, and value.

[0020] Furthermore, the converter structure is a converter-based dual-branch converter, comprising:

[0021] The queries for static features and dynamic templates in the image patch are input into two converters respectively, and different image patches are processed by two independent branches;

[0022] The processed image block enters the encoder of the multi-scale transformer and is then decoded to obtain the final target information.

[0023] Furthermore, the process of completing the occluded area includes:

[0024] After compensating the coordinate positions of the feature maps from previous frames, they are fused into the current frame. Then, the top-view area where the target detection result is located is regarded as a potential occlusion area. The location of the obstacle is determined to be either inside a blocky area or on the boundary line of the blocky area. If it is inside the area, no completion processing is required. If it is on the boundary of the area, the previous information feature maps and previous target detection results need to be retrieved. The position that is not occluded by dynamic obstacles is selected, and temporal inference is performed. After fusing with the result of the current frame, an enhanced static local map is obtained.

[0025] Furthermore, the post-processing for constraining target detection using generative local maps includes: querying the location of the target and determining whether the target is a vehicle; if it is a vehicle, the vehicle is traveling within the lane in the image plane without crossing the lane dividing lines on both sides, and the position of the vehicle is constrained by the generative map; the position of the vehicle's center point is corrected so that the vehicle does not intersect with the vector lane lines in the polygon of the bird's-eye view, the vehicle's orientation information is corrected to be aligned with the adjacent lane lines, and the vehicle's operating status is determined to be normal, correcting the vehicle's position and orientation; if it is not a vehicle, the pedestrian crossing dividing line is extracted, and the geographical area information is used to assist in the judgment of cyclists and pedestrians, with those located at the edge of the lane classified as cyclists and those on the pedestrian crossing classified as pedestrians.

[0026] Secondly, the technical solution adopted by the present invention is as follows: a system for enhancing target detection by multi-camera generating local map template understanding, comprising: a first processing module, which uses images from multiple vehicle-mounted cameras in the same frame as information sources to construct a generative local map within the vehicle's perception range; a second processing module, which, based on the generative local map, uses interactive learning and cross-attention enhancement between the generative local map and dynamic target detection through static environment templates, distinguishes whether static elements are occluded by obstacle information from target detection, completes the occluded areas, and constrains the position and state of dynamic obstacles through static elements to improve target detection; and an output module, which uses the generative local map to constrain post-processing of target detection, and outputs the semantic layer of static elements and the three-dimensional bounding box of dynamic elements.

[0027] Thirdly, the technical solution adopted by the present invention is: a computer-readable storage medium for storing one or more programs, wherein the one or more programs include instructions, which, when executed by a computing device, cause the computing device to perform any of the methods described above.

[0028] Fourthly, the technical solution adopted by the present invention is: a computing device comprising: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include instructions for performing any of the methods described above.

[0029] The present invention has the following advantages due to the adoption of the above technical solutions:

[0030] 1. This invention can simultaneously realize the bounding boxes of static map elements and dynamic objects using only a vehicle-mounted surround-view camera, achieving multi-task learning of two tasks.

[0031] 2. This invention can use a cross-attention mechanism to handle the constraints of dynamic targets on static environment templates and the occlusion of static templates by dynamic targets, thereby jointly enhancing the performance of local maps and target detection.

[0032] 3. This invention can obtain more interpretable reasoning about the appropriate state of obstacles in the environment through rule-based post-processing methods. Attached Figure Description

[0033] Figure 1 This is a flowchart of the multi-camera local map template generation and enhanced target detection method in this embodiment of the invention;

[0034] Figure 2 This is a flowchart of semantic segmentation in generative local map construction in an embodiment of the present invention;

[0035] Figure 3 This is a flowchart of the three-dimensional target detection process in an embodiment of the present invention;

[0036] Figure 4 This is a flowchart illustrating the implementation of the cross-attention module in this embodiment of the invention;

[0037] Figure 5 This is a flowchart of the post-processing of target detection with map information in an embodiment of the present invention;

[0038] Figure 6 This is a flowchart of the map completion process for areas obscured by obstacles in an embodiment of the present invention. Detailed Implementation

[0039] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the described embodiments of the present invention are within the scope of protection of the present invention.

[0040] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0041] To address the current lack of a constraint method that tightly couples dynamic objects with static elements, and the inevitable problem of dynamic objects occluding static map elements in traffic scenarios, this invention provides a method and system for enhancing target detection by generating local map templates using multiple cameras. This method utilizes computer vision to generate local maps through panoramic multi-camera views, and understands local static environment templates to enhance multi-camera target detection.

[0042] This invention utilizes a surround-view multi-camera system, using images from multiple vehicle-mounted cameras within the same frame as information sources. It shares a backbone and neck network between the bird's-eye view semantic segmentation network and the object detection network. The main focus is on multi-task learning of bird's-eye view semantic segmentation and object detection across multiple prediction heads. A cross-attention mechanism is then used to couple the constraints of dynamic targets within a static environment template with the obstruction caused by dynamic targets in the static template. Finally, a road orientation prediction head is used to constrain vehicle orientation information prediction, outputting a semantic layer of static features and 3D bounding boxes of dynamic features. This invention effectively uses a cross-attention mechanism to handle the constraints of dynamic targets within a static environment template and the occlusion of the static template by dynamic targets, jointly enhancing the performance of local mapping and object detection.

[0043] In one embodiment of the present invention, a method for enhancing target detection by generating local map templates using multiple cameras is provided. In this embodiment, as... Figure 1 As shown, the method includes the following steps:

[0044] 1) Using images from multiple vehicle-mounted cameras in the same frame as information sources, construct a generative local map within the vehicle's perception range, including a semantic surface layer and a separator layer, to achieve an understanding of the static environment template.

[0045] 2) Based on generative local maps, interactive learning and cross-attention enhancement between static environment template generative maps and dynamic object detection are used to identify whether static elements are occluded by object detection obstacle information, complete the occluded areas, and constrain the position and state of dynamic obstacles through static elements to improve object detection.

[0046] 3) Utilize generative local map constraints for post-processing of target detection to output semantic layers of static features and 3D bounding boxes of dynamic features, thereby enhancing the integrity of the local map generation process through target detection.

[0047] In step 1) above, the construction of the generative local map, including semantic polygon layers and separator layers, realizes the understanding of the static environment template, specifically including the following steps:

[0048] 1.1) As Figure 2 As shown, semantic segmentation is performed under a bird's-eye view, resulting in a dense semantic grid.

[0049] Specifically, semantic segmentation of static maps is a task performed under a bird's-eye view. The main objects to be segmented are planar road information, such as the location of sidewalks, carriageways, and crosswalks; and linear information, such as the boundary lines between sidewalks and lanes (i.e., guardrail bricks), lane boundaries (dashed lines, solid lines, double yellow lines), zebra crossings, etc.

[0050] The semantic segmentation method is as follows:

[0051] A semantic segmentation backbone network, such as DeeplabV3+ or EfficientNet, is applied to the image plane; then a two-dimensional to three-dimensional viewpoint transformation is performed, and the image is formed into a light-view frustum by estimating the depth of the pixels in the image.

[0052] By transferring the camera's intrinsic and extrinsic parameters to a 3D ground plane, a simple stacked convolutional network is applied to the bird's-eye view for final semantic segmentation.

[0053] Depth estimation networks can be implemented using networks such as PackNet.

[0054] 1.2) In monocular vision, for area map features, raster elements that are close in location and have the same semantic attributes are clustered into one class, and their outer envelope is defined by polygons; for line map features, a class of semantic attributes is fitted by polyline segments.

[0055] 1.3) When using multiple cameras, the extracted vector map features may overlap in different image planes, requiring stitching to avoid errors in edge observation after stitching. First, accurate intrinsic and extrinsic parameter calibration is performed. Then, vector data is correlated for the observed edges, and finally, smoothing is applied to obtain a panoramic 360° vector map.

[0056] In this embodiment, the result of semantic segmentation is a dense semantic raster, which requires post-processing to obtain the final vector map. Density-based noise-applied spatial clustering (DBSCAN) is used.

[0057] In step 2) above, if Figure 3 As shown, the target detection method based on surround-view cameras is as follows: Since multiple cameras are installed in a 360° surround view, the overlap area of ​​the camera's field of view is small. Therefore, target detection of surround-view cameras can be achieved by target detection of a monocular camera. Then, global non-maximum suppression is used to obtain the panoramic target detection result. Target detection of the monocular camera can use frameworks such as FCOS3D to directly regress a 3D bounding box from 2D observations, or it can use frameworks such as DD3D to first perform depth estimation and then convert it into a pseudo-LiDAR point cloud for 3D target detection. Finally, the 3D bounding box of the target is output. The attributes of the bounding box are (semantic category, length, width, height, center point coordinates x, y, z, yaw angle).

[0058] In step 2) above, multi-task learning of generative maps and dynamic object detection can be achieved when the semantic segmentation module of the generative map and the dynamic object detection share the same backbone network and neck network, but use different prediction heads. The main method is to weight the loss functions of each prediction head to obtain the final loss function, which is then backpropagated. The weight settings are also one of the learning parameters to dynamically adjust the optimization balance of the loss functions for multiple tasks. The loss function for static features consists of the classification loss of each smallest unit, characterized by the cross-entropy loss function. The loss function for dynamic features is jointly determined by the classification cross-entropy loss, the bounding box L1 norm loss, and the intersection-union ratio (IU / I) loss. The loss functions of the two sub-tasks are controlled by dynamic weights and jointly optimize the parameters of the backbone network.

[0059] In this embodiment, the implementation of interactive learning and cross-attention is as follows: Figure 4 As shown, specifically:

[0060] The cross-attention mechanism in the deep neural network enables the object detection framework to take into account information about the static geographic environment in which the object is located. A transformer structure is implemented in the prediction head. The output of the transformer structure is an object query, and the dimensions of the query are the number of hidden layers and the box encoding length. Global self-attention is formed in the transformer structure by changing the interaction matrix of the query, key, and value.

[0061] The converter structure is a dual-branch converter based on the converter, specifically as follows:

[0062] 2.1) Input the queries of static features and dynamic templates in the image patch into two converters respectively, and process different image patches using two independent branches;

[0063] 2.2) The processed image patch is input into the encoder of the multi-scale transformer and obtained through consistent decoding. The final target information is then obtained after decoding. The cross-attention mechanism enables image patches representing dynamic objects to notice cross-attention relationships with adjacent image patches representing static environmental features, thus implicitly constraining and determining the relative pose of the target in the environment.

[0064] In step 2) above, during local map updates, dynamic obstacles can obscure the continuity of static map features, resulting in discontinuities in some areas. For example... Figure 6As shown, for example, if a vehicle crosses the lane line, it will cause the lane line to break in the middle. The method to complete the occluded area is as follows: after the feature map of the previous multiple frames is compensated for the coordinate position, it is fused into the current frame. Then, the top view area where the target detection result is located is regarded as the potential occluded area. It is determined whether the obstacle is distributed inside a block area or on the boundary line of the block area. If it is distributed inside the area, no completion processing is required. If it is distributed on the boundary of the area, it is necessary to recall the previous information feature map and the previous target detection result, select the position that is not occluded by the dynamic obstacle, perform temporal inference, and fuse it with the result of the current frame to obtain an enhanced static local map.

[0065] In step 3) above, the driver simultaneously monitors the distribution of structured roads and the poses of dynamic traffic participants, making a preliminary judgment on the target's operational status. Similarly, this invention enables a multi-camera environmental perception system to simultaneously notice the coupling of static and dynamic elements. Because cameras lack texture depth information, errors often occur in estimating the depth and orientation of targets, especially when the vehicle's orientation differs from other vehicles, leading to numerous misjudgments. In the case of structured roads, for each perceived vehicle, the surrounding static environment, lane dividers to the left and right of the vehicle, and the lane area where the vehicle is located are analyzed.

[0066] Post-processing of target detection constrained by generative local maps, such as Figure 5 As shown, a typical vehicle operation scenario includes the following steps:

[0067] 3.1) Locate the target's location and determine if the target is a vehicle;

[0068] 3.2) If it is a vehicle, then the vehicle is traveling in the lane in the image plane without crossing the lane dividing lines on both sides, that is, the lane lines on both sides are continuous and unobstructed. The position of the vehicle is constrained by this generative map.

[0069] 3.3) Correct the position of the vehicle's center point so that the vehicle does not intersect with the vector lane lines in the polygon of the bird's-eye view, correct the vehicle's orientation information to be aligned with the adjacent lane lines, and determine whether the vehicle's running status is normal, and correct the vehicle's position and orientation.

[0070] In the post-processing correction, it is necessary to determine whether the vehicle's operating status is normal, which is mainly determined by the detection confidence level. Since visual detection has high detection accuracy and high detection confidence at close range (within 30 meters), if the vehicle's orientation is seriously inconsistent with the lane, this abnormal situation can be flagged. For example, if a vehicle stops abnormally due to a car accident on the road, this abnormality can be identified and the driver can be reminded to take over manually.

[0071] 3.4) If it is not a vehicle, extract the pedestrian crossing dividing line and use the geographic area information to assist in the judgment of cyclists and pedestrians. Those located at the edge of the lane are classified as cyclists, and those located on the pedestrian crossing are classified as pedestrians.

[0072] Specifically, in target detection, the similarity in appearance between cyclists and pedestrians often causes confusion for the detector. Since cyclists rarely ride on sidewalks, lane information has a weak constraint on cyclist location prediction. The main misclassification relies on the lane / sidewalk boundary to distinguish pedestrians from cyclists. For targets with low classification confidence (i.e., situations where the network struggles to distinguish between pedestrians and cyclists based on features), this embodiment assists in identifying cyclists and pedestrians using geographical area information.

[0073] In one embodiment of the present invention, a system for enhancing target detection by generating local map templates using multiple cameras is provided, comprising:

[0074] The first processing module uses images from multiple vehicle-mounted cameras in the same frame as information sources to construct a generative local map within the vehicle's perception range.

[0075] The second processing module, based on a generative local map, uses interactive learning and cross-attention enhancement between static environment template generative map and dynamic object detection. It uses object detection obstacle information to distinguish whether static elements are occluded, fills in the occluded area, and uses static elements to constrain the position and state of dynamic obstacles, thereby improving object detection.

[0076] The output module utilizes generative local map constraints for target detection post-processing to output semantic layers of static features and 3D bounding boxes of dynamic features.

[0077] In the first processing module mentioned above, the construction of the generative local map includes a semantic surface layer and a separator layer, enabling an understanding of the static environment template, including:

[0078] Semantic segmentation is performed under the bird's-eye view to obtain a dense semantic grid. Specifically, semantic segmentation involves applying a semantic segmentation backbone network to the image plane, then performing a 2D-to-3D perspective transformation to form a light-view cone. The image is then transferred to a 3D ground plane using the camera's intrinsic and extrinsic parameters, and a stacked convolutional network is applied to the bird's-eye view to perform the final semantic segmentation.

[0079] In monocular vision, for area map features, raster elements that are close in location and have the same semantic attribute are clustered into one class, and their outer envelope is defined by polygons; for line map features, a class of semantic attributes is fitted by polyline segments.

[0080] When using multiple cameras, accurate intrinsic and extrinsic parameter calibration is performed, vector data correlation is established for the observed edges, and then smoothing is applied to obtain a panoramic 360° vector map.

[0081] The implementation of interactive learning and cross-attention in the second processing module mentioned above is as follows:

[0082] Setting up a cross-attention mechanism enables the target detection framework to take into account information about the static geographic environment in which the target is located, and implements a transformer structure in the prediction head;

[0083] The output of the transformer structure is an object-level query, and the dimensions of the query are the number of hidden layers and the box encoding length. Global self-attention is formed in the transformer structure by changing the interaction matrix of the query, key, and value.

[0084] The transformer structure is a transformer-based dual-branch transformer. Specifically, the static features and dynamic template queries in the image patch are input into two transformers respectively, and different image patches are processed by two independent branches. The processed image patch enters the encoder of the multi-scale transformer and obtains consistent decoding. After decoding, the final target information is obtained.

[0085] In the second processing module mentioned above, the method for completing the occluded area is as follows: the feature map of the previous multiple frames is compensated for coordinate position and then fused into the current frame. Then, the top view area where the target detection result is located is regarded as the potential occluded area. The obstacle is located whether it is distributed inside a block area or on the boundary line of the block area. If it is distributed inside the area, no completion processing is required. If it is distributed on the boundary of the area, the previous information feature map and the previous target detection result need to be retrieved. The position that is not occluded by dynamic obstacles is selected, and temporal inference is performed and fused with the result of the current frame to obtain an enhanced static local map.

[0086] The above output module includes post-processing for target detection constrained by generative local maps, comprising:

[0087] Locate the target's location and determine if the target is a vehicle;

[0088] If it is a vehicle, then the vehicle is traveling within the lane in the image plane without crossing the lane divider lines on both sides, and the position of the vehicle is constrained by the generative map.

[0089] The vehicle's center point position is corrected so that the vehicle does not intersect with the vector lane lines in the polygon of the bird's-eye view. The vehicle's orientation information is corrected to be aligned with the adjacent lane lines. The vehicle's running status is judged to be normal, and the vehicle's position and orientation are corrected.

[0090] If it is not a vehicle, the pedestrian crossing dividing line is extracted, and the geographical area information is used to assist in the judgment of cyclists and pedestrians. Those located at the edge of the lane are classified as cyclists, and those located on the pedestrian crossing are classified as pedestrians.

[0091] The system provided in this embodiment is used to execute the above-described method embodiments. For specific processes and details, please refer to the above embodiments, which will not be repeated here.

[0092] In one embodiment of the present invention, a computing device is provided, which can be a terminal and may include: a processor, a communication interface, memory, a display screen, and an input device. The processor, communication interface, and memory communicate with each other via a communication bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and a computer program. When executed by the processor, the computer program implements a multi-camera local map template understanding-enhanced target detection method. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The communication interface is used for wired or wireless communication with external terminals. Wireless communication can be achieved through Wi-Fi, a management network, NFC (Near Field Communication), or other technologies. The display screen can be a liquid crystal display or an e-ink display. The input device can be a touch layer covering the display screen, or buttons, a trackball, or a touchpad mounted on the casing of the computing device, or an external keyboard, touchpad, or mouse. The processor can call logical instructions stored in the memory.

[0093] Furthermore, the logical instructions in the aforementioned memory can be implemented as software functional units and sold or used as independent products, and can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0094] In one embodiment of the present invention, a computer program product is provided, the computer program product including a computer program stored on a non-transitory computer-readable storage medium, the computer program including program instructions, and when the program instructions are executed by a computer, the computer is able to perform the methods provided in the above-described method embodiments.

[0095] In one embodiment of the present invention, a non-transitory computer-readable storage medium is provided, which stores server instructions that cause a computer to perform the methods provided in the above embodiments.

[0096] The computer-readable storage medium provided in the above embodiments has a similar implementation principle and technical effect to the above method embodiments, and will not be described again here.

[0097] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0098] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0099] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0100] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-camera generated local map template understanding augmented object detection method, characterized in that, The application relates to a method for constructing a generative local map in a self-perception range of a vehicle based on images of multiple vehicle-mounted cameras in the same frame as information sources. The method comprises the following steps: constructing a generative local map in a self-perception range of a vehicle based on images of multiple vehicle-mounted cameras in the same frame as information sources; based on the generative local map, interactive learning and cross attention are enhanced through interaction between a static environment template generative map and dynamic target detection; whether a static element is blocked is distinguished by target detection obstacle information; an occluded area is completed; the position and state of a dynamic obstacle are constrained by a static element to improve target detection; target detection post-processing is constrained by the generative local map; a semantic layer of a static element and a three-dimensional bounding box of a dynamic element are output. The implementation of the interactive learning and cross attention comprises the following steps: a cross attention mechanism is set to enable a target detection framework to pay attention to information of a static geographical environment where a target is located; and a transformer structure is realized in a prediction head. The output of the transformer structure is an object-level query, and the dimension of the query is the number of hidden layers and the length of a box code; in the transformer structure, global self-attention is formed by changing an interaction matrix of a query, a key and a value. The construction of the generative local map comprises a semantic surface layer and a separation line layer, and understanding of a static environment template is realized, which comprises the following steps: semantic segmentation is performed under a bird's eye view to obtain dense semantic grids; in monocular vision, positionally close raster with the same semantic attribute is clustered into a class, and a polygon frame is adopted to define an envelope surface; in a multiple-camera mode, accurate internal and external parameter calibration is performed, vector data correlation of observed edges is performed, and then a panoramic 360-degree vector map is obtained after smoothing. The semantic segmentation comprises the following steps: a semantic segmentation backbone network is applied to an image plane, and then two-dimensional to three-dimensional perspective conversion is performed to make the image form a light visual cone; the internal and external parameters of the camera are transferred to a three-dimensional ground plane, and a stacked convolutional network is performed in a bird's eye view to regress a final semantic segmentation. The transformer structure is a double-branch transformer based on a transformer, which comprises the following steps: static elements in an image block and a query of a dynamic template are respectively input into two transformers, and different image blocks are processed by two independent branches; the processed image blocks enter an encoder of a multi-scale transformer, and consistent decoding is obtained; and finally, target information is obtained after decoding.

2. The multi-camera generated partial map template understanding boosted object detection method of claim 1, wherein, The completion of the occluded area comprises the following steps: feature maps of previous multiple frames of information are fused to a current frame after coordinate position compensation; a bird's eye view area where a target detection result is located is regarded as a potential occluded area; whether the obstacle is distributed inside a block-shaped area or on a boundary line of the block-shaped area is located; if the obstacle is distributed inside the block-shaped area, no completion processing is needed; if the obstacle is distributed on the boundary line of the block-shaped area, previous information feature maps and previous target detection results are selected and time series inference is performed on positions not blocked by the dynamic obstacle to obtain an enhanced static local map after fusion of the current frame result. The target detection post-processing constrained by the generative local map comprises the following steps: the position of a target is queried, and whether the target is a vehicle is judged; if the target is a vehicle, the vehicle is driving in a lane in the image plane, and there is no lane division line on both sides of the vehicle; the position of the vehicle is constrained by the generative map. ​ ​ 3. The multi-camera generated partial map template understanding boosted object detection method of claim 2, wherein, ​ ​ ​ 4. The multi-camera generated partial map template understanding boosted object detection method of claim 1, wherein, ​ ​ ​ 5. The multi-camera generated partial map template understanding boosted object detection method of claim 1, wherein, ​ ​ 6. The multi-camera generated partial map template understanding boosted object detection method of claim 1, wherein, ​ ​ ​ The position of the vehicle center point is corrected so that the polygons of the vehicle in the bird's eye view do not intersect with the vector lane lines, the orientation information of the vehicle is corrected to be flush with the adjacent lane lines, and whether the running state of the vehicle is normal is determined to correct the position and orientation of the vehicle; If it is not a vehicle, the sidewalk dividing line is extracted, the judgment of the cyclist and the pedestrian is assisted according to the geographic area information, the classification of the position at the edge of the lane is the cyclist, and the classification of the position on the sidewalk is the pedestrian.

7. A system for multi-camera generated local map template understanding boosted object detection, characterized in that, Comprise: The first processing module takes the images of multiple vehicle-mounted cameras in the same frame as the information source, and constructs a generative local map within the self-perception range of the vehicle; The second processing module generates a generative map based on the generative local map, and enhances the interactive learning and cross-attention of the static environment template and the dynamic target detection, distinguishes whether the static element is blocked by the target detection obstacle information, completes the blocked area, and perfects the target detection by constraining the position and state of the dynamic obstacle by the static element; The output module uses the generative local map to constrain the post-processing of the target detection, and outputs the semantic layer of the static element and the three-dimensional bounding box of the dynamic element; The implementation of the interactive learning and cross-attention comprises: The cross-attention mechanism is set to enable the target detection framework to pay attention to the information of the static geographic environment where the target is located, and the transformer structure is realized in the prediction head; The output of the transformer structure is an object-level query, and the dimension of the query is the number of hidden layers and the length of the box code. In the transformer structure, the global self-attention is formed by changing the interaction matrix of the query, key and value.

8. A computer-readable storage medium storing one or more programs, the one or more programs comprising instructions that when executed by a computer cause the computer to perform a method of any of claims 1-7. The one or more programs include instructions that, when executed by a computing device, cause the computing device to perform any of the methods of claims 1-6.

9. A computing device, comprising: Comprise: One or more processors, memories and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs comprise instructions for executing any of the methods of claims 1-6.

Images