Multi-mode parking detection system based on image-text fusion enhancement

Abstract

The invention discloses a multi-modal parking detection system based on image-text fusion enhancement, which is characterized in that a multi-view camera inputs image information, a camera feature extraction module extracts image features of the multi-view camera, an image feature space conversion module fuses images of the multi-view camera to a bev view angle, and the image-text fusion enhancement is realized. Realizing conversion from multi-view image features to bev features; inputting text modal information, and extracting user intention text features by a text feature extraction module; the multi-modal feature fusion module is used for fully fusing multi-modal features through the multi-channel feature fusion module after the text features and the bev features are obtained; and the multi-mode decoder module is used for outputting a parking space with a specific intention of a user based on a decoding structure of transformer. According to the method, the text information is introduced to represent the specific parking intention of the user, and the optimal recommended parking space with the specific user intention is searched for the user through interactive fusion of the text information and the image information.

Description

Technical Field

[0001] The present invention relates to an automatic parking assistance technology, and in particular to a multimodal parking detection system based on image-text fusion enhancement. Background Art

[0002] The automatic parking assistance system can provide users with convenient and fast automatic parking functions in parking scenarios, especially reducing manual operations in complex scenarios, avoiding miscontrol and vehicle collision losses caused by poor driving skills. Camera-based parking perception detection is an important part of the parking assistance system, providing basic information such as parking spaces / obstacles / drivable areas for subsequent planning, positioning, control and other modules.

[0003] The traditional parking perception subsystem is based on multiple fisheye cameras. After acquiring the multi-view fisheye camera images, it uses surround stitching to generate a 360-degree surround stitching image, and performs related target perception tasks on the surround stitching image. Users need to manually enter or click the parking space they want to park in the interactive interface, which is somewhat inconvenient. Summary of the invention

[0004] In order to solve the deficiencies in the prior art, an object of the present invention is to provide a multimodal parking detection system based on image-text fusion enhancement.

[0005] To achieve the purpose of the present invention, the technical solution adopted by the present invention is: A multimodal parking detection system based on image-text fusion enhancement, comprising a multi-view camera, a CNN-based camera feature extraction module, an image feature space conversion module, text modal information, a text feature extraction module, a multimodal feature mixing module, and a multimodal decoder module; The multi-view camera inputs image information, the CNN-based camera feature extraction module extracts the multi-view camera image features, and the image feature space conversion module fuses the multi-view camera image to the bev perspective to achieve the conversion of multi-view image features to bev features; Input text modal information, and the text feature extraction module uses the clip model to extract the user's intended text features; The multimodal feature fusion module obtains the text features and bev grid features, and then the multimodal features are fully integrated through a multi-channel feature fusion module; the multimodal decoder module, based on the transformer decoding structure, finally outputs the parking space output with the user's specific intention.

[0006] Furthermore, the bev space-related loss is introduced after the multimodal feature fusion module, and the intention target-related loss is introduced after the multimodal decoder module.

[0007] Furthermore, the CNN-based camera feature extraction module uses a CNN-based backbone network to extract features from camera input images and uses a large-scale open source parking scene dataset for pre-training. The backbone parameters are shared by multiple cameras. Finally, each camera outputs feature layers of two scales, C4 and C3, for subsequent learning.

[0008] Furthermore, the image feature space conversion module uses a fully connected layer to realize implicit conversion of the feature space. Each bev grid feature is composed of a weighted combination of the entire image view features. The feature values ​​after conversion by the fully connected layer are reconverted and added to the corresponding positions to obtain the bev grid feature. Each camera is implemented with independent fully connected layer parameters. For the C3 and C4 feature outputs of each camera, an independent bev grid generation path is used to finally obtain bev feature maps bf_scale0 and bf_scale1 of two resolutions.

[0009] Furthermore, the multimodal feature fusion module includes a text-to-bev pathway and a bev-to-text pathway; in the text-to-bev pathway, the bev grid features are used to generate K and V features, and the text features generate Q, and new bev grid features are obtained through the hybrid feature module, which are represented as new_bev_feature_scale0 and new_bev_feature_scale1; In the bev-to-text pathway, text features generate K and V, bev grid features generate Q, and finally a new text feature is obtained, denoted as new_text_feature.

[0010] Furthermore, in the hybrid feature module, text features and bev grid features are fused through cross attention.

[0011] Furthermore, in the multimodal decoder module, bf_scale0 and bf_scale1 in the image feature space conversion module constitute a multi-scale bev grid feature; after the multi-scale bev grid feature passes through the multimodal feature fusion module, the fused multi-scale bev grid feature and the text feature are obtained respectively; In the multimodal decoder module, N groups of query features fq are designed to learn and fit features with content representation during the training process; the decoder module is composed of multiple stacked transformer structures. After passing through the decoder module, the feature expression for the intended parking space output is finally obtained.

[0012] Furthermore, in the target detection framework based on the transformer structure, a hybrid guided query position embedding generator is used to query the initial generation of PE embedding; this module contains two parts of guidance for generation: text features and anchor boxes of bev grid features.

[0013] Furthermore, the specific steps for initializing and generating PE embedding are as follows: The new_bev_feature obtained after fusion is guided by the loss function in the bev space to generate multiple locations with higher responses, marked as num_npos grid locations; according to the size of the conference value, the topk grid points are selected to obtain their positions and corresponding bev grid features; Generate anchor boxes at the positions corresponding to the topk bev features, and convert the anchor boxes into pe embeddings through position encoding operations; Text features are introduced as additional supplementary features of the initialized pe embedding. Specifically, the high-frequency user text prompts in the parking scenario are used to form a text prompt setting, and the corresponding text features are extracted to obtain the text feature set of the setting, which is recorded as PV. The topk bev grid features are dot-producted with the PV set to find the most relevant text feature for each grid. After obtaining the grid corresponding anchor box pe and text features respectively, the two parts are superimposed together for initialization of query PE embedding in each module in the two-layer decoding structure.

[0014] The beneficial effect of the present invention lies in that, compared with the prior art, the present invention adopts a solution similar to bevdet, avoids explicit image stitching, and uses a fast and easy-to-deploy method in the feature space to perform multi-perspective feature fusion and generate bev space features; it can simultaneously provide detection of parking spaces / obstacles, etc. on the front view and bev map and extraction of drivable areas.

[0015] The present invention introduces text information to represent the user's specific parking intention. Through the interactive fusion of text information and image information, the best recommended parking space with specific user intention can be searched for the user, avoiding the inconvenience of manual interaction in certain scenarios.

[0016] The present invention provides a parking detection solution that integrates text intention and image features in a parking scenario; the hybrid guided query generator part fully considers the comprehensiveness of text features and conventional position embedding, and has better initialization and convergence effects; the module structure of text and image fusion has the characteristics of cross-reciprocation, and can finally form image-aware text features and text-ware image features on the two paths of image and text, which is an important part of the multimodal model. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 This is a schematic diagram of a 4-way surround fisheye camera; Figure 2 is a schematic diagram of a multimodal parking detection system based on image-text fusion enhancement according to the present invention; Figure 3 This is a schematic diagram of the camera feature extraction module based on CNN; Figure 4 It is a schematic diagram of the image feature space conversion module; Figure 5 It is a schematic diagram of the multimodal feature fusion module; Figure 6 This is a schematic diagram of the hybrid feature module; Figure 7 It is a schematic diagram of the multimodal decoder module; Figure 8 It is a schematic diagram of the hybrid guided query position embedding generator of the transformer structure; Fig. 9 This is a schematic diagram of the bev space related loss; Fig.10 It is a schematic diagram of the loss related to the intention target. DETAILED DESCRIPTION

[0018] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of this application.

[0019] like Figure 1As shown, the present invention uses 4 surround fisheye cameras, which are installed at the illustrated positions of the vehicle, including the front camera, the rear camera, the left camera, and the right camera. The FOV angles of the adjacent 2 cameras have a certain overlap area; after joint calibration, the 4 cameras can form a complete 360-degree surround view mosaic.

[0020] like Figure 2 As shown, the multimodal parking detection system based on image-text fusion enhancement of the present invention includes a multi-view camera, a CNN-based camera feature extraction module, an image feature space conversion module, text modal information, a text feature extraction module, a multimodal feature mixing module, and a multimodal decoder module; the camera is used as the main sensor, and the text modal information is used as an auxiliary input, and feature extraction operations are performed on the input information respectively; after passing through the multimodal feature mixing module, a transformer-based decoder structure is used to finally output a parking space output with user-specific intentions.

[0021] With the help of the basic large model capability based on the transformer structure, the long-range modeling of the feature space of the BEV (panoramic) perspective makes the boundary of the drivable area more refined, and the improvement of the transformer's attention mechanism also makes the feature interaction and fusion between multiple modalities more reasonable. The present invention integrates the BEV image space features and the text features that reflect the user's intentions, and combines the application scenarios of automatic parking perception to achieve a variety of specific tasks such as parking spaces / obstacles / drivable areas.

[0022] At the same time, at the loss design level, multiple auxiliary losses in different feature spaces are introduced, including intent target related loss and bev space related loss, so that the network can achieve faster convergence effect during the learning process.

[0023] like Figure 3 As shown in the figure, the camera feature extraction module based on CNN uses a backbone network based on CNN to extract features from the input images of 4 cameras, and uses a large-scale open source parking scene dataset for pre-training, so that the network has a better sensitivity to the texture features of the parking scene. Among them, the backbone parameters are shared by multiple cameras; finally, each camera outputs two scale feature layers of C4 and C3 for subsequent learning. C4 is a 16-fold downsampling feature layer, and C3 is an 8-fold downsampling feature layer.

[0024] The text feature extraction module uses the encoder_text (encoding module) provided by the clip model (a multimodal pre-trained neural network model) to obtain the features corresponding to the user's intended text.

[0025] like Figure 4 As shown, the image feature space conversion module fuses the multi-view camera image features into the bev view, thereby realizing the conversion from the multi-view image space to the bev feature space.

[0026] The present invention uses a deployment-friendly calculation method to simplify the complexity of feature space conversion. Considering that once the 4-way camera is installed and calibrated, the intrinsic parameters and the extrinsic parameters relative to the ground are fixed, the mapping relationship from ground pixels to bev is fixed, but the pixels above the ground plane vary with depth. Therefore, the present invention uses a fully connected layer to realize the implicit conversion of feature space. Each bev grid feature is composed of a weighted combination of the entire image view features. The feature values ​​after the conversion of the fully connected layer are reconverted and added to the corresponding positions to obtain the bev grid features. The fully connected layer is equivalent to global attention. Through the iteration of the training process, it drives the network to adaptively complete the feature space conversion of the corresponding position.

[0027] The four cameras are implemented with independent fully connected layer parameters. For the C3 and C4 feature outputs of each camera, independent bev grid generation paths are used to finally obtain bev feature maps of two resolutions, bf_scale0 and bf_scale1.

[0028] like Figure 5 As shown in FIG. 1 , after the multimodal feature fusion module obtains the text features and the bev grid features, the multimodal features are fully integrated through the multi-channel feature fusion module. During the transmission process, the text features and the bev grid features will absorb each other's useful features.

[0029] In the text-to-bev pathway, bev grid features are used to generate K and V features, text features generate Q, and new bev grid features are obtained through the hybrid feature module, represented as new_bev_feature_scale0 and new_bev_feature_scale1; similarly, in the bev-to-text pathway, text features generate K and V, bev grid features generate Q, and finally new text features are obtained, represented as new_text_feature.

[0030] like Figure 6 As shown in the figure, the hybrid feature module in text-to-bev is given, and the text features and bev grid features are fused through cross attention.

[0031] like Figure 7 As shown in the figure, the multimodal decoder module is iteratively optimized layer by layer, and bf_scale0 and bf_scale1 in the image feature space conversion module constitute a multi-scale bev feature layer. After the multi-scale bev features pass through the multimodal feature fusion module, the fused multi-scale bev features and text features can be obtained respectively. In the multimodal decoder module, N groups of query features fq are designed to learn to fit features with content representation during the training process. The decoder module is composed of multiple transformer structures stacked together. After passing through the decoder module, the feature expression for the intended parking space output is finally obtained.

[0032] After the query feature fq passes through the scale0 decoder, the refined_fq feature is obtained; this feature is used to generate the input query feature of the scale1 decoder. Through the layer-by-layer iterative optimization mechanism, the query feature can be increasingly focused on the parking space or obstacle target with precise semantics and location.

[0033] Generally, in the transformer-based object detection framework, the query PE embedding is initialized with a randomly generated vector, which will make the subsequent training process extremely slow. Figure 8 As shown, the present invention uses a hybrid guided query position embedding generator for the generation of PE. The module includes two parts of text features and anchor boxes of BEV features for guidance.

[0034] The new_bev_feature obtained after fusion generates multiple positions with higher responses in the bev space through the guidance of the loss function, which are marked as num_npos grid positions. According to the size of the conference value, the topk grid points are selected to obtain their positions and corresponding bev features. First, the anchorbox is generated for the positions corresponding to the topk bev features, and the anchorbox is converted into peembedding through position encoding. Secondly, considering that the targets at the grid positions with high conference are associated with certain text semantics, text features are introduced as additional supplementary features of the initialized pe embedding. The specific operation is to use the high-frequency user text prompts in the parking scene to form a text prompt set, and use the clip model to extract the corresponding text features for each text prompt of the set to obtain the text feature set of the set, which is recorded as PV. By performing a dot product operation on the topk bev grid features and the PV set, the text feature with the highest correlation is found for each grid. After obtaining the grid corresponding anchor box PE and text feature respectively, the two parts are superimposed together for initializing the query PE embedding of each module in the 2-layer decoding structure.

[0035] Loss consists of multiple parts. Bev space related loss, after the multimodal feature fusion module obtains new_bev_feature and new_text_feature, the loss function is introduced to learn grid conference response and grid semantics, such as Fig. 9 shown.

[0036] Intent target related loss, after the multimodal decoder module obtains the decoded refined query feature, it uses this feature to learn parking spaces / obstacles / drivable areas with user intentions, such as Fig.10 shown.

[0037] The applicant of the present invention has made a detailed explanation and description of the implementation examples of the present invention in conjunction with the drawings in the specification. However, those skilled in the art should understand that the above implementation examples are only preferred implementation schemes of the present invention, and the detailed description is only to help readers better understand the spirit of the present invention, but not to limit the scope of protection of the present invention. On the contrary, any improvements or modifications based on the inventive spirit of the present invention should fall within the scope of protection of the present invention.

Claims

1. A multimodal parking detection system based on image-text fusion enhancement, characterized in that: Including multi-view camera, CNN-based camera feature extraction module, image feature space conversion module, text modal information, text feature extraction module, multi-modal feature mixing module, and multi-modal decoder module; The multi-view camera inputs image information, the CNN-based camera feature extraction module extracts the multi-view camera image features, and the image feature space conversion module fuses the multi-view camera image to the bev perspective to achieve the conversion of multi-view image features to bev features; Input text modal information, and the text feature extraction module uses the clip model to extract the user's intended text features; Multimodal feature fusion module, after obtaining text features and bev grid features, the multimodal features are fully integrated through a multi-channel feature fusion module; The multimodal decoder module, based on the transformer decoding structure, finally outputs the parking space output with user-specific intentions.

2. The multimodal parking detection system based on image-text fusion enhancement according to claim 1 is characterized in that: The bev space-related loss is introduced after the multimodal feature fusion module, and the intention target-related loss is introduced after the multimodal decoder module.

3. The multimodal parking detection system based on image-text fusion enhancement according to claim 1 is characterized in that: The CNN-based camera feature extraction module uses a CNN-based backbone network to extract features from camera input images and uses a large-scale open source parking scene dataset for pre-training. The backbone parameters are shared by multiple cameras. Finally, each camera outputs feature layers of two scales, C4 and C3, for subsequent learning.

4. The multimodal parking detection system based on image-text fusion enhancement according to claim 3, characterized in that: The image feature space conversion module uses a fully connected layer to realize implicit conversion of the feature space. Each bev grid feature is composed of a weighted combination of the entire image view features. The feature values ​​after conversion by the fully connected layer are reconverted and added to the corresponding positions to obtain the bev grid feature. Each camera is implemented with independent fully connected layer parameters. For the C3 and C4 feature outputs of each camera, an independent bev grid generation path is used to finally obtain bev feature maps bf_scale0 and bf_scale1 of two resolutions.

5. The multimodal parking detection system based on image-text fusion enhancement according to claim 4, characterized in that: Multimodal feature fusion module, including text-to-bev pathway and bev-to-text pathway; in the text-to-bev pathway, bev grid features are used to generate K and V features, and text features generate Q. New bevgrid features are obtained through the hybrid feature module, represented as new_bev_feature_scale0 and new_bev_feature_scale1; In the bev-to-text pathway, text features generate K and V, bev grid features generate Q, and finally a new text feature is obtained, denoted as new_text_feature.

6. The multimodal parking detection system based on image-text fusion enhancement according to claim 5, characterized in that: In the hybrid feature module, text features and bev grid features are fused through cross attention.

7. The multimodal parking detection system based on image-text fusion enhancement according to claim 5, characterized in that: In the multimodal decoder module, bf_scale0 and bf_scale1 in the image feature space conversion module constitute the multi-scale bev grid features; after the multi-scale bev grid features are passed through the multimodal feature fusion module, the fused multi-scale bev grid features and text features are obtained respectively; In the multimodal decoder module, N sets of query features fq are designed to learn and fit features with content representation during training; The decoder module is composed of multiple stacked transformer structures. After passing through the decoder module, the feature expression for the intended parking space output is finally obtained.

8. The multimodal parking detection system based on image-text fusion enhancement according to claim 7, characterized in that: In the target detection framework based on the transformer structure, a hybrid guided query position embedding generator is used to query the initial generation of PE embedding; this module contains two parts of guidance for generation: text features and anchor boxes of bev grid features.

9. The multimodal parking detection system based on image-text fusion enhancement according to claim 8, characterized in that: Query the specific steps of initialization generation of PE embedding: The new_bev_feature obtained after fusion is guided by the loss function in the bev space to generate multiple locations with higher responses, marked as num_npos grid locations; according to the size of the conference value, the topk grid points are selected to obtain their positions and corresponding bev grid features; Generate anchor boxes at the positions corresponding to the topk bev features, and convert the anchor boxes into pe embeddings through position encoding operations; Text features are introduced as additional supplementary features of the initialized pe embedding. Specifically, the high-frequency user text prompts in the parking scenario are used to form a text prompt setting, and the corresponding text features are extracted to obtain the text feature set of the setting, which is recorded as PV. The topk bev grid features are dot-producted with the PV set to find the most relevant text feature for each grid. After obtaining the grid corresponding anchor box pe and text features respectively, the two parts are superimposed together for initialization of query PE embedding in each module in the two-layer decoding structure.

Images