Multimodal dynamic fusion perception method and device for common sense memory

Abstract

The invention provides a multi-modal dynamic fusion perception method and device for common sense memory. The method comprises the following steps: acquiring scene text information, environment information and agent state information from a multi-vision RGB image at the current moment; obtaining first perception text information from a perception common sense memory database based on the scene text information; aggregating the scene text information, the continuous multi-frame environment information and agent state information at the current moment and before, and the first sensing text information to generate prompt information; performing feature extraction on the multi-vision RGB image to obtain a first BEV feature, and performing feature extraction on the point cloud data to obtain a second BEV feature; performing fusion processing on the first BEV feature and the second BEV feature to obtain a dynamic cooperation feature; and processing the prompt information and the dynamic cooperation features by using a visual language large model to obtain a perception result at the current moment. The comprehensive sensing precision of the intelligent agent can be improved.

Description

Technical Field

[0001] The present application relates to the field of autonomous driving technology, and in particular to a multimodal dynamic fusion perception method and device for common sense memory. Background Art

[0002] At present, the dynamic fusion methods of multimodal perception mainly include: 1) multimodal fusion based on dynamic network architecture or uncertainty estimation to achieve higher perception accuracy and range; 2) multimodal fusion based on cross-attention, using the attention mechanism to learn multimodal perception interaction and dynamically assign weights to features of different modalities to improve the fusion effect, but the real-time cost is relatively high; 3) using large language models to improve the effect of visual understanding.

[0003] The above methods all ignore the idea of ​​human perceptual memory and limit the adequacy of environmental perception. Summary of the invention

[0004] In view of this, the present application provides a multimodal dynamic fusion perception method and device for common sense memory to solve the above technical problems.

[0005] In a first aspect, the present application provides a multimodal dynamic fusion perception method for common sense memory, including:

[0006] Obtain the multi-visual RGB image and point cloud data collected by the agent at the current moment;

[0007] Obtain scene text information, environment information, and agent state information from the multi-visual RGB image at the current moment;

[0008] Obtaining first perception text information from a perception common sense memory database based on the scene text information;

[0009] Aggregate scene text information, environment information and agent state information of the current moment and previous consecutive frames, and first perception text information to generate prompt information;

[0010] Perform feature extraction on the multi-visual RGB image at the current moment to obtain a first BEV feature, and perform feature extraction on the point cloud data at the current moment to obtain a second BEV feature;

[0011] The first BEV feature and the second BEV feature are fused to obtain a dynamic collaboration feature;

[0012] The prompt information and the dynamic collaboration features are processed using a large visual language model to obtain a perception result at the current moment, wherein the perception result includes: second perception text information and visual features.

[0013] In a possible implementation, the perceptual common sense memory database stores a plurality of perceptual text information; the method further includes:

[0014] calculating a similarity value between the second perceived text information and each perceived text information;

[0015] When at least one similarity value is smaller than a preset threshold, the second perceived text information is stored in the perceived common sense memory database.

[0016] In a possible implementation, obtaining first perception text information from a perception common sense memory database based on scene text information includes:

[0017] Calculate the similarity between the scene text information and each perceived text information;

[0018] Arrange similar values ​​in descending order to generate a similar value sequence;

[0019] The K perceived text information corresponding to the first K similarity values ​​in the similarity value sequence are used as the first perceived text information.

[0020] In a possible implementation, the environmental information at the current moment includes: weather and lighting; the agent state information includes: speed and turning angle.

[0021] In a possible implementation, feature extraction is performed on the multi-visual RGB image at the current moment to obtain a first BEV feature, and feature extraction is performed on the point cloud data at the current moment to obtain a second BEV feature, including:

[0022] Use the ResNet101 network to extract features from N visual RGB images and obtain N visual feature maps:

[0023] The deformable cross attention network is used to process N visual feature maps to obtain the first BEV feature

[0024]

[0025] in, Indicates that the j-th three-dimensional point X j Projection to the i-th visual feature map The result; Q is a set of learnable BEV parameters, DeformAttn(·,·) represents the deformable cross attention network;

[0026] The PointPillars model is used to extract features from point cloud data to obtain the point cloud feature map F LiDR ;

[0027] Use the deformable cross attention mechanism to the point cloud feature map FLiDR Processing is performed to obtain the second BEV feature

[0028]

[0029] Among them, P(X j ,F LiDR ) represents the jth three-dimensional point X j Projection to point cloud feature map F LiDR result.

[0030] In a possible implementation, the first BEV feature and the second BEV feature are fused to obtain a dynamic collaboration feature, including:

[0031] The first BEV features and second BEV features Splicing is performed along the channel to obtain the third BEV feature;

[0032] The third BEV feature is processed by using an adaptive global average pooling operation to obtain a first global perception vector;

[0033] The first global perception vector is reduced in dimension using a 1x1 convolutional layer to obtain the second global perception vector.

[0034] The second global perception vector is processed using the sigmoid (·) function to obtain the global weight w;

[0035] Based on the global weight w, the self-attention mechanism is used to focus on the first BEV feature. and second BEV features Fusion to obtain dynamic collaboration features

[0036]

[0037] Among them, Q is the query vector of the triple of the self-attention mechanism, K is the key vector of the triple of the self-attention mechanism, and V is the value vector of the triple of the self-attention mechanism; w Q , w k and w v All represent learnable parameters; Concat(·,·) represents the concatenation function; d is the dimension of the key vector, and softmax(·) is the normalization function.

[0038] In a second aspect, the embodiment of the present application provides a multimodal dynamic fusion perception device for common sense memory, including:

[0039] An acquisition unit, used to acquire the multi-visual RGB image and point cloud data collected by the agent at the current moment;

[0040] A first processing unit, used to obtain scene text information, environment information and agent state information from the multi-visual RGB image at the current moment;

[0041] A second processing unit, configured to obtain first perception text information from a perception common sense memory database based on the scene text information;

[0042] A generating unit, configured to aggregate scene text information, environment information and agent state information of a current moment and a plurality of consecutive frames before and after, and first perceived text information to generate prompt information;

[0043] A feature extraction unit is used to extract features from the multi-visual RGB image at the current moment to obtain a first BEV feature, and to extract features from the point cloud data at the current moment to obtain a second BEV feature;

[0044] A fusion unit, used for fusing the first BEV feature and the second BEV feature to obtain a dynamic collaboration feature;

[0045] The perception unit is used to process the prompt information and the dynamic collaboration features using the visual language large model to obtain the perception result at the current moment, and the perception result includes: the second perception text information and the visual features.

[0046] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method of the embodiment of the present application when executing the computer program.

[0047] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, a method of the embodiment of the present application is implemented.

[0048] The method of the present application solves the problem of incomplete multimodal perception fusion caused by lack of feedback of experiential perception common sense by establishing a perceptual common sense memory database to store historical perceptual experience; it aggregates the acquired weather, lighting, speed, corner and other information and historical perception information to obtain aggregated prompt information, provides contextual information for visual language large model reasoning learning, and obtains accurate multimodal fusion perception. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the specific implementation methods of the present application or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0050] Figure 1 A technical roadmap of a multimodal dynamic fusion perception method for common sense memory provided in an embodiment of the present application;

[0051] Figure 2 A flowchart of a multimodal dynamic fusion perception method for common sense memory provided in an embodiment of the present application;

[0052] Figure 3 A schematic diagram of the process of aligning and fusing image features and point cloud features provided in an embodiment of the present application;

[0053] Figure 4 A functional structure diagram of a multimodal dynamic fusion perception device for common sense memory provided in an embodiment of the present application;

[0054] Figure 5 A functional structure diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.

[0056] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for which protection is sought, but merely represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0057] First, the design concept of the embodiments of the present application is briefly introduced.

[0058] At present, dynamic multimodal perception fusion technology is mostly aimed at those without common sense memory, and has the following problems:

[0059] The first is the lack of common sense perception experience: the lack of feedback from human experience perception common sense leads to incomplete perception in an open environment with small sample data, which limits the perception accuracy in an open environment, and then leads to poor effectiveness of downstream decision-making and planning tasks, causing insecurity in the field of intelligence.

[0060] The second is insufficient dynamic collaboration between modalities: the inability to perceive multimodal information dynamically and collaboratively like humans will make it impossible to fully utilize the complementary information between modalities, causing the intelligent agent to obtain incorrect environmental perception in an open environment and make wrong decisions.

[0061] In view of the above-mentioned deficiencies and in view of the way in which humans use common sense memory to dynamically perceive multimodal information in the environment, the present application provides a multimodal dynamic fusion perception method for common sense memory, which is characterized by comprising: obtaining multi-visual RGB images and point cloud data collected by the agent at the current moment; obtaining scene text information, environmental information and agent state information from the multi-visual RGB images at the current moment; obtaining first perception text information from the perception common sense memory database based on the scene text information; aggregating the scene text information, environmental information and agent state information of multiple consecutive frames at the current moment and before, and the first perception text information to generate prompt information; performing feature extraction on the multi-visual RGB image at the current moment to obtain a first BEV feature, and performing feature extraction on the point cloud data at the current moment to obtain a second BEV feature; fusing the first BEV feature and the second BEV feature to obtain a dynamic collaborative feature; using a large visual language model to process the prompt information and the dynamic collaborative feature to obtain a perception result at the current moment, the perception result including: second perception text information and visual features; the specific technical route is as follows Figure 1 shown.

[0062] Compared with the prior art, the advantages of the present invention are:

[0063] 1. This application adopts a fusion structure of intelligent agent multi-perspective, lidar, weather lighting, speed / corner multi-modal data, establishes a perceptual common sense memory database, expands the data domains such as weather information, speed, corners, etc. at the current moment, can supplement richer features, and provide a solid foundation for the accuracy of decision-making.

[0064] 2. This application adopts a large multimodal fusion model of visual language that connects visual features, text, weather lighting, and speed / corners, combines perceptual common sense memory and dynamic collaboration between modalities, conducts multi-perspective reasoning learning, obtains fused multimodal perceptual features, and can improve overall perception accuracy.

[0065] After introducing the application scenarios and design concepts of the embodiments of the present application, the technical solutions provided by the embodiments of the present application are described below.

[0066] like Figure 2 As shown, the present application implements a multimodal dynamic fusion perception method for common sense memory, including:

[0067] Step 101: Obtain the multi-visual RGB image and point cloud data collected by the agent at the current moment;

[0068] Among them, the multi-visual RGB image is collected by a multi-eye camera, which includes a forward-view camera, a left-view camera, a right-view camera and a rear-view camera, which can collect 360-degree image information around the intelligent body without blind spots.

[0069] The multi-vision RGB images include the front view image, left view image, right view image, rear view image, other vehicle view image and road side view image of the vehicle.

[0070] Step 102: Obtain scene text information, environment information, and agent state information from the multi-vision RGB image at the current moment;

[0071] Step 103: Obtaining first perception text information from a perception common sense memory database based on the scene text information;

[0072] For example, the perceptual common sense memory database is based on the idea of ​​human perceptual common sense memory, stores historical common sense perceptual features, and serves as empirical perception to help the visual-language large model learn reasoning, obtain accurate environmental perception at the current moment, and improve cognitive ability.

[0073] Step 104: Aggregate the scene text information, the current moment and the previous multiple consecutive frames of environmental information and agent state information, and the first perception text information to generate prompt information;

[0074] Preferably, the environment information and agent state information are usually selected from 5 consecutive frames.

[0075] Step 105: extracting features from the multi-visual RGB image at the current moment to obtain a first BEV feature, and extracting features from the point cloud data at the current moment to obtain a second BEV feature;

[0076] Step 106: Fusing the first BEV feature and the second BEV feature to obtain a dynamic collaboration feature;

[0077] In real-life scenarios, the key to complementary advantages between modalities is full collaboration between modalities. Therefore, enhancing dynamic collaboration between modalities is the basis for intelligent agents to accurately perceive the environment. By enhancing dynamic collaboration between modalities, the problem of insufficient utilization of complementary information between multiple modalities is solved. For example, the dynamic collaboration between image perception features and LiDAR perception features guided by visual images as the main modality is achieved.

[0078] Step 107: Use the visual language large model to process the prompt information and the dynamic collaboration features to obtain the perception result at the current moment, wherein the perception result includes: the second perception text information and the visual features.

[0079] Preferably, the visual language large model adopts the LLAMA2 large model.

[0080] The method of this embodiment solves the problem of incomplete multimodal perception fusion due to lack of feedback of experience perception common sense by establishing a perception common sense memory database to store historical perception experience. The acquired weather illumination, speed, corner and other information and historical perception information are aggregated to obtain aggregated prompt information, provide context information for the reasoning learning of the visual language large model, and obtain accurate multimodal fusion perception.

[0081] In some embodiments, the perceptual common sense memory database stores a plurality of perceptual text information; the method further comprises:

[0082] Calculating a similarity value between the second perceived text information and each perceived text information; wherein the perceived text information corresponds to a vector value;

[0083] When at least one similarity value is smaller than a preset threshold, the second perceived text information is stored in the perceived common sense memory database.

[0084] This embodiment uses an incremental method to store the current perceived text information into the common sense memory database, and can effectively avoid duplication and redundancy of perceived text information.

[0085] In a possible implementation, obtaining first perception text information from a perception common sense memory database based on scene text information includes:

[0086] Calculate the similarity between the scene text information and each perceived text information;

[0087] Arrange similar values ​​in descending order to generate a similar value sequence;

[0088] The K perceived text information corresponding to the first K similarity values ​​in the similarity value sequence are used as the first perceived text information.

[0089] This embodiment is based on the idea of ​​human perceptual common sense memory, establishes a perceptual common sense memory database, stores historical common sense perceptual features, and uses them as experience perception to help the visual-language large model learn reasoning and obtain accurate environmental perception at the current moment, thereby improving the model's cognitive ability and solving the problem of inaccurate model perception information due to lack of historical perceptual experience.

[0090] In some embodiments, the environmental information at the current moment includes: weather and lighting; the agent state information includes: speed and turning angle.

[0091] In some embodiments, feature extraction is performed on the multi-visual RGB image at the current moment to obtain a first BEV feature, and feature extraction is performed on the point cloud data at the current moment to obtain a second BEV feature, including:

[0092] Use the ResNet101 network to extract features from N visual RGB images and obtain N visual feature maps:

[0093] The deformable cross attention network is used to process N visual feature maps to obtain the first BEV feature

[0094]

[0095] in, Indicates that the j-th three-dimensional point X j Projection to the i-th visual feature map The result; Q is a set of learnable BEV parameters, DeformAttn(·,·) represents the deformable cross attention network;

[0096] The PointPillars model is used to extract features from point cloud data to obtain the point cloud feature map F LiDR ;

[0097] Use the deformable cross attention mechanism to the point cloud feature map F LiDR Processing is performed to obtain the second BEV feature

[0098]

[0099] Among them, P(X j ,F LiDR ) represents the jth three-dimensional point X j Projection to point cloud feature map F LiDR result.

[0100] Since the camera image features and radar features are not in the same coordinate system, the camera image perception feature F img and the lidar feature F LiDR Not in the same coordinate system, this embodiment refers to the BEV encoding technology to align the BEV features of the two modalities, enhance the dynamic collaboration of the features between the modalities, and better complement the difference information between the modalities, such as Figure 3 shown.

[0101] In some embodiments, the first BEV feature and the second BEV feature are fused to obtain a dynamic collaboration feature, including:

[0102] The first BEV features and second BEV features Splicing is performed along the channel to obtain the third BEV feature;

[0103] The third BEV feature is processed by using an adaptive global average pooling operation to obtain a first global perception vector;

[0104] The first global perception vector is reduced in dimension using a 1x1 convolutional layer to obtain the second global perception vector.

[0105] The second global perception vector is processed using the sigmoid (·) function to obtain the global weight w;

[0106] Based on the global weight w, the self-attention mechanism is used to focus on the first BEV feature. and second BEV features Fusion to obtain dynamic collaboration features

[0107]

[0108] Among them, Q is the query vector of the triple of the self-attention mechanism, K is the key vector of the triple of the self-attention mechanism, and V is the value vector of the triple of the self-attention mechanism; w Q , w k and w v All represent learnable parameters; Concat(·,·) represents the concatenation function; d is the dimension of the key vector, and softmax(·) is the normalization function.

[0109] In the above process, the weight w is used to adaptively enhance the camera perception features and radar perception features, and the two enhanced modalities are fused. Then, the enhanced camera perception features are used as Q to guide the dynamic collaborative fusion between the two modalities to obtain the final dynamic collaborative perception between the modalities, thereby solving the problem of perception errors caused by insufficient collaboration between multiple modalities.

[0110] Based on the above embodiments, the present application provides a multi-modal dynamic fusion perception device for common sense memory, see Figure 4 As shown, the multimodal dynamic fusion perception device 200 for common sense memory provided in the embodiment of the present application at least includes:

[0111] The first acquisition unit 201 is used to acquire the multi-visual RGB image and point cloud data collected by the agent at the current moment;

[0112] The first processing unit 202 is used to obtain scene text information, environment information and agent state information from the multi-vision RGB image at the current moment;

[0113] The second processing unit 203 is used to obtain first perception text information from the perception common sense memory database based on the scene text information;

[0114] A generating unit 204 is used to aggregate the scene text information, the environment information and agent state information of the current moment and the previous consecutive multiple frames, and the first perception text information to generate prompt information;

[0115] The feature extraction unit 205 is used to perform feature extraction on the multi-visual RGB image at the current moment to obtain a first BEV feature, and perform feature extraction on the point cloud data at the current moment to obtain a second BEV feature;

[0116] A fusion unit 206, configured to fuse the first BEV feature and the second BEV feature to obtain a dynamic collaboration feature;

[0117] The perception unit 207 is used to process the prompt information and the dynamic collaboration feature by using the visual language large model to obtain the perception result at the current moment, and the perception result includes: the second perception text information and the visual feature.

[0118] It should be noted that the principle of solving the technical problem of the multimodal dynamic fusion perception device 200 for common sense memory provided in the embodiment of the present application is similar to the method provided in the embodiment of the present application. Therefore, the implementation of the multimodal dynamic fusion perception device 200 for common sense memory provided in the embodiment of the present application can refer to the implementation of the method provided in the embodiment of the present application, and the repeated parts will not be repeated.

[0119] Based on the above embodiments, the present application also provides an electronic device, referring to Figure 5 As shown, the electronic device 300 provided in the embodiment of the present application includes at least: a processor 301, a memory 302, and a computer program stored in the memory 302 and executable on the processor 301. When the processor 301 executes the computer program, the multimodal dynamic fusion perception method for common sense memory provided in the embodiment of the present application is implemented.

[0120] The electronic device 300 provided in the embodiment of the present application may further include a bus 303 connecting different components (including the processor 301 and the memory 302). The bus 303 represents one or more of several types of bus structures, including a memory bus, a peripheral bus, a local bus, and the like.

[0121] The memory 302 may include a readable medium in the form of a volatile memory, such as a random access memory (RAM) 3021 and / or a cache memory 3022 , and may further include a read-only memory (ROM) 3023 .

[0122] The memory 302 may also include a program tool 3025 having a set (at least one) of program modules 3024, the program modules 3024 including but not limited to: an operating subsystem, one or more applications, other program modules and program data, each of which or some combination may include the implementation of a network environment.

[0123] The electronic device 300 may also communicate with one or more external devices 304 (e.g., keyboards, remote controls, etc.), and may also communicate with one or more devices that enable a user to interact with the electronic device 300 (e.g., mobile phones, computers, etc.), and / or communicate with any device that enables the electronic device 300 to communicate with one or more other electronic devices 300 (e.g., routers, modems, etc.). Such communication may be performed through an input / output (I / O) interface 305. Furthermore, the electronic device 300 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network adapter 306. Figure 5 As shown, the network adapter 306 communicates with other modules of the electronic device 300 via the bus 303. It should be understood that although Figure 5 Not shown, other hardware and / or software modules may be used in conjunction with the electronic device 300, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, disk arrays (Redundant Arrays of Independent Disks, RAID) subsystems, tape drives, and data backup storage subsystems.

[0124] It should be noted that Figure 5 The electronic device 300 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0125] The embodiment of the present application also provides a computer-readable storage medium, which stores computer instructions, and when the computer instructions are executed by the processor, the multimodal dynamic fusion perception method for common sense memory provided by the embodiment of the present application is implemented. Specifically, the executable program can be built-in or installed in the electronic device 300, so that the electronic device 300 can implement the multimodal dynamic fusion perception method for common sense memory provided by the embodiment of the present application by executing the built-in or installed executable program.

[0126] The multimodal dynamic fusion perception method for common sense memory provided in the embodiment of the present application can also be implemented as a program product, which includes a program code. When the program product can be run on an electronic device 300, the program code is used to enable the electronic device 300 to execute the multimodal dynamic fusion perception method for common sense memory provided in the embodiment of the present application.

[0127] The program product provided in the embodiments of the present application may adopt any combination of one or more readable media, wherein the readable medium may be a readable signal medium or a readable storage medium, and the readable storage medium may be but is not limited to an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any combination of the above. Specifically, more specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, RAM, ROM, Erasable Programmable Read Only Memory (EPROM), optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0128] The program product provided in the embodiment of the present application may adopt a CD-ROM and include program code, and may also be run on a computing device. However, the program product provided in the embodiment of the present application is not limited thereto. In the embodiment of the present application, the readable storage medium may be any tangible medium containing or storing a program, and the program may be used by or in combination with an instruction execution system, apparatus, or device.

[0129] It should be noted that, although several units or subunits of the device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided into multiple units to be embodied.

[0130] In addition, although the operations of the method of the present application are described in a specific order in the drawings, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0131] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present application and are not intended to limit it. Although the present application is described in detail with reference to the embodiments, a person skilled in the art should understand that any modification or equivalent replacement of the technical solution of the present application does not depart from the spirit and scope of the technical solution of the present application and should be included in the scope of the claims of the present application.

Claims

1. A multimodal dynamic fusion perception method for common sense memory, characterized in that: include: Obtain the multi-visual RGB image and point cloud data collected by the agent at the current moment; Obtain scene text information, environment information, and agent state information from the multi-visual RGB image at the current moment; Obtaining first perception text information from a perception common sense memory database based on the scene text information; Aggregate scene text information, environment information and agent state information of the current moment and previous consecutive frames, and first perception text information to generate prompt information; Perform feature extraction on the multi-visual RGB image at the current moment to obtain a first BEV feature, and perform feature extraction on the point cloud data at the current moment to obtain a second BEV feature; The first BEV feature and the second BEV feature are fused to obtain a dynamic collaboration feature; The prompt information and the dynamic collaboration features are processed using a large visual language model to obtain a perception result at the current moment, wherein the perception result includes: second perception text information and visual features.

2. The multimodal dynamic fusion perception method for common sense memory according to claim 1, characterized in that: The perceptual common sense memory database stores a plurality of perceptual text information; the method further comprises: calculating a similarity value between the second perceived text information and each perceived text information; When at least one similarity value is smaller than a preset threshold, the second perceived text information is stored in the perceived common sense memory database.

3. The multimodal dynamic fusion perception method for common sense memory according to claim 2 is characterized in that: Obtaining first perception text information from a perception common sense memory database based on the scene text information includes: Calculate the similarity between the scene text information and each perceived text information; Arrange similar values ​​in descending order to generate a similar value sequence; The K perceived text information corresponding to the first K similarity values ​​in the similarity value sequence are used as the first perceived text information.

4. The multimodal dynamic fusion perception method for common sense memory according to claim 1, characterized in that: The environmental information at the current moment includes: weather and lighting; the intelligent agent state information includes: speed and turning angle.

5. The multimodal dynamic fusion perception method for common sense memory according to claim 1, characterized in that: Perform feature extraction on the multi-visual RGB image at the current moment to obtain the first BEV feature, perform feature extraction on the point cloud data at the current moment to obtain the second BEV feature, including: Use the ResNet101 network to extract features from N visual RGB images and obtain N visual feature maps: The deformable cross attention network is used to process N visual feature maps to obtain the first BEV feature in, Indicates that the j-th three-dimensional point X j Projection to the i-th visual feature map The result; Q is a set of learnable BEV parameters, DeformAttn(·,·) represents the deformable cross attention network; The PointPillars model is used to extract features from point cloud data to obtain the point cloud feature map F LiDR ; Use the deformable cross attention mechanism to the point cloud feature map F LiDR Processing is performed to obtain the second BEV feature Among them, P(X j ,F LiDR ) represents the jth three-dimensional point X j Projection to point cloud feature map F LiDR result.

6. The multimodal dynamic fusion perception method for common sense memory according to claim 5, characterized in that: The first BEV feature and the second BEV feature are fused to obtain dynamic collaboration features, including: The first BEV features and second BEV features Splicing is performed along the channel to obtain the third BEV feature; The third BEV feature is processed by using an adaptive global average pooling operation to obtain a first global perception vector; The first global perception vector is reduced in dimension using a 1x1 convolutional layer to obtain the second global perception vector. The second global perception vector is processed using the sigmoid (·) function to obtain the global weight w; Based on the global weight w, the self-attention mechanism is used to focus on the first BEV feature. and second BEV features Fusion to obtain dynamic collaboration features Among them, Q is the query vector of the triple of the self-attention mechanism, K is the key vector of the triple of the self-attention mechanism, and V is the value vector of the triple of the self-attention mechanism; w Q , w k and w v All represent learnable parameters; Concat(·,·) represents the concatenation function; d is the dimension of the key vector, and softmax(·) is the normalization function.

7. A multimodal dynamic fusion perception device for common sense memory, characterized in that: include: An acquisition unit, used to acquire the multi-visual RGB image and point cloud data collected by the agent at the current moment; A first processing unit, used to obtain scene text information, environment information and agent state information from the multi-visual RGB image at the current moment; A second processing unit, configured to obtain first perception text information from a perception common sense memory database based on the scene text information; A generating unit, configured to aggregate scene text information, environment information and agent state information of a current moment and a plurality of consecutive frames before and after, and first perceived text information to generate prompt information; A feature extraction unit is used to extract features from the multi-visual RGB image at the current moment to obtain a first BEV feature, and to extract features from the point cloud data at the current moment to obtain a second BEV feature; A fusion unit, used for fusing the first BEV feature and the second BEV feature to obtain a dynamic collaboration feature; The perception unit is used to process the prompt information and the dynamic collaboration features using the visual language large model to obtain the perception result at the current moment, and the perception result includes: the second perception text information and the visual features.

8. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and when the computer instructions are executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images