A Multimodal Dynamic Fusion Perception Method and Device for Commonsense Memory

Abstract

The present application provides a multi-modal dynamic fusion perception method and device for common sense memory. The method includes: obtaining scene text information, environmental information, and agent state information from the multi-vision RGB image at the current moment; obtaining first perception text information from the perception common sense memory database based on the scene text information; aggregating the scene text information, environmental information and agent state information of the current moment and the previous consecutive multiple frames, and the first perception text information to generate a prompt message; extracting features from the multi-vision RGB image to obtain a first BEV feature, and extracting features from the point cloud data to obtain a second BEV feature; performing fusion processing on the first BEV feature and the second BEV feature to obtain a dynamic collaboration feature; using a vision-language large model to process the prompt message and the dynamic collaboration feature to obtain the perception result at the current moment. The present application can improve the comprehensive perception accuracy of the agent.

Description

Technical Field

[0001] This application relates to the field of autonomous driving technology, and particularly to a multi-modal dynamic fusion perception method and device for common sense memory. Background Art

[0002] Currently, multi-modal perception dynamic fusion methods mainly include: 1) multi-modal fusion based on dynamic network architecture or uncertainty estimation to achieve higher perception accuracy and range; 2) multi-modal fusion based on cross-attention, using the attention mechanism to learn multi-modal perception interaction, dynamically assigning weights to features of different modalities, thereby improving the fusion effect, but with a relatively high cost of real-time performance; 3) using large language models to improve the effect of visual understanding.

[0003] The above methods all ignore the idea of human perception memory, restricting the sufficiency of environmental perception. Summary of the Invention

[0004] In view of this, this application provides a multi-modal dynamic fusion perception method and device for common sense memory to solve the above technical problems.

[0005] In a first aspect, an embodiment of this application provides a multi-modal dynamic fusion perception method for common sense memory, including:

[0006] Obtain the multi-visual RGB images and point cloud data at the current moment collected by the intelligent agent;

[0007] Obtain scene text information, environmental information, and intelligent agent state information from the multi-visual RGB images at the current moment;

[0008] Obtain the first perception text information from the perception common sense memory database based on the scene text information;

[0009] Aggregate the scene text information, environmental information and intelligent agent state information of the current moment and the previous consecutive multiple frames, and the first perception text information to generate prompt information;

[0010] Extract features from the multi-visual RGB images at the current moment to obtain the first BEV feature, and extract features from the point cloud data at the current moment to obtain the second BEV feature;

[0011] Perform fusion processing on the first BEV feature and the second BEV feature to obtain a dynamic collaboration feature;

[0012] Use the vision-language large model to process the prompt information and the dynamic collaboration feature to obtain the perception result at the current moment, where the perception result includes: the second perception text information and visual features.

[0013] In a possible implementation, multiple perceptual text information is stored in the perceptual common sense memory database; the method further includes:

[0014] Calculate the similarity value between the second perceptual text information and each perceptual text information;

[0015] When at least one similarity value is less than a preset threshold, store the second perceptual text information in the perceptual common sense memory database.

[0016] In a possible implementation, obtaining the first perceptual text information from the perceptual common sense memory database based on the scenario text information includes:

[0017] Calculate the similarity value between the scenario text information and each perceptual text information;

[0018] Sort the similarity values in descending order to generate a similarity value sequence;

[0019] Take the K perceptual text information corresponding to the top K similarity values in the similarity value sequence as the first perceptual text information.

[0020] In a possible implementation, the environmental information at the current moment includes: weather and light; the intelligent agent state information includes: speed and turning angle.

[0021] In a possible implementation, extracting features from the multi-visual RGB image at the current moment to obtain the first BEV feature, and extracting features from the point cloud data at the current moment to obtain the second BEV feature, includes:

[0022] Use the ResNet101 network to extract features from N visual RGB images to obtain N visual feature maps:

[0023] Use the deformable cross-attention network to process the N visual feature maps to obtain the first BEV feature

[0024]

[0025] Among them, Indicates the result of projecting the j-th 3D point X j onto the i-th visual feature map ; Q is a set of learnable BEV parameters, and DeformAttn(·,·) represents the deformable cross-attention network;

[0026] Use the PointPillars model to extract features from the point cloud data to obtain the point cloud feature map F LiDR ;

[0027] Use the deformable cross-attention mechanism for the point cloud feature map FLiDR Process to obtain the second BEV feature

[0028]

[0029] where P(X j , F LiDR ) represents the result of projecting the j-th 3D point X j onto the point cloud feature map F LiDR .

[0030] In one possible implementation, the first BEV feature and the second BEV feature are fused to obtain a dynamic collaboration feature, including:

[0031] Concatenate the first BEV feature and the second BEV feature along the channel to obtain a third BEV feature;

[0032] Process the third BEV feature using an adaptive global average pooling operation to obtain a first global perception vector;

[0033] Use a 1x1 convolutional layer to reduce the dimension of the first global perception vector to obtain a second global perception vector,

[0034] Process the second global perception vector using the sigmoid(·) function to obtain a global weight w;

[0035] Based on the global weight w, use the self-attention mechanism to fuse the first BEV feature and the second BEV feature to obtain a dynamic collaboration feature

[0036]

[0037] where Q is the query vector of the triple of the self-attention mechanism, K is the key vector of the triple of the self-attention mechanism, V is the value vector of the triple of the self-attention mechanism; w Q , w k and w v all represent learnable parameters; Concat(·,·) represents the concatenation function; d is the dimension of the key vector, and softmax(·) is the normalization function.

[0038] In a second aspect, an embodiment of the present application provides a multi-modal dynamic fusion perception device for common sense memory, including:

[0039] An acquisition unit for acquiring the multi-visual RGB image and point cloud data collected by the agent at the current moment;

[0040] A first processing unit for obtaining scene text information, environmental information, and agent state information from a multi-vision RGB image at the current moment;

[0041] A second processing unit for obtaining first perceptual text information from a perceptual common sense memory database based on the scene text information;

[0042] A generating unit for aggregating the scene text information, environmental information and agent state information of the current moment and the previous consecutive multiple frames, and the first perceptual text information to generate a prompt message;

[0043] A feature extraction unit for extracting features from a multi-vision RGB image at the current moment to obtain a first BEV feature, and extracting features from point cloud data at the current moment to obtain a second BEV feature;

[0044] A fusion unit for performing fusion processing on the first BEV feature and the second BEV feature to obtain a dynamic cooperation feature;

[0045] A perception unit for using a vision-language large model to process the prompt message and the dynamic cooperation feature to obtain a perception result at the current moment, where the perception result includes: second perceptual text information and visual features.

[0046] In a third aspect, an embodiment of the present application provides an electronic device, including: a memory, a processor, and a computer program stored on the memory and executable on the processor, and when the processor executes the computer program, the method of the embodiment of the present application is implemented.

[0047] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is run by a processor, the method of the embodiment of the present application is implemented.

[0048] The method of the present application solves the problem of incomplete multi-modal perception fusion caused by the lack of feedback of empirical perception common sense by establishing a perceptual common sense memory database to store historical perception experiences; aggregates the obtained information such as weather, light, speed, corners, etc. and historical perception information to obtain aggregated prompt information, provides context information for the inference learning of the vision-language large model, and obtains accurate multi-modal fusion perception. Description of the Drawings

[0049] To more clearly illustrate the specific embodiments of the present application or the technical solutions in the prior art, the following will briefly introduce the drawings required for the description of the specific embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0050] Figure 1 It is a technical route block diagram of the multi-modal dynamic fusion perception method for common sense memory provided by an embodiment of the present application;

[0051] Figure 2 It is a flowchart of the multi-modal dynamic fusion perception method for common sense memory provided by an embodiment of the present application;

[0052] Figure 3 It is a schematic diagram of the process of aligning and fusing image features and point cloud features provided by an embodiment of the present application;

[0053] Figure 4 It is a functional structure diagram of the multi-modal dynamic fusion perception device for common sense memory provided by an embodiment of the present application;

[0054] Figure 5 It is a functional structure diagram of the electronic device provided by an embodiment of the present application. Specific Embodiments

[0055] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the following will clearly and completely describe the technical solutions in the embodiments of the present application with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some, but not all, of the embodiments of the present application. Usually, the components of the embodiments of the present application described and illustrated in the drawings here can be arranged and designed in various different configurations.

[0056] Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the present application to be protected, but merely represents the selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present application.

[0057] First, a brief introduction to the design concept of the embodiments of the present application will be given.

[0058] Currently, most dynamic multi-modal perception fusion technologies are oriented towards those without common sense memory and have the following problems:

[0059] First, the lack of common sense perception experience: The lack of feedback on human experience-based common sense perception leads to incomplete perception in the open environment with few-shot data, limiting the perception accuracy in the open environment. This in turn results in poor effectiveness of downstream tasks such as decision-making and planning, causing insecurity in the intelligent field.

[0060] Second, the lack of dynamic cooperation between modalities: Without being able to dynamically cooperate like humans to perceive multi-modal information, the complementary information between modalities cannot be fully utilized, leading to incorrect environmental perception by the intelligent agent in the open environment and causing incorrect decisions.

[0061] In view of the above deficiencies, and considering the way humans use common sense memory to dynamically perceive multi-modal information in the environment, this application provides a multi-modal dynamic fusion perception method for common sense memory, which is characterized by including: obtaining the multi-visual RGB images and point cloud data of the current moment collected by the intelligent agent; obtaining scene text information, environmental information, and intelligent agent state information from the multi-visual RGB images of the current moment; obtaining the first perception text information from the perception common sense memory database based on the scene text information; aggregating the scene text information, environmental information and intelligent agent state information of the current moment and the previous consecutive multi-frames, and the first perception text information to generate a prompt message; extracting features from the multi-visual RGB images of the current moment to obtain the first BEV feature, and extracting features from the point cloud data of the current moment to obtain the second BEV feature; performing fusion processing on the first BEV feature and the second BEV feature to obtain a dynamic cooperation feature; using a vision-language large model to process the prompt message and the dynamic cooperation feature to obtain the perception result of the current moment, and the perception result includes: the second perception text information and visual features; the specific technical route is as Figure 1 shown.

[0062] Compared with the prior art, the advantages of this invention are:

[0063] 1. This application adopts a fusion structure of multi-perspective, lidar, weather illumination, speed / corner multi-modal data of the intelligent agent, establishes a perception common sense memory database, expands the data domains such as weather information, speed, and corner at the current moment, and can supplement richer features, providing a solid foundation for the accuracy of decision-making.

[0064] 2. This application adopts a vision-language multi-modal fusion large model that connects visual features, text, weather illumination, and speed / corner, combines perception common sense memory and dynamic cooperation between modalities, and performs multi-perspective inference learning to obtain fused multi-modal perception features, which can improve the overall perception accuracy.

[0065] After introducing the application scenario and design concept of the embodiments of this application, the technical solutions provided by the embodiments of this application will be described below.

[0066] AsFigure 2 As shown in Figure 2 , the present application provides a multi-modal dynamic fusion perception method for common sense memory, including:

[0067] Step 101: Obtain the multi-visual RGB image and point cloud data at the current moment collected by the agent;

[0068] Among them, the multi-visual RGB image is collected by a multi-camera. The multi-camera includes a forward-view camera, a left-side view camera, a right-side view camera, and a rear-view camera, and can collect 360-degree panoramic image information around the agent.

[0069] The multi-visual RGB image includes the front-view image, left-side view image, right-side view image, rear-view image, other vehicle view image, and roadside view image of the ego vehicle.

[0070] Step 102: Obtain the scene text information, environmental information, and agent state information from the multi-visual RGB image at the current moment;

[0071] Step 103: Obtain the first perception text information from the perception common sense memory database based on the scene text information;

[0072] Exemplarily, the perception common sense memory database is based on the idea of human perception common sense memory, stores historical common sense perception features, and serves as empirical perception to help the vision-language large model learn and reason, obtain accurate environmental perception at the current moment, and improve cognitive ability.

[0073] Step 104: Aggregate the scene text information, environmental information and agent state information of the current moment and the previous continuous multi-frames, and the first perception text information to generate a prompt message;

[0074] Preferably, the environmental information and agent state information are usually selected for 5 consecutive frames.

[0075] Step 105: Extract features from the multi-visual RGB image at the current moment to obtain the first BEV feature, and extract features from the point cloud data at the current moment to obtain the second BEV feature;

[0076] Step 106: Perform fusion processing on the first BEV feature and the second BEV feature to obtain a dynamic collaboration feature;

[0077] In the real world, the key to complementary advantages between modalities is full collaboration between modalities. Therefore, enhancing dynamic collaboration between modalities is the basis for an agent to accurately perceive the environment. By enhancing dynamic collaboration between modalities, the problem of insufficient utilization of complementary information between multi-modalities is solved. Exemplarily, the visual image is used as the main modality to guide the dynamic collaboration between the image perception feature and the LiDAR perception feature.

[0078] Step 107: Process the prompt information and dynamic collaboration features using a vision-language large model to obtain the perception result at the current moment. The perception result includes: second perception text information and visual features.

[0079] Preferably, the vision-language large model uses the LLAMA2 large model.

[0080] The method of this embodiment solves the problem of incomplete multimodal perception fusion caused by the lack of feedback on empirical perception common sense by establishing a perception common sense memory database to store historical perception experiences. Aggregate the obtained weather, light, speed, corner and other information and historical perception information to obtain aggregated prompt information, providing context information for the inference learning of the vision-language large model, and obtaining accurate multimodal fusion perception.

[0081] In some embodiments, multiple perception text information is stored in the perception common sense memory database; the method further includes:

[0082] Calculate the similarity value between the second perception text information and each perception text information; wherein, the perception text information corresponds to a vector value;

[0083] When at least one similarity value is less than a preset threshold, store the second perception text information in the perception common sense memory database.

[0084] This embodiment stores the current perception text information in the common sense memory database in an incremental manner, and can effectively avoid duplication and redundancy of perception text information.

[0085] In a possible implementation, obtaining the first perception text information from the perception common sense memory database based on the scene text information includes:

[0086] Calculate the similarity value between the scene text information and each perception text information;

[0087] Sort the similarity values in descending order to generate a similarity value sequence;

[0088] Take the K perception text information corresponding to the first K similarity values in the similarity value sequence as the first perception text information.

[0089] This embodiment establishes a perception common sense memory database based on the idea of human perception common sense memory, stores historical common sense perception features, and uses them as empirical perception to help the vision-language large model learn and reason, obtaining accurate environmental perception at the current moment, improving the cognitive ability of the model, and solving the problem of inaccurate model perception information caused by the lack of historical perception experience.

[0090] In some embodiments, the environmental information at the current moment includes: weather and light; the intelligent agent state information includes: speed and corner.

[0091] In some embodiments, feature extraction is performed on the multi-vision RGB image at the current moment to obtain the first BEV feature, and feature extraction is performed on the point cloud data at the current moment to obtain the second BEV feature, including:

[0092] Using the ResNet101 network to perform feature extraction on N vision RGB images to obtain N vision feature maps:

[0093] Using the deformable cross-attention network to process the N vision feature maps to obtain the first BEV feature

[0094]

[0095] where, represents the result of projecting the j-th 3D point X j onto the i-th vision feature map ; Q is a set of learnable BEV parameters, and DeformAttn(·,·) represents the deformable cross-attention network;

[0096] Using the PointPillars model to perform feature extraction on the point cloud data to obtain the point cloud feature map F LiDR ;

[0097] Using the deformable cross-attention mechanism to process the point cloud feature map F LiDR to obtain the second BEV feature

[0098]

[0099] where, P(X j ,F LiDR ) represents the result of projecting the j-th 3D point X j onto the point cloud feature map F LiDR .

[0100] Since the camera image features and lidar features are not in the same coordinate system, the camera image perception feature F img and the lidar feature F LiDR at the agent side are not in the same coordinate system. In this embodiment, referring to the BEV encoding technology, the BEV features of these two modalities are aligned to enhance the dynamic cooperation of features between modalities, so as to better complement the differential information between modalities, as Figure 3 shown.

[0101] In some embodiments, fusion processing is performed on the first BEV feature and the second BEV feature to obtain the dynamic cooperation feature, including:

[0102] Taking the first BEV feature and the second BEV feature Perform splicing along the channel to obtain a third BEV feature;

[0103] Use an adaptive global average pooling operation to process the third BEV feature to obtain a first global perception vector;

[0104] Use a 1x1 convolutional layer to reduce the dimension of the first global perception vector to obtain a second global perception vector,

[0105] Use the sigmoid(·) function to process the second global perception vector to obtain a global weight w;

[0106] Based on the global weight w, use the self-attention mechanism for the first BEV feature and the second BEV feature Perform fusion to obtain a dynamic collaboration feature

[0107]

[0108] Among them, Q is the query vector of the triple of the self-attention mechanism, K is the key vector of the triple of the self-attention mechanism, V is the value vector of the triple of the self-attention mechanism; w Q , w k and w v All represent learnable parameters; Concat(·,·) represents the splicing function; d is the dimension of the key vector, and softmax(·) is the normalization function.

[0109] In the above process, the weight w is used to adaptively enhance the camera perception feature and the radar perception feature, and fuse the two enhanced modalities. Subsequently, the enhanced camera perception feature is used as Q to guide the dynamic collaborative fusion between the two modalities, and the final inter-modal dynamic collaborative perception is obtained, thereby solving the problem of perception errors caused by insufficient collaboration between multiple modalities.

[0110] Based on the above embodiments, an embodiment of the present application provides a multi-modal dynamic fusion perception device for common sense memory. Refer to Figure 4 As shown, the multi-modal dynamic fusion perception device 200 provided by the embodiment of the present application at least includes:

[0111] A first acquisition unit 201, configured to acquire the multi-visual RGB image and point cloud data collected by the agent at the current moment;

[0112] A first processing unit 202, configured to obtain scene text information, environmental information, and agent state information from the multi-visual RGB image at the current moment;

[0113] A second processing unit 203, configured to obtain first perceptual text information from a perceptual common sense memory database based on the scenario text information;

[0114] A generating unit 204, configured to aggregate the scenario text information, the environmental information and the agent state information of multiple consecutive frames before and at the current moment, and the first perceptual text information to generate a prompt message;

[0115] A feature extraction unit 205, configured to extract features from a multi-vision RGB image at the current moment to obtain a first BEV feature, and extract features from the point cloud data at the current moment to obtain a second BEV feature;

[0116] A fusion unit 206, configured to perform fusion processing on the first BEV feature and the second BEV feature to obtain a dynamic collaboration feature;

[0117] A perception unit 207, configured to use a vision-language large model to process the prompt message and the dynamic collaboration feature to obtain a perception result at the current moment, where the perception result includes: second perceptual text information and visual features.

[0118] It should be noted that the principle of the multi-modal dynamic fusion perception device 200 for common sense memory provided in the embodiments of the present application to solve technical problems is similar to the method provided in the embodiments of the present application. Therefore, the implementation of the multi-modal dynamic fusion perception device 200 for common sense memory provided in the embodiments of the present application can refer to the implementation of the method provided in the embodiments of the present application, and the repeated parts will not be elaborated.

[0119] Based on the above embodiments, the embodiments of the present application also provide an electronic device. Referring to Figure 5 As shown, the electronic device 300 provided in the embodiments of the present application at least includes: a processor 301, a memory 302, and a computer program stored on the memory 302 and executable on the processor 301. When the processor 301 executes the computer program, it implements the multi-modal dynamic fusion perception method for common sense memory provided in the embodiments of the present application.

[0120] The electronic device 300 provided in the embodiments of the present application may further include a bus 303 connecting different components (including the processor 301 and the memory 302). Among them, the bus 303 represents one or more of several types of bus structures, including a memory bus, a peripheral bus, a local bus, etc.

[0121] The memory 302 may include a readable medium in the form of a volatile memory, such as a random access memory (RAM) 3021 and / or a cache memory 3022, and may further include a read-only memory (ROM) 3023.

[0122] The memory 302 may also include a program tool 3025 having a set (at least one) of program modules 3024, and the program modules 3024 include but are not limited to: an operating subsystem, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment.

[0123] The electronic device 300 may also communicate with one or more external devices 304 (such as a keyboard, a remote control, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 300 (such as a mobile phone, a computer, etc.), and / or communicate with any device that enables the electronic device 300 to communicate with one or more other electronic devices 300 (such as a router, a modem, etc.). Such communication may be performed through an input / output (I / O) interface 305. And, the electronic device 300 may also communicate with one or more networks (such as a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network adapter 306. As Figure 5 shown, the network adapter 306 communicates with other modules of the electronic device 300 through the bus 303. It should be understood that although Figure 5 not shown in the figure, other hardware and / or software modules may be used in combination with the electronic device 300, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, redundant arrays of independent disks (RAID) subsystems, tape drives, and data backup storage subsystems, etc.

[0124] It should be noted that Figure 5 the electronic device 300 shown is only an example, and should not impose any limitation on the functions and usage scope of the embodiments of the present application.

[0125] The embodiments of the present application also provide a computer-readable storage medium, which stores computer instructions, and when the computer instructions are executed by a processor, the multi-modal dynamic fusion perception method for common sense memory provided by the embodiments of the present application is implemented. Specifically, the executable program may be built-in or installed in the electronic device 300, so that the electronic device 300 can implement the multi-modal dynamic fusion perception method for common sense memory provided by the embodiments of the present application by executing the built-in or installed executable program.

[0126] The multi-modal dynamic fusion perception method for common sense memory provided by the embodiments of the present application can also be implemented as a program product. The program product includes program code. When the program product runs on an electronic device 300, the program code is used to enable the electronic device 300 to execute the multi-modal dynamic fusion perception method for common sense memory provided by the embodiments of the present application.

[0127] The program product provided by the embodiments of the present application can adopt any combination of one or more readable media. Among them, the readable media can be a readable signal medium or a readable storage medium. The readable storage medium can be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. Specifically, more specific examples (non-exhaustive list) of the readable storage medium include: an electrical connection with one or more wires, a portable disk, a hard disk, RAM, ROM, an erasable programmable read-only memory (EPROM), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0128] The program product provided by the embodiments of the present application can adopt a CD-ROM and include program code, and can also run on a computing device. However, the program product provided by the embodiments of the present application is not limited to this. In the embodiments of the present application, the readable storage medium can be any tangible medium that contains or stores a program, and the program can be used by or combined with an instruction execution system, apparatus, or device.

[0129] It should be noted that although several units or subunits of the device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to the embodiments of the present application, the features and functions of the two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided and embodied by multiple units.

[0130] In addition, although the operations of the method of the present application are described in a specific order in the drawings, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, some steps can be omitted, multiple steps can be combined into one step for execution, and / or one step can be decomposed into multiple steps for execution.

[0131] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application and not to limit them. Although the present application has been described in detail with reference to the embodiments, those of ordinary skill in the art should understand that any modification or equivalent replacement of the technical solutions of the present application does not depart from the spirit and scope of the technical solutions of the present application, and they should all be covered within the scope of the claims of the present application.

Claims

1. A multimodal dynamic fusion perception method for common sense memory, characterized by: include: Obtain the multi-visual RGB image and point cloud data collected by the agent at the current moment; Obtain scene text information, environment information, and agent state information from the current multi-visual RGB image; Obtaining first perception text information from a perception common sense memory database based on the scene text information; Aggregate scene text information, environmental information and agent state information of the current moment and previous consecutive frames, and first perception text information to generate prompt information; Perform feature extraction on the multi-visual RGB image at the current moment to obtain the first BEV feature, and perform feature extraction on the point cloud data at the current moment to obtain the second BEV feature; Fusing the first BEV feature and the second BEV feature to obtain a dynamic collaboration feature; Using the visual language large model to process the prompt information and the dynamic collaboration features to obtain a perception result at the current moment, the perception result including: second perception text information and visual features; The perceptual common sense memory database stores a plurality of perceptual text information; the method further comprises: calculating a similarity value between the second perceived text information and each perceived text information; When at least one similarity value is less than a preset threshold, the second perceived text information is stored in the perceived common sense memory database; Obtaining first perception text information from a perception common sense memory database based on the scene text information includes: Calculate the similarity between the scene text information and each perceived text information; Arrange similar values in descending order to generate a similar value sequence; The K perception text information corresponding to the first K similarity values in the similarity value sequence are used as the first perception text information; Perform feature extraction on the multi-visual RGB image at the current moment to obtain the first BEV feature, and perform feature extraction on the point cloud data at the current moment to obtain the second BEV feature, including: Use the ResNet101 network to extract features from N visual RGB images and obtain N visual feature maps: 、 、 ; The deformable cross attention network is used to process N visual feature maps to obtain the first BEV feature : in, Indicates that the j-th three-dimensional point Projected to the i-th visual feature map the result; is a set of learnable BEV parameters, represents a deformable crisscross attention network; Use the PointPillars model to extract features from point cloud data and obtain a point cloud feature map ; Using deformable cross attention mechanism to focus on point cloud feature maps Processing is performed to obtain the second BEV feature : in, Indicates that the j-th three-dimensional point Projection to point cloud feature map results.

2. The multimodal dynamic fusion perception method for common sense memory according to claim 1 is characterized in that: The environmental information at the current moment includes: weather and lighting; the intelligent body state information includes: speed and turning angle.

3. The multimodal dynamic fusion perception method for common sense memory according to claim 1 is characterized in that: The first BEV feature and the second BEV feature are fused to obtain dynamic collaborative features, including: The first BEV features and second BEV features Splicing is performed along the channel to obtain the third BEV feature; The third BEV feature is processed using an adaptive global average pooling operation to obtain a first global perception vector; The first global perception vector is reduced in dimension using a 1x1 convolutional layer to obtain the second global perception vector. use The function processes the second global perception vector to obtain the global weight ; Based on global weight , using the self-attention mechanism to focus on the first BEV feature and second BEV features Fusion to obtain dynamic collaborative features : in, is the query vector of the triplet of the self-attention mechanism, is the key vector of the triplet of the self-attention mechanism, is the value vector of the triplet of the self-attention mechanism; , Both represent learnable parameters; represents the splicing function; is the dimension of the key vector, is the normalization function.

4. A multimodal dynamic fusion perception device for common sense memory, characterized by: include: An acquisition unit, used to acquire the multi-visual RGB image and point cloud data collected by the agent at the current moment; A first processing unit is used to obtain scene text information, environment information and agent state information from the multi-visual RGB image at the current moment; A second processing unit is configured to obtain first perception text information from a perception common sense memory database based on the scene text information; A generation unit, configured to aggregate scene text information, environmental information and agent state information of a current moment and a plurality of previous consecutive frames, and first perception text information to generate prompt information; A feature extraction unit is used to perform feature extraction on the multi-visual RGB image at the current moment to obtain a first BEV feature, and to perform feature extraction on the point cloud data at the current moment to obtain a second BEV feature; A fusion unit, configured to fuse the first BEV feature and the second BEV feature to obtain a dynamic collaboration feature; A perception unit, configured to process the prompt information and the dynamic collaboration features using a large visual language model to obtain a perception result at a current moment, the perception result including: second perception text information and visual features; The perceptual common sense memory database stores a plurality of perceptual text information; the device further comprises: a storage unit specifically configured to: calculating a similarity value between the second perceived text information and each perceived text information; When at least one similarity value is less than a preset threshold, the second perceived text information is stored in the perceived common sense memory database; The second processing unit is specifically configured to: Calculate the similarity between the scene text information and each perceived text information; Arrange similar values in descending order to generate a similar value sequence; The K perception text information corresponding to the first K similarity values in the similarity value sequence are used as the first perception text information; The feature extraction unit is specifically used to: Use the ResNet101 network to extract features from N visual RGB images and obtain N visual feature maps: 、 、 ; The deformable cross attention network is used to process N visual feature maps to obtain the first BEV feature : in, Indicates that the j-th three-dimensional point Projected to the i-th visual feature map the result; is a set of learnable BEV parameters, represents a deformable crisscross attention network; Use the PointPillars model to extract features from point cloud data and obtain a point cloud feature map ; Using deformable cross attention mechanism to focus on point cloud feature maps Processing is performed to obtain the second BEV feature : in, Indicates that the j-th three-dimensional point Projection to point cloud feature map results.

5. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 3 when executing the computer program.

6. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and when the computer instructions are executed by a processor, the method according to any one of claims 1 to 3 is implemented.

Images