Intelligent agent brain-like SLAM (Simultaneous Localization and Mapping) and end-side environment exploration method and system

Abstract

The invention relates to an agent brain-like SLAM (Simultaneous Localization and Mapping) and end-side environment exploration method and system, belongs to three-dimensional scene construction and artificial intelligence detection and identification, and solves the problems of low precision, high computing resource consumption, inaccurate target sound source positioning and insufficient visualization effect during end-side environment exploration in the existing map construction process. Comprising the following steps: acquiring IMU data, a camera image, DVS event sequence data and a sound source signal around an intelligent agent; characterizing the DVS event sequence data into a normalized voxel grid through frequency coding, then extracting features, fusing the features with the features of the camera image to obtain fused features, and obtaining the pose of the intelligent agent based on the fused features and IMU (Inertial Measurement Unit) data; constructing a cognitive map by using a spatial position characterization mechanism based on grid cell coding in combination with a camera image and DVS data after voxel grid normalization on the basis of the pose of the intelligent agent; and inputting a sound source signal into the trained sound listening and position distinguishing model to obtain azimuth information of a sound source relative to the intelligent agent in the cognitive map.

Description

Technical Field

[0001] The present invention relates to the field of three-dimensional scene construction and artificial intelligence detection and recognition technology, and in particular to an intelligent body brain-like SLAM and terminal-side environment exploration method and system. Background Art

[0002] SLAM, or simultaneous localization and mapping, is a technology that uses sensor data (such as cameras, lidar, etc.) to enable robots or unmanned vehicles to simultaneously determine their own positions and build scene maps in unknown environments. SLAM technology can determine the exact position of a vehicle in an unknown environment in real time and build an environmental map. Based on the environmental map built by SLAM, the system can plan paths and avoid collisions or conflicts with obstacles such as pedestrians and other vehicles. SLAM provides real-time environmental perception capabilities, allowing vehicles to adjust paths in a timely manner to ensure safe driving.

[0003] However, existing technologies mainly solve the environmental perception and situation recognition problems of vision or laser fusion inertial measurement units. Considering the complexity and diversity of the task environment, it is often difficult to obtain or adapt to accurate environmental information only through laser or visual sensors; as a passive sensor, the imaging effect of the camera depends greatly on the scene or target and environmental conditions. In addition, the number of image pixels collected by image data, especially high-resolution cameras, is large, which increases the computing consumption while providing rich perception information. Therefore, the existing SLAM and environmental exploration target detection and recognition algorithms face the problems of large computational complexity, high resource consumption, and low parallelism, and cannot adapt to the real-time requirements of mobile scenes on the end side. On the other hand, since the camera images the entire scene within the field of view, the target that needs to be focused on during the end-side environmental exploration process also contains most of the background area. Therefore, the traditional visual perception map is insufficient in highlighting and visualizing the environmental situation. Summary of the invention

[0004] In view of the above analysis, the embodiments of the present invention aim to provide an intelligent brain-like SLAM and end-side environment exploration method and system to solve the problems of low accuracy and high computing resource consumption in the existing visual SLAM map construction process, inaccurate target sound source positioning during end-side environment exploration, and insufficient visualization effect.

[0005] The purpose of the present invention is mainly achieved through the following technical solutions:

[0006] The present invention provides an intelligent agent brain-like SLAM and terminal-side environment exploration method, comprising the following steps:

[0007] Obtain IMU data, camera images, DVS event sequence data, and a sound source signal around the agent;

[0008] The DVS event sequence data is characterized into a normalized voxel grid through frequency encoding, and features are extracted, which are fused with the features of the camera image to obtain fused features, and the position and posture of the intelligent body are obtained based on the fused features and the IMU data;

[0009] Based on the position and posture of the intelligent agent, the camera image and the DVS data after the normalized voxel grid are combined, and a spatial position representation mechanism based on grid cell coding is used to construct a cognitive map;

[0010] The sound source signal is input into a trained sound positioning model to obtain the position information of the sound source relative to the intelligent body in the cognitive map; wherein the sound positioning model performs enhanced hybrid coding on the sound source signal, extracts sound features through a cascaded pulse neural network, and then uses a multi-layer perceptron classifier to obtain the position information of the sound source relative to the intelligent body.

[0011] Furthermore, the DVS event sequence data is characterized by frequency encoding into a normalized voxel grid and features are extracted, including:

[0012] Based on the spatial resolution of the DVS sensor and the number of preset time sub-windows, a voxel grid unit is constructed;

[0013] Traversing the DVS event sequence data, and classifying each DVS event data into a corresponding voxel grid unit based on the coordinates and timestamp of the DVS event data;

[0014] Calculating the event frequency in each voxel grid unit and normalizing the event frequency to obtain a three-dimensional voxel grid;

[0015] ORB features are extracted from the three-dimensional voxel grid to obtain the ORB features of the DVS event sequence data.

[0016] Furthermore, the event frequency in each voxel grid unit is calculated using the following formula:

[0017]

[0018] Among them, f x,y,k represents the event frequency in the voxel grid unit at the position (x, y) and the k-th time sub-window; N represents the total number of events in the DVS event sequence data; (x i ,y i ) represents the pixel coordinates of the ith event of the DVS event sequence data; t i The timestamp of the ith event of the DVS event sequence data; t k and t k+1 Respectively represent the start and end time of the kth time sub-window; δ() represents the position (x, y) and time window [tk ,t k+1 ] The total number of events in ].

[0019] Furthermore, obtaining the position and posture of the intelligent body includes:

[0020] Use ORB feature extraction method to obtain ORB features of camera images;

[0021] Adding and fusing the ORB features of the DVS event sequence data and the ORB features of the camera image to obtain a fused ORB feature;

[0022] Integrate the IMU data to obtain a preliminary pose of the intelligent body;

[0023] Based on the preliminary posture of the intelligent agent and the fused ORB features, Kalman filtering is used to perform state estimation optimization to obtain the posture of the intelligent agent.

[0024] Furthermore, the method of constructing a cognitive map based on the position and posture of the intelligent body, combining the camera image and the DVS data after the normalized voxel grid, and using a spatial position representation mechanism based on grid cell coding includes:

[0025] The posture of the intelligent body is input into a grid cell group representation model for path integration to obtain the position information of the intelligent body; wherein the grid cell group representation model includes a plurality of grid cells, each of which is connected to a plurality of position cells, and is used to provide a spatial position encoding through an activation mode of the grid cells, and the position cells are activated at a specific spatial position to convert the spatial position encoding into the position information of the intelligent body in space;

[0026] The visual templates of the camera image and the DVS data after the normalized voxel grid are extracted and fused respectively to obtain the visual template representing the current environment, which is combined with the position information of the intelligent agent to form a node of the cognitive map, and the nearest node is searched to establish an association to complete the construction of the cognitive map.

[0027] Furthermore, the position information of the intelligent agent is obtained by inputting the position of the intelligent agent into the grid cell group representation model for path integration, including:

[0028] Based on the position and posture of the agent, the grid cell population representation vector is updated using the following path integral formula:

[0029]

[0030] Where h represents the time step in the path integration process; I represents the identity matrix; θ i represents the direction information in the position of the intelligent body at the jth time step; △r jrepresents the position change information of the agent at the jth time step; B(θ j ) represents the grid cell at position θ j The activation intensity; g 0 represents the initial grid cell population representation vector;

[0031] Based on the grid cell population characterization vector, the place cell spike rate matrix is ​​obtained using the following formula:

[0032]

[0033] Where M represents the number of grid cells; w l represents the connection weight matrix between the lth grid cell and the place cell; represents the pulse firing rate of the lth grid cell representing the vector of the grid cell population;

[0034] The largest place cell in the place cell pulse emission rate matrix is ​​selected as the position information of the agent in space.

[0035] Furthermore, the sound location model includes an input layer, a sound feature extraction layer and a sound source localization layer; wherein,

[0036] The input layer is used to perform enhanced hybrid coding on the received sound source signal to obtain a pulse signal;

[0037] The sound feature extraction layer includes a cascaded pulse neural network, which is used to extract sound features from the pulse signal obtained from the input layer; wherein the cascaded pulse neural network includes a liquid state machine network and a time attention module connected in sequence;

[0038] The sound source localization layer includes a multi-layer perceptual classifier, which is used to identify the direction of the sound source according to the sound characteristics.

[0039] Furthermore, the intelligent body is equipped with 6 microphones arranged in a ring to form a microphone array, which is used to obtain sound source signals around the intelligent body;

[0040] The step of performing enhanced hybrid coding on the received sound source signal to obtain a sound pulse signal comprises:

[0041] Based on the sound source signal obtained by each microphone, the phase difference between the microphones is obtained using the following formula:

[0042]

[0043] Among them, f s represents the sampling frequency of the sound source signal; λ represents the wavelength of the sound wave; c represents the speed of sound; t m and t nRespectively represent the time between the sound source reaching the mth microphone and the nth microphone;

[0044] The phase difference between the microphones is time-encoded using the following formula:

[0045]

[0046] Among them, T0 represents the time scale factor; α represents the time adjustment coefficient;

[0047] Based on the time coding between the microphones, a time coding matrix of the microphone array is obtained and normalized to obtain a normalized time coding matrix;

[0048] The phase difference between the microphones is frequency encoded using the following formula:

[0049]

[0050] Among them, F0 represents the frequency scale factor; β represents the frequency adjustment coefficient;

[0051] Based on the frequency coding between the microphones, a frequency coding matrix of the microphone array is obtained and normalized to obtain a normalized frequency coding matrix;

[0052] The normalized time coding matrix and the normalized frequency coding matrix are weightedly fused to obtain a sound pulse signal.

[0053] Furthermore, the neurons of the liquid state machine network of the sound feature extraction layer are aLIF neuron models, which are used to simulate the dynamic behavior of the neural network in the biological brain to process the sound source signal that changes with time, and obtain the feature vector of the sound source signal at each time step;

[0054] In the time attention module of the sound feature extraction layer, a weight is calculated for the feature vector of the sound source signal at each time step, and the feature vector of each time step is weightedly fused to obtain a fused sound feature;

[0055] In the sound source localization layer, the fused sound features are sequentially passed through the hidden layer, ReLU activation layer, batch normalization layer and output layer of the multi-layer perceptual classifier to obtain the azimuth of the sound source signal relative to the agent; wherein the number of neurons in the output layer is a number of angle categories that evenly divide the 360-degree range.

[0056] The present invention also provides an intelligent agent brain-like SLAM and end-side environment exploration system, including:

[0057] The intelligent agent perception input system includes: integrated pickup, dynamic vision sensor, binocular RGB camera and IMU sensor, which are used to obtain the environmental information of the intelligent agent during the movement and the movement state of the intelligent agent itself;

[0058] A task machine system, used to perform brain-like SLAM and end-side environment exploration on the data obtained by the agent input system, build cognitive maps and perform target recognition;

[0059] The carrier-side image data transmission link is used to transmit the cognitive map constructed by the mission aircraft system and the identified targets to the ground command and control terminal to monitor and adjust the intelligent body.

[0060] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects:

[0061] 1. The method of constructing a cognitive map based on a brain-like SLAM map with multi-sensor fusion uses a dynamic visual sensor, a binocular RGB camera, and an IMU to perform visual odometer fusion calculations as position representation input based on a brain-inspired method, and adopts a brain-like SLAM method to construct a cognitive map, which has the characteristics of low computing latency, low computing power requirements, and adaptability to dynamically changing complex environments.

[0062] 2. Under resource-constrained conditions, the intelligent agent of the present invention constructs a map of the unknown environment while exploring targets, conducts efficient autonomous detection and multi-attribute recognition of key elements, and superimposes them on the map for presentation. Compared with traditional three-dimensional scene reconstruction methods, it highlights the rapid recognition and positioning of sensitive targets with low computing power resource consumption, helps to quickly gain attention to sensitive targets in unknown environments, and obtains the location information and attribute information of the target with the help of map construction.

[0063] 3. When exploring the end-side environment, the present invention takes into account the effect of sound positioning at different distances, simulates the working principle of the biological cochlea, uses a cascaded pulse neural network structure, combines a liquid state machine and a temporal attention mechanism, and realizes high-precision real-time positioning of the sound source. At the same time, it reduces the consumption of computing resources, enhances environmental adaptability, and intuitively maps the positioning results to the map, thereby providing the intelligent body with powerful situational awareness and decision-making support capabilities under resource-constrained conditions.

[0064] In the present invention, the above-mentioned technical solutions can also be combined with each other to achieve more preferred combination solutions. Other features and advantages of the present invention will be described in the subsequent description, and some advantages can become obvious from the description, or can be understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained through the contents particularly pointed out in the description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] The drawings are only for the purpose of illustrating particular embodiments and are not to be considered limiting of the present invention. Like reference symbols denote like components throughout the drawings.

[0066] Figure 1 It is a flow chart of an intelligent agent brain-like SLAM and terminal-side environment exploration method in an embodiment of the present invention;

[0067] Figure 2 It is a structural diagram of the intelligent agent brain-like SLAM and terminal-side environment exploration system in an embodiment of the present invention. DETAILED DESCRIPTION

[0068] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not used to limit the scope of the present invention.

[0069] A specific embodiment of the present invention discloses an intelligent agent brain-like SLAM and terminal-side environment exploration method, such as Figure 1 As shown, the steps S1 to S4 are included:

[0070] Step S1, obtaining IMU data, camera images, DVS event sequence data and a sound source signal around the intelligent body.

[0071] Specifically, in this embodiment, the sensors carried by the intelligent body abandon the high-resource-consuming laser radar and the high-computing-power and high-resource-consuming NVIDIA ORIN series chips, and use dynamic vision sensors, binocular RGB cameras, IMU sensors and integrated microphones to form sensor inputs; the computing chip uses the domestically produced Rockchip chip, with a typical power consumption of only 7W and a computing power of 6T@INT8, which can provide powerful computing capabilities while maintaining low power consumption.

[0072] The IMU data is the acceleration and angular velocity of the intelligent body during movement obtained by the IMU sensor.

[0073] The camera image is obtained by a binocular RGB camera and contains color information of the scene and disparity information for depth perception.

[0074] The dynamic vision sensor DVS is a new type of vision sensor inspired by biology. Unlike traditional cameras that capture images at a fixed frame rate, the dynamic vision sensor DVS detects the brightness change of each pixel. When the brightness change exceeds a certain threshold, it generates a pixel-level event output containing the pixel position, timestamp, and change polarity. The event data sequence can be defined as E = {(x i ,y i ,t i ,p i)|i=1,2,…,N}; where (x i ,y i ) represents the pixel coordinates of the ith event; t i represents the timestamp of the i-th event; p i Indicates the polarity of the i-th event; when the event brightness at time t is higher than the event brightness at the previous moment, the p of the event is recorded i = +1, indicating a positive event; when the brightness of the event at time t is lower than the brightness of the event at the previous time, the p of the event is recorded. i = -1, indicating a negative event; N indicates the total number of events.

[0075] The integrated microphone is an array of six microphones arranged in a ring, which acquires the phase difference of a sound source signal around the intelligent body reaching each microphone.

[0076] Step S2, the DVS event sequence data is represented by frequency encoding into a normalized voxel grid, features are extracted, and the features are fused with the features of the camera image to obtain fused features, and the position and posture of the intelligent body are obtained based on the fused features and the IMU data.

[0077] Specifically, the posture of the intelligent agent, including the position and attitude of the intelligent agent, is calculated by a multimodal fused visual inertial odometry; wherein the visual inertial odometry is a sensor fusion technology that integrates visual information and inertial measurement unit (IMU) data, receives acceleration and angular velocity from the inertial measurement unit (IMU) and camera images from a binocular RGB camera, and DVS event sequence data from a dynamic vision sensor, and performs feature tracking on the camera image and the DVS event sequence data and pre-integration on the IMU data, extracts key features that meet inter-frame correlation from the camera image and the DVS event sequence data, compares and correlates features between frames, and estimates the six-degree-of-freedom (position and attitude) motion of the object.

[0078] Furthermore, the DVS event sequence data is characterized into a normalized voxel grid through frequency encoding and then features are extracted, including steps S201 to S204:

[0079] Step S201: construct a voxel grid unit based on the spatial resolution of the DVS sensor and the preset number of time sub-windows.

[0080] Specifically, the spatial resolution of the DVS sensor is the smallest pixel unit that the sensor can distinguish, which determines the division accuracy of the voxel grid in the spatial dimension. The height and width of the voxel grid unit are the same as the spatial resolution of the DVS sensor to ensure that each voxel can accurately represent a pixel point in the DVS sensor.

[0081] The preset number of time sub-windows determines the division of the voxel grid in the time dimension. The time sub-window divides the entire observation time period T into multiple small time periods, and each time period corresponds to a time sub-window. According to the preset number of time sub-windows Vz and the observation time period T, the length △t of each time sub-window is obtained using the following formula:

[0082]

[0083] The spatial resolution and the number of temporal sub-windows are combined to construct a three-dimensional voxel grid. Each unit (voxel) of this grid corresponds to a pixel point of the DVS sensor in space and a temporal sub-window in time.

[0084] Exemplarily, the starting time of the kth time sub-window is t0+k*Δt, where t0 represents the starting time.

[0085] Step S202: traverse the DVS event sequence data, and classify each DVS event data into a corresponding voxel grid unit based on the coordinates and timestamp of the DVS event data.

[0086] Specifically, the DVS event sequence data is traversed, and if the timestamp of an event falls within the kth time sub-window, the event is mapped to the voxel grid unit corresponding to the spatial position (x, y) and the time sub-window k.

[0087] Step S203: Calculate the event frequency in each voxel grid unit, and normalize the event frequency to obtain a three-dimensional voxel grid.

[0088] Specifically, the event frequency in each voxel grid unit is obtained by calculating the number of events in each voxel unit.

[0089] Furthermore, the event frequency in each voxel grid unit is calculated using the following formula:

[0090]

[0091] Among them, f x,y,k represents the event frequency in the voxel grid unit at the position (x, y) and the k-th time sub-window; N represents the total number of events in the DVS event sequence data; (x i ,y i ) represents the pixel coordinates of the ith event of the DVS event sequence data; t i The timestamp of the ith event of the DVS event sequence data; t k and t k+1 Respectively represent the start and end time of the kth time sub-window; δ() represents the position (x, y) and time window [t k,t k+1 ];

[0092] Specifically, δ(x i =x,y i =y,t k ≤t i ≤t k+1 ) is an indicator function that checks whether the following three conditions are met simultaneously: the x coordinate x of the event i Whether it matches the x-coordinate of the voxel grid cell; the y-coordinate of the event i Whether it matches the y coordinate of the voxel grid cell; the timestamp t of the event i Whether it falls within the kth time subwindow, that is, the timestamp is in t k and t k+1 When all three conditions are met, the indicator function has a value of 1, and when one of the conditions is not met, the indicator function has a value of 0.

[0093] It should be noted that in order to make the data distribution consistent, the event frequency in the voxel grid needs to be normalized, and this embodiment uses the maximum-minimum normalization algorithm. The normalized event frequency is filled into the corresponding voxel grid unit to obtain a complete three-dimensional voxel grid.

[0094] Step S204: extracting ORB features from the three-dimensional voxel grid to obtain the ORB features of the DVS event sequence data.

[0095] Specifically, the FAST algorithm is applied on the two-dimensional slices of the three-dimensional voxel grid to quickly detect feature points. For each detected feature point, the BRIEF algorithm is used to generate a descriptor by comparing the brightness difference between the central pixel and a randomly selected pixel pair in its neighborhood, and a direction is assigned to achieve rotation invariance. A more accurate estimate is made for the position of each feature point, and a binary feature vector is constructed to obtain the ORB feature.

[0096] Furthermore, obtaining the position and posture of the intelligent body includes steps S211 to S214:

[0097] Step S211: Use the ORB feature extraction method to obtain the ORB features of the camera image.

[0098] Specifically, similar to the method of extracting ORB features from a three-dimensional voxel grid, FAST key point detection and BRIEF descriptor are applied to the camera image, while the direction of the feature points is considered to achieve invariance to rotation, so as to capture key information in the image and provide visual features for subsequent pose estimation.

[0099] Step S212: Add and fuse the ORB features of the DVS event sequence data and the ORB features of the camera image to obtain fused ORB features.

[0100] Specifically, the ORB features extracted from the DVS event sequence data and the ORB features extracted from the camera image are added and fused to combine the advantages of the two sensors, improve the robustness and richness of the features, and provide more comprehensive information for pose estimation.

[0101] Step S213: Integrate the IMU data to obtain a preliminary posture of the intelligent body.

[0102] Specifically, IMU data is the measurement data about the acceleration and angular velocity of the intelligent body. By integrating these data, the preliminary pose estimation of the intelligent body can be obtained, including position and attitude.

[0103] Step S214: Based on the preliminary posture of the intelligent body and the fused ORB features, Kalman filtering is used to perform state estimation optimization to obtain the posture of the intelligent body.

[0104] Specifically, Kalman filtering is a recursive state estimation algorithm that can combine prediction and observation data to provide more accurate pose estimation. During the optimization process, Kalman filtering uses the information of ORB feature matching to correct the integral drift of IMU, and uses IMU data to predict the movement of feature points to obtain an optimized and more accurate pose estimation of the intelligent body.

[0105] Step S3, based on the position and posture of the intelligent agent, combined with the camera image and the DVS data after the normalized voxel grid, a cognitive map is constructed using a spatial position representation mechanism based on grid cell coding, including:

[0106] The posture of the intelligent agent is input into a grid cell group representation model for path integration to obtain the position information of the intelligent agent; wherein the grid cell group representation model includes a plurality of grid cells, each grid cell is connected to a plurality of position cells, and is used to provide a spatial position encoding through an activation pattern of network cells, and the position cells are activated at a specific spatial position to convert the spatial position encoding into the position information of the intelligent agent in space.

[0107] Specifically, the grid cell group representation model is a biologically inspired computational model that mimics the characteristics of grid cells in the mammalian brain and is used to represent the spatial position of the agent in the brain-like SLAM system. In the hippocampus of mammals, grid cells are a type of neurons that have a specific response to spatial position and have a specific encoding of the spatial position of the environment where the agent is located. Each grid cell is activated at multiple specific positions in space, and these positions form a regular hexagonal grid pattern. The movement of the agent in space will cause the activation of different grid cells, forming a continuous spatial position encoding; position cells are also located in the hippocampus of mammals, and are interconnected with grid cells through neural networks. They are activated when the agent reaches a specific spatial position. Each position cell corresponds to a specific area in the environment. When the agent enters this area, the corresponding position cell will generate an action potential to identify the specific position.

[0108] It should be noted that in the grid cell group representation model, each grid cell is connected to several place cells, and its connection weight matrix can be regarded as the spatial discharge pattern matrix of the grid cell. The weight value is obtained through training with a large number of samples. The pulse firing rate of the grid cell determines the intensity of its discharge pattern. The pulse firing rate of the place cell can be obtained by accumulating the connections from the grid cell group to the place cell. The place cell with the largest pulse firing rate is the estimated position.

[0109] Furthermore, the position and posture of the intelligent body is input into the grid cell group representation model for path integration to obtain the position information of the intelligent body, including steps S31 to S33:

[0110] Step S31: Based on the position and posture of the intelligent agent, the grid cell group representation vector is updated using the following path integral formula:

[0111]

[0112] Where h represents the time step in the path integration process; I represents the identity matrix; θ j represents the direction information in the position of the intelligent body at the jth time step; △r j represents the position change information of the agent at the jth time step; B(θ j ) represents the grid cell at position θ j The activation intensity; g 0 Represents the initial grid cell population representation vector.

[0113] Specifically, the path integral calculates the position change of the agent over a period of time by accumulating the movement information of the agent. The initial position of the agent is set to x = (0,0), and the activation state of all grid cells at the initial position is set to an initial value, indicating that the agent has not moved yet; the corresponding grid cell group representation vector can be obtained as g 0 ; Then receive the self-motion information from the odometer, that is, the displacement △r at the jth time step j and direction information θ j , the cumulative product of matrix multiplication is used to update the grid cell group representation vector. In this way, the grid cell group representation vector continuously accumulates the displacement and direction information of each time step, thereby realizing continuous tracking and updating of the agent's position.

[0114] Step S32: Based on the grid cell population characterization vector, the place cell pulse firing rate matrix is ​​obtained using the following formula:

[0115]

[0116] Where M represents the number of grid cells; w l represents the connection weight matrix between the lth grid cell and the place cell; Represents the pulse firing rate of the lth grid cell in the representation vector of the grid cell group.

[0117] Specifically, the pulse firing rate matrix P of the place cell is calculated by obtaining the sum of the influence of each grid cell on the place cell. h , represents the activation state of all place cells at a given time step.

[0118] Step S33: Select the largest place cell in the place cell pulse emission rate matrix as the position information of the agent in space.

[0119] Specifically, place cells have a specific activation response to a particular spatial location. When the agent is at or close to a specific location, the corresponding place cells will be strongly activated. The place cells with the highest degree of activation usually represent the agent's current location or the most likely location.

[0120] Furthermore, the visual word bags of the camera image and the DVS data after the normalized voxel grid are extracted and fused respectively to obtain a visual template representing the current environment, which is combined with the position information of the agent to form a node of the cognitive map, and the nearest node is searched to establish an association to complete the construction of the cognitive map.

[0121] Specifically, errors will be generated in the process of path integration. The errors will accumulate over time, causing the positioning accuracy of the intelligent agent to gradually decrease over time. Therefore, visual information is introduced to match similar scenes, and visual templates are used to represent scenes and identify the similarity of scenes.

[0122] The visual information comes from the camera image and the DVS data after the normalized voxel grid. First, for the camera image, after converting it into a grayscale image, each row of the grayscale image is flattened into a one-dimensional vector and the vectors are connected, and a normalization operation is performed to obtain a camera image visual template; for the DVS data after the normalized voxel grid, similarly, each row is flattened into a one-dimensional vector and the vectors are connected, and a normalization operation is performed to obtain a DVS visual template; the camera image visual template and the DVS visual template are added and fused to obtain a visual template representing the current environment.

[0123] Furthermore, the cognitive map is composed of the position information represented by the place cells and grid cells, and the visual scene information represented by the visual template, forming an associative memory in the hippocampus of the brain, forming the u-th node N in the cognitive map. u = {VT u ,P u}; Among them, VT u For node N u The associated visual template, P u YesN u When the agent moves, a new map node N is created. v , and search for the nearest node N in the cognitive map u , a connection is established between the two to obtain the posture difference. At this time, through the map nodes and node connections, we construct a topological expression of the relative position and relative relationship of the landmark points in the environment, and complete the construction of the cognitive map.

[0124] Step S4, inputting the sound source signal into the trained sound positioning model to obtain the position information of the sound source relative to the intelligent body in the cognitive map; wherein the sound positioning model performs enhanced hybrid coding on the sound source signal, extracts the sound features through a cascaded pulse neural network, and then uses a multi-layer perceptron classifier to obtain the position information of the sound source relative to the intelligent body.

[0125] Specifically, in this embodiment, combining visual and auditory information can provide a more comprehensive environmental perception, identify targets around the intelligent agent based on sound source signals, conduct terminal-side environmental exploration, identify the location of the target, and mark it on the cognitive map constructed above, which can help the intelligent agent understand its surrounding environment more comprehensively.

[0126] Furthermore, the sound location model includes an input layer, a sound feature extraction layer and a sound source localization layer.

[0127] Specifically, sound positioning is a problem of estimating the position of a sound source relative to a microphone array based on recorded multi-channel microphone signals. In this embodiment, the intelligent agent is equipped with 6 microphones arranged in a ring to form a microphone array for acquiring sound source signals around the intelligent agent.

[0128] The input layer is used to perform enhanced hybrid coding on the received sound source signal to obtain a pulse signal.

[0129] Specifically, in the input layer, the received sound source signal is subjected to enhanced hybrid coding, which combines time coding and frequency coding compared to single-mode coding, and can simultaneously capture the time delay characteristics and frequency changes of the sound signal, providing more comprehensive sound source information to improve the performance and accuracy of the subsequent sound feature extraction layer.

[0130] Furthermore, the step of performing enhanced hybrid coding on the received sound source signal to obtain a sound pulse signal includes:

[0131] Based on the sound source signal obtained by each microphone, the phase difference between the microphones is obtained using the following formula:

[0132]

[0133] Among them, f s represents the sampling frequency of the sound source signal; λ represents the wavelength of the sound wave; c represents the speed of sound; t m and t n They represent the time between the sound source arriving at the mth microphone and the nth microphone respectively.

[0134] Specifically, during the sound propagation process, there will be different time delays when the sound reaches microphones at different positions. The phase difference between the sound source and the microphones can be obtained through the time difference between the sound source and the microphones. The phase difference reflects the spatial distribution characteristics of the sound source.

[0135] Furthermore, the phase difference is time-coded and frequency-coded respectively to obtain a time coding matrix and a frequency coding matrix of the microphone array, and they are normalized and then weightedly fused to obtain a sound pulse signal.

[0136] Specifically, time coding is used for the phase difference to capture the time delay characteristics of the signal, and frequency coding is used to enhance the sensitivity to frequency changes. The combination of the two is used to improve the accuracy of sound source positioning.

[0137] Furthermore, the phase difference between the microphones is time-encoded using the following formula:

[0138]

[0139] Among them, T0 represents the time scale factor; α represents the time adjustment coefficient.

[0140] Specifically, the time coding maps the phase difference to the pulse time, and uses a nonlinear function to map the phase difference to a time scale to enhance the ability to distinguish complex sound sources. T0 represents the reference time when the pulse occurs without any phase difference, which is used to control the time range. For example, it can be set to 1 millisecond; α is used to control the influence of the phase difference on the pulse time, which determines the sensitivity of the time coding to the phase difference. A larger value will make the time coding more sensitive to the change of the phase difference, while a smaller value will make the time coding less sensitive to the change of the phase difference. For example, it can be set to 0.1.

[0141] Furthermore, based on the time coding between the microphones, a time coding matrix of the microphone array is obtained and normalized to obtain a normalized time coding matrix.

[0142] Specifically, the generated time coding matrix represents the pulse time between each pair of microphones; the time coding matrix is ​​normalized to reduce the influence of different dimensions and numerical ranges on subsequent feature extraction. Preferably, this embodiment adopts the maximum-minimum normalization method.

[0143] Furthermore, the phase difference between the microphones is frequency encoded using the following formula:

[0144]

[0145] Among them, F0 represents the frequency scale factor; β represents the frequency adjustment coefficient.

[0146] Specifically, the frequency encoding maps the phase difference to the pulse frequency, and uses the inverse tangent function for mapping to compress the influence of frequency drift. F0 is used to control the frequency range. A smaller value will result in a lower frequency of the frequency encoding, and a larger value will result in a higher frequency of the frequency encoding. For example, it can be set to 100Hz; β is used to control the influence of the phase difference on the frequency encoding and enhance the response to frequency changes. A larger value will make the frequency encoding more sensitive to changes in the phase difference, and a smaller value will make the frequency encoding less sensitive to changes in the phase difference. For example, it can be set to 0.05.

[0147] Furthermore, based on the frequency coding between the microphones, a frequency coding matrix of the microphone array is obtained and normalized to obtain a normalized frequency coding matrix.

[0148] Specifically, the generated frequency coding matrix represents the pulse frequency between each pair of microphones; the frequency coding matrix is ​​normalized to reduce the influence of different dimensions and numerical ranges on subsequent feature extraction. Preferably, this embodiment adopts a maximum-minimum normalization method.

[0149] Furthermore, the normalized time coding matrix and the normalized frequency coding matrix are weightedly fused to obtain a sound pulse signal.

[0150] Specifically, by performing weighted summation on the time coding matrix and the frequency coding matrix, the importance of the time information and the frequency information are considered at the same time, and a fused coding matrix is ​​obtained.

[0151] It should be noted that at each time step t mic , from the fused coding matrix, use the following formula to extract the data related to the 6 microphone channels:

[0152] Input SNN [0,t mic ,d]=E d,(d+tmic)mod 6

[0153] Among them, t mic Indicates the tth mic time steps; d represents the d-th microphone channel, d∈[0,5]; E represents the fused encoding matrix, d,(d+t mic ) mod 6 means that the time step t and channel number d can loop through the 6 microphone channels when extracting the fused data.

[0154] The final input SNN [1,T mic ,6] as a sound pulse signal with dimensions (1,T mic ,6), where 1 represents a single sample; T mic Represents the total number of time steps; 6 represents the number of microphones. The sound pulse signal provides a comprehensive representation of time and frequency information for the subsequent feature extraction layer, enabling it to more effectively extract the sound features of the sound source.

[0155] Furthermore, the sound feature extraction layer includes a cascaded pulse neural network, which is used to extract sound features from the pulse signal obtained from the input layer; wherein the cascaded pulse neural network includes a liquid state machine network and a temporal attention module connected in sequence.

[0156] Specifically, the liquid state machine network is a type of pulse neural network, which simulates the dynamic behavior of the neural network in the biological brain and is used to process signals that change over time; the liquid state machine is composed of a large number of randomly connected neurons, the activities of these neurons propagate over time, and form complex spatiotemporal patterns through the connections between neurons.

[0157] The temporal attention module cascaded after the liquid state machine is used to enhance the processing and analysis capabilities of timing information and complex dynamic signals, and to perform weighted time pooling on the output of the liquid network layer to enhance the processing capabilities of time-sensitive signals.

[0158] Furthermore, the neurons of the liquid state machine network of the sound feature extraction layer are aLIF neurons, which are used to simulate the dynamic behavior of the neural network in the biological brain to process the sound source signal that changes with time and obtain the feature vector of the sound source signal at each time step.

[0159] Specifically, in this embodiment, the liquid state machine network integrates a total of 3,600 neurons, with excitatory neurons accounting for 70% and inhibitory neurons accounting for 30%, simulating the processing mechanism of the biological brain. Among them, excitatory neurons are used to transmit signals and enhance network activity, and inhibitory neurons are used to inhibit excessive activation and prevent the network from entering an unstable state.

[0160] Furthermore, in the temporal attention module of the sound feature extraction layer, a weight is calculated for the feature vector of the sound source signal at each time step, and the feature vector of each time step is weightedly fused to obtain a fused sound feature.

[0161] Specifically, the liquid state machine generates a state vector at each time step Represents the activation state of the network at this time step; the unnormalized weights of each time step are calculated using the following linear transformation:

[0162]

[0163] Among them, W a represents the weight matrix; b a Represents the bias vector.

[0164] The unnormalized weights of each time step are calculated by the softmax function Normalize to get the normalized weight for each time step The formula is:

[0165]

[0166] Based on the normalized weight The states of all time steps are weighted summed to obtain the fused sound features.

[0167] It should be noted that through the temporal attention module, the model can dynamically assign different weights to the states of different time steps, which can effectively capture the key information in the time series. Compared with traditional pooling, the temporal attention mechanism makes the network more sensitive to important time segments and can improve the classification accuracy of the model. The attention module using linear transformation will not increase a lot of computational overhead, but can greatly improve the performance of the model on long time series.

[0168] Furthermore, the sound source localization layer includes a multi-layer perceptual classifier for identifying the direction of the sound source according to the sound characteristics.

[0169] Specifically, the multi-layer perceptron classifier is a feed-forward artificial neural network that performs classification tasks by learning complex nonlinear features of input data.

[0170] Furthermore, in the sound source localization layer, the fused sound features are sequentially passed through the hidden layer, ReLU activation layer, batch normalization layer and output layer of the multi-layer perceptual classifier to obtain the azimuth of the sound source signal relative to the agent; wherein the number of neurons in the output layer is a number of angle categories that evenly divide the 360-degree range.

[0171] Specifically, the hidden layer is composed of multiple neurons, each neuron receives features from the input layer through a weighted connection, and processes these inputs through an activation function to generate outputs, which are used to extract higher-level feature representations from the fused sound features. The number of neurons in the hidden layer is a hyperparameter, and is exemplarily set to 5000 in this embodiment.

[0172] The ReLU activation layer includes a ReLU activation function, which is used to perform an element-by-element nonlinear transformation on the output of the hidden layer.

[0173] The batch normalization layer is used to normalize the output of the layer to reduce internal covariate shift, including calculating the mean and variance.

[0174] The output layer uses a softmax activation function, and each neuron in the output layer corresponds to an angle category, which is used to generate the final classification result, that is, the probability of the azimuth of the sound source signal relative to the agent. In this embodiment, the output layer outputs 18 categories at an interval of 20 degrees, and each category is spaced 20 degrees apart.

[0175] It should be noted that the following method is used to train the sound location recognition model:

[0176] A sound source data set with known sound source azimuths as labels is used as a training data set; a training set in the training data set is loaded, the sound location identification model is trained using a cross entropy loss function, and the parameters are updated by back propagation using a stochastic gradient descent method to obtain a trained sound location identification model.

[0177] Another embodiment of the present invention discloses an intelligent brain-like SLAM and terminal-side environment exploration system, such as Figure 2 As shown, including:

[0178] The intelligent agent perception input system includes: integrated pickup, dynamic vision sensor, binocular RGB camera and IMU sensor, which are used to obtain the environmental information of the intelligent agent during the movement and the movement state of the intelligent agent itself;

[0179] A task machine system, used to perform brain-like SLAM and end-side environment exploration on the data obtained by the agent input system, build cognitive maps and perform target recognition;

[0180] The carrier-side image data transmission link is used to transmit the cognitive map constructed by the mission aircraft system and the identified targets to the ground command and control terminal to monitor and adjust the intelligent body.

[0181] The above system embodiment and method embodiment are based on the same principle, and their related parts can be used for reference, and can achieve the same technical effect. The specific implementation process refers to the above embodiment, which will not be repeated here.

[0182] The intelligent body brain-like SLAM and end-side environment exploration system also includes a ground command terminal, which includes: a command-end image and data transmission link, control buttons and function buttons, which are used to receive data transmitted by the carrier-end image and data transmission link, and display them in the form of a map on the visual interface. At the same time, through the control buttons, the operator can remotely control the actions of the intelligent body.

[0183] It should be noted that in order to ensure performance under resource-constrained conditions and at the same time ensure that the power system and structure of the intelligent agent can support the weight and operating requirements of additional sensors, the intelligent agent's robotic car has been carefully adjusted and optimized, including adjusting the appropriate positions and angles of multiple sensors based on test feedback to obtain the best field of view and data quality; and ensuring that the robotic car's control system is flexible and reliable to adapt to different test environments and task requirements.

[0184] In summary, the intelligent agent brain-like SLAM and terminal-side environment exploration method and system of the embodiment of the present invention have the following beneficial effects:

[0185] 1. The method of constructing a cognitive map based on a brain-like SLAM map with multi-sensor fusion uses a dynamic visual sensor, a binocular RGB camera, and an IMU to perform visual odometer fusion calculations as position representation input based on a brain-inspired method, and adopts a brain-like SLAM method to construct a cognitive map, which has the characteristics of low computing latency, low computing power requirements, and adaptability to dynamically changing complex environments.

[0186] 2. Under resource-constrained conditions, the intelligent agent of the present invention constructs a map of the unknown environment while exploring targets, conducts efficient autonomous detection and multi-attribute recognition of key elements, and superimposes them on the map for presentation. Compared with traditional three-dimensional scene reconstruction methods, it highlights the rapid recognition and positioning of sensitive targets with low computing power resource consumption, helps to quickly gain attention to sensitive targets in unknown environments, and obtains the location information and attribute information of the target with the help of map construction.

[0187] 3. When exploring the end-side environment, the present invention takes into account the effect of sound positioning at different distances, simulates the working principle of the biological cochlea, uses a cascaded pulse neural network structure, combines a liquid state machine and a temporal attention mechanism, and realizes high-precision real-time positioning of the sound source. At the same time, it reduces the consumption of computing resources, enhances environmental adaptability, and intuitively maps the positioning results to the map, thereby providing the intelligent body with powerful situational awareness and decision-making support capabilities under resource-constrained conditions.

[0188] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.

Claims

1. An intelligent agent brain-like SLAM and end-side environment exploration method, characterized in that: The steps include: Obtain IMU data, camera images, DVS event sequence data, and a sound source signal around the agent; The DVS event sequence data is characterized into a normalized voxel grid through frequency encoding, and features are extracted, which are fused with the features of the camera image to obtain fused features, and the position and posture of the intelligent body are obtained based on the fused features and the IMU data; Based on the position and posture of the intelligent agent, the camera image and the DVS data after the normalized voxel grid are combined, and a spatial position representation mechanism based on grid cell coding is used to construct a cognitive map; The sound source signal is input into a trained sound positioning model to obtain the position information of the sound source relative to the intelligent body in the cognitive map; wherein the sound positioning model performs enhanced hybrid coding on the sound source signal, extracts sound features through a cascaded pulse neural network, and then uses a multi-layer perceptron classifier to obtain the position information of the sound source relative to the intelligent body.

2. The method according to claim 1, characterized in that: The DVS event sequence data is characterized by frequency encoding into a normalized voxel grid and features are extracted, including: Based on the spatial resolution of the DVS sensor and the number of preset time sub-windows, a voxel grid unit is constructed; Traversing the DVS event sequence data, and classifying each DVS event data into a corresponding voxel grid unit based on the coordinates and timestamp of the DVS event data; Calculating the event frequency in each voxel grid unit and normalizing the event frequency to obtain a three-dimensional voxel grid; ORB features are extracted from the three-dimensional voxel grid to obtain the ORB features of the DVS event sequence data.

3. The method according to claim 2, characterized in that: The event frequency in each voxel grid cell is calculated using the following formula: Among them, f x,y,k represents the event frequency in the voxel grid unit at the position (x, y) and the k-th time sub-window; N represents the total number of events in the DVS event sequence data; (x i ,y i ) represents the pixel coordinates of the ith event of the DVS event sequence data; t i The timestamp of the ith event of the DVS event sequence data; t k and t k+1 Respectively represent the start and end time of the kth time sub-window; δ() represents the position (x, y) and time window [t k ,t k+1 ] The total number of events in ].

4. The method according to claim 3, characterized in that: The obtaining of the position and posture of the intelligent body includes: Use ORB feature extraction method to obtain ORB features of camera images; Adding and fusing the ORB features of the DVS event sequence data and the ORB features of the camera image to obtain a fused ORB feature; Integrate the IMU data to obtain a preliminary pose of the intelligent body; Based on the preliminary posture of the intelligent agent and the fused ORB features, Kalman filtering is used to perform state estimation optimization to obtain the posture of the intelligent agent.

5. The method according to any one of claims 1 to 4, characterized in that: The method of constructing a cognitive map based on the position and posture of the intelligent body, combining the camera image and the DVS data after the normalized voxel grid, and using a spatial position representation mechanism based on grid cell coding includes: The posture of the intelligent body is input into a grid cell group representation model for path integration to obtain the position information of the intelligent body; wherein the grid cell group representation model includes a plurality of grid cells, each of which is connected to a plurality of position cells, and is used to provide a spatial position encoding through an activation mode of the grid cells, and the position cells are activated at a specific spatial position to convert the spatial position encoding into the position information of the intelligent body in space; The visual templates of the camera image and the DVS data after the normalized voxel grid are extracted and fused respectively to obtain the visual template representing the current environment, which is combined with the position information of the intelligent agent to form a node of the cognitive map, and the nearest node is searched to establish an association to complete the construction of the cognitive map.

6. The method according to claim 5, characterized in that: The step of inputting the position and posture of the intelligent body into the grid cell group representation model for path integration to obtain the position information of the intelligent body includes: Based on the position and posture of the agent, the grid cell population representation vector is updated using the following path integral formula: Where h represents the time step in the path integration process; I represents the identity matrix; θ i represents the direction information in the position of the intelligent body at the jth time step; △r j represents the position change information of the agent at the jth time step; B(θ j ) represents the grid cell at position θ j The activation intensity; g 0 represents the initial grid cell population representation vector; Based on the grid cell population characterization vector, the place cell spike rate matrix is ​​obtained using the following formula: Where M represents the number of grid cells; w l represents the connection weight matrix between the lth grid cell and the place cell; represents the pulse firing rate of the lth grid cell representing the vector of the grid cell population; The largest place cell in the place cell pulse emission rate matrix is ​​selected as the position information of the agent in space.

7. The method according to claim 1, characterized in that: The sound location model includes an input layer, a sound feature extraction layer and a sound source localization layer; wherein, The input layer is used to perform enhanced hybrid coding on the received sound source signal to obtain a pulse signal; The sound feature extraction layer includes a cascaded pulse neural network, which is used to extract sound features from the pulse signal obtained from the input layer; wherein the cascaded pulse neural network includes a liquid state machine network and a time attention module connected in sequence; The sound source localization layer includes a multi-layer perceptual classifier, which is used to identify the direction of the sound source according to the sound characteristics.

8. The method according to claim 7, characterized in that: The intelligent body is equipped with 6 microphones arranged in a ring to form a microphone array, which is used to obtain sound source signals around the intelligent body; The step of performing enhanced hybrid coding on the received sound source signal to obtain a sound pulse signal comprises: Based on the sound source signal obtained by each microphone, the phase difference between the microphones is obtained using the following formula: Among them, f s represents the sampling frequency of the sound source signal; λ represents the wavelength of the sound wave; c represents the speed of sound; t m and t n Respectively represent the time between the sound source reaching the mth microphone and the nth microphone; The phase difference between the microphones is time-encoded using the following formula: Among them, T0 represents the time scale factor; α represents the time adjustment coefficient; Based on the time coding between the microphones, a time coding matrix of the microphone array is obtained and normalized to obtain a normalized time coding matrix; The phase difference between the microphones is frequency encoded using the following formula: Among them, F0 represents the frequency scale factor; β represents the frequency adjustment coefficient; Based on the frequency coding between the microphones, a frequency coding matrix of the microphone array is obtained and normalized to obtain a normalized frequency coding matrix; The normalized time coding matrix and the normalized frequency coding matrix are weightedly fused to obtain a sound pulse signal.

9. The method according to any one of claims 7-8, characterized in that: The neurons of the liquid state machine network of the sound feature extraction layer are aLIF neuron models, which are used to simulate the dynamic behavior of the neural network in the biological brain to process the sound source signal that changes with time, and obtain the feature vector of the sound source signal at each time step; In the time attention module of the sound feature extraction layer, a weight is calculated for the feature vector of the sound source signal at each time step, and the feature vector of each time step is weightedly fused to obtain a fused sound feature; In the sound source localization layer, the fused sound features are sequentially passed through the hidden layer, ReLU activation layer, batch normalization layer and output layer of the multi-layer perceptual classifier to obtain the azimuth of the sound source signal relative to the agent; wherein the number of neurons in the output layer is a number of angle categories that evenly divide the 360-degree range.

10. An intelligent agent brain-like SLAM and end-side environment exploration system, characterized in that: include: The intelligent agent perception input system includes: integrated pickup, dynamic vision sensor, binocular RGB camera and IMU sensor, which are used to obtain the environmental information of the intelligent agent during the movement and the movement state of the intelligent agent itself; A task machine system, used to perform brain-like SLAM and end-side environment exploration on the data obtained by the agent input system, build cognitive maps and perform target recognition; The carrier-side image data transmission link is used to transmit the cognitive map constructed by the mission aircraft system and the identified targets to the ground command and control terminal to monitor and adjust the intelligent body.

Images