Spatial data sensing method and device, equipment and storage medium

Abstract

The embodiment of the invention provides a spatial data sensing method and device, equipment and a storage medium, and relates to the technical field of data processing. According to the method, image feature data and spectrum feature data are subjected to data alignment to obtain image alignment features and spectrum alignment features; carrying out self-attention calculation to obtain a frequency spectrum attention feature and an image attention feature, and carrying out cross attention fusion to obtain a spatial global feature; and finally, performing feature weighting on the image content features and the spatial global features to obtain spatial fusion features, and decoding the spatial fusion features to obtain a spatial perception image. And carrying out data alignment operation to avoid scale deviation among data of different dimensions. And performing cross attention calculation on the aligned data to integrate information among different modal data, enhancing richness and distinction degree of feature expression, and introducing image content features corresponding to global information through a weighting mode, so that the information content contained in a spatial perception image and the accuracy of the spatial perception image are integrally improved, and the accuracy of spatial perception is further improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to spatial data perception methods, devices, equipment and storage media. Background Art

[0002] In the process of spatial perception, multimodal fusion of echo data and image data, combined with the precise distance and depth information provided by echo data and the rich texture and color information provided by image data, can significantly improve the accuracy of environmental understanding and decision-making. This technology is widely used in fields such as autonomous driving, robot navigation, augmented reality and intelligent monitoring.

[0003] However, the related technology mainly adopts a simple splicing method for the data fusion of the above two modalities. Although the data processing efficiency is high, this method fails to fully utilize the unique advantages of each modality, ignores the complementary information between different modalities, and lacks the use of global information, resulting in unsatisfactory fusion effect and low accuracy of the generated spatial perception data. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to propose a spatial data perception method, device, equipment and storage medium to improve the accuracy of spatial perception data obtained by multimodal data fusion.

[0005] To achieve the above-mentioned purpose, a first aspect of an embodiment of the present application proposes a spatial data perception method, comprising:

[0006] Acquire image feature data, spectrum feature data and image content features of the target space, wherein the image content features and the image feature data are obtained based on the visual image data;

[0007] Performing data alignment on the image feature data and the spectrum feature data to obtain corresponding image alignment features and spectrum alignment features;

[0008] Performing self-attention calculation on the spectrum alignment feature to obtain a spectrum attention feature, performing self-attention calculation on the image alignment data to obtain an image attention feature, and performing cross-attention fusion on the spectrum attention feature and the image attention feature to obtain a spatial global feature;

[0009] The image content features and the spatial global features are feature weighted to obtain spatial fusion features, and the spatial fusion features are decoded to obtain a spatial perception image of the target space.

[0010] In some embodiments, the cross-attention fusion of the spectrum attention feature and the image attention feature to obtain the spatial global feature includes:

[0011] Performing a first cross-fusion operation according to the image attention feature and the spectrum attention feature to obtain an image spectrum fusion feature;

[0012] Performing a second cross fusion operation according to the spectrum attention feature and the image attention feature to obtain a spectrum image fusion feature;

[0013] The spatial global feature is obtained by performing feature combination according to the image spectrum fusion feature and the spectrum image fusion feature.

[0014] In some embodiments, performing a first cross-fusion operation according to the image attention feature and the spectrum attention feature to obtain an image spectrum fusion feature includes:

[0015] Obtaining a first cross-query weight, a first cross-key weight, and a first cross-value weight corresponding to the first cross-fusion operation;

[0016] Obtain a first query intermediate parameter corresponding to the product of the first cross-query weight and the image attention feature, a first key intermediate parameter corresponding to the product of the first cross-key weight and the spectrum attention feature, and a first value intermediate parameter corresponding to the product of the first cross-value weight and the spectrum attention feature;

[0017] Attention calculation is performed according to the first query intermediate parameter, the first key intermediate parameter, and the first value intermediate parameter to obtain the image spectrum fusion feature.

[0018] In some embodiments, performing a second cross-fusion operation according to the spectrum attention feature and the image attention feature to obtain a spectrum image fusion feature includes:

[0019] Obtaining a second cross-query weight, a second cross-key weight, and a second cross-value weight corresponding to the second cross-fusion operation;

[0020] Obtain a second query intermediate parameter corresponding to the product of the second cross-query weight and the spectral attention feature, a second key intermediate parameter corresponding to the product of the second cross-key weight and the image attention feature, and a second value intermediate parameter corresponding to the product of the second cross-value weight and the image attention feature;

[0021] Attention calculation is performed according to the second query intermediate parameter, the second key intermediate parameter and the second value intermediate parameter to obtain the spectrum image fusion feature.

[0022] In some embodiments, the step of combining the image spectrum fusion feature and the spectrum image fusion feature to obtain the spatial global feature includes:

[0023] Performing a feature extraction operation of a multi-layer perceptron on the image spectrum fusion features to obtain local features of the image;

[0024] Performing a feature extraction operation of a multi-layer perceptron on the fusion features of the spectrum image to obtain local features of the spectrum;

[0025] The spatial global feature is obtained by combining the local frequency spectrum feature and the local image feature.

[0026] In some embodiments, performing self-attention calculation on the spectrum alignment feature to obtain the spectrum attention feature includes:

[0027] Obtaining a first query weight, a first key weight, and a first value weight corresponding to the spectrum alignment feature;

[0028] Performing attention calculation according to the first query weight, the first key weight, and the first value weight to obtain a spectrum intermediate feature;

[0029] The spectrum alignment feature and the spectrum intermediate feature are superimposed to obtain the spectrum attention feature.

[0030] In some embodiments, aligning the image feature data with the spectrum feature data to obtain corresponding image alignment features and spectrum alignment features includes:

[0031] Obtaining a size parameter of block embedding, and performing block embedding on the image feature data and the spectrum feature data based on the size parameter to obtain corresponding image embedding features and spectrum embedding features;

[0032] Obtaining an image weight, and performing linear mapping based on the product of the image weight and the image embedding feature to obtain the image alignment feature;

[0033] A spectrum weight is obtained, and based on the product of the spectrum weight and the spectrum embedding feature, a linear mapping is performed to obtain the spectrum alignment feature.

[0034] In some embodiments, the acquiring of image feature data, spectrum feature data, and image content features of the target space includes:

[0035] Acquiring echo data and visual image data of the target space;

[0036] Performing short-time Fourier transform on the echo data to obtain a spectrum diagram, and inputting the spectrum diagram into an echo encoder for encoding to obtain the spectrum feature data;

[0037] Sending the visual image data to a visual encoder for encoding to obtain the image feature data;

[0038] Content features are extracted from the visual image data to obtain the image content features.

[0039] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application proposes a spatial data sensing device, including:

[0040] Data acquisition module: used to acquire image feature data, spectrum feature data and image content features of the target space, wherein the image content features and the image feature data are obtained based on the visual image data;

[0041] Data alignment module: used for aligning the image feature data and the spectrum feature data to obtain corresponding image alignment features and spectrum alignment features;

[0042] A self-cross attention module: used to perform self-attention calculation on the spectrum alignment feature to obtain a spectrum attention feature, perform self-attention calculation on the image alignment data to obtain an image attention feature, and perform cross-attention fusion on the spectrum attention feature and the image attention feature to obtain a spatial global feature;

[0043] Perception data generation module: used for performing feature weighting on the image content features and the spatial global features to obtain spatial fusion features, and decoding the spatial fusion features to obtain a spatial perception image of the target space.

[0044] To achieve the above objectives, a third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method described in the first aspect is implemented.

[0045] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a storage medium, which is a storage medium. The storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0046] The spatial data perception method, device, equipment and storage medium proposed in the embodiment of the present application obtain the image feature data, spectrum feature data and image content feature of the target space, align the image feature data and the spectrum feature data, and obtain the corresponding image alignment feature and spectrum alignment feature; perform self-attention calculation on the spectrum alignment feature to obtain the spectrum attention feature, perform self-attention calculation on the image alignment data to obtain the image attention feature, and perform cross-attention fusion on the spectrum attention feature and the image attention feature to obtain the spatial global feature; perform feature weighting on the image content feature and the spatial global feature to obtain the spatial fusion feature, and decode the spatial fusion feature to obtain the spatial perception image of the target space. In the embodiment of the present application, the multi-dimensional information in the target space is first obtained, and then the image feature data and the spectrum feature data are aligned to avoid the scale deviation between the data of different dimensions and improve the accuracy of the subsequent attention calculation process. Then the aligned data is cross-attention calculated to integrate the information between the data of different modalities to enhance the richness and discrimination of the feature expression. Finally, the image content feature corresponding to the global information is introduced in a weighted manner to improve the amount of information contained in the spatial perception image and its accuracy as a whole, thereby improving the accuracy of spatial perception. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 It is a flow chart of the spatial data perception method provided in an embodiment of the present application.

[0048] Figure 2 It is a flowchart of obtaining image feature data, spectrum feature data and image content features of a target space provided by an embodiment of the present application.

[0049] Figure 3 It is a flowchart of aligning image feature data and spectrum feature data to obtain corresponding image alignment features and spectrum alignment features provided by an embodiment of the present application.

[0050] Figure 4 It is a schematic diagram of the cross-fusion process provided in an embodiment of the present application.

[0051] Figure 5 This is a flowchart of performing self-attention calculation on spectrum alignment features to obtain spectrum attention features provided in an embodiment of the present application.

[0052] Figure 6 It is a flowchart of cross-attention fusion of spectral attention features and image attention features to obtain spatial global features provided by an embodiment of the present application.

[0053] Figure 7It is a flowchart provided in an embodiment of the present application for performing a first cross-fusion operation based on spectral attention features and image attention features to obtain image spectral fusion features.

[0054] Figure 8 It is a flowchart of performing a second cross-fusion operation based on spectral attention features and image attention features to obtain spectral image fusion features, provided in an embodiment of the present application.

[0055] Fig. 9 It is a flow chart of obtaining spatial global features by combining features according to image spectrum fusion features and spectrum image fusion features provided in an embodiment of the present application.

[0056] Fig.10 It is an overall flow chart of the spatial data perception method provided in the embodiment of the present application.

[0057] Fig.11 This is a structural block diagram of a spatial data sensing device provided in yet another embodiment of the present application.

[0058] Fig.12 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0060] It should be noted that although the functional modules are divided in the device schematic and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart.

[0061] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0062] In the process of spatial perception, multimodal fusion of echo data and image data, combined with the precise distance and depth information provided by echo data and the rich texture and color information provided by image data, can significantly improve the accuracy of environmental understanding and decision-making. This technology is widely used in fields such as autonomous driving, robot navigation, augmented reality and intelligent monitoring.

[0063] However, the data fusion of the above two modalities in the related art mainly adopts a simple splicing method. Although the data processing efficiency is high, this method can utilize some common features of the two modalities, but fails to fully utilize the unique advantages of each modality. Each modality contains rich and complementary features that the other cannot provide. For example, echo signals have unique advantages in detecting scene depth and spatial distribution of objects, while images can provide rich texture and color information. The simple splicing method in the related art does not effectively utilize this differentiated information, resulting in insufficient feature expression after fusion.

[0064] In addition, the fusion process in related technologies ignores the alignment information between different modalities. The alignment problem is very important for jointly learning the global information of different modalities. For example, there may be temporal and spatial offsets between echo information and image information, and if this offset is not processed, it may cause errors when information is fused. The splicing operation lacks a mechanism to process these offsets or differences, and directly stacks information from different modalities together, which cannot effectively align or integrate multimodal information, limiting the ability to express features.

[0065] In addition, the fusion process of related technologies lacks the use of global information. In complex computer vision tasks, understanding and utilizing global contextual information is key, such as in depth estimation or object detection. Since splicing is simply superimposing features in the channel dimension, the model cannot understand the global connection between modalities in this way. The lack of such global information fusion often leads to unsatisfactory fusion results and low accuracy of the generated spatial perception data.

[0066] Based on this, the embodiments of the present application provide a spatial data perception method, device, equipment and storage medium. First, multi-dimensional information in the target space is obtained, and then data alignment operations are performed on the image feature data and the spectrum feature data to avoid scale deviations between data of different dimensions and improve the accuracy of subsequent attention calculation processes. Then, the aligned data is cross-attention calculated to integrate information between different modal data to enhance the richness and discrimination of feature expression. Finally, the image content features corresponding to the global information are introduced in a weighted manner to improve the amount of information contained in the spatial perception image and its accuracy as a whole, thereby improving the accuracy of spatial perception.

[0067] The embodiments of the present application provide a spatial data perception method, apparatus, device and storage medium, which are specifically illustrated by the following embodiments. First, the spatial data perception method in the embodiments of the present application is described.

[0068] The spatial data perception method provided in the embodiment of the present application relates to the field of data processing technology. The spatial data perception method provided in the embodiment of the present application can be applied to a terminal, can be applied to a server side, or can be a computer program running in a terminal or a server side. For example, a computer program can be a native program or software module in an operating system; it can be a local (Native) application (Application, APP), that is, a program that needs to be installed in the operating system to run, such as a client that supports spatial data perception, that is, a program that can be run only by downloading it to a browser environment; it can also be a small program that can be embedded in any APP. In short, the above-mentioned computer program can be any form of application, module or plug-in. Among them, the terminal communicates with the server via a network. The spatial data perception method can be executed by a terminal or a server, or by a terminal and a server in collaboration.

[0069] In some embodiments, the terminal may be a smart phone, a tablet computer, a laptop computer, a desktop computer or a smart watch, etc. In addition, the terminal may also be an intelligent vehicle-mounted device. The intelligent vehicle-mounted device applies the spatial data perception method of this embodiment to provide related services and enhance the driving experience. The server may be an independent server, or it may be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), and big data and artificial intelligence platforms; it may also be a service node in a blockchain system, and each service node in the blockchain system forms a peer-to-peer (Peer To Peer, P2P) network, and the P2P protocol is an application layer protocol running on the Transmission Control Protocol (Transmission Control Protocol, TCP) protocol. The terminal and the server may be connected via Bluetooth, Universal Serial Bus (Universal Serial Bus, USB) or network communication connection methods, which are not limited in this embodiment.

[0070] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0071] The following describes the spatial data perception method in an embodiment of the present application.

[0072] Figure 1 is an optional flow chart of the spatial data perception method provided in the embodiment of the present application. Figure 1 The method may include but is not limited to steps 110 to 140. It can also be understood that this embodiment Figure 1 The order of step 110 to step 140 is not specifically limited, and the order of steps can be adjusted or some steps can be reduced or added according to actual needs.

[0073] Step 110: Obtain image feature data, spectrum feature data, and image content features of the target space.

[0074] In one embodiment, the target space is the area that needs to be spatially perceived, and the image content features and image feature data are obtained based on the visual image data. Figure 2 , Figure 2 This is a flowchart of obtaining image feature data, spectrum feature data, and image content features of a target space provided by an embodiment of the present application, which specifically includes the following steps:

[0075] Step 210: Acquire echo data and visual image data of the target space.

[0076] In one embodiment, an RGB image is acquired as visual image data in the target space through an image acquisition device such as a camera, and a scanning signal is sent out using a sound source located at the same position and direction as the camera. The scanning signal can be an audio signal of 20Hz-20KHz with a period of 3ms. Then, an echo receiving device consistent with the position of the sound source is used to acquire echo data corresponding to the scanning signal. After obtaining the echo data and visual image data, multi-dimensional information in the target space needs to be acquired based on this. It can be understood that the embodiment of the present application sets the direction of the sound source and the echo receiving device to be consistent, and prevents overlap between the echo data and the scanning signal through a wide frequency spectrum and a short duration response.

[0077] Step 220: Perform short-time Fourier transform on the echo data to obtain a spectrum diagram, and input the spectrum diagram into an echo encoder for encoding to obtain spectrum feature data.

[0078] In one embodiment, short-time Fourier transform is a method of converting echo data in the time domain into a two-dimensional representation spectrogram of time and frequency, which is achieved by sliding a window over the echo data and calculating the Fourier transform at each position. Therefore, the spectrogram is represented as:

[0079]

[0080] Among them, STFT represents short-time Fourier transform, S represents spectrogram, x(t) represents echo data, w(t) represents window function, m represents time index, indicating the center position of window function, T represents time step, indicating the step size of each moving window, ω represents frequency variable, and j represents imaginary unit. Specifically, the shape of the spectrogram is represented by S∈R T×F×2 , where 2 indicates binaural sound is used and F is the frequency dimension.

[0081] In one embodiment, after obtaining the spectrogram, the spectrogram is input into an echo encoder for encoding to obtain spectrum feature data. The spectrogram is a two-dimensional matrix, and the echo encoder is composed of three convolutional layers. The convolution kernel size of the three convolutional layers is 3×3, the stride size is 1×1, and the number of channels of each layer is 128, 256 and 512 respectively. Specifically, in the first convolutional layer, 128 3×3 filters are used to perform convolution operations on the input spectrogram to generate 128 feature maps, each of which is a local feature representation of the spectrogram. Next, the second convolutional layer uses 256 3×3 filters to perform convolution operations on the output of the first convolution layer to generate 256 feature maps, which further extract higher-level features. Finally, 512 3×3 filters are used to perform convolution operations on the output of the second convolution layer to generate 512 feature maps to obtain the highest-level feature representation, which contains the richest information. After three layers of convolution of the echo encoder, the spectrum graph is feature extracted to obtain 512 feature graphs as spectrum feature data.

[0082] Step 230: Send the visual image data to the visual encoder for encoding to obtain image feature data.

[0083] In one embodiment, the visual encoder consists of five convolutional layers. For each layer, the convolution kernel size is 3x3, the stride is 1×1, and the number of channels in the five layers is set to 64, 128, 256, 512, and 512, respectively. After the visual image data is sent into the visual encoder, the first convolution layer uses 64 3×3 filters to convolve the input image to generate 64 feature maps. The second convolution layer uses 128 3×3 filters to convolve the output of the first convolution layer to generate 128 feature maps. The third convolution layer uses 256 3×3 filters to convolve the output of the second convolution layer to generate 256 feature maps. The fourth convolution layer uses 512 3×3 filters to convolve the output of the third convolution layer to generate 512 feature maps. The last convolution layer again uses 512 3×3 filters to convolve the output of the fourth convolution layer to generate 512 feature maps. After five layers of convolution, 512 feature maps are obtained as image feature data. The image feature data not only contains the spatial information of the visual image data, but also captures the feature representations at different levels and scales.

[0084] Step 240: extracting content features from the visual image data to obtain image content features.

[0085] In one embodiment, a convolutional neural network can be used as a content extraction network. The content extraction network is used to extract basic visual information including edges, textures, colors, etc. from visual image data to form image content features. These image content features, as global information, can reflect the shape, structure, and surface features of objects in the target space.

[0086] Step 120: aligning the image feature data and the spectrum feature data to obtain corresponding image alignment features and spectrum alignment features.

[0087] In one embodiment, considering that data alignment is very important for jointly learning global information of different modalities, the embodiment of the present application needs to use an alignment operation to avoid the possible temporal and spatial offsets between echo information and image information. Figure 3 , Figure 3 The present invention provides a flowchart of aligning image feature data and spectrum feature data to obtain corresponding image alignment features and spectrum alignment features, which specifically includes the following steps:

[0088] Step 310: Obtain a size parameter of block embedding, and perform block embedding on the image feature data and the spectrum feature data based on the size parameter to obtain corresponding image embedding features and spectrum embedding features.

[0089] In one embodiment, patch embedding is a method of converting high-dimensional data into low-dimensional data by dividing the data into multiple small blocks and performing a pooling operation on each small block, thereby reducing the dimension of the data. In this embodiment, the size parameter of the block embedding is denoted as p, which is a hyperparameter that is trained or set according to the actual calculation situation. For image feature data and spectral feature data, both are composed of 512 feature maps, so they are divided into p×p sizes to obtain a series of image blocks, each of which is a small two-dimensional matrix, and the corresponding image embedding features or spectral feature data are obtained according to all image blocks.

[0090] Step 320: Obtain image weights, and perform linear mapping based on the product of the image weights and the image embedding features to obtain image alignment features.

[0091] Step 330: Obtain spectrum weights, and perform linear mapping based on the product of the spectrum weights and the spectrum embedding features to obtain spectrum alignment features.

[0092] In one embodiment, the image embedding features and the spectrum embedding features are mapped to a common latent space by linear mapping. In this latent space, the spectrum alignment features and the image alignment features have the same dimension and similar distribution, and the expression form is more unified, which is convenient for subsequent processing and fusion under a unified framework. Among them, the linear mapping converts the input data into output data through a linear transformation matrix. At this time, the image alignment feature is expressed as:

[0093] I′=IW i

[0094] Among them, I′ represents the image alignment feature, I represents the image embedding feature, and W i Represents the image weight.

[0095] The spectrum alignment feature is expressed as:

[0096] E′=EW e

[0097] Among them, E′ represents the spectrum alignment feature, E represents the spectrum embedding feature, and W e represents the spectrum weight.

[0098] Step 130: Perform self-attention calculation on the spectrum alignment features to obtain spectrum attention features, perform self-attention calculation on the image alignment data to obtain image attention features, and perform cross-attention fusion on the spectrum attention features and the image attention features to obtain spatial global features.

[0099] In one embodiment, referring to Figure 4 , Figure 4 Schematic diagram of the cross-fusion process provided by the embodiment of the present application. First, the spectrum alignment feature E′ and the image alignment data I′ are respectively regularized (Layer Normalization, LN), and then self-attention calculation is performed after regularization. Based on the residual connection method, the spectrum attention features Z are obtained respectively. e and image attention feature Z i Then, we focus on the spectral feature Z e and image attention feature Z i Regularization processing is performed separately, and then cross-fusion is performed after regularization processing. Based on the residual connection method, the image spectrum fusion feature Z is obtained respectively. i→e and spectrum image fusion feature Z e→i Then the image spectrum fusion feature Z i→e and spectrum image fusion feature Z e→i After regularization, the images are sent to a multi-column convolution (MLP) for feature extraction to obtain the cross-feature Z of the spectrum image.e→i ′ and image spectrum cross feature Z i→e ′, and finally, based on the residual connection method, the spectrum image cross feature Z is superimposed e→i ′ and spectrum image fusion feature Z e→i Get the spectrum extraction feature F e , superimposed image spectrum cross feature Z i→e ′ and image spectrum fusion feature Z i→e Get the image extraction feature F i , extract feature F from the spectrum e And image extraction features F i Combine features to obtain the spatial global feature F joint .

[0100] Below Figure 4 The process is introduced in detail.

[0101] In one embodiment, referring to Figure 5 , Figure 5 This is a flow chart of performing self-attention calculation on the spectrum alignment feature to obtain the spectrum attention feature provided by an embodiment of the present application, which specifically includes the following steps:

[0102] Step 510: Obtain a first query weight, a first key weight, and a first value weight corresponding to the spectrum alignment feature.

[0103] Step 520: Perform attention calculation according to the first query weight, the first key weight and the first value weight to obtain the intermediate features of the spectrum.

[0104] Step 530: Superimpose the spectrum alignment feature and the spectrum intermediate feature to obtain the spectrum attention feature.

[0105] In one embodiment, the first query weight is expressed as: The first key weight is expressed as: The first value weight is expressed as: Therefore, the intermediate features of the spectrum are expressed as:

[0106]

[0107]

[0108]

[0109]

[0110] Among them, Z e ′ represents the intermediate features of the spectrum, and the softmax function is used to ensure the normalization of the weights. k1It can be understood that the first query weight, the first key weight, the first value weight and the dimension d k1 These are all trainable parameters.

[0111] Next, the spectrum alignment feature and the spectrum intermediate feature are superimposed through residual connection to obtain the spectrum attention feature, which contains the long-distance dependency in the echo data. The spectrum attention feature Z e It is expressed as:

[0112] Z=Z′+E′

[0113] ee

[0114] In one embodiment, referring to Figure 4 And the above spectral attention feature Z e The calculation process of performing self-attention calculation on the image alignment feature to obtain the image attention feature specifically includes: obtaining the second query weight, the second key weight and the second value weight corresponding to the image alignment feature, performing attention calculation according to the second query weight, the second key weight and the second value weight to obtain the image intermediate feature, and superimposing the image alignment feature and the image intermediate feature to obtain the image attention feature.

[0115] Among them, the second query weight is expressed as: The second key weight is expressed as: The second value weight is expressed as: Therefore, the intermediate features of the image are expressed as:

[0116]

[0117]

[0118]

[0119]

[0120] Among them, Z i ′ represents the intermediate features of the image, and the softmax function is used to ensure the normalization of the weights. k2 It can be understood that the second query weight, the second key weight, the second value weight and the dimension d k2 These are all trainable parameters.

[0121] Next, the image alignment feature and the image intermediate feature are superimposed through residual connection to obtain the image attention feature, which contains the long-distance dependency in the visual image data. The image attention feature Z i It is expressed as:

[0122] Z i =Z i ′+I′

[0123] In one embodiment, referring to Figure 4 , get the spectral attention feature Z e and image attention feature Z i After that, the spectral attention feature Z e and image attention feature Z i Regularization is performed separately, and cross-fusion is performed after regularization. Figure 6 , Figure 6 This is a flow chart of performing cross-attention fusion of spectral attention features and image attention features to obtain spatial global features provided by an embodiment of the present application, which specifically includes the following steps:

[0124] Step 610: Perform a first cross-fusion operation based on the image attention feature and the spectrum attention feature to obtain an image spectrum fusion feature.

[0125] In one embodiment, referring to Figure 7 , Figure 7 The present invention provides a flowchart of performing a first cross-fusion operation based on spectrum attention features and image attention features to obtain image spectrum fusion features, which specifically includes the following steps:

[0126] Step 710: Obtain a first cross-query weight, a first cross-key weight, and a first cross-value weight corresponding to a first cross-fusion operation.

[0127] In one embodiment, the cross-fusion operation of cross-fusion of the spectral attention features on the image attention features is called the first cross-fusion operation. In this case, the first cross-query weight is represented by W i→e Q , the first cross-bond weight is denoted as W i→e K , the first cross-value weight is denoted as W i→e V .

[0128] Step 720: Obtain the first query intermediate parameter corresponding to the product of the first cross-query weight and the image attention feature, the first key intermediate parameter corresponding to the product of the first cross-key weight and the spectral attention feature, and the first value intermediate parameter corresponding to the product of the first cross-value weight and the spectral attention feature.

[0129] In one embodiment, the first query intermediate parameter Q i→e It is expressed as:

[0130] Q i→e =Z iW i→e Q

[0131] The first key intermediate parameter K i→e It is expressed as:

[0132] K i→e =Z e W i→e K

[0133] The first value intermediate parameter V i→e It is expressed as:

[0134] V i→e =Z e W i→e V

[0135] Step 730: Perform attention calculation according to the first query intermediate parameter, the first key intermediate parameter and the first value intermediate parameter to obtain image spectrum fusion features.

[0136] In one embodiment, attention calculation is first performed according to the first query intermediate parameter, the first key intermediate parameter and the first value intermediate parameter to obtain the image spectrum fusion intermediate feature Z i→e1 , expressed as:

[0137]

[0138] Among them, d k3 It can be understood that the first cross query weight, the first cross key weight, the first cross value weight and the dimension d k3 These are all trainable parameters.

[0139] Next, the image attention feature Z is connected through residual connection i Fusion with image spectrum Z i→e1 The intermediate features are superimposed to obtain the image spectrum fusion feature Z i→e It is expressed as:

[0140] Z i→e =Z i→e1 +Z i

[0141] Step 620: Perform a second cross-fusion operation based on the spectrum attention feature and the image attention feature to obtain a spectrum image fusion feature.

[0142] In one embodiment, referring to Figure 8 , Figure 8The present invention provides a flowchart of performing a second cross-fusion operation according to the spectrum attention feature and the image attention feature to obtain the spectrum image fusion feature, which specifically includes the following steps:

[0143] Step 810: Obtain a second cross-query weight, a second cross-key weight, and a second cross-value weight corresponding to the second cross-fusion operation.

[0144] In one embodiment, the cross-fusion operation of cross-fusion of the image attention features on the spectrum attention features is called the second cross-fusion operation. In this case, the second cross-query weight is expressed as W e→i Q , the second cross-bond weight is denoted as W e→i K , the second cross-value weight is denoted as W e→i V .

[0145] Step 820: Obtain the second query intermediate parameter corresponding to the product of the second cross-query weight and the spectral attention feature, the second key intermediate parameter corresponding to the product of the second cross-key weight and the image attention feature, and the second value intermediate parameter corresponding to the product of the second cross-value weight and the image attention feature.

[0146] In one embodiment, the second query intermediate parameter Q e→i It is expressed as:

[0147] Q e→i =Z e W e→i Q

[0148] Second key intermediate parameter K e→i It is expressed as:

[0149] K e→i =Z i W e→i K

[0150] The second value intermediate parameter V e→i It is expressed as:

[0151] V e→i =Z i W e→i V

[0152] Step 830: Perform attention calculation according to the second query intermediate parameter, the second key intermediate parameter and the second value intermediate parameter to obtain the spectrum image fusion feature.

[0153] In one embodiment, attention calculation is first performed according to the second query intermediate parameter, the second key intermediate parameter and the second value intermediate parameter to obtain the spectrum image fusion intermediate feature Z e→i1 , expressed as:

[0154]

[0155] Among them, d k4 It can be understood that the second cross query weight, the second cross key weight, the second cross value weight and the dimension d k4 These are all trainable parameters.

[0156] Next, the spectral attention feature Z is connected through residual connection e Fusion of intermediate feature Z with spectrum image e→i1 Superposition is performed to obtain the spectrum image fusion feature Z e→i It is expressed as:

[0157] Z e→i =Z e→i1 +Z e

[0158] Through the above cross fusion operation, the image spectrum fusion feature Z i→e and spectrum image fusion feature Z e→i It contains the correlation information of another modality data.

[0159] Step 630: Combining features according to the image spectrum fusion features and the spectrum image fusion features to obtain spatial global features.

[0160] In one embodiment, referring to Figure 4 , get the image spectrum fusion feature Z i→e and spectrum image fusion feature Z e→i Then, the image spectrum is fused with feature Z i→e and spectrum image fusion feature Z e→i After regularization, the image spectrum fusion features and the spectrum image fusion features obtained by regularization are combined to obtain the spatial global features. Fig. 9 , Fig. 9 The flowchart of obtaining the spatial global feature by combining features according to the image spectrum fusion feature and the spectrum image fusion feature provided by the embodiment of the present application specifically includes the following steps:

[0161] Step 910: Perform a feature extraction operation of a multi-layer perceptron on the image spectrum fusion features to obtain local features of the image.

[0162] Step 920: Perform a feature extraction operation of a multi-layer perceptron on the fusion features of the spectrum image to obtain local features of the spectrum.

[0163] Step 930: Combining the local features of the spectrum and the local features of the image to obtain the global features of the space.

[0164] In one embodiment, the image spectrum fusion feature and the spectrum image fusion feature are respectively sent to a multi-layer perceptron for feature extraction to obtain a spectrum image cross feature Z e→i ′ and image spectrum cross feature Z i→e ′, and then based on the residual connection method, superimpose the spectrum image cross feature Z e→i ′ and spectrum image fusion feature Z e→i Get the spectrum extraction feature F e , superimposed image spectrum cross feature Z i→e ′ and image spectrum fusion feature Z i→e Get the image extraction feature F i Next, the spectrum is extracted by feature concatenation, feature superposition or patch merging. e And image extraction features F i Combine features to obtain the spatial global feature F joint In one embodiment, patch merging is similar to the previous block embedding and linear mapping process, which divides the corresponding feature data into small blocks and flattens the blocks, and then uses a linear layer to map the flattened vectors into a high-dimensional space.

[0165] Considering that visual image features contain visual information corresponding to the target space, and echo data can provide additional spatial information (especially the distance information between the object and the sensor), the importance of spatial information in the perception process is particularly prominent in complex scenes or under poor lighting conditions. Therefore, the embodiments of the present application improve the understanding and expression capabilities of multimodal data in the spatial perception process by capturing and integrating image spectrum fusion features and spectrum image fusion features at different spatial scales. At the same time, by utilizing combination methods such as patch merging, local spectral features and local image features can be effectively synthesized into larger-scale spatial global features, thereby capturing multi-scale feature relationships, while maintaining a lower feature dimension, and enhancing the sensitivity and understanding of multi-scale information in the feature extraction process.

[0166] In one embodiment, the above-mentioned data alignment operation process and the self-cross-attention calculation process (including the self-attention calculation and the cross-fusion process) can be repeated multiple times as a whole process, and each repeated execution process can capture more complex inter-modal relationships and feature interactions at a higher level, thereby improving the overall feature expression capability. At the same time, through multiple self-cross-attentions, a feature representation containing multiple scales can be formed, and these scales can represent abstract features at different levels, such as low-level local features and high-level global features. And through reasonable patch merging, these feature information of different scales can be fused together to obtain spatial global features, so that the multi-modal data corresponding to the visual image data and the echo data can be more comprehensively understood and processed at multiple levels in the spatial data perception process, thereby improving the accuracy of the spatial fusion results.

[0167] Step 140: perform feature weighting on the image content features and the spatial global features to obtain spatial fusion features, and decode the spatial fusion features to obtain a spatial perception image of the target space.

[0168] In one embodiment, after obtaining the spatial global features, it is necessary to further introduce image content features corresponding to the global information to improve the accuracy of the perception results. The spatial fusion features are expressed as:

[0169] F fusion =αF img +βF joint

[0170] Among them, F img represents the image content feature, α represents the first weight corresponding to the image content feature, and F joint represents the spatial global feature, β represents the second weight corresponding to the spatial global feature, and F fusion It is understandable that the first weight and the second weight can be set according to actual needs.

[0171] In one embodiment, the spatial fusion features obtained through the above process can enhance the expressiveness of the fusion features, ensure that the spatial information of the target space can be fully understood and utilized, and thus accurately infer the spatial structure corresponding to the target space. Next, the decoder can be used to decode the spatial fusion features, perform various downstream tasks, and obtain a spatial perception image of the target space. Among them, the downstream tasks can be scene space perception tasks such as depth estimation and object detection.

[0172] In one embodiment, referring to Fig.10 , Fig.10 It is an overall flow chart of the spatial data perception method provided in the embodiment of the present application.

[0173] First, the echo data and visual image data of the target space are acquired, and the echo data is short-time Fourier transformed to obtain a spectrogram. The spectrogram is input into the echo encoder for encoding to obtain spectrum feature data. Meanwhile, the visual image data is input into the visual encoder for encoding to obtain image feature data. Finally, content features are extracted from the visual image data to obtain image content features.

[0174] Next, multiple data alignment operations and self-cross-attention calculation processes are performed (the figure shows 3 times as an example, and this example does not limit the number of executions). It should be clear that the data alignment operation process covers block embedding and linear mapping. Taking the first data alignment operation process as an example, after segmenting the different modal data through patch embedding, linear mapping is used to map them to the same high-dimensional feature space. Specifically: the size parameter of the block embedding is obtained, and the image feature data and the spectrum feature data are block-embedded according to the size parameter to obtain the corresponding image embedding features and spectrum embedding features, and then the image alignment features are obtained by multiplying the image weights and the image embedding features, and the spectrum alignment features are obtained by multiplying the spectrum weights and the spectrum embedding features, and then the two are linearly mapped to achieve effective alignment of the multimodal information of the visual image data and the echo data, avoiding the modal mismatch problem that may be caused by ordinary splicing methods.

[0175] Then, in the self-cross-attention process, the spectrum alignment features are self-attentioned to obtain the spectrum attention features, the image alignment data are self-attentioned to obtain the image attention features, and the spectrum attention features and the image attention features are cross-attention fused to obtain the spatial global features, making full use of the complementary information between different modal data to improve the fusion effect. Finally, the image content features and the spatial global features are feature weighted to obtain the spatial fusion features, and the spatial fusion features are decoded to obtain the spatial perception image of the target space, which significantly improves the accuracy and robustness of depth perception in spatial perception. In general, the self-attention and cross-attention mechanisms are used to align and associate in the feature space to promote the fusion of image and echo representations. This mechanism synergistically establishes the correlation between different features and produces a cohesive interaction between the visual and echo domains.

[0176] The technical solution provided by the embodiment of the present application is to obtain the image feature data, spectrum feature data and image content features of the target space, align the image feature data and the spectrum feature data, and obtain the corresponding image alignment features and spectrum alignment features; perform self-attention calculation on the spectrum alignment features to obtain the spectrum attention features, perform self-attention calculation on the image alignment data to obtain the image attention features, and perform cross-attention fusion on the spectrum attention features and the image attention features to obtain the spatial global features; perform feature weighting on the image content features and the spatial global features to obtain the spatial fusion features, and decode the spatial fusion features to obtain the spatial perception image of the target space. In the embodiment of the present application, the multi-dimensional information in the target space is first obtained, and then the image feature data and the spectrum feature data are aligned to avoid the scale deviation between the data of different dimensions and improve the accuracy of the subsequent attention calculation process. Then the aligned data is cross-attention calculated to integrate the information between the data of different modalities to enhance the richness and discrimination of the feature expression. Finally, the image content features corresponding to the global information are introduced in a weighted manner to improve the amount of information contained in the spatial perception image and its accuracy as a whole, thereby improving the accuracy of spatial perception.

[0177] The present application also provides a spatial data sensing device, which can implement the above-mentioned spatial data sensing method. Fig.11 , the device comprises:

[0178] Data acquisition module 1110: used to acquire image feature data, spectrum feature data and image content features of the target space, where both the image content features and image feature data are obtained based on the visual image data.

[0179] The data alignment module 1120 is used to align the image feature data and the spectrum feature data to obtain corresponding image alignment features and spectrum alignment features.

[0180] Self-cross-attention module 1130: used to perform self-attention calculation on spectrum alignment features to obtain spectrum attention features, perform self-attention calculation on image alignment data to obtain image attention features, and perform cross-attention fusion on spectrum attention features and image attention features to obtain spatial global features.

[0181] The perception data generation module 1140 is used to perform feature weighting on the image content features and the spatial global features to obtain spatial fusion features, and decode the spatial fusion features to obtain a spatial perception image of the target space.

[0182] In some embodiments, the self-cross-attention module 1130 is used to:

[0183] Performing a first cross-fusion operation according to the image attention feature and the spectrum attention feature to obtain an image spectrum fusion feature;

[0184] Performing a second cross fusion operation according to the spectrum attention feature and the image attention feature to obtain a spectrum image fusion feature;

[0185] The spatial global feature is obtained by performing feature combination according to the image spectrum fusion feature and the spectrum image fusion feature.

[0186] In some embodiments, the self-cross-attention module 1130 is further configured to:

[0187] Obtaining a first cross-query weight, a first cross-key weight, and a first cross-value weight corresponding to the first cross-fusion operation;

[0188] Obtain a first query intermediate parameter corresponding to the product of the first cross-query weight and the image attention feature, a first key intermediate parameter corresponding to the product of the first cross-key weight and the spectrum attention feature, and a first value intermediate parameter corresponding to the product of the first cross-value weight and the spectrum attention feature;

[0189] Attention calculation is performed according to the first query intermediate parameter, the first key intermediate parameter, and the first value intermediate parameter to obtain the image spectrum fusion feature.

[0190] In some embodiments, the self-cross-attention module 1130 is further configured to:

[0191] Obtaining a second cross-query weight, a second cross-key weight, and a second cross-value weight corresponding to the second cross-fusion operation;

[0192] Obtain a second query intermediate parameter corresponding to the product of the second cross-query weight and the spectral attention feature, a second key intermediate parameter corresponding to the product of the second cross-key weight and the image attention feature, and a second value intermediate parameter corresponding to the product of the second cross-value weight and the image attention feature;

[0193] Attention calculation is performed according to the second query intermediate parameter, the second key intermediate parameter and the second value intermediate parameter to obtain the spectrum image fusion feature.

[0194] In some embodiments, the self-cross-attention module 1130 is further configured to:

[0195] Performing a feature extraction operation of a multi-layer perceptron on the image spectrum fusion features to obtain local features of the image;

[0196] Performing a feature extraction operation of a multi-layer perceptron on the fusion features of the spectrum image to obtain local features of the spectrum;

[0197] The spatial global feature is obtained by combining the local frequency spectrum feature and the local image feature.

[0198] In some embodiments, the self-cross-attention module 1130 is further configured to:

[0199] Obtaining a first query weight, a first key weight, and a first value weight corresponding to the spectrum alignment feature;

[0200] Performing attention calculation according to the first query weight, the first key weight, and the first value weight to obtain a spectrum intermediate feature;

[0201] The spectrum alignment feature and the spectrum intermediate feature are superimposed to obtain the spectrum attention feature.

[0202] In some embodiments, the data alignment module 1120 is used to:

[0203] Obtaining a size parameter of block embedding, and performing block embedding on the image feature data and the spectrum feature data based on the size parameter to obtain corresponding image embedding features and spectrum embedding features;

[0204] Obtaining an image weight, and obtaining the image alignment feature based on the product of the image weight and the image embedding feature;

[0205] A spectrum weight is obtained, and the spectrum alignment feature is obtained based on the product of the spectrum weight and the spectrum embedding feature.

[0206] In some embodiments, the data acquisition module 1110 is used to:

[0207] Acquiring echo data and visual image data of the target space;

[0208] Performing short-time Fourier transform on the echo data to obtain a spectrum diagram, and inputting the spectrum diagram into an echo encoder for encoding to obtain the spectrum feature data;

[0209] Sending the visual image data to a visual encoder for encoding to obtain the image feature data;

[0210] Content features are extracted from the visual image data to obtain the image content features.

[0211] The specific implementation of the spatial data sensing device of this embodiment is basically the same as the specific implementation of the above-mentioned spatial data sensing method, and will not be repeated here.

[0212] The present application also provides an electronic device, including:

[0213] at least one memory;

[0214] at least one processor;

[0215] at least one program;

[0216] The program is stored in the memory, and the processor executes the at least one program to implement the spatial data perception method implemented in the present application. The electronic device can be any intelligent terminal including a mobile phone, a tablet computer, a personal digital assistant (PDA), a car computer, etc.

[0217] See also Fig.12 , Fig.12 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0218] The processor 1201 may be implemented by a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0219] The memory 1202 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1202 can store an operating system and other applications. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 1202, and the processor 1201 calls and executes the spatial data perception method of the embodiment of this application;

[0220] Input / output interface 1203, used to implement information input and output;

[0221] Communication interface 1204, used to realize communication interaction between the device and other devices, which can be realized by wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.); and

[0222] A bus 1205 that transmits information between various components of the device (e.g., the processor 1201, the memory 1202, the input / output interface 1203, and the communication interface 1204);

[0223] The processor 1201 , the memory 1202 , the input / output interface 1203 and the communication interface 1204 are connected to each other in communication within the device via the bus 1205 .

[0224] An embodiment of the present application also provides a storage medium, which is a storage medium that stores a computer program. When the computer program is executed by a processor, the above-mentioned spatial data perception method is implemented.

[0225] As a non-transitory storage medium, the memory can be used to store non-transitory software programs and non-transitory computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transitory memory, such as at least one disk storage device, a flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0226] The spatial data perception method, device, equipment and storage medium proposed in the embodiment of the present application obtain the image feature data, spectrum feature data and image content feature of the target space, align the image feature data and the spectrum feature data, and obtain the corresponding image alignment feature and spectrum alignment feature; perform self-attention calculation on the spectrum alignment feature to obtain the spectrum attention feature, perform self-attention calculation on the image alignment data to obtain the image attention feature, and perform cross-attention fusion on the spectrum attention feature and the image attention feature to obtain the spatial global feature; perform feature weighting on the image content feature and the spatial global feature to obtain the spatial fusion feature, and decode the spatial fusion feature to obtain the spatial perception image of the target space. In the embodiment of the present application, the multi-dimensional information in the target space is first obtained, and then the image feature data and the spectrum feature data are aligned to avoid the scale deviation between the data of different dimensions and improve the accuracy of the subsequent attention calculation process. Then the aligned data is cross-attention calculated to integrate the information between the data of different modalities to enhance the richness and discrimination of the feature expression. Finally, the image content feature corresponding to the global information is introduced in a weighted manner to improve the amount of information contained in the spatial perception image and its accuracy as a whole, thereby improving the accuracy of spatial perception.

[0227] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0228] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0229] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0230] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0231] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0232] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0233] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0234] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0235] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0236] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store programs.

[0237] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A spatial data perception method, characterized in that: include: Acquire image feature data, spectrum feature data and image content features of the target space, wherein the image content features and the image feature data are both obtained based on the visual image data; Performing data alignment on the image feature data and the spectrum feature data to obtain corresponding image alignment features and spectrum alignment features; Performing self-attention calculation on the spectrum alignment feature to obtain a spectrum attention feature, performing self-attention calculation on the image alignment data to obtain an image attention feature, and performing cross-attention fusion on the spectrum attention feature and the image attention feature to obtain a spatial global feature; The image content features and the spatial global features are feature weighted to obtain spatial fusion features, and the spatial fusion features are decoded to obtain a spatial perception image of the target space.

2. The spatial data perception method according to claim 1, characterized in that: The cross-attention fusion of the spectrum attention feature and the image attention feature to obtain a spatial global feature includes: Performing a first cross-fusion operation according to the image attention feature and the spectrum attention feature to obtain an image spectrum fusion feature; Performing a second cross fusion operation according to the spectrum attention feature and the image attention feature to obtain a spectrum image fusion feature; The spatial global feature is obtained by performing feature combination according to the image spectrum fusion feature and the spectrum image fusion feature.

3. The spatial data perception method according to claim 2, characterized in that: The performing a first cross-fusion operation according to the image attention feature and the spectrum attention feature to obtain an image spectrum fusion feature includes: Obtaining a first cross-query weight, a first cross-key weight, and a first cross-value weight corresponding to the first cross-fusion operation; Obtain a first query intermediate parameter corresponding to the product of the first cross-query weight and the image attention feature, a first key intermediate parameter corresponding to the product of the first cross-key weight and the spectrum attention feature, and a first value intermediate parameter corresponding to the product of the first cross-value weight and the spectrum attention feature; Attention calculation is performed according to the first query intermediate parameter, the first key intermediate parameter, and the first value intermediate parameter to obtain the image spectrum fusion feature.

4. The spatial data perception method according to claim 2, characterized in that: The performing a second cross fusion operation according to the spectrum attention feature and the image attention feature to obtain a spectrum image fusion feature includes: Obtaining a second cross-query weight, a second cross-key weight, and a second cross-value weight corresponding to the second cross-fusion operation; Obtain a second query intermediate parameter corresponding to the product of the second cross-query weight and the spectral attention feature, a second key intermediate parameter corresponding to the product of the second cross-key weight and the image attention feature, and a second value intermediate parameter corresponding to the product of the second cross-value weight and the image attention feature; Attention calculation is performed according to the second query intermediate parameter, the second key intermediate parameter and the second value intermediate parameter to obtain the spectrum image fusion feature.

5. The spatial data perception method according to claim 2, characterized in that: The step of combining features according to the image spectrum fusion feature and the spectrum image fusion feature to obtain the spatial global feature includes: Performing a feature extraction operation of a multi-layer perceptron on the image spectrum fusion features to obtain local features of the image; Performing a feature extraction operation of a multi-layer perceptron on the fusion features of the spectrum image to obtain local features of the spectrum; The spatial global feature is obtained by combining the local frequency spectrum feature and the local image feature.

6. The spatial data perception method according to claim 3, characterized in that: The performing self-attention calculation on the spectrum alignment feature to obtain the spectrum attention feature includes: Obtaining a first query weight, a first key weight, and a first value weight corresponding to the spectrum alignment feature; Performing attention calculation according to the first query weight, the first key weight, and the first value weight to obtain a spectrum intermediate feature; The spectrum alignment feature and the spectrum intermediate feature are superimposed to obtain the spectrum attention feature.

7. The spatial data perception method according to claim 1, characterized in that: The step of aligning the image feature data with the spectrum feature data to obtain corresponding image alignment features and spectrum alignment features includes: Obtaining a size parameter of block embedding, and performing block embedding on the image feature data and the spectrum feature data based on the size parameter to obtain corresponding image embedding features and spectrum embedding features; Obtaining an image weight, and performing linear mapping based on the product of the image weight and the image embedding feature to obtain the image alignment feature; A spectrum weight is obtained, and based on the product of the spectrum weight and the spectrum embedding feature, a linear mapping is performed to obtain the spectrum alignment feature.

8. The spatial data perception method according to any one of claims 1 to 7, characterized in that: The step of acquiring image feature data, spectrum feature data, and image content features of the target space includes: Acquiring echo data and visual image data of the target space; Performing short-time Fourier transform on the echo data to obtain a spectrum diagram, and inputting the spectrum diagram into an echo encoder for encoding to obtain the spectrum feature data; Sending the visual image data to a visual encoder for encoding to obtain the image feature data; Content features are extracted from the visual image data to obtain the image content features.

9. A spatial data sensing device, characterized in that: include: Data acquisition module: used to acquire image feature data, spectrum feature data and image content features of the target space, wherein the image content features and the image feature data are obtained based on the visual image data; Data alignment module: used for aligning the image feature data and the spectrum feature data to obtain corresponding image alignment features and spectrum alignment features; A self-cross attention module: used to perform self-attention calculation on the spectrum alignment feature to obtain a spectrum attention feature, perform self-attention calculation on the image alignment data to obtain an image attention feature, and perform cross-attention fusion on the spectrum attention feature and the image attention feature to obtain a spatial global feature; Perception data generation module: used for performing feature weighting on the image content features and the spatial global features to obtain spatial fusion features, and decoding the spatial fusion features to obtain a spatial perception image of the target space.

10. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the spatial data perception method according to any one of claims 1 to 8 when executing the computer program.

11. A storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the spatial data perception method according to any one of claims 1 to 8 is implemented.

Images