Perception method, device, equipment, medium and computer program product

Abstract

The invention provides a sensing method, device and equipment, a medium and a computer program product, and relates to the technical field of artificial intelligence. The method comprises the steps of obtaining visual data, collected by a plurality of visual sensors, of a first scene at multiple moments within a preset duration, converting the visual data into second features under the same visual angle and under the same coordinate system, fusing the second features at the same moment into fused features, connecting the fused features at the multiple moments and extracting spatiotemporal features, and finally, decoding the spatial-temporal characteristics, and obtaining the occupancy states of the scene grids of the first scene at the current moment and the future time. A fusion scheme based on intermediate features is adopted, information of all modes is fused in a unified feature space, the method is not limited by information modes and the number of sensors and is easy to expand, and therefore the problem that in the prior art, the application scene of a sensing method is limited is solved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a sensing method, device, equipment, medium, and computer program product. Background Technology

[0002] Completing a perception and prediction scheme based on a single vision sensor generally involves several steps: data feature extraction, feature decoding, outputting perception results, and outputting prediction results. Existing high-performance methods typically employ deep learning-based methods for feature extraction, using convolutional neural networks and other techniques to extract features from the data. Feature decoding generally uses a task-specific decoding structure, primarily implemented through neural networks. Outputting perception results typically uses a multi-head output mode, outputting the elements of the perception result. For example, in object detection, this involves outputting the class probability of each pixel and the regression parameters of the corresponding anchor box, including the box size offset and rotation angle.

[0003] Existing methods mostly employ a cascaded workflow when performing perception and prediction tasks. This involves first obtaining the perception result, then tracking the target to derive its historical trajectory, and finally predicting the target's potential future location based on that trajectory. This approach is susceptible to the amplification of cascaded errors, making prediction performance highly dependent on the accuracy of the perception results. Errors or inaccuracies in the perception results are further amplified in the prediction results. Furthermore, existing perception and prediction models based on target detection, target tracking, and trajectory prediction cannot successfully perceive objects not seen during system training, hindering their generalization to open scenes.

[0004] Existing pre-fusion schemes cannot effectively fuse raw data from different modalities, limiting their application scenarios. Current mid-fusion schemes, based on simple operations like addition and averaging, apply the same operation to every pixel of the feature, failing to distinguish the importance of information from different regions. This not only fails to highlight important information but also introduces noise, diluting crucial data. Transformer-based fusion strategies, due to their computational complexity, often involve long computation times, high resource consumption, and difficulty in achieving convergence during training. Post-fusion schemes only provide result-level information during fusion, and results derived from single-sensor information often exhibit poor performance and errors. Post-fusion struggles to correct these errors, and result-level fusion requires strict localization information, making it less robust and less resistant to localization errors. Summary of the Invention

[0005] The purpose of this invention is to provide a sensing method, device, equipment, medium, and computer program product to solve the problem that the application scenarios of sensing and prediction methods in the prior art are limited.

[0006] To achieve the above objectives, embodiments of the present invention provide a sensing method, comprising:

[0007] Acquire visual data collected by multiple visual sensors within a preset time period; wherein, the visual sensors are used to collect visual data of a first scene;

[0008] The visual data of each visual sensor at multiple moments within the preset time period are respectively converted into a first feature under the target viewpoint, and a second feature of the first feature under the target coordinate system is obtained.

[0009] The second features corresponding to multiple vision sensors at the same time are fused to obtain the fused features corresponding to the time.

[0010] The fused features corresponding to the multiple time points are connected and feature extraction is performed to obtain spatiotemporal features;

[0011] Using a pre-trained decoding model, the spatiotemporal features are decoded to obtain the occupancy status of the scene grid in the first scene at the target time; wherein, the decoding model is used to perform semantic analysis on the spatiotemporal features to obtain the displacement vector of the dynamic target in the first scene, and obtain the occupancy status of the dynamic target on the scene grid at the target time based on the displacement vector; the target time includes future time after the current time.

[0012] Optionally, the method further includes:

[0013] Acquire visual data from the multiple visual sensors at a specific moment within the preset time period;

[0014] Extract visual features from the visual data of each of the aforementioned visual sensors corresponding to one of the aforementioned time points;

[0015] Based on the visual features, the visual data corresponding to one of the moments is converted into a first feature under the target viewpoint.

[0016] Optionally, in the method, the plurality of visual sensors are of at least two types;

[0017] Extracting visual features from the visual data of each of the aforementioned visual sensors corresponding to a specific moment, including:

[0018] The visual features of the visual data of the visual sensor at a certain moment are extracted using a feature encoder corresponding to the type of the visual sensor.

[0019] Optionally, the method, wherein obtaining the second feature of the first feature in the target coordinate system includes:

[0020] Based on the position and orientation information of the visual sensor corresponding to the visual data and the target coordinate system, the first feature is transformed to obtain the second feature.

[0021] Optionally, the method, wherein feature fusion is performed on the second features corresponding to the multiple visual sensors at the same time to obtain the fused features corresponding to the time, includes:

[0022] Randomly select one feature from the second feature as the main feature;

[0023] The main feature is connected to each feature in the second feature at the same time to obtain multiple first connection features;

[0024] Each of the first connection features is input into a convolutional neural network to obtain the fusion weight of the second feature corresponding to each of the first connection features;

[0025] The fusion feature corresponding to each second feature is obtained by multiplying it by the corresponding fusion weight and summing the results.

[0026] Optionally, the method, wherein inputting each of the first connection features into a convolutional neural network to obtain the fusion weights of the second features corresponding to each of the first connection features, includes:

[0027] Each of the first connection features is input into the convolutional neural network with a continuous number of convolutional kernels of size 1*1, and the channel dimension of the first connection feature is reduced to 1 to obtain the calculation result of each of the first connection features.

[0028] The calculation results of the plurality of first connection features are all normalized to obtain the fusion weight of the second feature corresponding to each first connection feature.

[0029] Optionally, the method, wherein concatenating the fused features corresponding to the plurality of time points and extracting features to obtain spatiotemporal features, includes:

[0030] The fusion features at the multiple time points are connected to obtain a second connection feature;

[0031] The second connection feature is downsampled in both spatial and temporal dimensions to obtain the first spatiotemporal feature at multiple resolutions;

[0032] The first spatiotemporal features at the multiple resolutions are processed by upsampling and skip connections to obtain the spatiotemporal features.

[0033] Optionally, the method, wherein decoding the spatiotemporal features using a pre-trained decoding model to obtain the occupancy state of the scene grid of the first scene at the target time, includes:

[0034] Using the decoding model, semantic analysis is performed on the spatiotemporal features to obtain the displacement vector of the dynamic target in the first scene;

[0035] The occupancy status of the dynamic target on the scene grid of the first scene at the target time is obtained based on the displacement vector;

[0036] Based on the occupancy status of the scene grid in the first scene by the dynamic target at the target time, the occupancy status of the scene grid in the first scene at the target time is obtained.

[0037] Optionally, in the method, the decoding model includes one or more of a semantic segmentation module, an instance center prediction module, an instance offset prediction module, and an instance flow prediction module.

[0038] The semantic segmentation module is used to predict the probability that the pixel corresponding to the scene grid in the first scene is occupied by the dynamic target.

[0039] The instance center prediction module is used to predict the probability that the pixel is an instance center; the instance center is the geometric center of the dynamic target;

[0040] The instance offset prediction module is used to predict a two-dimensional vector pointing to the center of the instance.

[0041] The instance flow prediction module is used to predict the movement vector of the dynamic target.

[0042] To achieve the above objectives, embodiments of the present invention provide a sensing device, comprising:

[0043] The first acquisition module is used to acquire visual data collected by multiple visual sensors within a preset time period; wherein, the visual sensors are used to collect visual data of a first scene;

[0044] The second acquisition module is used to convert the visual data of each visual sensor at multiple moments within the preset time period into a first feature under the target viewpoint, and obtain a second feature of the first feature under the target coordinate system.

[0045] The first processing module is used to perform feature fusion on the second features corresponding to the multiple vision sensors at the same time to obtain the fused features corresponding to the time.

[0046] The second processing module is used to connect the fused features corresponding to the multiple time points respectively and extract features to obtain spatiotemporal features;

[0047] The third processing module is used to decode the spatiotemporal features using a pre-trained decoding model to obtain the occupancy status of the scene grid in the first scene at the target time; wherein, the decoding model is used to perform semantic analysis on the spatiotemporal features to obtain the displacement vector of the dynamic target in the first scene, and obtain the occupancy status of the dynamic target on the scene grid at the target time based on the displacement vector; the target time includes future time after the current time.

[0048] To achieve the above objectives, embodiments of the present invention provide a network device, including: a processor, a memory, and a program or instructions stored in the memory and executable on the processor; wherein, when the processor executes the program or instructions, it implements the sensing method as described above.

[0049] To achieve the above objectives, embodiments of the present invention provide a readable storage medium having a program or instructions stored thereon, wherein the program or instructions, when executed by a processor, implement the steps in the sensing method as described above.

[0050] To achieve the above objectives, embodiments of the present invention provide a computer program product, which includes computer instructions that, when executed by a processor, implement the steps of the perception method described above.

[0051] The beneficial effects of the above-described technical solution of the present invention are as follows:

[0052] This invention, in its embodiments, acquires visual data from multiple visual sensors at multiple moments within a preset time period for a first scene. The visual data is then converted into second features from the same viewpoint and coordinate system. These second features from the same moment are fused into a fused feature. The fused features from multiple moments are connected to extract spatiotemporal features. Finally, the spatiotemporal features are decoded to obtain the scene grid occupancy status of the first scene at the current moment and in future times. Employing a fusion scheme based on intermediate features, information from various modalities is fused in a unified feature space, unrestricted by information modalities or the number of sensors, and easily scalable, thus solving the problem of limited application scenarios in existing perception methods. Attached Figure Description

[0053] Figure 1 This is a schematic diagram of the sensing method described in an embodiment of the present invention;

[0054] Figure 2 This is a flowchart of the sensing method described in an embodiment of the present invention;

[0055] Figure 3This is a schematic diagram illustrating the acquisition of fusion features by the perception method described in an embodiment of the present invention;

[0056] Figure 4 This is a schematic diagram of the sensing device described in an embodiment of the present invention. Detailed Implementation

[0057] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0058] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of the invention. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification do not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments.

[0059] In various embodiments of the present invention, it should be understood that the sequence number of each process described below does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0060] In addition, the terms "system" and "network" are often used interchangeably in this article.

[0061] In the embodiments provided by this invention, it should be understood that "B corresponding to A" means that B is associated with A, and B can be determined based on A. However, it should also be understood that determining B based on A does not mean that B is determined solely based on A; B can also be determined based on A and / or other information.

[0062] For ease of understanding, the following describes some aspects of the embodiments of the present invention:

[0063] like Figure 1 As shown, an embodiment of the present invention provides a sensing method, which includes:

[0064] Step S10: Obtain visual data collected by multiple visual sensors within a preset time period; wherein, the visual sensors are used to collect visual data of the first scene;

[0065] It should be noted that, as Figure 2As shown, in step 201, several consecutive frames of LiDAR point clouds and camera images from different LiDARs and cameras are input into the system. This means that within the preset time period, multiple visual sensors acquire visual data of the first scene. At each keyframe, the LiDAR and camera simultaneously acquire visual data within the first scene. Considering that the visual sensors acquire visual data for T+1 consecutive time periods, the LiDAR point cloud collected by each visual sensor at time t is... The camera image is , .

[0066] Step S20: Convert the visual data of each visual sensor at multiple moments within the preset time period into a first feature under the target viewpoint, and obtain a second feature of the first feature under the target coordinate system;

[0067] It should be noted that, as Figure 2 As shown, in step 202, the corresponding features of each LiDAR point cloud and image are extracted using a LiDAR point cloud feature encoder and an image feature encoder, respectively. In step 203, all features are converted to a bird's-eye view, that is, the visual data of each visual sensor at multiple moments within the preset time period are converted into the first feature under the target view. In step 204, all features are converted to the same coordinate system, that is, the second feature of the first feature under the target coordinate system is obtained.

[0068] Step S30: Perform feature fusion on the second features corresponding to the multiple vision sensors at the same time to obtain the fused features corresponding to the time.

[0069] It should be noted that, as Figure 2 As shown, in step 205, all lidar point cloud features and image features are fused through a feature fusion model, that is, the second features corresponding to multiple vision sensors at the same time are fused to obtain the fused features.

[0070] Step S40: Connect the fused features corresponding to the multiple time points respectively and extract the features to obtain spatiotemporal features;

[0071] It should be noted that, as Figure 2 As shown, in step 206, a spatiotemporal feature extraction model is used to extract spatiotemporal features from the fused features, that is, the fused features corresponding to the multiple time points are connected and feature extraction is performed to obtain spatiotemporal features. Based on the fused features within the past T+1 time points (i.e., the multiple time points), a spatiotemporal feature extraction model is used to extract the spatiotemporal features of that time period (i.e., the preset duration) from the fused features of consecutive T+1 frames.

[0072] Step S50: Using a pre-trained decoding model, the spatiotemporal features are decoded to obtain the occupancy status of the scene grid in the first scene at the target time; wherein, the decoding model is used to perform semantic analysis on the spatiotemporal features to obtain the displacement vector of the dynamic target in the first scene, and to obtain the occupancy status of the dynamic target on the scene grid at the target time based on the displacement vector; the target time includes future time after the current time;

[0073] It should be noted that, as Figure 2 As shown, in step 207, the spatiotemporal features are decoded using a decoder to obtain the perception and prediction results. That is, the spatiotemporal features are decoded using a pre-trained decoding model to obtain the occupancy status of the scene grid of the first scene at the target time.

[0074] In this embodiment, visual data of the first scene at multiple moments collected by multiple visual sensors are acquired within the preset time period. The visual data is converted into second features under the same viewpoint and coordinate system. The second features at the same moment are fused into a fused feature. The fused features at multiple moments are connected and the spatiotemporal features are extracted. Finally, the spatiotemporal features are decoded to obtain the occupancy status of the scene grid of the first scene at the current moment and in future times. By adopting a fusion scheme based on intermediate features, information from various modalities is fused in a unified feature space, which is not limited by information modalities or the number of sensors, and is easily expandable, thereby solving the problem of limited application scenarios of existing perception methods.

[0075] Optionally, the method further includes:

[0076] Acquire visual data from the multiple visual sensors at a specific moment within the preset time period;

[0077] Extract visual features from the visual data of each of the aforementioned visual sensors corresponding to one of the aforementioned time points;

[0078] Based on the visual features, the visual data corresponding to one of the moments is converted into a first feature under the target viewpoint.

[0079] In this embodiment, at each keyframe moment (i.e., the multiple moments), the LiDAR and the camera (i.e., the multiple visual sensors) simultaneously acquire perception data (i.e., the visual data) within the scene (the first scene). Considering that the visual sensors acquire visual data for T+1 consecutive moments, the LiDAR point cloud collected by each visual sensor at moment t is... The camera image is , For each LiDAR point cloud, a point cloud feature encoder is used. (A commonly used LiDAR point cloud encoder, Pointpillar, can be used) to extract point cloud features (i.e., the visual features). For each image, an image feature encoder is used. (The commonly used image feature extraction network ResNet can be used to extract the two-dimensional features of the image and then project them into three-dimensional space using the Lift-Splat Shoot (LSS) algorithm.) Image features (i.e., the visual features) are extracted, and after obtaining the visual features, all the visual features are transformed to a bird's-eye view (i.e., the target view). This process can be represented as follows:

[0080] , n= m= , .

[0081] in, Indicates time t from the point cloud The first feature is extracted and transformed into the target viewpoint. Indicates time t from the image The first feature is extracted and transformed into the target viewpoint.

[0082] Optionally, in the method, the plurality of visual sensors are of at least two types;

[0083] Extracting visual features from the visual data of each of the aforementioned visual sensors corresponding to a specific moment, including:

[0084] The visual features of the visual data of the visual sensor at a certain moment are extracted using a feature encoder corresponding to the type of the visual sensor.

[0085] In this embodiment, the plurality of visual sensors are of at least two types; LiDAR and cameras are used as examples here, but other types of visual sensors are also applicable. N LiDARs and M cameras are deployed within the first scene. Each visual sensor has its position and orientation information, i.e., its degrees of freedom of movement along the three Cartesian coordinate axes. and rotational degrees of freedom about these three coordinate axes For each LiDAR point cloud, a point cloud feature encoder is used. (A commonly used LiDAR point cloud encoder, Pointpillar, can be used) to extract point cloud features (i.e., the visual features). For each image, an image feature encoder is used. (The commonly used image feature extraction network ResNet can be used to extract the two-dimensional features of the image and then project them into three-dimensional space using the Lift-Splat Shoot (LSS) algorithm.) Image features (i.e., the visual features) are extracted, and after obtaining the visual features, all the visual features are transformed to a bird's-eye view (i.e., the target view). This process can be represented as follows:

[0086] , n= m= , .

[0087] in, Indicates time t from the point cloud The first feature is extracted and transformed into the target viewpoint. Indicates time t from the image The first feature is extracted and transformed into the target viewpoint.

[0088] Optionally, the method, wherein step S20 includes:

[0089] Based on the position and orientation information of the visual sensor corresponding to the visual data and the target coordinate system, the first feature is transformed to obtain the second feature.

[0090] In this embodiment, after obtaining the first feature from the target viewpoint, all the first features are unified to the same target coordinate system through coordinate transformation based on the position and pose information of the corresponding visual sensor. Specifically, each coordinate in the first feature is first rotated and then translated. For example, let the rotation matrix from coordinate system i to coordinate system j be... The translation vector is Then the features in coordinate system i After being transferred to coordinate system j, it can be represented as + .make They represent the points from the point cloud at time t, respectively. and in the image The second feature is extracted and transformed to the target viewpoint and unified to the same target coordinate system.

[0091] Optionally, the method, wherein step S30 includes:

[0092] Randomly select one feature from the second feature as the main feature;

[0093] The main feature is connected to each feature in the second feature at the same time to obtain multiple first connection features;

[0094] Each of the first connection features is input into a convolutional neural network to obtain the fusion weight of the second feature corresponding to each of the first connection features;

[0095] The fusion feature corresponding to each second feature is obtained by multiplying it by the corresponding fusion weight and summing the results.

[0096] In this embodiment, such as Figure 3 As shown, in step 301, a feature is randomly selected from the second features as the main feature, and the main feature is compared with each feature in the second features at the same time (e.g., ...). Figure 3 Features 1, 2, and m+n are concatenated to obtain multiple first connection features. A corresponding second feature of any visual sensor is arbitrarily selected as the main feature. The main feature is then concatenated pairwise with all second features, including itself, along the channel dimension to obtain multiple first connection features, such as... Figure 3 The connection features are from connection feature 1, connection feature 2 to connection feature m+n. For example... Figure 3 In step 302, each of the first connection features is input into the convolutional neural network to obtain the fusion weights (such as...). Figure 3 (The fusion weights are 1, 2, and so on up to m+n). Figure 3 In step 303, each second feature is multiplied by its corresponding fusion weight and then summed to obtain the fusion feature at that time. The fusion weight represents the importance of the corresponding position in the feature map when each second feature participates in the fusion. Based on the fusion weight, all the second features are weighted and summed to obtain the fusion result of the second features from each visual sensor at each time (i.e., the fusion feature). This process can be represented as follows: ,in This indicates the above fusion process. This represents the fusion result (i.e., the fused feature) of the features of each of the visual sensors at time t.

[0097] Optionally, the method, wherein inputting each of the first connection features into a convolutional neural network to obtain the fusion weights of the second features corresponding to each of the first connection features, includes:

[0098] Each of the first connection features is input into the convolutional neural network with a continuous number of convolutional kernels of size 1*1, and the channel dimension of the first connection feature is reduced to 1 to obtain the calculation result of each of the first connection features.

[0099] The calculation results of the plurality of first connection features are all normalized to obtain the fusion weight of the second feature corresponding to each first connection feature.

[0100] In this embodiment, such as Figure 3 As shown, in step 302, the first connection features are input into a model consisting of multiple 1*1 convolutional neural networks with softmax normalization to obtain the fusion weights. The convolutional neural network can be four consecutive 1*1 convolutional neural networks. The first connection features are downsampled to reduce the channel dimension to 1 to obtain the calculation result. The softmax function is used to normalize the calculation results corresponding to all the first connection features to obtain the fusion weights. The adaptively calculated fusion weights measure the importance of features in different regions, highlighting important features and suppressing irrelevant and noisy features during feature fusion. Compared to existing techniques that directly add or average, this method more effectively fuses key information and improves performance. Compared to existing methods that use transformers for fusion, it saves significant computational resources and is easier to train.

[0101] Optionally, the method, wherein step S40 includes:

[0102] The fusion features at the multiple time points are connected to obtain a second connection feature;

[0103] The second connection feature is downsampled in both spatial and temporal dimensions to obtain the first spatiotemporal feature at multiple resolutions;

[0104] The first spatiotemporal features at the multiple resolutions are processed by upsampling and skip connections to obtain the spatiotemporal features.

[0105] In this embodiment, based on the fused features over the past T+1 time periods (i.e., the multiple time periods), a spatiotemporal feature extraction model is used to extract the spatiotemporal features for that time period (the preset duration) from the fused features of consecutive T+1 frames. The spatiotemporal feature extraction model can refer to the spatiotemporal pyramid network design in Motion Network (MotionNet), obtaining the first spatiotemporal features at multiple resolutions by downsampling the connected second connection features in both spatial and temporal dimensions. Then, starting from the highest-dimensional feature, a continuous process of linear interpolation upsampling, connection with the first spatiotemporal feature of the previous dimension, and computation through a convolutional network (i.e., upsampling combined with skip connections) is used to obtain the extracted spatiotemporal features. The specific structure of the downsampling network consists of one 3D convolutional layer and two 2D convolutional layers, each followed by a batch normalization layer and a Rectified Linear Unit (ReLU) function. The spatiotemporal feature extraction process can be represented as follows: , in This represents the spatiotemporal characteristics used to generate the output at time t. This represents the spatiotemporal feature extraction process, and concat represents the feature concatenation operation.

[0106] Optionally, the method, wherein step S50 includes:

[0107] Using the decoding model, semantic analysis is performed on the spatiotemporal features to obtain the displacement vector of the dynamic target in the first scene;

[0108] The occupancy status of the dynamic target on the scene grid of the first scene at the target time is obtained based on the displacement vector;

[0109] Based on the occupancy status of the scene grid in the first scene by the dynamic target at the target time, the occupancy status of the scene grid in the first scene at the target time is obtained.

[0110] In this embodiment, to achieve the prediction function, the decoding model generates four results for both the current time and the next g times (i.e., the target time). The decoding process is represented as follows: ,in, The output at time t represents the semantic segmentation results, including the current time and the semantic segmentation results for the next g times (i.e., the target time). Instance center prediction results Instance offset prediction results Instance flow prediction results , The above decoding process is described above. The occupancy status of the scene grid in the first scene at the target time is obtained by the output result.

[0111] Specifically, candidate center points are extracted from the instance center results at the current time using a non-maximum suppression method. The pixels corresponding to the scene grid are grouped to the nearest candidate center points using the instance offset results at the current time, forming an instance mask. The instance mask is combined with the semantic segmentation results to assign a category label to each instance (i.e., the dynamic target), thereby obtaining the perception result at the current time (i.e., the occupancy state of the scene grid). Then, by applying the instance flow prediction results to the instance centers at the current time, the predicted perception result for future time steps (i.e., the occupancy state of the scene grid at future time steps) is obtained.

[0112] Optionally, in the method, the decoding model includes one or more of a semantic segmentation module, an instance center prediction module, an instance offset prediction module, and an instance flow prediction module.

[0113] The semantic segmentation module is used to predict the probability that the pixel corresponding to the scene grid in the first scene is occupied by the dynamic target.

[0114] The instance center prediction module is used to predict the probability that the pixel is an instance center; the instance center is the geometric center of the dynamic target;

[0115] The instance offset prediction module is used to predict a two-dimensional vector pointing to the center of the instance.

[0116] The instance flow prediction module is used to predict the movement vector of the dynamic target.

[0117] In this embodiment, the decoding model includes four decoding heads, which correspond to the semantic segmentation module, the instance center prediction module, the instance offset prediction module, and the instance flow prediction module, respectively, and each is composed of two layers of convolutional neural networks.

[0118] The training loss function of the perception method described in this embodiment is the sum of the loss functions between the outputs of the four decoders and their corresponding ground truth values. Specifically, the four loss functions are a segmentation loss function, an instance center loss function, an instance offset loss function, and an instance flow loss function. The segmentation loss is a pixel-wise binary cross-entropy loss function, the instance center loss function is a pixel-wise mean squared error loss (L2) function, and the instance offset loss function and instance flow loss function are both pixel-wise mean absolute error loss (L1) functions. The loss function at future time steps will decay exponentially through the parameter φ (which can be set to 0.75). When optimizing the parameters based on the loss functions, the Adam optimizer is used, where two hyperparameters can be set to 0.9 and 0.99.

[0119] Compared to traditional methods, the aforementioned fusion mechanism can intelligently distinguish key regions and suppress noise interference, while its computational efficiency is significantly better than the Transformer scheme. This embodiment of the invention employs a spatiotemporal pyramid network to uniformly process temporal features, and outputs current perception and future prediction results in parallel through a four-task decoder, avoiding cascading errors. In particular, using grid occupancy state as the output target reduces dependence on object categories and effectively detects unknown obstacles, significantly improving robustness and practicality in complex environments.

[0120] It should be noted that the embodiments of the present invention extract bird's-eye view features from LiDAR point clouds and camera images, unify them into the same coordinate system through coordinate transformation, and innovatively adopt an adaptive spatial importance weight fusion model to fuse multimodal features. Then, through a spatiotemporal pyramid network, temporal features are extracted, and finally, a four-task decoder (semantic segmentation, instance center / offset / stream prediction) outputs scene instance segmentation and motion trajectory prediction for the current and future times. This achieves efficient fusion and spatiotemporal prediction of multi-sensor data, significantly improves perception accuracy and computational efficiency, and is particularly suitable for real-time environmental perception and prediction in complex scenarios.

[0121] Furthermore, in this embodiment of the invention, arbitrary visual sensor features are selected as the main features and connected to each visual sensor feature in pairs along the channel dimension. Then, each connected feature is input into a 4-layer continuous 1×1 convolutional network to reduce the channel dimension to 1 dimension. Softmax normalization is then applied to the dimensionality reduction result to obtain spatial adaptive feature fusion weights. Finally, all features are weighted and summed according to the fusion weights. This method can effectively highlight key information and suppress noise compared to traditional methods, and is also more computationally efficient than the Transformer scheme.

[0122] The advantages of the embodiments of the present invention include:

[0123] 1. Existing solutions typically deploy sensors on moving objects such as vehicles, whose positions change dynamically. Real-time positioning information is required for accurate information fusion, and their performance is easily affected by positioning errors. In the embodiments of this invention, the visual sensor is fixed in position and does not require real-time positioning, thus reducing the impact of positioning errors on the results. It is especially suitable for perception and prediction tasks within fixed areas.

[0124] 2. Existing front-fusion based schemes cannot achieve direct fusion of point cloud data and image data, while back-fusion based schemes are susceptible to noise. The scheme proposed in this invention adopts a fusion scheme based on intermediate features, in which information of each modality is fused in a unified feature space, which is not limited by information modalities and the number of sensors, and is easy to expand.

[0125] 3. Existing fusion schemes use simple summation, averaging, and maximum / minimum value feature fusion strategies, which cannot distinguish the importance of information in different regions. Alternatively, they use transformer-based fusion strategies, which have long computation times and are difficult to train. The perception method described in this invention uses a simple convolutional neural network to calculate the spatial importance of different regions as the fusion weights of features. This not only highlights important information and suppresses irrelevant information and noise, but also avoids complex calculations and difficult training.

[0126] 4. In existing solutions, the perception and prediction tasks are performed in series, which can cause cascading errors. Errors in perception are amplified in the prediction task. The embodiments of the present invention output perception and prediction results simultaneously based on the same spatiotemporal features, reducing cascading errors. Furthermore, the two tasks can achieve mutual enhancement through common features.

[0127] 5. In this embodiment of the invention, the occupancy status of each grid in the scene is used as the perception and prediction result. It does not need to care about the specific category of the object occupying the space, but focuses more on the occupancy status that has a greater impact on the scene's safety, and can effectively predict some unknown objects.

[0128] This invention, through the fixed deployment of multi-source heterogeneous sensors (LiDAR + camera), avoids the impact of mobile platform positioning errors on fusion accuracy, making it particularly suitable for fixed-area monitoring scenarios. Secondly, it employs a fusion strategy based on intermediate features, innovatively designing an adaptive spatial importance weight calculation module. Feature-level fusion is achieved through a 1x1 convolutional network and Softmax normalization, overcoming both the modal difference problem of pre-fusion and the noise sensitivity defect of post-fusion. Compared to traditional methods, this invention intelligently distinguishes key regions, suppresses noise interference, and achieves significantly better computational efficiency than the Transformer approach. A spatiotemporal pyramid network is used to uniformly process temporal features, and four-task decoders output current perception and future prediction results in parallel, avoiding cascading errors. In particular, using the occupancy state of the scene grid as the output target reduces dependence on object categories and effectively detects unknown obstacles, significantly improving robustness and practicality in complex environments.

[0129] like Figure 4 As shown, to achieve the above objectives, embodiments of the present invention provide a sensing device, comprising:

[0130] The first acquisition module 401 is used to acquire visual data collected by multiple visual sensors within a preset time period; wherein, the visual sensors are used to collect visual data of the first scene;

[0131] The second acquisition module 402 is used to convert the visual data of each visual sensor at multiple moments within the preset time period into a first feature under the target viewpoint, and obtain a second feature of the first feature under the target coordinate system.

[0132] The first processing module 403 is used to perform feature fusion on the second features corresponding to the multiple vision sensors at the same time to obtain the fused features corresponding to the time.

[0133] The second processing module 404 is used to connect the fused features corresponding to the multiple time points respectively and extract features to obtain spatiotemporal features;

[0134] The third processing module 405 is used to decode the spatiotemporal features using a pre-trained decoding model to obtain the occupancy status of the scene grid in the first scene at the target time; wherein, the decoding model is used to perform semantic analysis on the spatiotemporal features to obtain the displacement vector of the dynamic target in the first scene, and obtain the occupancy status of the dynamic target on the scene grid at the target time based on the displacement vector; the target time includes future time after the current time.

[0135] Optionally, the device further includes:

[0136] The third acquisition module is used to acquire visual data from the multiple visual sensors at one moment within the preset time period.

[0137] The fourth processing module is used to extract visual features of the visual data of each of the visual sensors at a given moment.

[0138] The fifth processing module is used to convert the visual data corresponding to one of the moments into a first feature under the target viewpoint based on the visual features.

[0139] Optionally, in the aforementioned device, the plurality of visual sensors are of at least two types;

[0140] The fourth processing module includes:

[0141] The first processing unit is used to extract visual features of the visual data of the visual sensor at a certain time using a feature encoder corresponding to the type of the visual sensor.

[0142] Optionally, in the aforementioned apparatus, the second acquisition module 402 includes:

[0143] The first acquisition unit is used to perform coordinate transformation on the first feature based on the position and pose information of the visual sensor corresponding to the visual data and the target coordinate system, and acquire the second feature.

[0144] Optionally, in the aforementioned apparatus, the first processing module 403 includes:

[0145] The second processing unit is used to randomly select one feature from the second features as the main feature;

[0146] The second acquisition unit is used to connect the main feature with each feature in the second feature at the same time to acquire multiple first connection features;

[0147] The third acquisition unit is used to input each of the first connection features into the convolutional neural network and acquire the fusion weight of the second feature corresponding to each of the first connection features;

[0148] The fourth acquisition unit is used to multiply each of the second features by the corresponding fusion weight and then sum them to obtain the fusion feature corresponding to the time.

[0149] Optionally, in the aforementioned apparatus, the third acquisition unit comprises:

[0150] The first acquisition component is used to input each of the first connection features into the convolutional neural network with a continuous plurality of convolutional kernels of size 1*1, reduce the channel dimension of the first connection features to 1, and acquire the calculation result of each of the first connection features.

[0151] The second acquisition component is used to normalize the calculation results of the plurality of first connection features and obtain the fusion weight of the second feature corresponding to each first connection feature.

[0152] Optionally, in the aforementioned apparatus, the second processing module 404 includes:

[0153] The fifth acquisition unit is used to connect the fusion features at the multiple time points to obtain a second connection feature;

[0154] The sixth acquisition unit is used to downsample the second connection feature in the spatial and temporal dimensions to acquire the first spatiotemporal feature at multiple resolutions;

[0155] The seventh acquisition unit is used to process the first spatiotemporal features at the multiple resolutions by upsampling and combining them with skip connections to acquire the spatiotemporal features.

[0156] Optionally, in the aforementioned apparatus, the third processing module 405 includes:

[0157] The eighth acquisition unit is used to perform semantic analysis on the spatiotemporal features using the decoding model to obtain the displacement vector of the dynamic target in the first scene;

[0158] The ninth acquisition unit is used to acquire the occupancy status of the scene grid of the first scene by the dynamic target at the target time based on the displacement vector;

[0159] The tenth acquisition unit is used to acquire the occupancy status of the scene grid of the first scene at the target time based on the occupancy status of the scene grid of the first scene by the dynamic target at the target time.

[0160] Optionally, in the aforementioned apparatus, the decoding model includes one or more of a semantic segmentation module, an instance center prediction module, an instance offset prediction module, and an instance flow prediction module.

[0161] The semantic segmentation module is used to predict the probability that the pixel corresponding to the scene grid in the first scene is occupied by the dynamic target.

[0162] The instance center prediction module is used to predict the probability that the pixel is an instance center; the instance center is the geometric center of the dynamic target;

[0163] The instance offset prediction module is used to predict a two-dimensional vector pointing to the center of the instance.

[0164] The instance flow prediction module is used to predict the movement vector of the dynamic target.

[0165] It should be noted that the apparatus provided in this embodiment of the invention can implement all the method steps implemented in the above method embodiment and can achieve the same technical effect. Therefore, the parts and beneficial effects that are the same as those in the method embodiment will not be described in detail here.

[0166] To achieve the above objectives, embodiments of the present invention provide a network device, including: a processor, a memory, and a program or instructions stored in the memory and executable on the processor; wherein, when the processor executes the program or instructions, it implements the sensing method as described above.

[0167] To achieve the above objectives, embodiments of the present invention provide a readable storage medium having a program or instructions stored thereon, wherein the program or instructions, when executed by a processor, implement the steps in the sensing method as described above.

[0168] To achieve the above objectives, embodiments of the present invention provide a computer program product, which includes computer instructions that, when executed by a processor, implement the steps of the perception method described above.

[0169] It should be further noted that the terminals described in this specification include, but are not limited to, smartphones, tablets, etc., and many of the functional components described are referred to as modules in order to emphasize the independence of their implementation.

[0170] In this embodiment of the invention, the module can be implemented in software so that it can be executed by various types of processors. For example, an identified executable code module may include one or more physical or logical blocks of computer instructions, which may be constructed as objects, procedures, or functions. Nevertheless, the executable code of the identified module does not need to be physically located together, but may include different instructions stored in different bits, which, when logically combined, constitute the module and achieve the module's intended purpose.

[0171] In practice, an executable code module can be a single instruction or many instructions, and can even be distributed across multiple different code segments, different programs, and across multiple memory devices. Similarly, operational data can be identified within the module and can be implemented in any suitable form and organized within any suitable data structure. This operational data can be collected as a single dataset or distributed across different locations (including different storage devices), and can exist, at least in part, solely as electronic signals within the system or network.

[0172] When a module can be implemented using software, considering the current level of hardware technology, modules that can be implemented in software can be implemented using hardware circuits by those skilled in the art to achieve the corresponding functions, without considering cost. These hardware circuits include conventional very-large-scale integrated circuits (VLSI) or gate arrays, as well as existing semiconductors such as logic chips and transistors, or other discrete components. Modules can also be implemented using programmable hardware devices, such as field-programmable gate arrays, programmable array logic, and programmable logic devices.

[0173] The exemplary embodiments described above are with reference to the accompanying drawings. Many different forms and embodiments are feasible without departing from the spirit and teachings of the invention. Therefore, the invention should not be construed as limiting the exemplary embodiments set forth herein. Rather, these exemplary embodiments are provided to make the invention complete and convey the scope of the invention to those skilled in the art. In these drawings, component dimensions and relative dimensions may be exaggerated for clarity. The terminology used herein is for the purpose of describing particular exemplary embodiments only and is not intended to be limiting. As used herein, unless clearly indicated otherwise, the singular forms “a,” “an,” and “the” are intended to include all such forms. It will be further understood that the terms “comprising” and / or “including”, when used in this specification, indicate the presence of the stated features, integers, steps, operations, components, and / or elements, but do not exclude the presence or addition of one or more other features, integers, steps, operations, components, and / or groups thereof. Unless otherwise indicated, when stated, a range of values ​​includes the upper and lower limits of the range and any subranges in between.

[0174] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A sensing method, characterized in that, include: Acquire visual data collected by multiple visual sensors within a preset time period; wherein, the visual sensors are used to collect visual data of a first scene; The visual data of each visual sensor at multiple moments within the preset time period are respectively converted into a first feature under the target viewpoint, and a second feature of the first feature under the target coordinate system is obtained. The second features corresponding to multiple vision sensors at the same time are fused to obtain the fused features corresponding to the time. The fused features corresponding to the multiple time points are connected and feature extraction is performed to obtain spatiotemporal features; Using a pre-trained decoding model, the spatiotemporal features are decoded to obtain the occupancy status of the scene grid in the first scene at the target time; wherein, the decoding model is used to perform semantic analysis on the spatiotemporal features to obtain the displacement vector of the dynamic target in the first scene, and obtain the occupancy status of the dynamic target on the scene grid at the target time based on the displacement vector; the target time includes future time after the current time.

2. The method according to claim 1, characterized in that, The method further includes: Acquire visual data from the multiple visual sensors at a specific moment within the preset time period; Extract visual features from the visual data of each of the aforementioned visual sensors corresponding to one of the aforementioned time points; Based on the visual features, the visual data corresponding to one of the moments is converted into a first feature under the target viewpoint.

3. The method according to claim 2, characterized in that, The plurality of visual sensors are of at least two types; Extracting visual features from the visual data of each of the aforementioned visual sensors corresponding to a specific moment, including: The visual features of the visual data of the visual sensor at a certain moment are extracted using a feature encoder corresponding to the type of the visual sensor.

4. The method according to claim 1, characterized in that, Obtaining the second feature of the first feature in the target coordinate system includes: Based on the position and orientation information of the visual sensor corresponding to the visual data and the target coordinate system, the first feature is transformed to obtain the second feature.

5. The method according to claim 1, characterized in that, The second features corresponding to multiple vision sensors at the same time are fused to obtain the fused features corresponding to the time, including: Randomly select one feature from the second feature as the main feature; The main feature is connected to each feature in the second feature at the same time to obtain multiple first connection features; Each of the first connection features is input into a convolutional neural network to obtain the fusion weight of the second feature corresponding to each of the first connection features; The fusion feature corresponding to each second feature is obtained by multiplying it by the corresponding fusion weight and summing the results.

6. The method according to claim 5, characterized in that, Each of the first connectivity features is input into a convolutional neural network to obtain the fusion weights of the second features corresponding to each of the first connectivity features, including: Each of the first connection features is input into the convolutional neural network with a continuous number of convolutional kernels of size 1*1, and the channel dimension of the first connection feature is reduced to 1 to obtain the calculation result of each of the first connection features. The calculation results of the plurality of first connection features are all normalized to obtain the fusion weight of the second feature corresponding to each first connection feature.

7. The method according to claim 1, characterized in that, The fused features corresponding to the multiple time points are concatenated and feature extraction is performed to obtain spatiotemporal features, including: The fusion features at the multiple time points are connected to obtain a second connection feature; The second connection feature is downsampled in both spatial and temporal dimensions to obtain the first spatiotemporal feature at multiple resolutions; The first spatiotemporal features at the multiple resolutions are processed by upsampling and skip connections to obtain the spatiotemporal features.

8. The method according to claim 1, characterized in that, Using a pre-trained decoding model, the spatiotemporal features are decoded to obtain the occupancy state of the scene grid in the first scene at the target time, including: Using the decoding model, semantic analysis is performed on the spatiotemporal features to obtain the displacement vector of the dynamic target in the first scene; The occupancy status of the dynamic target on the scene grid of the first scene at the target time is obtained based on the displacement vector; Based on the occupancy status of the scene grid in the first scene by the dynamic target at the target time, the occupancy status of the scene grid in the first scene at the target time is obtained.

9. The method according to claim 1, characterized in that, The decoding model includes one or more of the following: semantic segmentation module, instance center prediction module, instance offset prediction module, and instance flow prediction module; The semantic segmentation module is used to predict the probability that the pixel corresponding to the scene grid in the first scene is occupied by the dynamic target. The instance center prediction module is used to predict the probability that the pixel is an instance center; the instance center is the geometric center of the dynamic target; The instance offset prediction module is used to predict a two-dimensional vector pointing to the center of the instance. The instance flow prediction module is used to predict the movement vector of the dynamic target.

10. A sensing device, characterized in that, include: The first acquisition module is used to acquire visual data collected by multiple visual sensors within a preset time period; wherein, the visual sensors are used to collect visual data of a first scene; The second acquisition module is used to convert the visual data of each visual sensor at multiple moments within the preset time period into a first feature under the target viewpoint, and obtain a second feature of the first feature under the target coordinate system. The first processing module is used to perform feature fusion on the second features corresponding to the multiple vision sensors at the same time to obtain the fused features corresponding to the time. The second processing module is used to connect the fused features corresponding to the multiple time points respectively and extract features to obtain spatiotemporal features; The third processing module is used to decode the spatiotemporal features using a pre-trained decoding model to obtain the occupancy status of the scene grid in the first scene at the target time; wherein, the decoding model is used to perform semantic analysis on the spatiotemporal features to obtain the displacement vector of the dynamic target in the first scene, and obtain the occupancy status of the dynamic target on the scene grid at the target time based on the displacement vector; the target time includes future time after the current time.

11. A network device, comprising: A processor, a memory, and a program or instructions stored in the memory and executable on the processor; characterized in that, when the processor executes the program or instructions, it implements the sensing method as described in any one of claims 1-9.

12. A readable storage medium having a program or instructions stored thereon, characterized in that, When the program or instructions are executed by the processor, they implement the steps in the perception method as described in any one of claims 1-9.

13. A computer program product, characterized in that, It includes computer instructions that, when executed by a processor, implement the steps of the sensing method as described in any one of claims 1-9.

Images