Multi-sensor information fusion method and device for multi-target recognition scenarios

Abstract

This invention provides a multi-sensor information fusion method and apparatus for multi-target recognition scenarios, relating to the field of intelligent sensing technology. The method includes: acquiring multi-sensor point cloud data for each target to be identified through a sensor system; performing polarization slicing on the multi-sensor point cloud data to obtain corresponding multi-path information slices; performing spatiotemporal synthetic polarization coding on the multi-path information slices to obtain a synthetic coding stream formed by multiple polarization codes; and calling a spatiotemporal neural network model to extract multi-scale semantic information of the target to be identified from the synthetic coding stream for alignment evaluation of each target. This invention addresses the shortcomings of existing multi-sensor systems in multi-target recognition scenarios, such as low information coding efficiency, prolonged data processing time, and severe information synthesis distortion.

Description

Technical Field

[0001] This invention relates to the field of intelligent sensing technology, and in particular to a method and apparatus for multi-sensor information fusion in multi-target recognition scenarios. Background Technology

[0002] The Industry Brain refers to an intelligent decision-making system that empowers industrial transformation and upgrading, driven by the digital transformation scenarios of industrial clusters and utilizing next-generation information technologies such as artificial intelligence. It is built upon a framework of "industrial big data and ten industry-specific models." Its core functions focus on constructing industry application scenarios covering areas such as industry finance, industrial intelligence, and supply-demand matching, forming the technological foundation for the large-scale promotion of intelligent digital factories. This system achieves local data processing through a "cloud-edge-device" collaborative architecture, encompassing various vertical industries such as fault prediction and decision optimization, and is particularly effective for multi-target recognition in complex dynamic scenarios.

[0003] In multi-target recognition scenarios, a sensor system is mounted around the target to be identified to collect multi-sensor point cloud data of the target's surface. This multi-sensor point cloud data is then processed in real time to identify and understand the characteristics of the target, thereby making alignment assessment decisions, such as performing target tracking, target classification, and multi-target feature fusion.

[0004] In existing technologies, multi-sensor systems for multi-target recognition scenarios suffer from the following drawbacks: low information encoding efficiency, prolonged data processing time, and severe information synthesis distortion. These drawbacks stem from the limitations of traditional methods in multi-sensor point cloud data fusion encoding techniques and the impact of random perturbations in complex dynamic scenes. First, the number of targets to be identified is enormous, and they are in a complex dynamic environment with high-speed movement, which may also be subject to environmental interference. Here, the encoding efficiency drops significantly when processing high-dimensional heterogeneous data generated by multiple sensors. Second, errors may be introduced into the feature extraction and fusion encoding stages of multi-sensor point cloud data, making it difficult to fully retain the effective information from each sensor, resulting in poor perception capabilities, especially severe distortion in high-speed dynamic scenes. Summary of the Invention

[0005] This invention provides a multi-sensor information fusion method and apparatus for multi-target recognition scenarios, which solves the defects of existing multi-sensor systems in multi-target recognition scenarios, such as low information encoding efficiency, long data processing time, and severe information synthesis distortion.

[0006] This invention provides a multi-sensor information fusion method for multi-target recognition scenarios, comprising:

[0007] The sensor system acquires multi-sensor point cloud data for each target to be identified, and the sensor system consists of millimeter-wave radar, infrared sensors and optical cameras.

[0008] Polarization slicing is performed on the multi-sensor point cloud data to obtain corresponding multi-path information slices;

[0009] The multi-path information slices are subjected to spatiotemporal synthetic polar coding to obtain a synthetic coding stream formed by multi-polar codes;

[0010] A spatiotemporal neural network model is invoked to extract multi-scale semantic information of the target to be identified from the synthetic encoded stream. The multi-scale semantic information is used to perform alignment evaluation on each target to be identified.

[0011] In some embodiments, prior to performing polarization slicing on the multi-sensor point cloud data, the method further includes:

[0012] Transform the sensor point cloud coordinate system corresponding to the multi-sensor point cloud data into a point cloud coordinate system centered on the target to be identified;

[0013] Motion compensation is performed on the non-instantaneously sampled sensor point cloud data in the multi-sensor point cloud data;

[0014] For sensor point cloud data collected by sensors with polarization characteristics, the polarization reference plane of the point cloud is uniformly aligned.

[0015] In some embodiments, the polarization slicing process performed on the multi-sensor point cloud data to obtain corresponding multi-path information slices includes:

[0016] The multi-sensor point cloud data is divided into slices by calling a simple linear iterative clustering algorithm to obtain multiple superpixel slices;

[0017] The superpixel slices are encrypted using a shared key, and polarization feature data is extracted for each encrypted superpixel slice to obtain the corresponding multi-channel information slice. The polarization feature data includes polarization features under the HH channel, HV channel, and VV channel.

[0018] In some embodiments, the slice granularity of the superpixel slice is negatively correlated with the scene dynamic index of the target to be identified, and the calculation process of the scene dynamic index includes:

[0019] The motion velocity of the target to be identified is obtained, including angular velocity, acceleration, and the relative velocity of the target to be identified relative to a stationary reference object;

[0020] The relative velocity, the angular velocity, and the acceleration are weighted by preset relative velocity weight, angular velocity weight, and acceleration weight respectively, and the weighted results are summed to obtain the scene dynamic index.

[0021] In some embodiments, performing spatiotemporal synthesis polar coding on the multi-path information slices to obtain a synthesized coded stream formed by multi-polar codes includes:

[0022] According to the preset coding strategy, the multi-path information slices are subjected to multiplicative repeated spreading processing, and the spreading sequence with the largest minimum square Euclidean distance is selected from the obtained spreading sequence as the synthetic coding stream.

[0023] The encoding strategy includes a rate adaptation strategy and a spatiotemporal correlation strategy;

[0024] The spatiotemporal correlation strategy is used to extract spatiotemporal correlation features between codewords when generating codewords for a spread spectrum sequence through multiplicative repetitive spread spectrum processing.

[0025] The rate adaptation strategy is used to adjust the coding rate during multiplicative repetitive spread spectrum processing according to the channel environment of the target to be identified. The coding rate is positively correlated with the signal quality value of the channel environment.

[0026] In some embodiments, the alignment evaluation process for the target to be identified includes:

[0027] Locally update the environment map of the target to be identified;

[0028] The target to be identified is classified or located and tracked.

[0029] The present invention also provides a multi-sensor information fusion device for multi-target recognition scenarios, comprising:

[0030] The multi-sensor information perception module is used to acquire multi-sensor point cloud data of each target to be identified through a sensor system, which consists of a millimeter-wave radar, an infrared sensor, and an optical camera.

[0031] The channel polarization slicing module is used to perform polarization slicing processing on the multi-sensor point cloud data to obtain corresponding multi-channel information slices.

[0032] The slice synthesis coding module is used to perform spatiotemporal synthesis polar coding on the multi-channel information slices to obtain a synthetic coding stream formed by multi-polar codes;

[0033] A multi-scale processing module is used to call a spatiotemporal neural network model to extract multi-scale semantic information of the target to be identified from the synthetic coding stream. The multi-scale semantic information is used to perform alignment evaluation on each target to be identified.

[0034] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the multi-sensor information fusion method for multi-target recognition scenarios as described above.

[0035] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the multi-sensor information fusion method for multi-target recognition scenarios as described above.

[0036] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the multi-sensor information fusion method for multi-target recognition scenarios as described above.

[0037] The present invention provides a multi-sensor information fusion method and apparatus for multi-target recognition scenarios. Polar coding is applied to the fusion process of multi-sensor point cloud data. For the multi-sensor point cloud data of the target to be identified, polarimetric slicing is performed first, followed by encoding of the slices to construct a multi-source polar code. Compared to the traditional method of directly encoding and fusing using deep learning models, this method not only has better real-time encoding efficiency but also achieves efficient fusion of sensor point cloud data. Furthermore, by extracting features from the synthesized coded stream using a spatiotemporal neural network model, multi-scale semantic information of the target to be identified can be obtained simultaneously from both temporal and spatial perspectives. This ensures the coherent expression of the multi-sensor point cloud data stream, enhances the perception capability of multiple targets to be identified, and improves the accuracy and efficiency of target recognition. It overcomes the shortcomings of traditional methods, such as severe information synthesis distortion and the reliance on only instantaneous observation data for environmental perception and target recognition. Attached Figure Description

[0038] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced one by one below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0039] Figure 1 This is a flowchart illustrating the multi-sensor information fusion method for multi-target recognition scenarios provided by the present invention.

[0040] Figure 2 This is a schematic diagram of the structure of the multi-sensor information fusion device for multi-target recognition scenarios provided by the present invention.

[0041] Figure 3 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0042] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0043] It should be noted that the multi-sensor information fusion method for multi-target recognition scenarios provided by this invention can be applied to quality inspection systems for industrial products. In high-speed production line scenarios, the sensor system is mounted on the quality inspection system, and the target to be identified can be industrial products moving at high speed on the production line, such as industrial parts. The final alignment evaluation of each target includes target classification or localization tracking, which means real-time quality inspection of multiple industrial products to detect defects, such as cracks.

[0044] The following description, in conjunction with the accompanying drawings, describes the multi-sensor information fusion method and apparatus for multi-target recognition scenarios according to the present invention. Figure 1 This is a flowchart illustrating the multi-sensor information fusion method for multi-target recognition scenarios provided by the present invention, as shown below. Figure 1 As shown, the method includes the following steps 101 to 104.

[0045] Step 101: Acquire multi-sensor point cloud data for each target to be identified through the sensor system.

[0046] Here, the sensor system is used to perceive each target in the scene in real time, scanning each target at different angles and collecting corresponding multi-sensor point cloud data. The sensor system consists of millimeter-wave radar, infrared sensors, and optical cameras. The collected multi-sensor point cloud data specifically includes radar reflection signals collected by the millimeter-wave radar, infrared images collected by the infrared sensors, and point cloud data processed from spectral information collected by the optical cameras. The sensors in the system can be classified according to their polarization characteristics. For example, millimeter-wave radar is a polarized sensor, infrared sensors have certain polarization characteristics under the polarization state of infrared radiation, and optical cameras are non-polarized sensors. This classification facilitates the subsequent extraction of polarization features from the sensor point cloud data using different methods.

[0047] When a sensor system acquires data, spatiotemporal alignment is required. At the hardware level, a precise clock synchronization signal protocol is used to trigger all sensors to acquire data simultaneously. The clock synchronization signal can be distributed via the IEEE1588-2008 (PTPv2) protocol. At the software level, the residual time difference is corrected by a corresponding motion compensation algorithm. In this way, the synchronization accuracy of each sensor can be controlled within 5ms, which meets the requirements of multi-target recognition scenarios.

[0048] Furthermore, the target to be identified is in motion within the scene. Therefore, when collecting multi-sensor point cloud data, the data acquisition range of the sensor system can be controlled based on the target's motion speed. Motion speed and data acquisition range are negatively correlated: the faster the motion, the narrower the data acquisition range, and vice versa.

[0049] Step 102: Perform polarization slicing on the multi-sensor point cloud data to obtain the corresponding multi-channel information slices.

[0050] Next, polarimetric slicing is performed on the multi-sensor point cloud data to obtain corresponding multi-path information slices. Each information slice represents a local region with similar polarization characteristics, which is a portion of the environment where the target to be identified is located. However, before performing polarimetric slicing on the multi-sensor point cloud data, preprocessing is required. The preprocessing process includes temporal alignment, spatial alignment, and polarization characteristic alignment, which will be explained in detail below.

[0051] In terms of spatial alignment, the sensor point cloud coordinate system corresponding to the multi-sensor point cloud data is transformed into a point cloud coordinate system centered on the target to be identified. Due to the differences in hardware characteristics of each sensor, the structure of the collected sensor point cloud data varies. Here, the point cloud coordinate system transformation can be achieved based on preset calibration data, such as using a calibration board with feature points or directly capturing calibration points.

[0052] For time alignment, motion compensation is performed on the non-instantaneously sampled sensor point cloud data in the multi-sensor point cloud data. This motion compensation primarily targets the non-instantaneously sampled sensor point cloud data, whose acquisition time deviates from that of other instantaneously sampled sensors. Therefore, a B-spline interpolation algorithm is further used to perform motion compensation on the non-instantaneously sampled sensor point cloud data to reconstruct the synchronization time data, ensuring that the sensor point cloud data acquired by all sensors are synchronized at all times.

[0053] In terms of polarization alignment, for sensor point cloud data collected by sensors with polarization characteristics, the reference plane of polarization is unified. This is done by adjusting the polarization state of the sensor point cloud data to a set direction and unifying the reference plane of polarization.

[0054] This invention addresses the challenge of asynchronous point cloud data from multiple sensors in high-speed dynamic environments by using a spatiotemporal registration mechanism to align sensor point cloud data. Furthermore, polarization characteristic alignment ensures that the polarization states of all sensor point cloud data with polarization characteristics are identical, providing a data foundation for subsequent polarization feature extraction.

[0055] After the above preprocessing, the multi-sensor point cloud data can be further processed by polarization slicing, as explained below.

[0056] First, a simple linear iterative clustering algorithm is used to segment the multi-sensor point cloud data, resulting in multiple superpixel slices. The initial cluster center spacing of the simple linear iterative clustering algorithm is set to 1.5 times the superpixel slice granularity. The superpixel slice granularity ranges from 10×10 pixels to 100×100 pixels, and here it can be set to 30×30 pixels. The slice type for the superpixel slices is a hybrid mode of vector slices and feature slices to balance segmentation accuracy and efficiency.

[0057] Regarding the setting of superpixel slice granularity, this embodiment of the invention designs an automatic granularity adjustment mechanism, which determines the granularity based on the current scene dynamics of the target to be identified. The stronger the dynamics, the finer the slice granularity; conversely, the more stable the high-speed pipeline scene, the coarser the slice granularity. Scene dynamics are measured by the Scene Dynamics Index, denoted as DI. The slice granularity of the superpixel slice is negatively correlated with the Scene Dynamics Index of the target to be identified. That is, the larger the Scene Dynamics Index, the stronger the dynamics of the high-speed pipeline scene, and the smaller the slice granularity of the superpixel slice; conversely, the smaller the Scene Dynamics Index, the more stable the high-speed pipeline scene, and the larger the slice granularity of the superpixel slice.

[0058] The scene dynamic index (DI) can be determined by the motion speed. The calculation process of the scene dynamic index is described below.

[0059] First, obtain the motion velocity of the target to be identified, including angular velocity. Acceleration *a* and the relative velocity *v* of the target relative to a stationary reference object. Angular velocity. The relative velocity v can be measured by setting up visual inertial navigation equipment and a velocimeter, and the acceleration a can be calculated in real time by the relative velocity over a certain period of time.

[0060] Furthermore, each is then processed using preset relative velocity weights. Angular velocity weight and acceleration weights For relative velocity v and angular velocity We weight the acceleration 'a' and sum the weighted results to obtain the scene dynamic index DI, expressed as the following formula:

[0061] (1)

[0062] Of course, relative speed weight Angular velocity weight and acceleration weights All of these are adaptively adjusted, and their values ​​can range from 0 to 1. When the target to be identified is in uniform linear motion, the relative velocity weight will be adaptively adjusted to a medium level, such as 0.5, while the other weights, such as angular velocity and acceleration, will be adaptively adjusted to a lower level, such as below 0.4 or even close to 0. At this time, the calculated scene dynamic index DI will be relatively small. The more stable the target to be identified is, the larger the granularity of the superpixel slice will be.

[0063] When the target to be identified is in an accelerated motion state and is not in a linear motion (such as frequently changing the direction of motion), the angular velocity weight and acceleration weight will be adaptively adjusted to be higher, for example, 0.6 and 0.7. The corresponding relative velocity weight will also be adaptively adjusted to be higher, for example, 0.8. At this time, the calculated scene dynamic index DI will be relatively large. The stronger the dynamics of the target to be identified, the smaller the granularity of the superpixel slice will be.

[0064] In this embodiment of the invention, the granularity of superpixel slices is adjusted according to the dynamic level of the high-speed pipeline scene, so that the data processing can adapt to the motion environment. This adaptive mechanism effectively balances the processing accuracy and computational overhead of processing sensor point cloud data.

[0065] Furthermore, the superpixel slices are encrypted using a shared key, and polarization features are extracted for each encrypted superpixel slice to obtain the corresponding multi-path information slices.

[0066] To prevent superpixel slices from being stolen or tampered with during information transmission, a shared key is used to encrypt and encode the superpixel slices after they are divided. A differentiated encryption strategy is employed, particularly for the freeze bits and information bits, ensuring security without affecting the normal decoding process. Furthermore, enhanced protection is implemented for some critical slices by adding an extra 10% of redundant check bits.

[0067] Finally, there is the polarization feature extraction process, which extracts polarization feature data for each encrypted superpixel slice to obtain the corresponding multi-path information slice.

[0068] Polarization feature data specifically includes polarization features in the HH, HV, and VV channels. For sensor point cloud data acquired by sensors with polarization characteristics, polarization scattering matrices in the corresponding HH, HV, and VV channels can be directly constructed based on the corresponding superpixel slices. Examples include radar reverse signals and infrared images under polarization states. For sensor point cloud data acquired by sensors without polarization characteristics, such as optical cameras, superpixel slices are formed from the acquired spectral data. Multispectral information can be used for feature-level fusion to simulate the polarization scattering matrix and extract equivalent polarization parameters. The polarization scattering matrix is ​​specifically a 9-dimensional polarization feature vector, including: scattering matrix elements, polarization entropy, inverse entropy, anisotropy degree, etc.

[0069] After extracting polarization features, the parameters of the corresponding polarization features can be labeled on the corresponding superpixel slices. The parameters include the scattering matrix, polarization entropy, etc., thus forming multi-path information slices, which facilitates subsequent synthesis and encoding to achieve data fusion.

[0070] This invention achieves efficient processing of multi-sensor point cloud data by dividing the data into slices and further using encryption encoding and polarization feature extraction. The slice division process effectively preserves data feature details and suppresses noise, while encryption encoding further ensures the security of the multi-sensor point cloud data.

[0071] Step 103: Perform spatiotemporal synthesis polar coding on the multi-path information slices to obtain a synthetic coding stream formed by multi-polar codes.

[0072] Here, by performing spatiotemporal synthetic polarization coding on multi-channel information slices, and by constructing multi-polarization codes, the multi-sensor point cloud data after polarization slicing is deeply fused to generate a synthetic coded stream with strong error correction capability and high transmission efficiency.

[0073] In some embodiments, spatiotemporal synthesis polar coding of multi-path information slices can be used to obtain a synthetic coding stream formed by multi-polar codes. This can be achieved in the following ways, which are described in detail below.

[0074] According to the preset coding strategy, the multi-channel information slices are multiplicatively repeated spread spectrum processing, and the spread spectrum sequence with the largest minimum square Euclidean distance is selected from the obtained spread spectrum sequence as the synthetic coding stream.

[0075] Here, a spatiotemporal synthesis polarization coding method based on multiplicative repetition is used to construct multivariate polarization codes for slices. This process involves two aspects: first, employing a multiplicative repetition spreading method based on Euclidean distance optimization; and second, mapping multi-channel information slices to a spatiotemporal coding matrix through multiplicative repetition spreading processing. In specific implementation, this embodiment of the invention employs a differentiated coding protection strategy. Specifically, based on the differences in importance of the sensor point cloud data corresponding to the multi-channel information slices, the required coding resources for multiplicative repetition spreading processing are rationally allocated. For example, the spectral information acquired by the optical camera is more important than the infrared image; therefore, more coding resources will be allocated to the multiplicative repetition spreading processing for the multi-channel information slices formed from the spectral information.

[0076] In addition, the multiplicative repetitive spread spectrum processing is performed according to a preset coding strategy, which is used to constrain the multiplicative repetitive spread spectrum processing process.

[0077] The encoding strategy includes a rate adaptation strategy and a spatiotemporal correlation strategy. The spatiotemporal correlation strategy is used to extract the spatiotemporal correlation features between codewords when generating codewords for the spread spectrum sequence through multiplicative repetitive spread spectrum processing. As a result, the spatiotemporal coding matrix obtained by mapping not only includes the measurement values ​​of the sensor point cloud data corresponding to the information slices, but also embeds the spatiotemporal correlation relationships between the various measurement values.

[0078] The rate adaptation strategy adjusts the coding rate during multiplicative repetitive spread spectrum processing based on the channel environment of the target to be identified. The coding rate is positively correlated with the signal quality of the channel environment. Here, an adaptive rate matching strategy is employed during coding in multiplicative repetitive spread spectrum processing. This strategy dynamically adjusts the coding parameters based on the channel conditions of the target's environment, thereby controlling the coding rate. Higher signal quality in the channel environment indicates better channel conditions, allowing for a corresponding increase in coding speed and data transmission rate. Conversely, lower signal quality indicates worse channel conditions and more severe signal interference, leading to a decrease in coding speed and data transmission rate. This enhances the error correction capability of the coding, achieving optimal coding performance.

[0079] After multiplicative repetitive spreading, the multi-information slices form a multi-polar code with a code length N of 256 and an information bit K of 128.

[0080] This invention employs a rate adaptation strategy and a spatiotemporal correlation strategy to implement multiplicative repetitive spreading processing. On one hand, this preserves the spatiotemporal correlation characteristics of the data, particularly reducing data reconstruction errors when receiving multi-sensor point cloud data, especially when the target to be identified is moving at high speed. On the other hand, it adapts to different channel conditions, balancing coding rate and error correction capabilities to achieve optimal coding performance. Furthermore, by using multiplicative repetitive spreading processing to construct multi-polar codes, compared to traditional binary polar codes, it exhibits a lower error plane and stronger resistance to burst interference during encoding.

[0081] Step 104: Call the spatiotemporal neural network model to extract multi-scale semantic information of the target to be identified from the synthetic coding stream. The multi-scale semantic information is used to perform alignment evaluation of the target to be identified.

[0082] This describes the process of parsing and understanding the synthetic encoded stream, extracting multi-scale semantic information to achieve a multi-granular understanding of the target to be identified.

[0083] Specifically, a spatiotemporal neural network model is invoked to extract features from the synthesized encoded stream. This model comprises multiple parallel network branches, each extracting spatiotemporal semantic features at different scales. Scale is measured by the frequency of feature changes, and the extracted spatiotemporal semantic features at different scales include both high-frequency and low-frequency features of the target to be identified. Finally, an attention module is designed to fuse the spatiotemporal semantic features extracted from each network branch at different scales, yielding multi-scale semantic information of the target to be identified. This multi-scale semantic information specifically includes the spatiotemporal contextual semantic features of the target to be identified, used for alignment evaluation of the detection model.

[0084] To improve the efficiency of the spatiotemporal neural network model, this embodiment of the invention performs model compression and acceleration on the spatiotemporal neural network model. By means of quantization and pruning, the size of the spatiotemporal neural network model is compressed while ensuring the accuracy of the model. The compressed spatiotemporal neural network model is then used to extract multi-scale semantic information.

[0085] Multi-scale semantic information is used for alignment evaluation of each target to be identified. The alignment evaluation process includes two aspects: first, locally updating the environmental map of the target to be identified; and second, classifying or locating and tracking the target.

[0086] Here, an environmental map composed of data points is constructed for the scene where the target to be identified is located. The environmental map can be constructed using an incremental kd-tree. The environmental map of the target to be identified is locally updated by extracting multi-scale semantic information, that is, only the data points of the changed parts of the environmental map are updated. The update frequency is set to 100Hz, which can significantly reduce the computational cost.

[0087] Furthermore, the extracted multi-scale semantic information can also be used for target recognition, specifically including target classification or localization tracking of the target, such as classifying industrial products with quality defects, or determining the real-time position of the target, generating the corresponding motion trajectory, and identifying whether the target may collide in order to avoid obstacles.

[0088] In this embodiment of the invention, the understanding process of the target to be identified is realized by extracting multi-scale semantic information. The spatiotemporal neural network model enables the extracted features to have coherent expression, enhances the alignment evaluation effect of the target to be identified, achieves good performance phenotype in target classification or localization tracking, and can significantly reduce the resource consumption required for target identification by locally updating the environmental map of the target to be identified.

[0089] When the method of this invention is applied to the perception system of a high-speed pipeline, a multi-sensor system is mounted on the perception system. Assuming that the switching time of the high-speed pipeline is 30-40ms, the method of this invention can locate and track the high-speed pipeline. The method of this invention successfully controls the tracking error of the high-speed pipeline to within 0.12m, the threat identification accuracy reaches 99.7%, and the end-to-end data processing delay is only 4.3ms, which fully meets the stringent requirements of the switching step within 10ms.

[0090] When the method of this invention is applied to a quality inspection system for industrial products, the sensor system is mounted on the quality inspection system, and the industrial products are industrial parts. The production line where the industrial products are located operates at a speed of 2000 pieces per minute. The sensor system synchronously triggers data acquisition, collecting comprehensive information such as images of the surface, three-dimensional shape, and internal structure of the industrial parts for target classification and detection of defects in the industrial products. The method of this invention can increase the inspection speed of industrial parts from 800 pieces per minute to 2000 pieces per minute, meeting the needs of the production line. The false detection rate is less than 0.1%, and the missed detection rate is close to zero. It can successfully identify many minute defects (such as 50μm-level surface cracks) that are difficult to detect by traditional methods, significantly improving the level of product quality control.

[0091] In the coding process of multi-sensor information fusion, the coding efficiency of this invention can reach over 85%, which is a significant improvement compared to traditional coding methods. Regarding the bit error rate, the method of this invention also performs well, especially under poor channel conditions, maintaining a high level of accuracy. The error rate is on the order of magnitude.

[0092] The following describes the multi-sensor information fusion device for multi-target recognition scenarios provided by the present invention. The multi-sensor information fusion device for multi-target recognition scenarios described below and the multi-sensor information fusion method for multi-target recognition scenarios described above can be referred to in correspondence with each other.

[0093] like Figure 2 As shown, the multi-sensor information fusion device for multi-target recognition scenarios specifically includes: a multi-sensor information perception module 201, a channel polarization slicing module 202, a slice synthesis encoding module 203, and a multi-scale processing module 204. Specifically, the multi-sensor information perception module 201 is used to acquire multi-sensor point cloud data for each target to be identified through a sensor system, which consists of a millimeter-wave radar, an infrared sensor, and an optical camera; the channel polarization slicing module 202 is used to perform polarization slicing processing on the multi-sensor point cloud data to obtain corresponding multi-channel information slices; the slice synthesis encoding module 203 is used to perform spatiotemporal synthesis polarization encoding on the multi-channel information slices to obtain a synthetic encoding stream formed by multi-polarization codes; the multi-scale processing module 204 is used to call a spatiotemporal neural network model to extract multi-scale semantic information of the target to be identified from the synthetic encoding stream, and the multi-scale semantic information is used for alignment evaluation of each target to be identified.

[0094] It should be noted that the beneficial effects of the multi-sensor information fusion device in the multi-target recognition scenario here correspond to those of the multi-sensor information fusion method in the multi-target recognition scenario mentioned above. Therefore, the beneficial effects of the multi-sensor information fusion device in the multi-target recognition scenario will not be elaborated here.

[0095] Figure 3 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 3As shown, the electronic device may include a processor 310, a communications interface 320, a memory 330, and a communication bus 340, wherein the processor 310, communications interface 320, and memory 330 communicate with each other via the communication bus 340. The processor 310 can call logical instructions in the memory 330 to execute a multi-sensor information fusion method for a multi-target recognition scenario. This method includes: acquiring multi-sensor point cloud data for each target to be identified through a sensor system, wherein the sensor system consists of a millimeter-wave radar, an infrared sensor, and an optical camera; performing polarimetric slicing processing on the multi-sensor point cloud data to obtain corresponding multi-channel information slices; performing spatiotemporal synthetic polarimetric coding on the multi-channel information slices to obtain a synthetic coding stream formed by multi-polar codes; and calling a spatiotemporal neural network model to extract multi-scale semantic information of the target to be identified from the synthetic coding stream, wherein the multi-scale semantic information is used for alignment evaluation of each target to be identified.

[0096] Furthermore, the logical instructions in the aforementioned memory 330 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0097] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the multi-sensor information fusion method for multi-target recognition scenarios provided by the above methods. The method includes: acquiring multi-sensor point cloud data of each target to be identified through a sensor system, wherein the sensor system consists of a millimeter-wave radar, an infrared sensor, and an optical camera; performing polarimetric slicing processing on the multi-sensor point cloud data to obtain corresponding multi-path information slices; performing spatiotemporal synthetic polarimetric coding on the multi-path information slices to obtain a synthetic coding stream formed by multi-polar codes; and calling a spatiotemporal neural network model to extract multi-scale semantic information of the target to be identified from the synthetic coding stream, wherein the multi-scale semantic information is used to perform alignment evaluation on each target to be identified.

[0098] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements a multi-sensor information fusion method for multi-target recognition scenarios provided by the methods described above. The method includes: acquiring multi-sensor point cloud data of each target to be identified through a sensor system, wherein the sensor system consists of a millimeter-wave radar, an infrared sensor, and an optical camera; performing polarimetric slicing processing on the multi-sensor point cloud data to obtain corresponding multi-path information slices; performing spatiotemporal synthetic polarimetric coding on the multi-path information slices to obtain a synthetic coding stream formed by multi-polar codes; and calling a spatiotemporal neural network model to extract multi-scale semantic information of the target to be identified from the synthetic coding stream, wherein the multi-scale semantic information is used for alignment evaluation of each target to be identified.

[0099] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0100] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0101] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-sensor information fusion method for multi-target recognition scenarios, characterized in that, The method comprises the steps of: acquiring multi-sensor point cloud data of each target to be identified by a sensor system composed of a millimeter wave radar, an infrared sensor and an optical camera; performing polarization slicing processing on the multi-sensor point cloud data to obtain corresponding multi-path information slices; performing space-time synthesis polarization coding on the multi-path information slices to obtain a synthesis coding stream formed by a multi-element polarization code; calling a space-time neural network model to extract multi-scale semantic information of the target to be identified from the synthesis coding stream, and the multi-scale semantic information is used for alignment evaluation of each target to be identified; the polarization slicing processing on the multi-sensor point cloud data to obtain corresponding multi-path information slices comprises: calling a simple linear iterative clustering algorithm to perform slice division on the multi-sensor point cloud data to obtain a plurality of super-pixel slices; encrypting and coding the super-pixel slices by a shared key, and extracting polarization feature data for each encrypted and coded super-pixel slice to obtain corresponding multi-path information slices, wherein the polarization feature data comprises polarization features in HH, HV and VV channels; the space-time synthesis polarization coding on the multi-path information slices to obtain a synthesis coding stream formed by a multi-element polarization code comprises: According to a preset coding strategy, the multi-path information slices are subjected to multiplicative repeated spread spectrum processing, and the spread spectrum sequence with the maximum least square Euclidean distance is selected from the obtained spread spectrum sequences as the synthesis coding stream; wherein the coding strategy comprises a rate adaptation strategy and a space-time correlation strategy; the space-time correlation strategy is used to extract the space-time correlation features between the code words when the multiplicative repeated spread spectrum processing generates the code words of the spread spectrum sequence; the rate adaptation strategy is used to adjust the coding rate in the multiplicative repeated spread spectrum processing according to the channel environment of the target to be identified, and the coding rate is positively correlated with the signal quality value of the channel environment.

2. The multi-sensory information fusion method for multi-target recognition scenarios according to claim 1, characterized in that, Before the polarization slicing processing on the multi-sensor point cloud data, the method further comprises: transforming the sensor point cloud coordinate system corresponding to the multi-sensor point cloud data into a point cloud coordinate system centered on the target to be identified; motion compensation is performed on the non-instantaneous sampled sensor point cloud data in the multi-sensor point cloud data; for the sensor point cloud data collected by the sensor with polarization characteristics, the reference plane of polarization is unified.

3. The multi-sensory information fusion method for multi-target recognition scenarios according to claim 1, characterized in that, The slice granularity size of the super-pixel slice is negatively correlated with the scene dynamic index of the target to be identified, and the calculation process of the scene dynamic index comprises: obtaining the motion speed of the target to be identified, wherein the motion speed comprises angular velocity, acceleration and relative speed of the target to be identified relative to a stationary reference; respectively weighting the relative speed, the angular velocity and the acceleration by preset relative speed weight, angular velocity weight and acceleration weight, and summing the weighted results to obtain a scene dynamic index.

4. The multi-sensory information fusion method for multi-target recognition scenarios according to claim 1, characterized in that, The alignment evaluation process of the target to be identified comprises: locally updating the environment map of the target to be identified; target classification or positioning tracking is performed on the target to be identified.

5. A multi-sensor information fusion device for a multi-target recognition scenario, characterized in that, The method comprises the steps of: A multi-sensor information perception module is configured to acquire multi-sensor point cloud data of each target to be identified through a sensor system composed of a millimeter wave radar, an infrared sensor, and an optical camera. A channel polarization slicing module is configured to perform polarization slicing processing on the multi-sensor point cloud data to obtain corresponding multi-path information slices. A slice synthesis encoding module is configured to perform spatio-temporal synthesis polarization encoding on the multi-path information slices to obtain a synthesis encoding stream formed by a multi-element polarization code. A multi-scale processing module is configured to call a spatio-temporal neural network model to extract multi-scale semantic information of the target to be identified from the synthesis encoding stream, and the multi-scale semantic information is used for alignment evaluation of each target to be identified. The polarization slicing processing on the multi-sensor point cloud data to obtain corresponding multi-path information slices includes: A simple linear iterative clustering algorithm is called to perform slice division on the multi-sensor point cloud data to obtain a plurality of super-pixel slices. The super-pixel slices are encrypted and encoded by a shared key, and polarization feature data is extracted for each encrypted and encoded super-pixel slice to obtain corresponding multi-path information slices, wherein the polarization feature data includes polarization features in HH, HV, and VV channels. The spatio-temporal synthesis polarization encoding on the multi-path information slices to obtain a synthesis encoding stream formed by a multi-element polarization code includes: According to a preset encoding strategy, the multi-path information slices are subjected to multiplicative repeated spread spectrum processing, and the spread spectrum sequence with the maximum least square Euclidean distance is selected from the obtained spread spectrum sequences as the synthesis encoding stream. The encoding strategy includes a rate adaptation strategy and a spatio-temporal correlation strategy. The spatio-temporal correlation strategy is used to extract the spatio-temporal correlation features between code words when the multiplicative repeated spread spectrum processing generates the code words of the spread spectrum sequence. The rate adaptation strategy is used to adjust the encoding rate during the multiplicative repeated spread spectrum processing according to the channel environment of the target to be identified, and the encoding rate and the signal quality value of the channel environment are in a positive correlation relationship.

6. An electronic device comprising a memory, a processor, and a computer program stored on the memory and running on the processor, characterized in that, The processor executes the computer program to implement the multi-sensor information fusion method for a multi-target identification scene as claimed in any one of claims 1 to 4.

7. A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the multi-sensor information fusion method for a multi-target identification scene as claimed in any one of claims 1 to 4.

8. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the multi-sensor information fusion method for a multi-target identification scene as claimed in any one of claims 1 to 4.

Images