Real-time image processing method for 3D and AI visual sensing visible light movement

Abstract

The present invention relates to the field of image processing technology, and discloses a real-time image processing method for a 3D and AI visual sensing visible light movement. The method comprises: performing dual-modal channel image acquisition through an ISP image signal processing unit and a GPU acceleration unit to obtain a 3D image data stream and a visible light image data stream; performing light intensity detection and standardization processing to obtain standardized dual-modal image data; inputting a dynamic convolutional neural network for feature fusion to obtain a multi-modal fusion feature map; performing spatiotemporal contrast learning on the multi-modal fusion feature map to obtain a first feature map; inputting the first feature map into a graph structure redundancy assessment model for feature selection to obtain a second feature map; performing GPU parallel decoding and pipeline processing on the second feature map, and outputting a real-time processed image. The present invention realizes efficient real-time processing capability and meets the requirements of real-time image processing response speed.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a real-time image processing method for a 3D and AI visual sensing visible light movement. Background Art

[0002] Traditional image processing methods usually use fixed algorithms and parameter configurations, which are difficult to adapt to dynamically changing scene requirements. Especially in practical application scenarios with changing lighting conditions and complex target features, existing technologies often have problems such as poor image quality, insufficient feature extraction, and insufficient real-time performance, which seriously restricts the practical value of visual sensing systems.

[0003] In addition, current visual sensors lack an effective fusion mechanism when processing 3D data and visible light data, making it difficult to fully utilize the complementary advantages of multimodal information. At the same time, in the process of feature extraction and target recognition, due to the lack of efficient feature selection and optimization mechanisms, redundant calculations and information loss are prone to occur, affecting the overall performance and processing efficiency of the system. Summary of the invention

[0004] The present invention provides a real-time image processing method for a 3D and AI visual sensing visible light movement, which realizes efficient real-time processing capability and meets the requirements of real-time image processing response speed.

[0005] In a first aspect, the present invention provides a real-time image processing method for a 3D and AI visual sensing visible light movement, the real-time image processing method for a 3D and AI visual sensing visible light movement comprising:

[0006] Dual-modality channel image acquisition is performed through the ISP image signal processing unit and the GPU acceleration unit to obtain a 3D image data stream and a visible light image data stream;

[0007] Performing light intensity detection and standardization processing on the 3D image data stream and the visible light image data stream to obtain standardized dual-modal image data;

[0008] Inputting the standardized bimodal image data into a dynamic convolutional neural network for feature fusion to obtain a multimodal fusion feature map;

[0009] Performing spatiotemporal contrast learning on the multimodal fusion feature map to obtain a first feature map;

[0010] Inputting the first feature graph into a graph structure redundancy assessment model for feature selection to obtain a second feature graph;

[0011] GPU parallel decoding and pipeline processing are performed on the second feature map to output a real-time processed image.

[0012] In a second aspect, the present invention provides a real-time image processing device for a 3D and AI visual sensing visible light core, the real-time image processing device for a 3D and AI visual sensing visible light core comprising:

[0013] An acquisition module is used to acquire dual-modal channel images through an ISP image signal processing unit and a GPU acceleration unit to obtain a 3D image data stream and a visible light image data stream;

[0014] A standardization processing module, used for performing light intensity detection and standardization processing on the 3D image data stream and the visible light image data stream to obtain standardized dual-modal image data;

[0015] A feature fusion module, used for inputting the standardized bimodal image data into a dynamic convolutional neural network for feature fusion to obtain a multimodal fusion feature map;

[0016] A contrastive learning module, used for performing spatiotemporal contrastive learning on the multimodal fusion feature map to obtain a first feature map;

[0017] A feature selection module, used for inputting the first feature graph into a graph structure redundancy assessment model for feature selection to obtain a second feature graph;

[0018] The output module is used to perform GPU parallel decoding and pipeline processing on the second feature map and output a real-time processed image.

[0019] The third aspect of the present invention provides a real-time image processing device for a 3D and AI vision sensing visible light movement, comprising: a memory and at least one processor, wherein the memory stores instructions; the at least one processor calls the instructions in the memory to enable the real-time image processing device for the 3D and AI vision sensing visible light movement to execute the above-mentioned real-time image processing method for the 3D and AI vision sensing visible light movement.

[0020] A fourth aspect of the present invention provides a computer-readable storage medium, which stores instructions, and when the computer-readable storage medium is run on a computer, it enables the computer to execute the above-mentioned real-time image processing method of the 3D and AI visual sensing visible light movement.

[0021] In the technical solution provided by the present invention, through the deep fusion of 3D imaging and visible light imaging, combined with the adaptive illumination system and the ISP processing unit, high-quality acquisition and processing of multimodal data are achieved, and the problem of unstable image quality under a single imaging mode is solved; a dynamic convolutional neural network and a dual attention mechanism are adopted to realize adaptive extraction and enhancement of features, and improve the system's feature expression capability for complex scenes; the introduction of spatiotemporal contrast learning and automatic labeling technology reduces the dependence on a large amount of labeled data, and at the same time ensures the stability of feature learning through consistency regularization constraints; through a graph structure redundancy evaluation model and an information gain sorting mechanism, intelligent screening and optimization of features are realized, redundant calculations are reduced, and system efficiency is improved; based on GPU parallel computing and pipeline architecture, efficient real-time processing capabilities are achieved, meeting the requirements of practical applications for system response speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0023] Figure 1 A schematic flow chart of a real-time image processing method for a 3D and AI visual sensing visible light movement provided in an embodiment of the present application;

[0024] Figure 2 A schematic block diagram of the structure of a real-time image processing device for a 3D and AI visual sensing visible light core provided in an embodiment of the present application;

[0025] Figure 3 A schematic block diagram of the structure of a real-time image processing device for a 3D and AI visual sensing visible light movement provided in an embodiment of the present application. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0027] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change based on actual conditions.

[0028] It should also be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0029] It should be further understood that the term “and / or” used in the specification and appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0030] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0031] See also Figure 1 , Figure 1 A flow chart of a real-time image processing method for a 3D and AI visual sensing visible light core provided in an embodiment of the present application is shown in FIG. Figure 1 As shown, the real-time image processing method of the 3D and AI visual sensing visible light movement provided in the embodiment of the present application includes steps S100 to S600.

[0032] Step S100: performing dual-modal channel image acquisition through the ISP image signal processing unit and the GPU acceleration unit to obtain a 3D image data stream and a visible light image data stream;

[0033] It is understandable that the execution subject of the present invention can be a real-time image processing device of a 3D and AI visual sensing visible light core, or a terminal or a server, which is not limited here. The embodiment of the present invention is described by taking a server as the execution subject as an example.

[0034] Specifically, the photoelectric performance parameters of the 3D imaging unit and the visible light imaging unit are matched to obtain the spectral matching parameters. The spectral matching parameters can achieve a balance between the 3D imaging unit and the visible light imaging unit in terms of spectral response, so that the subsequent data acquisition is consistent and reliable. Based on the spectral matching parameters, the adaptive lighting system is configured with lighting parameters to ensure that the lighting system can be adaptively adjusted under different lighting conditions to generate suitable adaptive lighting control parameters. According to the spectral matching parameters and the adaptive lighting control parameters, the 3D imaging unit, the visible light imaging unit and the adaptive lighting system are integrated and connected to obtain a core module. The integration process involves optimization of signal matching, lighting adjustment and multi-modal data collaboration between systems. Through a highly integrated approach, the 3D imaging unit and the visible light imaging unit maintain consistent lighting conditions during the data acquisition process. The entire core module is time-calibrated by a clock divider and a phase-locked loop to obtain a dual-modal acquisition timing signal, so that the 3D data and the visible light data are synchronized in time to avoid data mismatch problems caused by acquisition delays. The movement module and dual-modal acquisition timing signals are input into the ISP (image signal processing) unit. The ISP processes the received signals in real time through a parallel pipeline architecture to obtain raw image data. In this process, the ISP performs basic image processing, including image denoising, white balance adjustment, and gamma correction, to obtain high-quality raw image data. The raw image data is converted into a data format to obtain image data in a standard format. The standard format image data is input into the GPU acceleration unit. The GPU parses and reorganizes these data through the CUDA parallel computing framework to obtain parallel processing data. The parallel computing capability of CUDA is used to improve the processing efficiency of image data. Especially when facing large-scale data, GPU parallel processing can significantly shorten the computing time and improve the response speed of the entire system. The parallel processing data is channel-separated and the data stream is reconstructed to output 3D image data stream and visible light image data stream respectively. Through channel separation, it is ensured that 3D and visible light image data can be processed independently in the future, so that data of different modalities can be applied to different analysis tasks in a targeted manner.

[0035] Step S200, performing light intensity detection and standardization processing on the 3D image data stream and the visible light image data stream to obtain standardized dual-modal image data;

[0036] Specifically, the illumination intensity of the 3D image data stream is calculated to obtain the illumination intensity parameters of the 3D image, and at the same time, the ambient illumination analysis is performed on the visible light image data stream to obtain the visible light ambient illumination parameters. Based on the 3D image illumination intensity parameters and the visible light ambient illumination parameters, a dual-modal illumination compensation matrix operation is performed to obtain illumination compensation parameters. The calculation of the illumination compensation matrix can effectively balance the illumination differences between images of different modalities, so that during the fusion process, the 3D image and the visible light image are consistent in illumination conditions, avoiding feature matching difficulties and fusion errors caused by uneven illumination. According to the illumination compensation parameters, the multi-angle projection transformation matrix of the 3D image data stream and the visible light image data stream is calculated to obtain a multi-view transformation matrix. By performing multi-angle projection transformation on the image, the data of each view can be expressed in the same coordinate system. Based on the multi-view transformation matrix, the 3D image data stream and the visible light image data stream are spatially reprojected and feature point matched to obtain a stereo feature point set. Spatial reprojection calculation is to uniformly map images from different perspectives so that the features of the same object from different perspectives can be accurately matched in three-dimensional space. Feature point matching is performed to ensure that the feature points between different modalities and different perspectives can correspond and calibrate with each other to obtain a set of three-dimensional feature points. The set of three-dimensional feature points is input into the GPU acceleration unit, and three-dimensional space reconstruction and feature normalization operations are performed through the CUDA parallel computing architecture to obtain normalized feature data. The use of the CUDA parallel computing architecture improves the efficiency of three-dimensional reconstruction and normalization calculations, especially when facing a large number of three-dimensional feature points. Parallel computing can greatly shorten the calculation time and meet the requirements of real-time processing. When reconstructing three-dimensional space, each three-dimensional feature point calculates its position in three-dimensional space, and these positions are further normalized to ensure that different modal features have consistent expressions at the same scale. Noise filtering is performed on the normalized feature data to remove invalid or erroneous feature points caused by acquisition errors or environmental interference. The normalized feature data after noise filtering is subjected to depth-visible light feature fusion to obtain standardized dual-modal image data. By utilizing the complementary characteristics of 3D depth information and visible light information, the fused image data has higher expressiveness and accuracy in both geometric shapes and detail features, meeting the needs of multimodal perception in complex scenes.

[0037] Step S300, inputting the standardized bimodal image data into a dynamic convolutional neural network for feature fusion to obtain a multimodal fusion feature map;

[0038] Specifically, the 3D features and visible light features are respectively input into the encoder of the dynamic convolutional neural network. The encoder includes 4 dynamic convolutional layers. The low-level and intermediate features of the 3D and visible light data are effectively extracted through the convolutional layers to obtain the initial feature map. The initial feature map is input into the spatial attention submodule of the dynamic convolutional neural network. The spatial attention submodule includes a global average pooling layer and two convolutional layers. The ReLU activation function is used between the two convolutional layers to introduce nonlinear characteristics to enhance the network's ability to express features. The feature map is aggregated spatially by the global average pooling layer, and the attention score matrix of each spatial position is calculated to determine which areas are more important for feature expression. By calculating the attention score, the spatial attention weight is obtained, which is used to weight each spatial position in the initial feature map, thereby highlighting the key areas, suppressing irrelevant background information, and enhancing the spatial feature expression. At the same time, the initial feature map is input into the channel attention submodule of the dynamic convolutional neural network. The channel attention submodule includes two fully connected layers and a Sigmoid activation function. By learning the importance coefficient of each channel, the module can calculate a weight for each channel, which reflects the importance of each channel to the overall feature representation. The role of the Sigmoid activation function is to limit the weight between 0 and 1, so as to ensure that the influence of each channel is reasonable. The final channel attention weight is used to enhance the feature channels with stronger representation ability in specific tasks and suppress the channels that are irrelevant to the task or have greater interference. The initial feature map is weighted according to the spatial attention weight and the channel attention weight to generate the attention enhancement feature. The attention enhancement feature is input into the time delay self-attention mechanism layer. The time delay self-attention mechanism layer contains a temporal encoder and three linear transformation layers. The temporal encoder encodes the temporal correlation of the feature, and the three linear transformation layers perform linear transformation and projection on the time series data to generate a temporal correlation matrix. The temporal correlation matrix is ​​used to capture the temporal dependency and dynamic changes between multiple frames of images, so that the network can not only enhance features in space, but also effectively aggregate and model the information in the time dimension to obtain a more comprehensive feature representation. Based on the temporal correlation matrix, the attention enhancement feature is aggregated by temporal features to obtain the temporal fusion feature. The temporal fusion features are input into the decoder of the dynamic convolutional neural network for decoding. The decoder contains 4 deconvolution layers, each of which is equipped with batch normalization and ReLU activation functions. Batch normalization effectively speeds up the training process and stabilizes the convergence of the network, while the ReLU activation function introduces nonlinearity, allowing the decoder to gradually restore and improve the spatial resolution of the features to obtain multi-scale features. The multi-scale features are aligned and fused across modalities, and the 3D features and visible light features are cascaded through feature cascading operations, so that the features of different modalities are aligned and fused on the same spatial scale.In order to compress and refine the fused features, the cascaded features are processed using a 1×1 convolution operation. The 1×1 convolution can effectively compress the feature channels, reduce the amount of calculation while retaining useful information, and finally output a multimodal fusion feature map.

[0039] Step S400, performing spatiotemporal contrast learning on the multimodal fusion feature map to obtain a first feature map;

[0040] Specifically, data enhancement transformation is performed on the multimodal fusion feature map to obtain multimodal enhanced features. Through various operations such as rotation, flipping, scaling and color perturbation, the diversity of features is enriched, so that the model can generalize better and enhance its adaptability to different scenes and conditions. For the multimodal features after data enhancement, temporal frame sampling is performed to extract the temporal features in the continuous video sequence, and the spatiotemporal feature pairs are constructed in time order to obtain temporal positive sample pairs. The temporal positive sample pairs retain the continuity in time and can effectively capture the dynamic characteristics of object movement or scene changes. The temporal positive sample pairs are input into the twin network structure for feature encoding. The twin network structure contains two encoders with shared weights, each of which is composed of 3 convolutional layers and 2 fully connected layers. Through this structure, the input temporal features are encoded to obtain feature embedding vectors. The shared weight encoder ensures that the network can maintain consistent representation capabilities when processing features of different time frames or perspectives, so that the encoded feature embedding has high similarity between similar samples. The feature embedding vectors are contrastively learned to optimize the representation capabilities of the features. Contrastive learning calculates the maximum mutual information between positive sample pairs and the minimum mutual information between negative sample pairs through the InfoNCE loss function to obtain contrast features. The role of the InfoNCE loss function is to make similar samples (positive sample pairs) closer in the feature space, while dissimilar samples (negative sample pairs) are far away in the feature space. In this way, the differences between features in different time frames or different perspectives are highlighted, and the recognition ability of the model in dynamic spatiotemporal changes is improved. In order to enhance the distinguishability of contrast features, the feature space of contrast features is divided into K categories by the dynamic threshold clustering algorithm, and a unique label ID is assigned to each category to obtain labeled features. The dynamic threshold clustering algorithm can adaptively adjust the clustering threshold according to the feature distribution, realize the accurate division of the feature space, and ensure that there is sufficient distinction between different categories. The consistency of the labeled features is calculated, and the difference value between the features of different views in the same time series is calculated by the mean square error function to construct the consistency constraint feature. The consistency constraint helps to align the representation of the same object under different perspectives, so that the model can maintain the consistency of feature expression when processing different perspective inputs. The constructed consistency constraint features are input into the feature metric learning network to learn the similarity measurement function between features. The feature metric learning network contains a deep separable convolution layer and an attention mechanism. The deep separable convolution layer is used to improve computational efficiency and reduce the number of parameters, while the attention mechanism can focus on important feature areas and enhance the network's ability to capture key information. Through the feature metric learning network, the similarity between different features is effectively quantified and measured to obtain metric features. The metric features are reconstructed and mapped back to the original feature space through the deconvolution network to output the first feature map.The deconvolution network restores the features after encoding, contrastive learning, clustering and metric learning to the original spatial scale, so that the reconstructed features not only retain the original image information, but also integrate the high-level information obtained by multi-layer feature extraction and learning.

[0041] Step S500, inputting the first feature graph into a graph structure redundancy assessment model for feature selection to obtain a second feature graph;

[0042] Specifically, the Euclidean distance between the feature points in the first feature graph is calculated to determine the similarity and spatial relationship between the feature points. A distance threshold is set to determine whether to establish a connection between adjacent feature points, and a weighted relationship graph is constructed. In the relationship graph, the feature points are nodes, and the edges between the nodes represent the strength of the association between them through weights, forming a complex feature relationship network. The normalized Laplace matrix is ​​calculated for the weighted relationship graph. The feature propagation matrix is ​​constructed by multiplying the inverse matrix of the degree matrix with the adjacency matrix. The feature propagation matrix is ​​used to transmit feature information in the structure of the graph, so that the relationship between adjacent feature points is quantified and expressed, and the graph structure features are obtained. The normalized Laplace matrix helps to make the feature points stable and balanced during the propagation process, thereby improving the accuracy of feature selection. The Pearson correlation coefficient is used to calculate the correlation between each pair of feature nodes in the graph structure feature, and an N×N-dimensional feature correlation matrix is ​​constructed, where N is the total number of feature nodes. The Pearson correlation coefficient quantifies the linear correlation between two feature nodes, and the obtained feature correlation matrix reflects the similarity and correlation between each node. On this basis, the information entropy is calculated to evaluate the amount of information contained in each feature node, and the mutual information formula is used to calculate the contribution of each feature node to the overall feature distribution to obtain the contribution degree. Based on the contribution degree, the information gain sequence is constructed to obtain the importance vector of each feature node. Based on the feature importance vector, nodes with information gain values ​​lower than the average value of all features are eliminated, and only feature nodes with high information content are retained to obtain an optimized feature map. By eliminating feature nodes with low contribution, redundant features are effectively reduced, computational complexity is reduced, and key information important for feature fusion and subsequent analysis is retained. The optimized feature map is input into the residual connection network, and the feature expression ability is improved by residual connection. In the structure of the residual connection network, two convolutional layers and a short-circuit connection are set in each residual block. The role of the short-circuit connection is to directly transfer the low-level feature information to the high-level feature layer, avoid the gradient vanishing problem caused by the deep network structure, and enhance the model's ability to retain detailed features. In this way, the network can learn multi-level residual features, so that the feature map has a richer representation at different levels. The multi-level residual features are modeled as feature sequences to capture the temporal changes and correlations of the features and obtain temporal correlation features. The attention weight coefficients between the nodes of the temporal correlation features are calculated through the graph attention network. The graph attention network can focus on the most representative feature nodes by adaptively allocating weights, so that nodes with higher correlation can be considered when aggregating features. Based on the calculated attention weight coefficients, the neighborhood features are weighted aggregated so that the feature information is effectively integrated in both space and time, and the second feature map is finally output.

[0043] Step S600: Perform GPU parallel decoding and pipeline processing on the second feature map, and output a real-time processed image.

[0044] Specifically, according to the CUDA parallel computing framework, thread blocks are allocated to the second feature map, 16×16 threads are set for each thread block, and an independent feature data access space is allocated to each thread to form a GPU parallel computing grid. The structural design of the parallel computing grid can make full use of the parallel processing capability of the GPU, divide the feature map into multiple computing units to speed up data processing, and ensure that calculation and decoding can be performed quickly in real-time application scenarios. The constructed GPU parallel computing grid is input into the three-level pipeline scheduler. By setting a command queue control unit on the CPU side, the tasks of the three stages of decoding, processing and output are pipelined to obtain a pipeline task queue. The design of pipeline scheduling enables the tasks of each stage to be executed in parallel and in order, thereby improving the overall efficiency of the system. Through the pipeline mode, it is ensured that while the decoding operation is being performed, the processing tasks and output tasks are also performed in parallel, thereby improving the real-time performance of image processing. In the pipeline task queue, the feature data is decoded, and 8 parallel decoding units are set to perform dequantization and inverse transformation operations at the same time. Each decoding unit is responsible for restoring a specific target data block to obtain the original feature data. The decoded raw feature data is divided into multiple overlapping grids according to the spatial distribution. The overlap rate of each overlapping grid is set to 0.2. The overlapping division can reduce the edge missed detection in the target detection process. The preset target detection convolution kernel is used to extract the features of the overlapping grids to obtain a set of candidate regions. The function of the convolution kernel is to identify the key patterns in the feature data, so as to mark the candidate regions containing the target in the image. The non-maximum suppression operation is performed on the candidate regions. By calculating the intersection over union (IoU) between the candidate target frames and setting the IoU threshold to 0.5, the detection frames with an overlap rate higher than the threshold are merged to obtain the target detection frame. Non-maximum suppression can reduce repeated detection frames and retain only the most representative detection results to improve the accuracy of target detection and reduce redundant detection. The target detection frame is input into the multi-classification scoring network, and the probability score of each category is calculated by the Softmax activation function. For each target frame, the category it belongs to is determined according to the probability scores of different categories, and the confidence threshold is 0.75. Only target frames with scores higher than the threshold are considered valid target frames. For the effective target frame, target tracking calculation is performed, the target's motion state is predicted by the Kalman filter, and the target features are continuously updated in combination with the target's appearance model to obtain the target tracking frame. The Kalman filter can effectively predict the position of the target in real-time video processing, especially in the presence of motion blur or short-term occlusion, to maintain continuous tracking of the target. The target features are updated in combination with the appearance model to improve the target re-identification capability, so that the system can effectively track multiple targets in complex scenes.The target tracking frame is superimposed on the original image, and the image edge is smoothed by the image post-processing unit, and necessary annotation information is added, such as target category and confidence, so as to output a real-time processed image. Edge smoothing helps to improve the visual effect of the superimposed image, making the target annotation more naturally integrated with the background. At the same time, the added annotation information provides users with detailed recognition results of the target.

[0045] In the embodiment of the present invention, through the deep fusion of 3D imaging and visible light imaging, combined with the adaptive illumination system and the ISP processing unit, high-quality acquisition and processing of multimodal data are achieved, and the problem of unstable image quality under a single imaging mode is solved; a dynamic convolutional neural network and a dual attention mechanism are adopted to achieve adaptive extraction and enhancement of features, and improve the system's feature expression capability for complex scenes; the introduction of spatiotemporal contrast learning and automatic labeling technology reduces the dependence on a large amount of labeled data, and at the same time ensures the stability of feature learning through consistency regularization constraints; through a graph structure redundancy evaluation model and an information gain sorting mechanism, intelligent screening and optimization of features are achieved, redundant calculations are reduced, and system efficiency is improved; based on GPU parallel computing and pipeline architecture, efficient real-time processing capabilities are achieved, meeting the requirements of practical applications for system response speed.

[0046] In a specific embodiment, the process of executing step S100 may specifically include the following steps:

[0047] Matching the photoelectric performance parameters of the 3D imaging unit and the visible light imaging unit to obtain spectral matching parameters, and configuring the illumination parameters of the adaptive illumination system based on the spectral matching parameters to obtain adaptive illumination control parameters;

[0048] According to the spectrum matching parameters and the adaptive illumination control parameters, the 3D imaging unit, the visible light imaging unit and the adaptive illumination system are integrated and connected to obtain a core module, and the core module is time-calibrated through a clock divider and a phase-locked loop to obtain a dual-mode acquisition timing signal;

[0049] The movement module and dual-mode acquisition timing signals are input into the ISP image signal processing unit, and real-time signal processing is performed through a parallel pipeline architecture to obtain raw image data, and the raw image data is converted into a data format to obtain standard format image data;

[0050] The standard format image data is input into the GPU acceleration unit, and the data is parsed and reorganized through the CUDA parallel computing framework to obtain parallel processing data. The parallel processing data is then channel-separated and the data stream is reconstructed to output 3D image data stream and visible light image data stream respectively.

[0051] Specifically, the photoelectric performance parameters of the 3D imaging unit and the visible light imaging unit are matched to ensure that the two can work together under different lighting conditions. The photoelectric parameters used include spectral response range, signal gain, exposure time, and noise level. For example, assuming that the spectral response range of the 3D imaging unit is , and the spectral response range of the visible light imaging unit is In order to obtain similar imaging effects for the two in the same scene, find the overlapping spectral range between them . Through the spectral matching function Describe this spectral similarity, for example:

[0052] ;

[0053] in, and They represent the spectral response functions of the 3D imaging unit and the visible light imaging unit, respectively. and are the minimum and maximum wavelengths of the overlapping spectra of the two, respectively. By calculating , a parameter indicating the degree of spectral matching between the two is obtained, which is used to guide the subsequent lighting configuration. Based on the spectral matching parameter , configure the illumination parameters of the adaptive illumination system to ensure that the light source can adapt to the spectral requirements of the imaging unit under different environmental conditions. For example, suppose the illumination intensity of the adaptive illumination system is , while the optimal response intensity of the 3D imaging unit and the visible light imaging unit to light intensity is and , by adjusting the intensity of the lighting system, the output light intensity is matched with the response range of the two, which is expressed as:

[0054] ;

[0055] in, and is the weight coefficient, which depends on the result of the spectral matching function By adjusting and The value of ensures that the output of the adaptive lighting control system can maximize the response of the imaging unit and obtain the best image quality. The configured lighting parameters are the adaptive lighting control parameters. According to the spectral matching parameters and the adaptive lighting control parameters, the 3D imaging unit, the visible light imaging unit and the adaptive lighting system are integrated and connected to obtain the core module. During the integration process, the consistency of the physical connection, signal transmission path and power supply of each imaging unit and the lighting system is guaranteed. In order to ensure the time synchronization between the 3D imaging unit and the visible light imaging unit, the timing of the core module is calibrated by using a clock divider and a phase-locked loop. Assume that the clock signal of the 3D imaging unit is , the clock signal of the visible light imaging unit is , in order to achieve dual-mode synchronization, the output frequency of the clock divider and the phase-locked loop is expressed as:

[0056] ;

[0057] in, express and The greatest common divisor of is obtained in this way, and the timing signal of dual-modal acquisition is obtained to ensure the time consistency of the data collected by the imaging unit. The movement module and the dual-modal acquisition timing signal are input into the ISP (image signal processing) unit, and real-time signal processing is performed through the parallel pipeline architecture to obtain the original image data. The pipeline architecture of the ISP unit includes multiple image processing stages, such as noise reduction, white balance, gamma correction, etc. These steps can significantly improve the quality of the image. For example, for the gamma correction of each pixel, it is expressed as:

[0058] ;

[0059] in, represents the input pixel intensity, is the gamma value, is the output pixel intensity. Through pipeline parallel processing, ISP can perform multi-level processing on the input image data and finally obtain high-quality raw image data. The raw image data is converted into a data format to obtain image data in a standard format. For example, the raw RGB image data is converted into a YCbCr format to improve the processing capability of the chrominance and brightness components. Assume that the red, green, and blue components in the RGB image are , , , the standard format YCbCr conversion is expressed as:

[0060] ;

[0061] ;

[0062] ;

[0063] Through the above conversion formula, RGB data is converted to brightness ( ) and two chromaticities ( , ) components. Such conversion helps to separate brightness information and chrominance information more easily in subsequent processing. The standard format image data is input to the GPU acceleration unit, and the data is parsed and reorganized through the CUDA parallel computing framework to obtain parallel processing data. In the CUDA framework, the data is divided into multiple thread blocks, each of which contains 16x16 threads, and each thread is responsible for processing an independent data unit. Assume that the total data size is , the number of thread blocks is , the number of threads in each thread block is ,but:

[0064] ;

[0065] In the CUDA computing framework, all thread blocks are executed simultaneously to improve the efficiency of data processing. During the parsing and reorganization process, the GPU performs necessary preprocessing on the input data, such as downsampling, quantization, and rearrangement, to ensure that the data can be efficiently processed according to the requirements of subsequent operations. The parallel processing data is channel-separated and the data stream is reconstructed to output the 3D image data stream and the visible light image data stream respectively. The channel separation process is achieved by assigning different processing paths to different types of feature data. Assume that the 3D image data contains depth information , visible light images contain brightness information , then the channel separation process is expressed as:

[0066] ;

[0067] ;

[0068] in, and It is a mapping function for channel separation. Through separation, 3D data and visible light data can be processed independently. Data stream reconstruction rearranges data from different channels in time and space so that the data of the two modalities have the same reference coordinate system in the same time series, which is convenient for subsequent fusion and analysis.

[0069] In a specific embodiment, the process of executing step S200 may specifically include the following steps:

[0070] Calculating the illumination intensity of the 3D image data stream to obtain the 3D image illumination intensity parameter, and performing ambient illumination analysis on the visible light image data stream to obtain the visible light ambient illumination parameter;

[0071] Performing a dual-mode illumination compensation matrix operation based on the 3D image illumination intensity parameter and the visible light environment illumination parameter to obtain an illumination compensation parameter, and performing a multi-angle projection transformation matrix calculation on the 3D image data stream and the visible light image data stream according to the illumination compensation parameter to obtain a multi-viewing angle transformation matrix;

[0072] Perform spatial reprojection calculation and feature point matching on the 3D image data stream and the visible light image data stream according to the multi-view transformation matrix to obtain a three-dimensional feature point set;

[0073] The stereo feature point set is input into the GPU acceleration unit, and the three-dimensional space reconstruction and feature normalization operations are performed through the CUDA parallel computing architecture to obtain normalized feature data. The normalized feature data is then subjected to noise filtering and depth-visible light feature fusion to obtain standardized bimodal image data.

[0074] Specifically, the illumination characteristics of each 3D pixel are evaluated. Suppose the pixel of the 3D image data stream is , which contains depth information and reflection intensity. The key to calculating the illumination intensity is to estimate the illumination intensity of the entire scene by the reflected light intensity of each pixel. Assuming that the reflected light intensity of each pixel is , then the illumination intensity parameter of the entire 3D image is Calculated by averaging the light intensity of all pixels:

[0075] ;

[0076] in, is the number of all pixels in the 3D image, and Represent the width and height of the image respectively. The light intensity parameter reflects the average brightness level of the 3D image in the entire scene. At the same time, the ambient light analysis is performed on the visible light image data stream to obtain the ambient light parameters. For visible light images, the intensity and color temperature of ambient light are evaluated by analyzing the color histogram and pixel brightness of the image. Assume that the pixel brightness of the visible light image is , the ambient lighting parameters are calculated by taking the mean brightness of all pixels:

[0077] ;

[0078] in, is the number of pixels in the visible light image, Represents the brightness value of a pixel in a two-dimensional image. Describes the overall brightness level of a visible light scene and provides ambient lighting information for visible light images. Based on 3D image illumination intensity parameters and visible light ambient lighting parameters , perform dual-modal illumination compensation matrix operations to obtain illumination compensation parameters. Coordinate the imaging differences of the two modalities under different illumination conditions to improve the effect of feature fusion. Suppose the illumination compensation matrix is , calculated according to the following formula:

[0079] ;

[0080] in, Is a balance coefficient used to adjust the illumination difference between the two modalities. By calculating the illumination compensation matrix, the illumination intensity of the 3D image and the visible light image is compensated so that both are processed under similar illumination levels. The multi-angle projection transformation matrix is ​​calculated for the 3D image data stream and the visible light image data stream to obtain the multi-view transformation matrix. Assume that the transformation matrix is , through the illumination compensation matrix And the viewing angle parameters are defined:

[0081] ;

[0082] in, Represents a rotation matrix, which is used to rotate an image data stream from one view to another; is a translation matrix, used to translate the image in three-dimensional space; is the illumination compensation matrix calculated previously. Through multi-angle projection transformation, image representations under different viewing angles are obtained. According to the multi-view transformation matrix, the 3D image data stream and the visible light image data stream are spatially reprojected and feature point matched to obtain a set of three-dimensional feature points. In the process of spatial reprojection, the 3D image data is transformed according to the multi-view transformation matrix. Perform coordinate transformation to obtain new projection coordinates. Let the original 3D point coordinates be , then the transformed coordinates are :

[0083] ;

[0084] in, for The projection matrix is ​​used to realize the multi-view transformation of 3D space points. In this way, the feature points under different viewpoints are projected and matched to obtain a consistent set of three-dimensional feature points. The three-dimensional feature point set is input into the GPU acceleration unit, and the three-dimensional space reconstruction and feature normalization operations are performed through the CUDA parallel computing architecture to obtain normalized feature data. The three-dimensional reconstruction process is mainly based on the spatial position of the three-dimensional feature points to regenerate the three-dimensional representation of the scene. Set the coordinates of the three-dimensional feature points as , the goal of 3D reconstruction is to reconstruct the surface shape of the object through these feature points. Through CUDA's parallel computing, each thread is responsible for processing a feature point, and the 3D reconstruction is quickly completed using the position and depth information of the feature point. The parallel processing method improves the computing speed, especially in the scenario of large-scale feature points. After the 3D space reconstruction is completed, the feature data is normalized to ensure that the coordinates and intensity values ​​of all feature points are represented in a uniform scale. The normalization process is expressed as:

[0085] ;

[0086] in, , , are the mean values ​​of the feature point coordinates, , , are the standard deviations of the coordinates respectively. Through normalization, the errors caused by scale and offset are eliminated, making the feature data more stable and consistent. The normalized feature data is subjected to noise filtering and depth-visible light feature fusion to obtain standardized bimodal image data. The purpose of noise filtering is to remove invalid feature points caused by acquisition errors or environmental interference, and use statistical methods such as mean filtering or median filtering to smooth the coordinates and intensity of feature points to remove isolated noise points. In the process of feature fusion, the 3D depth features and visible light features are combined according to the spatial position to generate more expressive fusion features. For example, suppose the depth feature is , the visible light characteristics are , fusion features It is expressed as:

[0087] ;

[0088] in, and is the fusion coefficient, which is used to adjust the weights of the depth feature and the visible light feature. In this way, the fused features retain the depth information of the 3D image and contain the detail features of the visible light image, forming standardized dual-modal image data.

[0089] In a specific embodiment, the process of executing step S300 may specifically include the following steps:

[0090] The standardized bimodal image data is input into the encoder of the dynamic convolutional neural network according to the 3D features and the visible light features, respectively. The encoder includes 4 dynamic convolutional layers to obtain the initial feature map.

[0091] The initial feature map is input into the spatial attention submodule of the dynamic convolutional neural network. The spatial attention submodule consists of a global average pooling layer and two convolutional layers. The ReLU activation function is used between the two convolutional layers. The spatial attention weight is obtained by calculating the attention score matrix of each spatial position of the feature map.

[0092] The initial feature map is input into the channel attention submodule of the dynamic convolutional neural network. The channel attention submodule contains two fully connected layers and a Sigmoid activation function. The channel attention weight is obtained by learning the importance coefficient of each channel.

[0093] The initial feature map is weighted according to the spatial attention weight and the channel attention weight to obtain the attention-enhanced feature, and the attention-enhanced feature is input into the time-delayed self-attention mechanism layer, which contains a temporal encoder and three linear transformation layers to generate a temporal correlation matrix;

[0094] Based on the temporal correlation matrix, the attention enhancement features are aggregated to obtain temporal fusion features, and the temporal fusion features are input into the decoder of the dynamic convolutional neural network. The decoder contains 4 deconvolution layers, each of which uses batch normalization and ReLU activation function to obtain multi-scale features.

[0095] The multi-scale features are aligned and fused across modalities, and the 3D features and visible light features are merged through feature cascading and 1×1 convolution operations to output a multi-modal fusion feature map.

[0096] Specifically, the standardized bimodal image data is input into the encoder of the dynamic convolutional neural network according to the 3D features and visible light features. The encoder contains 4 dynamic convolutional layers to extract feature information from different modal data. Assume that the standardized 3D features are and visible light characteristics are , the input of the encoder is these two feature tensors. Each dynamic convolution layer extracts features through convolution operation. Let the convolution kernel be , the output of the convolutional layer It is expressed as:

[0097] ;

[0098] in, Indicates The convolution kernel weight of the layer, * is the convolution operator, represents the input feature map, is the bias term. In the encoder, the convolution kernels of each layer are dynamic, that is, they are automatically adjusted according to the characteristics of the input data to extract features more flexibly and obtain the initial feature map . Map the initial feature The input is fed into the spatial attention submodule of the dynamic convolutional neural network to enhance the key spatial positions in the feature map, thereby more accurately locating the feature area of ​​the object. The spatial attention submodule contains a global average pooling layer and two convolutional layers, where the ReLU activation function is used between the two convolutional layers. The global average pooling layer is used to Perform spatial average calculation on each channel to obtain the global feature vector:

[0099] ;

[0100] in, and are the height and width of the input feature map respectively. The global feature vector It can capture the global information of all spatial positions in the feature map, and then convolve the global features through two convolutional layers to obtain the spatial attention score matrix :

[0101] ;

[0102] in, is the Sigmoid activation function, and are the weights of the first and second convolutional layers respectively. The spatial attention weight is obtained by calculating the attention score matrix for each spatial position: , used to highlight the key spatial area. At the same time, the initial feature map The input is sent to the channel attention submodule of the dynamic convolutional neural network. The channel attention submodule learns the importance of each channel and obtains the channel attention weight. This module contains two fully connected layers and a Sigmoid activation function. A global pooling operation is performed on the input feature map to obtain the channel description vector , and then transform the channel description through two fully connected layers to obtain the channel attention weight :

[0103] ;

[0104] in, and are the weights of the two fully connected layers, is the Sigmoid activation function, which limits the weight value between 0 and 1, so as to perform weighted calculation on the importance of each channel. and channel attention weights For the initial feature map Weighted to get attention enhancement features The weighted calculation is expressed as:

[0105] ;

[0106] in, Represents element-by-element multiplication, and by using both spatial and channel attention, the feature expression capabilities of key areas and key channels are enhanced. The input is sent to the time-delayed self-attention mechanism layer, which contains a time encoder and three linear transformation layers. The time encoder is used to capture the changing relationship of features in the time dimension, thereby generating time-related features. Let the time feature be , through the timing encoder Encode and get a time series representation:

[0107] ;

[0108] Then, the time series features are linearly projected through three linear transformation layers to obtain the time series correlation matrix :

[0109] ;

[0110] in, , , are the weights of the linear transformation layer. The temporal correlation matrix is ​​used to describe the correlation between different time frames, so that time-related features can be effectively aggregated in subsequent operations. , for attention enhancement features Aggregate time series features to obtain time series fusion features The time series feature aggregation is achieved through weighted averaging:

[0111] ;

[0112] Through the aggregation operation, the features in the time dimension are effectively combined, so that the feature expression is not only rich in space and channels, but also has dynamic characteristics in the time dimension. The input is sent to the decoder of the dynamic convolutional neural network. The decoder contains 4 deconvolution layers, each of which uses batch normalization and ReLU activation function. In the decoder, each deconvolution layer gradually enlarges the spatial dimension of the feature map through deconvolution operation and reconstructs the original resolution. Let the deconvolution kernel be , the output of the decoder It is expressed as:

[0113] ;

[0114] Among them, BatchNorm represents the batch normalization operation, and ReLU is the activation function. Through multi-layer deconvolution and batch normalization, multi-scale features are obtained. , these features contain rich information at different spatial scales. , perform cross-modal feature alignment and fusion to achieve full integration of 3D features and visible light features. and visible light characteristics Splice to get fusion features :

[0115] ;

[0116] Among them, [;] represents the feature cascade operation, in this way, the features of the two modes are combined in the channel dimension. In order to compress the feature channel and improve the computational efficiency, the cascaded features are fused through a 1×1 convolution operation to obtain a multi-modal fusion feature map :

[0117] ;

[0118] in, represents a 1×1 convolution kernel, is the bias term. Through 1×1 convolution, the feature information of 3D and visible light is effectively integrated and the number of channels is reduced, resulting in a more compact and information-rich feature representation.

[0119] In a specific embodiment, the process of executing step S400 may specifically include the following steps:

[0120] Perform data enhancement transformation on the multimodal fusion feature map to obtain multimodal enhanced features, perform time-series frame sampling on the multimodal enhanced features, extract time-series features from continuous video sequences, construct spatiotemporal feature pairs in time-series order, and obtain time-series positive sample pairs;

[0121] The time series positive sample pairs are input into the twin network structure, and the features are encoded through two encoders with shared weights. Each encoder contains three convolutional layers and two fully connected layers to obtain the feature embedding vector.

[0122] Perform contrastive learning on the feature embedding vectors, and use the InfoNCE loss function to maximize the mutual information between positive sample pairs and minimize the mutual information between negative sample pairs to obtain contrastive features;

[0123] The feature space of the contrast feature is divided into K categories through the dynamic threshold clustering algorithm, and a unique label ID is assigned to each category to obtain the labeled feature. The consistency of the labeled feature is calculated, and the difference value of the features of different views in the same time series is calculated through the mean square error function to construct the consistency constraint feature.

[0124] Input the consistency constraint features into the feature metric learning network, and learn the similarity measurement function between features through the feature metric learning network. The feature metric learning network includes a deep separable convolutional layer and an attention mechanism to obtain metric features;

[0125] The quantized features are reconstructed, the features are mapped back to the original feature space through the deconvolution network, and the first feature map is output.

[0126] Specifically, the multimodal fusion feature map is subjected to data enhancement transformation, and multimodal enhanced features are obtained through operations such as rotation, translation, flipping, scaling, and color perturbation. Data enhancement methods help improve the robustness and generalization ability of feature learning. The multimodal enhanced features are sampled in time series frames to extract time series features from continuous video sequences, thereby establishing spatiotemporal associations. Suppose the enhanced features of each frame in the video sequence are ,in Represents the index of the time frame. In order to perform time series analysis, the spatiotemporal feature pairs are constructed in chronological order. Assume that between consecutive time frames, adjacent frames are selected to form positive sample pairs, which are defined as , and obtain the time series positive sample pairs. The construction of positive sample pairs is to capture the feature changes between consecutive time frames, so that the model can better understand the relationship between dynamic features. Input the time series positive sample pairs into the twin network structure. The twin network structure consists of two encoders with shared weights, each of which contains 3 convolutional layers and 2 fully connected layers. Let the two inputs of the twin network be the features in the positive sample pair and , the convolutional layer output of each encoder is:

[0127] ;

[0128] in, , , are the weights of the three-layer convolution kernels, * represents the convolution operation, , , is the corresponding bias term, and ReLU is the activation function. The features are abstracted layer by layer through three convolutional layers to obtain a more expressive feature representation. The convolutional features are input into the fully connected layer for embedding. The weight of the fully connected layer is set to and , the feature embedding vector is:

[0129] ;

[0130] Through these two fully connected layers, a compact representation of the features is obtained, namely the feature embedding vector. The feature embedding vector is subjected to comparative learning. By maximizing the similarity between positive sample pairs and minimizing the similarity between negative sample pairs, the model's ability to discriminate features is improved. The InfoNCE loss function is used for optimization. Assume that the feature embedding of the positive sample pair is and , the negative sample pairs are other randomly sampled feature embeddings, then the InfoNCE loss is expressed as:

[0131] ;

[0132] in, Represents the similarity between features, calculated using cosine similarity. is the temperature coefficient, used to adjust the sensitivity of the similarity, Represents the number of negative samples. By maximizing the similarity between positive sample pairs, the model learns more temporal information and obtains contrast features. In order to classify the contrast features, the feature space is divided into categories, and assign a unique label ID to each category. Let the contrast feature be , obtained by dynamic threshold clustering The label vector of the categories The purpose of dynamic threshold clustering is to dynamically determine the clustering threshold according to the distribution of features in high-dimensional space, so as to adaptively adjust the clustering results. In this way, similar features are effectively divided into the same category to obtain labeled features. The consistency of the labeled features is calculated, and the difference between the features of different views in the same time series is calculated by the mean square error function, so as to construct the consistency constraint feature. Let the labeled feature be and , the mean square error is expressed as:

[0133] ;

[0134] in, The dimension of the feature is constructed by calculating the feature differences of different views in the same time series to build consistency constraints so that similar features maintain high consistency. The consistency constraint features are input into the feature metric learning network for similarity metric learning. The network contains a deep separable convolution layer and an attention mechanism. The purpose of deep separable convolution is to reduce the amount of calculation while maintaining the expressiveness of the features. Let the convolution kernel of deep separable convolution be , the input features are , the output features are:

[0135] ;

[0136] in, is the depth convolution kernel, is a point-by-point convolution kernel, which convolves each channel through depth convolution and then combines them in the channel dimension through point-by-point convolution, thereby reducing the number of parameters and computational complexity. At the same time, the attention mechanism highlights the key features by learning the importance between features. Let the attention weight be , the final metric feature is:

[0137] ;

[0138] in, represents element-by-element multiplication, and the attention weight increases the focus on key features to obtain more accurate metric features. The metric features are reconstructed and mapped back to the original feature space through the deconvolution network to output the first feature map. The deconvolution network consists of multiple layers of deconvolution, and each layer gradually increases the spatial size of the feature through deconvolution. Let the deconvolution kernel be , the input features are , the output feature map is:

[0139] ;

[0140] Through multi-layer deconvolution operations, the high-dimensional feature embedding is restored to the original image feature space, and finally the first feature map is output. .

[0141] In a specific embodiment, the process of executing step S500 may specifically include the following steps:

[0142] Calculate the Euclidean distance between feature points in the first feature graph, and build edge connections between adjacent feature points based on the distance threshold to obtain a weighted relationship graph;

[0143] The normalized Laplace matrix is ​​calculated for the weighted relationship graph, and the feature propagation matrix is ​​constructed by multiplying the inverse matrix of the degree matrix with the adjacency matrix to obtain the graph structure features;

[0144] The Pearson correlation coefficient is used to calculate the correlation between each pair of feature nodes in the graph structure feature, and an N×N-dimensional feature correlation matrix is ​​constructed, where N is the total number of feature nodes, to obtain the feature correlation value;

[0145] The information entropy of the feature correlation value is calculated, and the contribution of each feature node to the overall feature distribution is calculated by the mutual information formula to obtain the contribution degree. The information gain sequence is constructed according to the contribution degree to obtain the feature importance vector.

[0146] Feature selection is performed based on the feature importance vector, nodes with information gain values ​​lower than the average value of all features are removed, and feature nodes with high information content are retained to obtain an optimized feature graph;

[0147] The optimized feature map is input into the residual connection network. By setting two convolutional layers and a short-circuit connection in each residual block, the low-level feature information is directly transferred to the high-level feature layer to obtain multi-level residual features.

[0148] The multi-level residual features are modeled as feature sequences to obtain time-series correlation features. The attention weight coefficients between nodes of the time-series correlation features are calculated through the graph attention network. The neighborhood features are weightedly aggregated based on the attention weight coefficients to output the second feature graph.

[0149] Specifically, the Euclidean distance between feature points is calculated. Suppose the feature points in the first feature map are and , Euclidean distance Calculated by the following formula:

[0150] ;

[0151] in, and Represents the feature points and By calculating the Euclidean distance between each pair of feature points, we get the distance matrix between feature points. In order to construct a weighted relationship graph, we introduce a distance threshold. , if the feature point and The distance between Less than threshold , then establish an edge connection between the two feature points and divide the distance As the weight of the edge. Thus, a weighted relationship graph is constructed ,in, Represents a collection of feature nodes represents the set of edges, is the edge weight matrix. Calculate the normalized Laplace matrix of the weighted relationship graph. Construct the degree matrix , whose diagonal elements Representation and Node The sum of the weights of the connected edges:

[0152] ;

[0153] in, Representation Node and nodes The edge weights between . The adjacency matrix Defined as the connection relationship between nodes, if the node and nodes If there is an edge connection between ,otherwise . Normalized Laplacian matrix It is constructed by multiplying the inverse matrix of the degree matrix and the adjacency matrix. The formula is as follows:

[0154] ;

[0155] Through normalization, we get the feature propagation matrix, which is used to represent the relationship between feature points in the graph and their information propagation capabilities, and get the graph structure features. Through the graph structure features, we calculate the correlation between each pair of feature nodes. We use the Pearson correlation coefficient to measure the correlation between nodes. Let the feature node and nodes The eigenvectors of and , its correlation coefficient It is expressed as:

[0156] ;

[0157] in, and The eigenvectors are and In the The value of the dimension, and Represents the mean of the feature vector respectively. By calculating the Pearson correlation coefficient between each pair of feature nodes, we construct dimensional feature correlation matrix, where is the total number of feature nodes. The information entropy is calculated for the feature-related values ​​to measure the contribution of each feature node to the overall feature distribution. The correlation is , information entropy It is expressed as:

[0158] ;

[0159] in, Characteristic node The probability of correlation with other nodes is defined as Information Entropy Reflect feature nodes The contribution of each feature node to the overall feature distribution is calculated using the mutual information formula. :

[0160] ;

[0161] in, is the conditional entropy, which describes the conditional entropy of a node given other nodes. By calculating the contribution of each feature node, we get the information gain sequence, which is used to describe the importance of each node. We build a feature importance vector based on the information gain sequence, sort the feature nodes according to their contribution, and remove nodes whose information gain values ​​are lower than the average value of all features. Only feature nodes with high information content are retained to obtain the optimized feature graph. . The feature map will be optimized Input to the residual connection network to enhance the expressiveness of features. The residual connection network sets two convolutional layers and a short-circuit connection in each residual block to directly transfer low-level feature information to the high-level feature layer, alleviating the gradient vanishing problem in the deep network. Suppose the input feature is , the convolutional layer weight is and , the output of the residual block It is expressed as:

[0162] ;

[0163] Through residual connection, the multi-level information of features is maintained to obtain multi-level residual features. Feature sequence modeling is performed on multi-level residual features to capture the changing relationship of features in the time dimension and obtain time series correlation features. The attention weight coefficients between nodes in the time series correlation features are calculated through the graph attention network. Assume that the time series correlation feature is , the attention weight coefficient of the graph attention network is , calculated by the following formula:

[0164] ;

[0165] in, and is the weight matrix of query and key, is the dimension of the key, and softmax is used to normalize the weight coefficient. By calculating the attention weight coefficient, the importance of each feature node to the neighboring node in the feature aggregation process is determined. Based on the attention weight coefficient, the neighborhood features are weighted aggregated to obtain the aggregated features. :

[0166] ;

[0167] Through weighted aggregation, the neighborhood feature information can be effectively integrated to output the second feature map , which includes complex associations in time and space dimensions and the integration of neighborhood information.

[0168] In a specific embodiment, the process of executing step S600 may specifically include the following steps:

[0169] According to the CUDA parallel computing framework, thread blocks are allocated to the second feature map, 16×16 threads are set in each thread block, and an independent feature data access space is allocated to each thread to obtain a GPU parallel computing grid;

[0170] The GPU parallel computing grid is input into the three-level pipeline scheduler. By setting a command queue control unit on the CPU side, the tasks of the three stages of decoding, processing and output are pipelined to obtain a pipeline task queue.

[0171] Decode the feature data in the pipeline task queue, and set 8 parallel decoding units to perform inverse quantization and inverse transformation operations simultaneously. Each decoding unit is responsible for restoring the target data block to obtain the original feature data.

[0172] The original feature data is divided into multiple overlapping grids according to the spatial distribution. The overlapping rate of each overlapping grid is set to 0.2. The feature is extracted through the preset target detection convolution kernel to obtain a set of candidate regions.

[0173] Perform non-maximum suppression on the candidate region set, calculate the intersection over union ratio of the target frame and set the IoU threshold to 0.5, merge the detection frames with overlap rates higher than the threshold to obtain the target detection frame;

[0174] The target detection frame is input into the multi-classification scoring network, and the probability score of each category is calculated through the Softmax activation function. The confidence threshold is set to 0.75 to obtain the valid target frame, and the target tracking calculation is performed on the valid target frame. The target motion state is predicted through the Kalman filter, and the target features are updated in combination with the appearance model to obtain the target tracking frame.

[0175] The target tracking frame is superimposed on the original image, the edge is smoothed and annotation information is added through the image post-processing unit, and the real-time processed image is output.

[0176] Specifically, the second feature graph is allocated thread blocks according to the CUDA parallel computing framework. Each thread block is set to threads, which means that there are 256 threads in each thread block, which perform computing tasks in parallel to improve processing efficiency. Suppose the data of the second feature map is , whose size is ,in , and Represent the width, height and number of channels of the image respectively. By dividing the feature map into multiple The GPU parallel computing grid is input into the three-level pipeline scheduler, and the command queue control unit is set on the CPU side to perform pipeline scheduling on the tasks of the three stages of decoding, processing and output to obtain the pipeline task queue. The purpose of three-level pipeline scheduling is to achieve parallel work between different stages. Assume that the pipeline stages are decoding, processing and output respectively. The pipeline scheduler divides the operations of these stages into independent tasks and adds them to the task queue so that the tasks can be seamlessly connected between different computing units to achieve the purpose of improving processing efficiency. Through this pipeline mode, the system can perform feature processing and output tasks in parallel in other stages while decoding in a certain stage, thereby maximizing the use of computing resources. Decode the feature data in the pipeline task queue. By setting 8 parallel decoding units, these units perform inverse quantization and inverse transformation operations at the same time. Assume that the feature data block is , each decoding unit performs inverse quantization and inverse transformation operations on it. The inverse quantization operation is expressed as:

[0177] ;

[0178] in, represents the dequantization operation, is the quantization factor, and the original value of the data is restored by inverse quantization. The quantized data is converted into a spatial domain representation through inverse transformation. Let the inverse transformation matrix be , then the inverse transformation is expressed as:

[0179] ;

[0180] In this way, the original feature data is restored The original feature data is divided into multiple overlapping grids according to the spatial distribution, and the overlap rate of each overlapping grid is set to 0.2, so as to reduce the detection omissions at the target boundary. Assume that the size of the feature map is , the grid size is , then the size of the overlapping area is , ensuring that features near the boundary can be detected multiple times, improving the accuracy of target detection. Through the preset target detection convolution kernel, the overlapping grids are extracted to obtain a set of candidate regions. Assume that the convolution kernel is , then the convolution operation is expressed as:

[0181] ;

[0182] in, Represents a set of candidate regions, and uses convolution operations to identify regions containing targets in the feature map. For the set of candidate regions, non-maximum suppression operations are performed to remove redundant detection frames. The intersection over union (IoU) between target frames is calculated, and detection frames with an overlap ratio higher than a threshold are merged. Suppose two detection frames and The intersection-and-union ratio is:

[0183] ;

[0184] in, and Respectively represent the detection box and The area of represents the intersection of two regions, represents the union of two regions. If , then merge the two detection frames to get a more accurate target detection frame The target detection box is input into the multi-classification scoring network, and the probability score of each category is calculated through the Softmax activation function. Let the output of the classification scoring network be the vector , where each element Indicates that the target belongs to The probability of the class, then the Softmax function is expressed as:

[0185] ;

[0186] in, Indicates The score of the class, is the total number of categories. The output is converted into a probability distribution through the Softmax function, and the confidence threshold is set to 0.75. Only the target boxes with probability greater than the threshold are retained to obtain the valid target boxes. For the valid target boxes, target tracking calculation is performed. The Kalman filter is used to predict the motion state of the target. The state update equation of the Kalman filter is expressed as:

[0187] ;

[0188] ;

[0189] in, is the current state vector, is the control input, is the process noise, is the measured value, , and are the state transfer matrix, control matrix and observation matrix. The motion trajectory of the target is predicted through the Kalman filter, and the target features are updated in combination with the appearance model to obtain a stable target tracking frame. The target tracking frame is superimposed on the original image, and the image edge is smoothed by the image post-processing unit to improve the visual effect. The purpose of edge smoothing is to remove the sharp edges caused by target superposition, making the image more natural and coherent. Edge smoothing methods include Gaussian blur and bilateral filtering. Suppose the original image is , the target tracking box is , the image after smoothing is:

[0190] ;

[0191] Among them, © represents element-by-element multiplication, The mask of the target box, the area with a value of 1 indicates the location of the target, and a smooth transition is achieved through weighted addition. Annotation information, such as the category and confidence of the target, is added to the image to output the final real-time processed image.

[0192] The GPU parallel computing grid is input into the three-level pipeline scheduler, and the tasks of the three stages of decoding, processing and output are pipeline scheduled by setting a command queue control unit on the CPU side to obtain a pipeline task queue, including: dividing the GPU parallel computing grid according to decoding tasks, processing tasks and output tasks, setting independent task descriptors and execution identifiers for each type of task, and obtaining a task classification matrix; allocating memory space to each task in the task classification matrix, building a command buffer on the CPU side and setting read and write pointers to obtain a command storage structure; inputting the command storage structure into a synchronization control module, establishing a data transmission channel between the CPU and the GPU through the PCI-E bus, and obtaining a synchronous transmission channel; establish a task status table based on the synchronous transmission channel, mark the running state, waiting state and completion state of each task, and obtain the task status identification; allocate computing resources according to the task status identification, divide the GPU computing core into three stages of decoding, processing and output respectively, and obtain the resource allocation table; perform pipeline planning on the resource allocation table, set the task switching cycle to 2ms, divide the scheduling timing of GPU thread blocks, and obtain the pipeline scheduling table; input the pipeline scheduling table into the load controller, count and balance the number of tasks of the three-stage pipeline, and obtain the task allocation parameters; build a task execution queue based on the task allocation parameters, organize the execution order of tasks according to the first-in-first-out principle, and output the pipeline task queue.

[0193] The feature data in the pipeline task queue is decoded, and 8 parallel decoding units are set to perform inverse quantization and inverse transformation operations at the same time. Each decoding unit is responsible for restoring the target data block to obtain the original feature data, including: dividing the feature data in the pipeline task queue into blocks according to the data block size of 64×64 pixels, assigning a unique block identifier to each data block, and obtaining block feature data; distributing the block feature data, assigning the feature data to 8 decoding units according to the spatial adjacency of the data blocks, and obtaining a decoding task allocation table; inputting the data in the decoding task allocation table into the inverse quantization processing unit, and performing the inverse quantization on the bit depth, quantization parameter and quantization step size of the feature data. The method comprises the following steps: analyzing the feature data to obtain a quantization parameter matrix; performing inverse quantization calculation on the feature data based on the quantization parameter matrix, restoring the data by establishing a mapping relationship from the quantization value to the actual value, and obtaining inverse quantized data; inputting the inverse quantized data into the inverse transform processing unit, performing inverse discrete cosine transform and inverse wavelet transform on the data, and obtaining transform domain data; reconstructing the transform domain data, restoring the spatial structure of the original data by reconstructing the coefficient matrix, and obtaining reconstructed feature data; performing boundary processing on the reconstructed feature data, eliminating the splicing traces and sawtooth effects between data blocks, and obtaining smooth feature data; reorganizing the smooth feature data according to the original data format, merging the output data of the 8 decoding units, and outputting the original feature data.

[0194] See also Figure 2 , Figure 2 A schematic block diagram of the structure of a real-time image processing device 200 for a 3D and AI visual sensing visible light core provided in an embodiment of the present application is shown in FIG. Figure 2 As shown, the real-time image processing device 200 of the 3D and AI visual sensing visible light core includes:

[0195] The acquisition module 210 is used to perform dual-modal channel image acquisition through the ISP image signal processing unit and the GPU acceleration unit to obtain a 3D image data stream and a visible light image data stream;

[0196] A standardization processing module 220 is used to perform light intensity detection and standardization processing on the 3D image data stream and the visible light image data stream to obtain standardized dual-modal image data;

[0197] The feature fusion module 230 is used to input the standardized dual-modal image data into the dynamic convolutional neural network for feature fusion to obtain a multi-modal fusion feature map;

[0198] A contrast learning module 240 is used to perform spatiotemporal contrast learning on the multimodal fusion feature map to obtain a first feature map;

[0199] A feature selection module 250 is used to input the first feature graph into a graph structure redundancy assessment model for feature selection to obtain a second feature graph;

[0200] The output module 260 is used to perform GPU parallel decoding and pipeline processing on the second feature map, and output a real-time processed image.

[0201] Through the coordinated cooperation of the above components, through the deep fusion of 3D imaging and visible light imaging, combined with the adaptive lighting system and ISP processing unit, high-quality acquisition and processing of multimodal data are achieved, and the problem of unstable image quality under a single imaging mode is solved; the dynamic convolutional neural network and dual attention mechanism are adopted to realize the adaptive extraction and enhancement of features, and improve the system's feature expression ability for complex scenes; the introduction of spatiotemporal contrast learning and automatic labeling technology reduces the dependence on a large amount of labeled data, and at the same time ensures the stability of feature learning through consistency regularization constraints; through the graph structure redundancy evaluation model and information gain sorting mechanism, intelligent screening and optimization of features are realized, redundant calculations are reduced, and system efficiency is improved; based on GPU parallel computing and pipeline architecture, efficient real-time processing capabilities are achieved, meeting the requirements of practical applications for system response speed.

[0202] See also Figure 3 , Figure 3The present invention provides a schematic block diagram of the structure of a real-time image processing device 300 for a 3D and AI visual sensing visible light movement, wherein the real-time image processing device 300 for a 3D and AI visual sensing visible light movement comprises a processor 301 and a memory 302, wherein the processor 301 and the memory 302 are connected via a device bus 303, wherein the memory 302 may include a non-volatile storage medium and an internal memory.

[0203] The non-volatile storage medium can store a computer program. The computer program includes program instructions, which, when executed by the processor 301, can enable the processor 301 to execute any of the above-mentioned real-time image processing methods for 3D and AI visual sensing visible light cores.

[0204] The processor 301 is used to provide computing and control capabilities to support the operation of the real-time image processing device 300 of the entire 3D and AI visual sensing visible light movement.

[0205] The internal memory provides an environment for the operation of the computer program in the non-volatile storage medium. When the computer program is executed by the processor 301, the processor 301 can execute any of the above-mentioned real-time image processing methods of the 3D and AI visual sensing visible light movement.

[0206] Those skilled in the art will understand that Figure 3 The structure shown in the figure is only a block diagram of a part of the structure related to the present application scheme, and does not constitute a limitation on the real-time image processing device 300 of the 3D and AI visual sensing visible light movement involved in the present application scheme. The specific real-time image processing device 300 of the 3D and AI visual sensing visible light movement may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0207] It should be understood that the processor 301 may be a central processing unit (CPU), and the processor 301 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0208] It should be noted that technicians in the relevant field can clearly understand that for the convenience and conciseness of description, the specific working process of the real-time image processing device 300 of the 3D and AI vision sensing visible light movement described above can refer to the corresponding process of the real-time image processing method of the aforementioned 3D and AI vision sensing visible light movement, and will not be repeated here.

[0209] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by one or more processors, the one or more processors implement the real-time image processing method of the 3D and AI visual sensing visible light movement provided in the embodiment of the present application.

[0210] The computer-readable storage medium may be an internal storage unit of the real-time image processing device 300 of the 3D and AI visual sensing visible light movement of the aforementioned embodiment, such as a hard disk or memory of the real-time image processing device 300 of the 3D and AI visual sensing visible light movement. The computer-readable storage medium may also be an external storage device of the real-time image processing device 300 of the 3D and AI visual sensing visible light movement, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc., equipped with the real-time image processing device 300 of the 3D and AI visual sensing visible light movement.

[0211] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described systems, systems and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0212] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store program codes.

[0213] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A real-time image processing method for 3D and AI visual sensing visible light movement, characterized in that: include: Dual-modality channel image acquisition is performed through the ISP image signal processing unit and the GPU acceleration unit to obtain a 3D image data stream and a visible light image data stream; Performing light intensity detection and standardization processing on the 3D image data stream and the visible light image data stream to obtain standardized dual-modal image data; Inputting the standardized bimodal image data into a dynamic convolutional neural network for feature fusion to obtain a multimodal fusion feature map; Performing spatiotemporal contrast learning on the multimodal fusion feature map to obtain a first feature map; specifically comprising: performing data enhancement transformation on the multimodal fusion feature map to obtain multimodal enhanced features, performing time series frame sampling on the multimodal enhanced features, extracting time series features from continuous video sequences, constructing spatiotemporal feature pairs in time series order, and obtaining time series positive sample pairs; inputting the time series positive sample pairs into a twin network structure, encoding the features through two shared weight encoders, each of which includes three convolutional layers and two fully connected layers, to obtain a feature embedding vector; performing contrast learning on the feature embedding vector, and calculating the mutual information maximization between positive sample pairs and the mutual information between negative sample pairs through the InfoNCE loss function Minimize to obtain contrast features; divide the feature space of the contrast features into K categories through a dynamic threshold clustering algorithm, assign a unique label ID to each category to obtain labeled features, perform consistency calculation on the labeled features, calculate the difference values ​​of different view features under the same time series through a mean square error function, and construct consistency constraint features; input the consistency constraint features into a feature metric learning network, learn a similarity metric function between features through the feature metric learning network, and the feature metric learning network includes a deep separable convolutional layer and an attention mechanism to obtain metric features; perform feature reconstruction on the metric features, map the features back to the original feature space through a deconvolution network, and output a first feature map; The first feature graph is input into the graph structure redundancy assessment model for feature selection to obtain a second feature graph; specifically comprising: calculating the Euclidean distance between feature points in the first feature graph, and constructing edge connections between adjacent feature points based on a distance threshold to obtain a weighted relationship graph; performing normalized Laplace matrix calculation on the weighted relationship graph, constructing a feature propagation matrix by multiplying the inverse matrix of the degree matrix with the adjacency matrix to obtain graph structure features; calculating the correlation between each pair of feature nodes in the graph structure features by the Pearson correlation coefficient, and constructing an N×N-dimensional feature correlation matrix, where N is the total number of feature nodes, to obtain feature correlation values; performing information entropy calculation on the feature correlation values, and calculating the contribution of each feature node to the overall feature distribution by the mutual information formula, The contribution degree is obtained, and an information gain sequence is constructed according to the size of the contribution degree to obtain a feature importance vector; feature selection is performed according to the feature importance vector, nodes whose information gain values ​​are lower than the average value of all features are eliminated, and feature nodes with high information content are retained to obtain an optimized feature graph; the optimized feature graph is input into a residual connection network, and two convolutional layers and a short-circuit connection are set in each residual block to directly transfer low-level feature information to a high-level feature layer to obtain a multi-level residual feature; feature sequence modeling is performed on the multi-level residual feature to obtain a time series correlation feature, and the attention weight coefficient between nodes of the time series correlation feature is calculated through a graph attention network, and neighborhood features are weighted aggregated based on the attention weight coefficient to output a second feature graph; GPU parallel decoding and pipeline processing are performed on the second feature map to output a real-time processed image.

2. The real-time image processing method of the 3D and AI visual sensing visible light core according to claim 1, characterized in that: The dual-modal channel image acquisition is performed through the ISP image signal processing unit and the GPU acceleration unit to obtain a 3D image data stream and a visible light image data stream, including: Matching the photoelectric performance parameters of the 3D imaging unit and the visible light imaging unit to obtain spectral matching parameters, and configuring the illumination parameters of the adaptive illumination system based on the spectral matching parameters to obtain adaptive illumination control parameters; According to the spectrum matching parameters and the adaptive illumination control parameters, the 3D imaging unit, the visible light imaging unit and the adaptive illumination system are integrated and connected to obtain a core module, and the core module is time-calibrated through a clock divider and a phase-locked loop to obtain a dual-modal acquisition timing signal; Inputting the movement module and the dual-modal acquisition timing signal into the ISP image signal processing unit, performing real-time signal processing through a parallel pipeline architecture to obtain raw image data, and performing data format conversion on the raw image data to obtain standard format image data; The standard format image data is input into the GPU acceleration unit, and the data is parsed and reorganized through the CUDA parallel computing framework to obtain parallel processing data, and the parallel processing data is subjected to channel separation and data stream reconstruction to output a 3D image data stream and a visible light image data stream respectively.

3. The real-time image processing method of the 3D and AI visual sensing visible light core according to claim 2, characterized in that: The performing light intensity detection and standardization processing on the 3D image data stream and the visible light image data stream to obtain standardized dual-modal image data includes: Performing illumination intensity calculation on the 3D image data stream to obtain 3D image illumination intensity parameters, and performing ambient illumination analysis on the visible light image data stream to obtain visible light ambient illumination parameters; Performing a dual-modal illumination compensation matrix operation based on the 3D image illumination intensity parameter and the visible light environment illumination parameter to obtain an illumination compensation parameter, and performing a multi-angle projection transformation matrix calculation on the 3D image data stream and the visible light image data stream according to the illumination compensation parameter to obtain a multi-viewing angle transformation matrix; Performing spatial reprojection calculation and feature point matching on the 3D image data stream and the visible light image data stream according to the multi-view transformation matrix to obtain a stereo feature point set; The stereo feature point set is input into the GPU acceleration unit, and three-dimensional space reconstruction and feature normalization operations are performed through the CUDA parallel computing architecture to obtain normalized feature data, and noise filtering and depth-visible light feature fusion are performed on the normalized feature data to obtain standardized bimodal image data.

4. The real-time image processing method of the 3D and AI visual sensing visible light core according to claim 3 is characterized in that: The step of inputting the standardized bimodal image data into a dynamic convolutional neural network for feature fusion to obtain a multimodal fusion feature map comprises: Inputting the standardized bimodal image data into an encoder of a dynamic convolutional neural network according to 3D features and visible light features, respectively, the encoder comprising four dynamic convolutional layers to obtain an initial feature map; The initial feature map is input into the spatial attention submodule of the dynamic convolutional neural network, wherein the spatial attention submodule comprises a global average pooling layer and two convolutional layers, a ReLU activation function is used between the two convolutional layers, and the spatial attention weight is obtained by calculating the attention score matrix of each spatial position of the feature map; Inputting the initial feature map into the channel attention submodule of the dynamic convolutional neural network, wherein the channel attention submodule comprises two fully connected layers and a Sigmoid activation function, and obtaining the channel attention weight by learning the importance coefficient of each channel; The initial feature map is weighted according to the spatial attention weight and the channel attention weight to obtain an attention-enhanced feature, and the attention-enhanced feature is input into a time-delayed self-attention mechanism layer, wherein the time-delayed self-attention mechanism layer includes a temporal encoder and three linear transformation layers to generate a temporal correlation matrix; Performing temporal feature aggregation on the attention enhancement feature based on the temporal association matrix to obtain a temporal fusion feature, and inputting the temporal fusion feature into a decoder of the dynamic convolutional neural network, wherein the decoder comprises four deconvolution layers, each of which uses batch normalization and ReLU activation function to obtain multi-scale features; The multi-scale features are aligned and fused across modalities, 3D features and visible light features are merged through feature concatenation and 1×1 convolution operations, and a multi-modal fusion feature map is output.

5. The real-time image processing method of the 3D and AI visual sensing visible light core according to claim 1, characterized in that: The performing GPU parallel decoding and pipeline processing on the second feature map to output a real-time processed image includes: According to the CUDA parallel computing framework, thread blocks are allocated to the second feature graph, 16×16 threads are set in each thread block, and an independent feature data access space is allocated to each thread to obtain a GPU parallel computing grid; The GPU parallel computing grid is input into a three-stage pipeline scheduler, and a command queue control unit is set on the CPU end to perform pipeline scheduling on the tasks of the three stages of decoding, processing and output, so as to obtain a pipeline task queue; Decoding the feature data in the pipeline task queue, by setting 8 parallel decoding units to simultaneously perform inverse quantization and inverse transformation operations, each decoding unit is responsible for restoring the target data block, to obtain the original feature data; The original feature data is divided into a plurality of overlapping grids according to spatial distribution, the overlapping rate of each overlapping grid is set to 0.2, and feature extraction is performed through a preset target detection convolution kernel to obtain a set of candidate regions; A non-maximum suppression operation is performed on the candidate region set, by calculating the intersection over union ratio of the target frame and setting the IoU threshold to 0.5, and merging the detection frames whose overlap rate is higher than the threshold to obtain the target detection frame; The target detection frame is input into the multi-classification scoring network, the probability score of each category is calculated by the Softmax activation function, and the confidence threshold is set to 0.75 to obtain a valid target frame, and the target tracking calculation is performed on the valid target frame, the target motion state is predicted by the Kalman filter, and the target features are updated in combination with the appearance model to obtain the target tracking frame; The target tracking frame is superimposed on the original image, the edge is smoothed and annotation information is added through the image post-processing unit, and a real-time processed image is output.

6. A real-time image processing device for 3D and AI visual sensing visible light movement, characterized in that: A real-time image processing method for executing the 3D and AI visual sensing visible light core according to any one of claims 1 to 5, comprising: An acquisition module is used to acquire dual-modal channel images through an ISP image signal processing unit and a GPU acceleration unit to obtain a 3D image data stream and a visible light image data stream; A standardization processing module, used for performing light intensity detection and standardization processing on the 3D image data stream and the visible light image data stream to obtain standardized dual-modal image data; A feature fusion module, used for inputting the standardized bimodal image data into a dynamic convolutional neural network for feature fusion to obtain a multimodal fusion feature map; A contrastive learning module, used for performing spatiotemporal contrastive learning on the multimodal fusion feature map to obtain a first feature map; A feature selection module, used for inputting the first feature graph into a graph structure redundancy assessment model for feature selection to obtain a second feature graph; The output module is used to perform GPU parallel decoding and pipeline processing on the second feature map and output a real-time processed image.

7. A real-time image processing device for 3D and AI visual sensing visible light movement, characterized in that: The real-time image processing device of the 3D and AI visual sensing visible light core includes: a memory and at least one processor, wherein the memory stores instructions; The at least one processor calls the instructions in the memory so that the real-time image processing device of the 3D and AI vision sensing visible light movement executes the real-time image processing method of the 3D and AI vision sensing visible light movement as described in any one of claims 1-5.

8. A computer-readable storage medium having instructions stored thereon, characterized in that: When the instructions are executed by the processor, the real-time image processing method of the 3D and AI visual sensing visible light movement as described in any one of claims 1-5 is implemented.

Images