A deep learning-based stereo matching method and system

Abstract

The application discloses a kind of stereomatching method and system based on deep learning, comprising: build manufacturing scene data acquisition platform, synchronously collect stereoscopic image pair and real depth information, construct the training dataset containing left view, right view and real parallax graph;The stereomatching neural network model is constructed, which includes unary feature extraction, cost volume construction, context aggregation and parallax prediction module;Image is processed in turn through each module, extracts features, constructs cost volume, aggregates context information, calculates predicted parallax;Model is trained using training dataset and multi-task joint loss function, and is deployed on acquisition platform after reaching standard, processes real-time image and outputs parallax graph.The method introduces the visual attention module VABlock architecture of stacking, improves feature expression capability, combined with double-path aggregation and other modules, enhances the matching accuracy and robustness in complex scene, can realize manufacturing scene real-time accurate three-dimensional perception, provides data basis for subsequent tasks.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of stereo matching technology in the intelligent manufacturing and industrial automation scene, in particular to a stereo matching method and system based on deep learning. BACKGROUND

[0002] As a basic link of intelligent manufacturing systems, stereo matching technology estimates pixel-level disparity maps through binocular images, and then realizes high-precision spatial perception of the production environment, playing a key role in industrial applications such as robot navigation, automated detection, and production line monitoring. Accurate disparity estimation not only helps to obtain more detailed three-dimensional structure information, but also provides a data foundation and technical support for optimizing manufacturing processes and improving resource utilization efficiency. However, existing stereo matching methods still face many technical bottlenecks when faced with complex large-scale manufacturing scenarios. Most traditional methods rely on multi-view images or depth sensors and perform well in small-scale or structured scenes, but in large-scale and dynamically changing industrial environments, there are generally problems such as inaccurate feature extraction, low computational efficiency, and weak noise resistance, which seriously affect the accuracy and real-time performance of disparity estimation. Especially in highly complex manufacturing scenarios, such as the presence of a large number of mechanical arms, conveyors, metal surface equipment, and other objects, their material reflectivity is strong, texture is sparse, and geometric structure is complex, and are often accompanied by interference factors such as occlusion and light changes. These characteristics make it difficult for traditional matching methods based on local texture or edge features to effectively extract robust image representations, resulting in a significant increase in disparity estimation errors.

[0003] In this context, the stereo matching method for large-scale manufacturing scenarios is of great significance. In recent years, stereo matching technology based on deep learning has made significant progress, especially the method based on convolutional neural network (CNN), which has significantly improved the accuracy of disparity estimation. However, when faced with large-scale and complex manufacturing scenarios, CNN methods are still limited by their local receptive field and fixed structure, and there are certain limitations in handling long-distance dependencies and complex structure modeling. But when faced with large-scale and complex manufacturing scenarios, it is still difficult to guarantee high precision while meeting the requirement of high efficiency. Compared with traditional convolutional neural networks (CNN), state space model (SSM)-based models can capture more long-distance information interaction, breaking through the limitations of local receptive field of convolution operation; at the same time, unlike the quadratic computational complexity required by the self-attention mechanism in Transformer, SSM has the advantage of linear time complexity, which shows higher efficiency when processing high-resolution images or long sequence inputs. Therefore, it is urgent to develop a stereo matching method based on deep learning that can balance accuracy and efficiency, especially for efficient and robust disparity estimation in large-scale manufacturing scenarios. The specific implementation process has the following difficulties:

[0004] 1. Traditional stereo matching techniques usually rely on high computational complexity image processing and optimization algorithms. These methods often require dense matching calculations on a large number of input images, and the calculation process involves multiple steps such as image feature extraction, disparity calculation, image alignment, and depth map generation. In large-scale manufacturing scenarios, the number of images processed and the amount of data are huge, and low computational efficiency limits its application in large-scale manufacturing scenarios.

[0005] 2. Large-scale manufacturing scenarios usually contain a large number of details, different surface materials, and complex geometric structures, such as industrial equipment, production lines, and robotic arms, which may exhibit very different visual features under different viewing angles. Traditional feature extraction methods often rely on low-level features based on edges or textures. Although these methods can capture some basic information in the image, they may not be accurate or complete when dealing with scenes with complex structures and details, which can affect the stereo matching effect.

[0006] 3. In large-scale manufacturing scenarios, the features of local regions often have different scales and complexities, especially in areas with dramatic depth changes or many occlusions. The extraction of these local features is more difficult. Traditional stereo matching algorithms and feature extraction networks lack dynamic focusing on key areas, and cannot adjust attention allocation according to local or global image information, which can be easily disturbed by unimportant areas, leading to inaccurate feature extraction. SUMMARY

[0007] In order to solve the above technical problems, the application provides a stereo matching method and system based on deep learning.

[0008] The technical scheme adopted by the application to solve the technical problems is:

[0009] A stereo matching method based on deep learning, the method comprising the following steps:

[0010] S100: Building a large-scale manufacturing scene data acquisition platform, synchronously acquiring a stereo image pair and corresponding real depth information through a binocular camera and a laser radar, and constructing a training data set containing a left view, a right view and a real disparity map after calibration and calculation;

[0011] S200: Constructing a stereo matching neural network model, including a unary feature extraction module, a cost volume construction module, a context aggregation module and a disparity prediction module;

[0012] S300: Inputting the left view and the right view into the unary feature extraction module, which is constructed based on a stacked visual state space block, and performing deep semantic feature extraction through a selective scanning mechanism combined with a multi-head local attention, realizing joint modeling of local and global context information, and outputting feature maps of the left and right images;

[0013] S400: Inputting the left and right image feature maps into the cost volume construction module, constructing a cost volume for disparity search by calculating the grouping correlation between the features and combining feature connection;

[0014] S500: Inputting the cost volume into the context aggregation module, performing feature enhancement and aggregation on the cost volume through a double-branch structure containing a local detail extraction path and a global context modeling path, to fuse multi-scale context information;

[0015] S600: Inputting the aggregated cost volume into the disparity prediction module, adopting a decoding mechanism of classification and regression strategy fusion to calculate the final predicted disparity result;

[0016] S700: Supervising and training the model to optimize its parameters by using the training data set combined with a multi-task joint loss function until the difference between the predicted disparity result output by the model and the real disparity map is within a preset threshold, deploying the stereo matching neural network model optimized by training on the data acquisition platform, processing the real-time acquired stereo image pair, and outputting a disparity map for a three-dimensional vision task.

[0017] Preferably, S100 comprises:

[0018] S110: Fix the binocular camera and the laser radar on the mobile robot trolley, the relative position of the camera and the radar is fixed, adjust the installation position and angle to make the field of view face forward;

[0019] S120: Connect the binocular camera, the laser radar and the NVIDIA Jetson AGX Orin developer kit, power and data transmission for the camera and the radar, fix the NVIDIA Jetson AGX Orin development board on the trolley, install the matching acquisition card for data acquisition, and use the upper computer for remote communication and operation;

[0020] S130: Realize time synchronization of the binocular camera and the laser radar through the ROS node, so that the binocular camera and the laser radar synchronize data acquisition;

[0021] S140: Obtain multiple sets of scene data by wireless remote control of the mobile robot trolley, the binocular camera obtains image data, and the laser radar obtains point cloud data, and the internal parameter matrix and the distortion coefficient of the left and right cameras are obtained through internal parameter calibration;

[0022] S150: Joint calibration of the camera and the radar using the internal parameter matrix and the distortion coefficient, and the positional relationship between them is obtained, the three-dimensional point cloud obtained by the laser radar is projected onto the image to obtain the depth map, and the specific relationship is represented as:

[0023] ;

[0024] Wherein is the representation of the point in the left camera coordinate system, represents the x-axis coordinate component of the camera position in the three-dimensional rectangular coordinate system, represents the y-axis coordinate component of the camera position in the three-dimensional rectangular coordinate system, represents the z-axis coordinate component of the camera position in the three-dimensional rectangular coordinate system, is the representation of the point in the laser radar coordinate system, represents the transverse coordinate of the laser radar measurement point, represents the longitudinal coordinate of the laser radar measurement point, represents the depth coordinate of the laser radar measurement point, R is the rotation matrix, is the translation vector;

[0025] S160: Again use the parameters of the camera itself, the camera focal length , the baseline length , and calculate the real parallax map according to the following formula :

[0026] ;

[0027] S170: The left view, the right view and the disparity map are taken as a set of data, and the collected data is divided into a training set and a validation set according to a preset ratio, so as to complete the data set production of a large-scale manufacturing scene.

[0028] Preferably, the unary feature extraction module in S300 is implemented by stacking multiple visual attention modules VA Block, and S300 includes:

[0029] S310: The obtained stereo image pair is divided into small patch blocks by the stem module Stem Block, that is, after a series of convolution, batch normalization, ReLU activation and pooling operations, an output feature map with a size of is obtained , ;

[0030] S320: The feature map with a shape size of [B, H, W, C] , is converted into a shape size of [B, HxW, C], is processed by a depth separable convolution, and a convolution position encoding is introduced. The feature after position encoding enhancement is normalized by LN, and then a query vector Q, a key vector K and a value vector V are generated by linear transformation; wherein B represents the batch size, H and W represent the height and width of the feature map respectively, and C is the channel number;

[0031] S330: Q, K and V are divided into five parts, and a different kernel size aggregator is used to generate group agents in four of the parts, wherein three kernel sizes are set to 3, 5 and 7, and the other one does not use an aggregator, so that it becomes an identity mapping. At the same time, a branch with an aggregator but without attention is set. The four branches with attention are fused by a learnable weighting fusion mechanism, and then spliced and combined with the branch without attention;

[0032] S340: The output result is adaptively selected to enter a state space model SSM branch or a multi-layer perception MLP branch by a router mechanism, wherein the SSM branch is composed of layer normalization LN, linear mapping, depth separable convolution, activation function, selective scanning module, another layer of LN and linear mapping, forming a local-global joint space modeling path; the MLP branch is a feedforward network composed of two layers of linear transformation and activation function with residual connection, and finally outputs a feature map with the same shape size as the input;

[0033] S350: A corresponding number of VA Block are stacked to obtain feature outputs with the same size in stages 1 to 4, and the size is . The four-stage features are spliced in the channel dimension to form a unary feature map with a total channel number of . 、 .

[0034] Preferably, S400 comprises:

[0035] S410: evenly divide all the channels into groups along the channel dimension, so each group has channels, the th feature group 、 is composed of the th group of original features, the grouped correlation volume is calculated as:

[0036] ;

[0037] where represents the inner product, d represents the disparity level, represents the horizontal coordinate index of the feature point in the image, represents the vertical coordinate index of the feature point in the image;

[0038] S420: calculate the correlation for all feature groups and disparity levels, and pack all the correlation maps into a matching cost volume with a shape of , where represents the maximum disparity, corresponds to the maximum disparity of the feature; when the number of groups , the grouped correlation becomes full correlation;

[0039] S430: concatenate the left and right features directly to form a connection volume , combine the connection volume with the grouped correlation volume constructed in S410 to obtain the final cost volume .

[0040] Preferably, S500 comprises:

[0041] The local detail extraction path extracts fine-grained spatial features in the cost volume using grouped convolution and depth separable convolution, and the global context modeling path generates global context weights based on the guide features to model long-distance information for the down-sampled cost volume. The outputs of the two paths are weighted and fused through a channel attention mechanism to realize the collaborative optimization of local details and global structures, and finally obtain the output of the dual-path guided context aggregation module, i.e. the aggregated cost volume .

[0042] Preferably, S600 comprises:

[0043] ​S610: Feature transformation is performed on the aggregated cost volume, and two output branches are extracted through a set of lightweight convolutional layers: the classification branch output is represented as , and the regression branch output is represented as The classification branch output tensor S is normalized into a probability distribution by applying a softmax function in the disparity dimension:

[0044] ;

[0045] wherein denotes the probability distribution of the disparity value at the image coordinate , denotes the original score of the disparity value at the image coordinate , denotes the temporary disparity score traversed in the summation term, and denotes the maximum disparity.

[0046] The probability distribution reflects the likelihood of each disparity value at the pixel position .

[0047] S620: Then, the weighted average disparity value is calculated as:

[0048] ;

[0049] wherein denotes the weighted average disparity value.

[0050] S630: Meanwhile, the offset prediction value at the same position is obtained according to the regression branch output:

[0051] ;

[0052] wherein denotes the offset prediction value corresponding to the disparity value at the same position.

[0053] The final disparity prediction result is composed of the sum of the weighted average disparity and the offset:

[0054] .

[0055] Preferably, the multi-task joint loss function in S700 is specifically:

[0056] ;

[0057] wherein For classification loss, used to measure the accuracy of the model's probability prediction for different disparity values; For regression loss, used to optimize the sub-pixel offset in disparity prediction; For smoothness constraint, used to enhance the spatial continuity of the disparity map and reduce noise interference.

[0058] A stereo matching system based on deep learning, comprising:

[0059] A data set construction module is configured to build a large-scale manufacturing scene data acquisition platform, to synchronously acquire stereo image pairs and corresponding real depth information through a binocular camera and a laser radar, and to construct a training data set containing left views, right views and real disparity maps after calibration and calculation;

[0060] A model construction module is configured to construct a stereo matching neural network model, including a unary feature extraction module, a cost volume construction module, a context aggregation module and a disparity prediction module;

[0061] The unary feature extraction module is constructed based on a stacked visual state space block, configured to receive left and right views, to perform deep semantic feature extraction through a selective scanning mechanism combined with a multi-head local attention, to realize joint modeling of local and global context information, and to output feature maps of the left and right images;

[0062] The cost volume construction module is configured to receive left and right image feature maps, to construct a cost volume for disparity search by calculating the grouping correlation between the features and combining feature connection;

[0063] The context aggregation module is configured to receive the cost volume, to perform feature enhancement and aggregation on the cost volume through a double-branch structure containing a local detail extraction path and a global context modeling path, and to fuse multi-scale context information;

[0064] The disparity prediction module is configured to calculate the final predicted disparity result by using a decoding mechanism that combines classification and regression strategies to fuse the aggregated cost volume;

[0065] A training optimization module is configured to supervise the training of the model to optimize its parameters by using the training data set combined with a multi-task joint loss function until the difference between the predicted disparity result output by the model and the real disparity map is within a preset threshold, to deploy the stereo matching neural network model optimized through training on the data acquisition platform, to process the real-time acquired stereo image pairs, and to output a disparity map for three-dimensional vision tasks.

[0066] A computer device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the steps of a stereo matching method based on deep learning when executing the computer program.

[0067] A computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of a deep learning based stereo matching method.

[0068] The deep learning based stereo matching method and system effectively capture long-range dependencies and deep semantic information of images by introducing a unary feature extraction module constructed based on a stack of visual state space blocks, and significantly improve the feature expression capability. Combined with cost volume construction and a dual-path context aggregation module, local details and global context are fully fused, and the matching accuracy and robustness of the model in complex manufacturing scenes such as weak texture and occlusion are enhanced. The cost volume construction adopts a combination of grouping correlation calculation and feature connection, which effectively controls the computational complexity while maintaining rich matching information. The disparity prediction module integrates classification and regression strategies to achieve accurate disparity estimation; relying on a high-quality dataset constructed by synchronous calibration of binocular cameras and laser radars for training, the model has strong generalization ability, and the trained model can be deployed on an embedded platform to realize real-time and accurate three-dimensional perception of the manufacturing scene, providing a reliable data foundation for subsequent tasks such as stereo reconstruction and quality detection. BRIEF DESCRIPTION OF DRAWINGS

[0069] Figure 1 A flowchart of a deep learning based stereo matching method in an embodiment of the present application;

[0070] Figure 2 A VABlock structure diagram in an embodiment of the present application;

[0071] Figure 3 A deep learning based stereo matching model structure diagram in an embodiment of the present application. DETAILED DESCRIPTION

[0072] In order to enable personnel in the technical field to better understand the technical solutions of the present application, the present application will be further described in detail below with reference to the drawings.

[0073] In one embodiment, as shown in Figure 1 a deep learning based stereo matching method, the method comprising the following steps:

[0074] S100: Build a large-scale manufacturing scene data acquisition platform, synchronously acquire stereo image pairs and corresponding real depth information through binocular cameras and laser radars, and construct a training dataset containing left view, right view and real disparity map after calibration and calculation;

[0075] S200: Construct a stereo matching neural network model, including a unary feature extraction module, a cost volume construction module, a context aggregation module and a disparity prediction module;

[0076] S300: input the left view and the right view into a unary feature extraction module, which is constructed based on stacked visual state space blocks, performs deep semantic feature extraction through a selective scanning mechanism combined with multi-head local attention, realizes joint modeling of local and global context information, and outputs feature maps of the left and right images;

[0077] S400: input the left and right image feature maps into a cost volume construction module, construct a cost volume for disparity search by calculating the grouping correlation between the features and combining feature connection;

[0078] S500: input the cost volume into a context aggregation module, perform feature enhancement and aggregation on the cost volume through a double-branch structure containing a local detail extraction path and a global context modeling path, to fuse multi-scale context information;

[0079] S600: input the aggregated cost volume into a disparity prediction module, adopt a decoding mechanism that combines classification and regression strategies to calculate the final predicted disparity result;

[0080] S700: use a training data set combined with a multi-task joint loss function to supervise the training of the model to optimize its parameters until the difference between the predicted disparity result output by the model and the real disparity map is within a preset threshold, deploy the trained and optimized stereo matching neural network model on a data acquisition platform, process the real-time acquired stereo image pair, and output the disparity map for three-dimensional vision tasks.

[0081] The above-mentioned stereo matching method based on deep learning aims to solve the problems of low computational efficiency, inaccurate or incomplete extraction of complex structure and detail features, occlusion or illumination change, difficulty in multi-scale feature fusion, and lack of dynamic focusing on key areas in the current stereo matching method in a wide range of manufacturing scenarios.

[0082] In one embodiment, S100 includes:

[0083] S110: fix the zedx binocular camera and the laser radar on the mobile robot trolley, the relative position of the camera and the radar is fixed, adjust the installation position and angle to make the field of view face forward;

[0084] S120: connect the binocular camera, the laser radar and the NVIDIA Jetson AGX Orin developer kit, power and data transmit the camera and the radar, fix the NVIDIA Jetson AGX Orin development board on the trolley, install the matching acquisition card for data acquisition, and use the upper computer for remote communication and control;

[0085] S130: Time synchronization of binocular camera and laser radar is realized through a ROS (Robot Operating System) node, so that the binocular camera and the laser radar synchronously collect data;

[0086] S140: A plurality of groups of scene data are acquired by moving the robot car through wireless remote control, the binocular camera acquires image data, and the laser radar acquires point cloud data, and the intrinsic parameter matrix and the distortion coefficient of the left and right cameras are obtained through intrinsic parameter calibration;

[0087] S150: The intrinsic parameter matrix and the distortion coefficient are used for joint calibration of the camera and the radar (lidar_camera_calibration or matlab can be used), the positional relationship between them is obtained, the three-dimensional point cloud acquired by the laser radar is projected onto the image to obtain a depth map, and the specific relationship is represented as:

[0088] ;

[0089] wherein is a representation of a point in the left camera coordinate system, represents an x-axis coordinate component of the camera position in the three-dimensional rectangular coordinate system, represents a y-axis coordinate component of the camera position in the three-dimensional rectangular coordinate system, represents a z-axis coordinate component of the camera position in the three-dimensional rectangular coordinate system, is a representation of a point in the laser radar coordinate system, represents a transverse coordinate of a laser radar measurement point, represents a longitudinal coordinate of a laser radar measurement point, represents a depth coordinate of a laser radar measurement point, R is a rotation matrix, is a translation vector;

[0090] S160: The real parallax map is calculated by using the parameters of the camera itself, the camera focal length , the baseline length , and the following formula:

[0091] ;

[0092] S170: The left view, the right view, and the parallax map are taken as a group of data, the collected data are divided into a training set and a validation set according to a preset proportion, and the data set production of the large-scale manufacturing scene is completed.

[0093] Specifically, in this embodiment, 70% of the collected data is randomly divided into the training set, and the remaining 30% is taken as the validation set. Thus, the data set production of the large-scale manufacturing scene is completed, which is used for subsequent model performance evaluation.​

[0094] The stereo matching network can quickly and accurately generate a high-precision disparity map based on binocular images, and is widely applicable to intelligent manufacturing, industrial detection, equipment state monitoring and the like. The network can realize stable and efficient three-dimensional perception in a complex environment (such as light change, texture loss, and reflective surface), significantly improve the identification and response ability of a manufacturing system to equipment abnormalities, reduce manual inspection and maintenance costs, and promote the development of a production process towards automation and intelligence.

[0095] In one embodiment, as shown in FIG. 3, the one-dimensional feature extraction module in S300 is implemented by stacking multiple VA Blocks (Vision Attention Blocks), and S300 includes: Figure 2

[0096] S310: The obtained stereo image pair is divided into small patch blocks by the Stem Block, that is, after a series of convolution, batch normalization, ReLU (Rectified Linear Unit) activation and pooling operations, a feature map with an output size of

[0097] S320: The feature map with a shape size of [B, H, W, C] is converted into a shape size of [B, HxW, C], processed by a depth separable convolution (DSC, Depthwise Separable Convolution), and position encoding is introduced. The features after position encoding enhancement are normalized by LN (Layer Normalization), and then the query vector Q, the key vector K and the value vector V are generated by linear transformation; wherein B represents the batch size, H and W represent the height and width of the feature map respectively, and C is the number of channels; S330: Q, K and V are divided into five parts, and aggregators with different kernel sizes are used to generate group agents in four of the parts, wherein three kernel sizes are set to 3, 5 and 7, and the other part does not use an aggregator, so that it becomes an identity mapping; at the same time, a branch with an aggregator but without attention is set; the four branches with attention are fused by a learnable weighting fusion mechanism, and then spliced and combined with the branch without attention;

[0098]

[0099] ​​​​​​​S340: The output result is adaptively selected by a router mechanism to enter a SSM (State Space Model) branch or a MLP (Multi-layer Perceptron) branch, wherein the SSM branch is composed of layer normalization LN, linear mapping, depth separable convolution, an activation function, a selective scan module (Selective Scan module for 2D data, SS2D), and another layer of LN and linear mapping, forming a local-global joint space modeling path; the MLP branch is a feedforward network composed of two layers of linear transformation (1x1 convolution or full connection) containing residual connection and an activation function, and finally outputs a feature map with the same shape and size as the input;

[0100] S350: A corresponding number of VA Blocks are stacked to obtain feature outputs of stages 1 to 4 with the same size, and the size is

[0101] In this embodiment, the VA Blocks are stacked, and VA Blocks are stacked to obtain the feature of stage 1 Similarly, a number of VA Blocks are stacked respectively to obtain the feature output of stage 2 the feature output of stage 3 , and the feature output of stage 4 , and the size of the output of the four stages is .

[0102] Specifically, the above-mentioned one-dimensional feature extraction module for fusing state space modeling for feature enhancement utilizes the modeling capability of long-range dependency of the stacked VA Blocks to enhance the perception capability of the model to complex structures and long-distance context information in the manufacturing scene. While maintaining the computing efficiency, the module significantly improves the adaptability of the model to complex structures, reflective surfaces, and texture missing areas in the manufacturing scene.

[0103] In one embodiment, S400 includes:

[0104] S410: All channels are evenly divided into groups along the channel dimension, so that each group has channels, and the first​​​​​ a group of features 、 is composed of original features 、 , the grouped correlation volume is obtained as:

[0105] ;

[0106] wherein represents an inner product, d represents a disparity level, represents a horizontal coordinate index of a feature point in an image, represents a vertical coordinate index of a feature point in an image;

[0107] S420: calculate the correlation for all feature groups and disparity levels, and pack all the correlation maps into a matching cost volume with a shape of , wherein represents a maximum disparity, corresponds to the maximum disparity of a feature; when the number of groups , the grouped correlation becomes a full correlation;

[0108] S430: directly construct a connection volume by concatenating left and right features , combine the connection volume with the grouped correlation volume constructed in S410 to obtain a final cost volume .

[0109] In an embodiment, S500 includes:

[0110] The local detail extraction path extracts fine-grained spatial features in the cost volume by using grouped convolution and depth separable convolution, and the global context modeling path generates global context weights based on guide features to model long-distance information for the down-sampled cost volume. The outputs of the two paths are weighted and fused through a channel attention mechanism to realize collaborative optimization of local details and global structures, and finally obtain the output of the dual-path guided context aggregation module, i.e., the aggregated cost volume .

[0111] Specifically, the context aggregation module can effectively improve the structural expression capability of the cost volume and the disparity prediction accuracy without significantly increasing the computational burden, and is especially suitable for industrial manufacturing scenes with missing texture and severe occlusion.

[0112] In an embodiment, S600 includes:

[0113] S610: perform feature transformation on the aggregated cost volume, and extract two output branches through a group of lightweight convolution layers: the classification branch output is represented as ​, the regression branch output is represented as , the classification branch output tensor S Applying a softmax function in the disparity dimension normalizes it to a probability distribution:

[0114] ;

[0115] where denotes the probability distribution of disparity values at image coordinates , denotes the raw score of disparity values at image coordinates , denotes the temporary disparity score traversed in the summation term, denotes the maximum disparity;

[0116] This probability distribution reflects the likelihood of each disparity value at pixel position ;

[0117] S620: Then, the weighted average disparity value is calculated:

[0118] ;

[0119] where denotes the weighted average disparity value;

[0120] S630: At the same time, according to the regression branch output, the offset prediction value at the same position is obtained:

[0121] ;

[0122] where denotes the offset prediction value corresponding to disparity value at the same position;

[0123] S640: The final disparity prediction result is composed of the sum of the weighted average disparity and the offset:

[0124] .

[0125] Specifically, the complete model construction is as follows Figure 3The high-quality disparity map output by the model has wide application extension, and can be used not only for high-precision stereo reconstruction, but also for further serving various industrial visual tasks such as equipment state monitoring, surface defect detection, spatial deformation analysis, and key component pose estimation. By combining the binocular camera parameters and the three-dimensional space mapping relationship, the model provides a unified and reliable spatial perception basis for robot navigation, automatic quality inspection, and dynamic monitoring in intelligent manufacturing systems, and shows good task adaptability and engineering promotion value.

[0126] In one embodiment, the multi-task joint loss function in S700 is specifically:

[0127] ;

[0128] wherein, is a classification loss, used to measure the probability prediction accuracy of the model for different disparity values; is a regression loss, used to optimize the sub-pixel offset in disparity prediction; is a smoothness constraint, used to enhance the spatial continuity of the disparity map and reduce noise interference.

[0129] Specifically, the network learning rate, batch size, feature dimension, and other hyperparameters are adjusted according to the results to find the optimal local hyperparameter solution, so as to obtain the best-performing network model.

[0130] Further, the EndPoint Error (EPE) evaluation index is used to evaluate the gap between the predicted disparity map and the true disparity map, and the evaluation index is calculated; a threshold is set, when the EPE is higher than the threshold, the hyperparameters, optimizers, learning rates, etc. are adjusted to further improve the model until the EPE is lower than the set threshold; when the EPE is lower than the threshold, the prediction disparity map precision is considered to be up to standard, and the training is ended to obtain the trained model.

[0131] The data of a large-scale manufacturing scene can be captured in real time by a zedx binocular camera to obtain a disparity in real time, including the following steps: moving a trolley, synchronously collecting a left and right perspective stereo image pair from different perspectives using a zedx camera, and transmitting the collected image data in real time; then, inputting the collected preprocessed image pair into the trained stereo matching model to output a high-precision pixel-level disparity map. The disparity map can be further converted into depth information in combination with camera intrinsic information, used to reconstruct the three-dimensional structure of the manufacturing scene, and support real-time analysis and anomaly detection of the device surface state.

[0132] The stereo matching model constructed by the application has high efficiency, robustness and scalability. It is not only suitable for large-scale manufacturing scenes, but also can be extended to other complex industrial scenes (power, transportation, etc.), and has wide practical value.

[0133] The specific principle of the above-mentioned depth learning-based stereo matching method is as follows: left and right stereo image pairs of a manufacturing scene are captured from different angles by a binocular camera, and the constructed VA Block is used to enhance and extract features. Then, a cost volume is constructed, and the constructed cost volume is aggregated by a double-path guided context aggregation module to obtain a predicted disparity prediction result. Based on the predicted disparity map, not only can it be used for stereo reconstruction, but also can it be combined with the intrinsic parameters of the binocular camera to calculate the depth and generate an accurate three-dimensional point cloud, thereby completing the stereo reconstruction of the scene. The high-quality disparity map can also be directly applied to real-time monitoring of the state of the equipment to identify any abnormalities or potential problems on the surface of the equipment, such as surface wear, cracks or part misplacement. In this way, not only is high-precision stereo matching of key areas in the manufacturing scene achieved, but also comprehensive state monitoring support is provided to meet the high-precision reconstruction and real-time monitoring needs in the manufacturing scene, thereby providing an advanced technical solution for the intelligentization and digitization of the industrial manufacturing field.

[0134] The method provided by the present application can realize real-time processing of newly collected image data through a stereo matching model, high-precision disparity estimation and equipment state monitoring of a large-scale manufacturing scene. By real-time acquisition of left and right view image pairs by a binocular camera and inputting the image pairs into a stereo matching model that has been trained, a pixel-level high-precision disparity map can be quickly generated. The disparity map can be used to restore the depth information of the scene, thereby supporting continuous monitoring of the surface state of the manufacturing equipment. By comparing the real-time acquired disparity features with the standard state, abnormal changes or potential problems on the surface of the equipment, such as wear, cracks or part misplacement, can be effectively identified. The real-time monitoring mechanism helps to discover equipment abnormalities in a timely manner, reduces maintenance costs and avoids production stagnation or product quality problems caused by faults. At the same time, the spatial difference information output by the system can provide intuitive and accurate decision-making basis for the operation and maintenance team, thereby improving the efficiency of the equipment operation and the overall stability of the production line.

[0135] In one embodiment, a depth learning-based stereo matching system is also provided, comprising:

[0136] A data set construction module is configured to build a large-scale manufacturing scene data acquisition platform, to synchronously acquire stereo image pairs and corresponding real depth information by a binocular camera and a laser radar, and to construct a training data set containing left view, right view and real disparity map after calibration and calculation;

[0137] A model construction module is configured to construct a stereo matching neural network model, including a unary feature extraction module, a cost volume construction module, a context aggregation module and a disparity prediction module;

[0138] The unary feature extraction module is constructed based on stacked visual state space blocks. It is used to receive the left and right views and perform deep semantic feature extraction by combining selective scanning mechanism with multi-head local attention. It realizes joint modeling of local and global contextual information and outputs feature maps of the left and right images.

[0139] The cost volume construction module receives feature maps from the left and right images, calculates the grouping correlation between features and combines feature connections to construct the cost volume for disparity search.

[0140] The context aggregation module receives the cost volume and performs feature enhancement and aggregation on the cost volume through a two-branch structure containing local detail extraction paths and global context modeling paths, so as to fuse multi-scale context information.

[0141] The disparity prediction module is used to calculate the final predicted disparity result by using a decoding mechanism that fuses classification and regression strategies to calculate the aggregated cost volume.

[0142] The training and optimization module is used to supervise the training of the model by combining the training dataset with a multi-task joint loss function to optimize its parameters until the difference between the predicted disparity result output by the model and the real disparity map is within a preset threshold. The trained and optimized stereo matching neural network model is then deployed on the data acquisition platform to process the real-time acquired stereo image pairs and output disparity maps for use in 3D vision tasks.

[0143] For specific limitations regarding a deep learning-based stereo matching system, please refer to the limitations of a deep learning-based stereo matching method described above, which will not be repeated here. Each module in the aforementioned deep learning-based stereo matching system can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0144] A computer device includes a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of a deep learning-based stereo matching method.

[0145] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of a deep learning-based stereo matching method.

[0146] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the above-mentioned embodiments of each method. Any reference to memory, storage, database or other medium used in each embodiment provided by the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0147] The above describes in detail a stereo matching method and system based on deep learning provided by the present application. The principles and implementation modes of the present application are described by applying specific examples in this paper. The above description of the embodiments is only used to help understand the core idea of the present application. It should be pointed out that for those skilled in the art, without departing from the principles of the present application, the present application can be improved and modified in several ways, and these improvements and modifications also fall within the protection scope of the claims of the present application.

Claims

1. A stereo matching method based on deep learning, characterized in that, The method includes the following steps: S100: Build a large-scale manufacturing scene data acquisition platform. Simultaneously acquire stereo image pairs and corresponding real depth information through binocular cameras and LiDAR. After calibration and calculation, construct a training dataset containing left view, right view and real disparity map. S200: Construct a stereo matching neural network model, including a unary feature extraction module, a cost volume construction module, a context aggregation module, and a disparity prediction module; S300: Inputs the left and right views into the unary feature extraction module. This module is constructed based on stacked visual state space blocks and performs deep semantic feature extraction through a combination of selective scanning and multi-head local attention, achieving joint modeling of local and global contextual information and outputting feature maps of the left and right images. The unary feature extraction module in S300 is implemented by stacking multiple visual attention modules (VA Blocks). S300 includes: S310: Pair the acquired stereo image with... The main Stem Block is divided into smaller patch blocks, which are then subjected to a series of convolution, batch normalization, ReLU activation, and pooling operations to obtain an output of size [size missing]. Feature map , ; S320: Transfer the feature map with shape size [B, H, W, C] , The feature map is transformed into a shape and size of [B, H×W, C], processed by depthwise separable convolution, and then the convolutional position encoding is introduced. The position-encoded features are normalized using LN, and then a linear transformation is used to generate a query vector Q, a key vector K, and a value vector V. Here, B represents the batch size, H and W represent the height and width of the feature map, respectively, and C is the number of channels. S330: Divide Q, K, and V into five parts. Use aggregators of different kernel sizes to generate group agents on four of these parts. Three of these parts have kernel sizes of 3, 5, and 7, while the third part does not use an aggregator, making it an identity mapping. At the same time, a branch with an aggregator but without attention is set. The four branches with attention are then merged with the branch without attention through a learnable weighted fusion mechanism and then combined. S340: The output result is adaptively selected to enter the State Space Model (SSM) branch or the Multilayer Perceptron (MLP) branch through a router mechanism. The SSM branch consists of a normalized Linear N layer, a linear mapping, a depthwise separable convolution, an activation function, a selective scanning module, and another LN layer with a linear mapping, forming a local-global joint spatial modeling path. The MLP branch is a feedforward network consisting of two layers of linear transformations and activation functions with residual connections, and the final output is a feature map with the same shape and size as the input. S350: Stack the corresponding number of VA Blocks to obtain feature outputs of the same size for stages 1 to 4, with a size of [size missing]. The four-stage features are concatenated along the channel dimension to form a total of channels. Univariate feature map , ; S400: Input the feature maps of the left and right images into the cost volume construction module, and construct the cost volume for disparity search by calculating the group correlation between features and combining feature connections; S500: The cost volume is input to the context aggregation module. Through a dual-branch structure containing local detail extraction paths and global context modeling paths, the cost volume is enhanced and aggregated to fuse multi-scale context information. S600: The aggregated cost volume is input into the disparity prediction module, and a decoding mechanism that combines classification and regression strategies is used to calculate the final predicted disparity result. S700: Supervised training of the model is performed using the training dataset and a multi-task joint loss function to optimize its parameters until the difference between the predicted disparity result output by the model and the real disparity map is within a preset threshold. The trained and optimized stereo matching neural network model is then deployed on the data acquisition platform to process the real-time acquired stereo image pairs and output a disparity map for use in 3D vision tasks.

2. The method according to claim 1, characterized in that, S100 includes: S110: Fix the binocular camera and lidar to the mobile robot car. Fix the relative positions of the camera and lidar, and adjust the installation position and angle so that its field of view is facing forward. S120: Connects the binocular camera, LiDAR, and NVIDIA Jetson AGX Orin developer kit to provide power and data transmission to the camera and LiDAR. It also fixes the NVIDIA Jetson AGX Orin development board on the vehicle, installs the matching acquisition card for data acquisition, and uses a host computer for remote communication and control. S130: Time synchronization between the binocular camera and the LiDAR is achieved through ROS nodes, enabling the binocular camera and the LiDAR to acquire data synchronously. S140: Multiple sets of scene data are acquired by wirelessly controlling the movement of a mobile robot car. The binocular camera acquires image data, the lidar acquires point cloud data, and the intrinsic parameter matrix and distortion coefficient of the left and right cameras are obtained through intrinsic parameter calibration. S150: The camera and radar are jointly calibrated using intrinsic parameter matrices and distortion coefficients to obtain their positional relationship. The 3D point cloud acquired by the lidar is then projected onto the image using this positional relationship to obtain a depth map. The specific relationship is expressed as follows: ； in It is the representation of the point in the left camera coordinate system. This represents the x-axis coordinate components of the camera position in a three-dimensional Cartesian coordinate system. This represents the y-axis coordinate component of the camera position in a three-dimensional Cartesian coordinate system. This represents the z-axis coordinate component of the camera position in a three-dimensional Cartesian coordinate system. It is the representation of a point in the lidar coordinate system. This indicates the lateral coordinates of the lidar measurement point. This indicates the longitudinal coordinates of the lidar measurement point. R represents the depth coordinates of the lidar measurement point, and R is the rotation matrix. It is a translation vector; S160: Reusing the camera's own parameters, such as camera focal length. Baseline length The true disparity map is calculated using the following formula. : ； S170: The left view, right view, and disparity map are treated as a set of data. The collected data is divided into training set and validation set according to a preset ratio to complete the production of a dataset for a large-scale manufacturing scene.

3. The method according to claim 2, characterized in that, The S400 includes: S410: Divide all channels evenly along the channel dimension. There are groups, therefore each group has The first channel, the... Feature groups , It is based on the original features , The The grouped related volumes are calculated as follows: in d represents the inner product, and d represents the parallax level. This represents the horizontal coordinate index of a feature point in the image. This represents the vertical coordinate index of a feature point in the image; S420: Calculate the correlation for all feature groups and disparity levels, and package all correlation maps into a shape of... The matching cost volume, where Indicates the maximum parallax. The maximum disparity corresponding to the feature; when the number of groups Grouped correlation becomes perfect correlation; S430: Cascade left and right features to directly form a connecting volume. The connection volume is compared with the grouped related volume constructed by S410. Combined to obtain the final cost volume .

4. The method according to claim 3, characterized in that, The S500 includes: The local detail extraction path employs grouped convolution and depthwise separable convolution to extract fine-grained spatial features from the cost volume. The global context modeling path generates global context weights based on guided features, performing long-range information modeling on the downsampled cost volume. The outputs of the two paths are weighted and fused through a channel attention mechanism to achieve collaborative optimization of local details and global structure. Finally, the output of the dual-path guided context aggregation module is obtained, which is the aggregated cost volume. .

5. The method according to claim 4, characterized in that, The S600 includes: S610: Perform feature transformation on the aggregated cost volume, and extract two output branches through a set of lightweight convolutional layers: the classification branch output is represented as... The regression branch output is represented as Output tensor for classification branch S Apply the softmax function along the disparity dimension to normalize it into a probability distribution: ； in, Indicates the coordinates in the image For disparity values The probability distribution, Indicates the coordinates in the image For disparity values The original matching score, This represents the temporary parallax score traversed in the summation term. Indicates the maximum parallax; This probability distribution reflects the pixel location Below, the probability of each disparity value; S620: Then calculate the weighted average disparity value: ； in, This represents the weighted average disparity value; S630: Simultaneously, based on the regression branch output, the predicted offset value is obtained at the same location: ； in, This indicates the parallax value at the same location. The predicted offset value; S640: Final disparity prediction results It consists of the sum of weighted average disparity and offset: 。 6. The method according to claim 5, characterized in that, The joint loss function for multiple tasks in S700 is as follows: ； in, The classification loss is used to measure the accuracy of the model's probability predictions for different disparity values. The regression loss is used to optimize sub-pixel offsets in disparity prediction; Smoothness constraints are used to enhance the spatial continuity of disparity maps and reduce noise interference.

7. A deep learning-based stereo matching system that performs the method as described in any one of claims 1 to 6, characterized in that, include: The dataset construction module is used to build a large-scale manufacturing scene data acquisition platform. It synchronously acquires stereo image pairs and corresponding real depth information through binocular cameras and LiDAR. After calibration and calculation, it constructs a training dataset containing left view, right view and real disparity map. The model building module is used to build stereo matching neural network models, including a unary feature extraction module, a cost volume building module, a context aggregation module, and a disparity prediction module. The unary feature extraction module is constructed based on stacked visual state space blocks. It is used to receive the left and right views and perform deep semantic feature extraction by combining selective scanning mechanism with multi-head local attention. It realizes joint modeling of local and global contextual information and outputs feature maps of the left and right images. The cost volume construction module receives feature maps from the left and right images, calculates the grouping correlation between features and combines feature connections to construct the cost volume for disparity search. The context aggregation module receives the cost volume and performs feature enhancement and aggregation on the cost volume through a two-branch structure containing local detail extraction paths and global context modeling paths, so as to fuse multi-scale context information. The disparity prediction module is used to calculate the final predicted disparity result by using a decoding mechanism that fuses classification and regression strategies to take the aggregated cost volume. The training and optimization module is used to supervise the training of the model by combining the training dataset with a multi-task joint loss function to optimize its parameters until the difference between the predicted disparity result output by the model and the real disparity map is within a preset threshold. The trained and optimized stereo matching neural network model is then deployed on the data acquisition platform to process the real-time acquired stereo image pairs and output disparity maps for use in 3D vision tasks.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images