End-to-end multi-task stereo depth and semantic fusion visual perception system and method

Abstract

The application provides an end-to-end multi-task stereo depth and semantic fusion visual perception system and method. It comprises an image acquisition module for outputting image data carrying calibration parameter identification; a fusion preprocessing module for processing synchronous image pairs to obtain registered image pairs; a multi-task network module comprising a shared feature extraction layer, a depth estimation branch and a semantic segmentation branch; a multi-task coupling unit for applying consistency constraints to shared features and branch features, and transmitting geometric priors and class priors between branches; and an output interface module for packaging depth maps and semantic masks according to a preset data structure, and outputting them to the real-time control interface of an external execution control system through an industrial real-time communication interface for calling. The application can realize end-to-end low latency and deterministic output, time synchronization and calibration consistency maintenance, compact integration and simplified wiring, and stably provide high-precision depth maps and semantic masks to the control system.

Description

Technical Field

[0001] This application relates to the field of industrial embodied intelligence technology, and in particular to an end-to-end multi-task stereo depth and semantic fusion visual perception system and method. Background Technology

[0002] In industrial embodied intelligence scenarios, robotic arms, AGVs, and other automated guided vehicles (AGVs) rely on visual perception to complete environmental modeling, target recognition, and localization to meet the control requirements of production line cycle time and human-machine collaboration. The limited space, strong interference, and long continuous operation time in industrial settings place higher demands on the real-time performance and stability of the perception system.

[0003] Existing mainstream technologies typically adopt an architecture that separates 3D visual perception from visual AI recognition: the stereo / ToF camera generates depth or point cloud locally, an external industrial control computer completes decoding, preprocessing and model inference, and transmits the results to the robot controller or PLC via Ethernet, relying on independent geometric calibration and time synchronization mechanisms.

[0004] The aforementioned existing technologies still have the following problems: First, the number of devices and connections increases the number of hardware failure points, and the stability is insufficient under electromagnetic interference and vibration conditions; second, large data transmission across devices and multiple memory copies make it difficult to control end-to-end latency and jitter, and it is difficult to maintain the consistency of time base and external parameters; third, the split-type design has a large volume and heat dissipation footprint, which is not conducive to deployment on terminal or mobile platforms; fourth, the costs of installation, debugging, recalibration and version maintenance are high, and concurrent expansion is limited, making it difficult to meet the application requirements of high real-time, high stability and narrow space. Summary of the Invention

[0005] In view of this, embodiments of this application provide an end-to-end multi-task stereo depth and semantic fusion visual perception system and method to solve the problems of low stability of the existing technology with a separate architecture, difficulty in controlling end-to-end latency and synchronization caused by cross-device data transmission, and large space occupation due to complex size and wiring.

[0006] The first aspect of this application provides an end-to-end multi-task stereo depth and semantic fusion visual perception system, comprising: an image acquisition module for acquiring synchronous image pairs under a unified time base through a binocular imaging unit and outputting image data carrying calibration parameter identifiers; a fusion preprocessing module for performing geometric correction, noise suppression, and stereo registration on the synchronous image pairs according to the calibration parameters to obtain registered image pairs; a multi-task network module including a shared feature extraction layer, a depth estimation branch, and a semantic segmentation branch; wherein the shared feature extraction layer is used to generate a shared feature map from the registered image pairs, the depth estimation branch is used to construct a cost representation based on the shared feature map and output a depth map, and the semantic segmentation branch is used to perform multi-scale semantic parsing based on the shared feature map and output a semantic mask; a multi-task coupling unit connected to the shared feature extraction layer, the depth estimation branch, and the semantic segmentation branch, for applying consistency constraints to the shared features and branch features, and for transmitting geometric priors and class priors between branches; and an output interface module for encapsulating the depth map and semantic mask according to a preset data structure and outputting them to the real-time control interface of an external execution control system for invocation through an industrial real-time communication interface.

[0007] The second aspect of this application provides a visual perception method for an end-to-end multi-task stereo depth and semantic fusion visual perception system, comprising: acquiring synchronized image pairs by binocular imaging under a unified time base, and associating timestamps and calibration parameter identifiers for the synchronized image pairs; performing geometric correction and epipolar alignment on the synchronized image pairs according to the calibration parameters, performing noise suppression, and completing stereo registration based on correlation metrics and sub-pixel interpolation to obtain registered image pairs; performing weighted hierarchical convolutional encoding on the left and right views of the registered image pairs, and forming a shared feature map through cross-layer fusion; constructing a cost representation based on adaptive disparity range on the shared feature map, and weighting the cost representation. The system aggregates and refines the data through learnable sub-pixel thinning to generate continuous disparity, which is then converted into a depth map. Semantic features are generated based on multi-scale semantic parsing, and a semantic mask is decoded and output. The consistency measure between depth edges and semantic boundaries is calculated in the boundary neighborhood of the shared feature map. Geometric priors are input into the semantic decoding, and class priors are used to perform item-by-item weighting and incompatibility masking on the cost representation to jointly correct the boundary regions of the depth map and the semantic mask. The depth map and semantic mask, along with their corresponding timestamps, calibration parameter identifiers, and model version identifiers, are encapsulated according to a preset data structure and output to the real-time control interface of an external execution control system for invocation via an industrial real-time communication interface.

[0008] The above-described technical solutions adopted in the embodiments of this application can achieve the following beneficial effects:

[0009] The system includes an image acquisition module for acquiring synchronized image pairs using a binocular imaging unit at a unified time base and outputting image data carrying calibration parameter identifiers; a fusion preprocessing module for performing geometric correction, noise suppression, and stereo registration on the synchronized image pairs according to the calibration parameters to obtain registered image pairs; a multi-task network module including a shared feature extraction layer, a depth estimation branch, and a semantic segmentation branch; wherein the shared feature extraction layer generates a shared feature map from the registered image pairs, the depth estimation branch constructs a cost representation based on the shared feature map and outputs a depth map, and the semantic segmentation branch performs multi-scale semantic parsing based on the shared feature map and outputs a semantic mask; a multi-task coupling unit, connected to the shared feature extraction layer, depth estimation branch, and semantic segmentation branch, applies consistency constraints to the shared features and branch features and transmits geometric and class priors between branches; and an output interface module for encapsulating the depth map and semantic mask according to a preset data structure and outputting them to the real-time control interface of an external execution control system for invocation via an industrial real-time communication interface. This application enables end-to-end low latency and deterministic output, time synchronization and calibration consistency maintenance, compact integration and simplified wiring, and provides high-precision depth maps and semantic masks to the control system stably. Attached Figure Description

[0010] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 This is a schematic diagram of the overall structural framework of the end-to-end multi-task stereo depth and semantic fusion visual perception system provided in the embodiments of this application;

[0012] Figure 2 This is a schematic diagram of the structural composition of the end-to-end multi-task stereo depth and semantic fusion visual perception system provided in the embodiments of this application;

[0013] Figure 3 This is a flowchart illustrating the visual perception method of the end-to-end multi-task stereo depth and semantic fusion visual perception system provided in this application embodiment. Detailed Implementation

[0014] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0015] In the field of industrial embodied intelligence, robotic arms, AGVs, or other automated equipment rely on a visual perception device to clearly see the objects being operated on and the surrounding environment, interact with people, and perform intelligent operations. Ultimately, this enables human-machine collaboration on industrial production lines, solving pressing issues such as labor shortages in the field of intelligent manufacturing.

[0016] To address this issue, existing technical solutions often employ a two-part approach that separates 3D visual perception from visual AI recognition processing: the environmental depth perception calculation is performed within a separate stereo camera, while the visual AI recognition process is completed in a separate industrial control computer.

[0017] As robots and other embodied intelligent agents increasingly penetrate various sectors of human society, especially industrial production, the demands on the intelligence level of these agents are rising. Visual perception systems, as a crucial means of perception for embodied intelligent agents, hold a central position within the entire system and are key to realizing embodied intelligence.

[0018] Although existing technical solutions can also complete the functions of environmental perception, object classification and recognition, and localization of embodied intelligent agents, they also have the following defects: (1) The separation of stereo vision perception and AI recognition calculation leads to a decrease in the overall system stability due to too many devices and too many external connections in the system, which increases the risk of interruption of industrial production capacity; (2) The data transmission delay of the separate solution is large, which cannot meet the needs of timely response of embodied intelligence, and thus cannot be competent for some high real-time process tasks; (3) The workstation space is often limited in the industrial production environment, and the space occupied by the existing separate solution is too large, which makes it unusable in many narrow space environments.

[0019] This application aims to address the problems of low system stability, large data transmission latency, and excessive space occupation caused by the separation of 3D visual perception and visual AI recognition processing in the prior art. It provides an end-to-end multi-task stereo depth and semantic fusion visual perception system, which realizes the integrated processing of stereo depth perception and semantic recognition, improves system stability, reduces data transmission latency, and reduces space occupation, so as to meet the needs of high real-time performance, high stability, and narrow space operation in industrial production.

[0020] The overall structural framework and functions of each part of the end-to-end multi-task stereo depth and semantic fusion visual perception system provided in this application will be described below with reference to the accompanying drawings and embodiments. Figure 1 This is a schematic diagram of the overall structural framework of the end-to-end multi-task stereo depth and semantic fusion visual perception system provided in the embodiments of this application, as shown below. Figure 1 As shown, this end-to-end multi-task stereo depth and semantic fusion visual perception system may specifically include the following:

[0021] I. Image Acquisition Module

[0022] It consists of a binocular camera, a rigid base, a fine-tuning slide, and an FPGA synchronous triggering unit. It is used to simultaneously expose the left and right views under a unified time base and output synchronized image pairs; the base and fine-tuning slide are used for baseline adjustment and locking; the FPGA generates hardware triggers and timestamps, which are output along with calibration parameter identifiers with each frame.

[0023] II. Fusion Preprocessing Module

[0024] It consists of a preprocessing unit and a data buffer unit (e.g., 8MB DDR3). The preprocessing unit performs distortion correction, noise suppression, and stereo registration on the synchronized image pairs according to the calibration parameters, generating registered image pairs; the data buffer unit manages the registered image pairs, timestamps, and parameter version identifiers with a circular buffer, providing a constant-time data source to the downstream.

[0025] III. Multi-task network module

[0026] It includes a ResNet-50 feature extraction layer, a depth estimation branch, and a semantic segmentation branch. The feature extraction layer performs weight-sharing hierarchical encoding on the registered image pairs and fuses them across layers to generate a shared feature map; the depth estimation branch constructs a cost representation based on the shared feature map and outputs a depth map; the semantic segmentation branch performs multi-scale parsing on the shared feature map and outputs a semantic mask; the module has multi-task coupling logic set up to allow geometric priors and class priors to be passed between the two branches.

[0027] IV. Output Interface Module

[0028] It consists of an EtherCAT / PROFINET communication unit and dual redundant links. It is used to encapsulate depth maps and semantic masks according to a preset data structure and transmit them periodically on the industrial real-time bus via zero-copy DMA; when the main link fails, it switches to the backup link and maintains continuous data output to the external execution control system.

[0029] V. Power Module

[0030] It consists of a wide-voltage input unit and an isolated output unit. It receives 9-36V DC input and provides multiple electrically isolated regulated outputs to power the image acquisition module, fusion processing module, multi-task network module, and output module, and provides power-good and reset control signals to the relevant modules.

[0031] The system's data flow is sequentially connected in the order of "acquisition → preprocessing → multi-task inference → industrial interface output", and the power module provides unified power supply and timing control for each functional module.

[0032] The following is based on Figure 2 The diagram shown illustrates the overall system architecture. Combined with the accompanying drawings and specific embodiments, this application provides a detailed explanation of the specific structure and functions of the end-to-end multi-task stereo depth and semantic fusion visual perception system. Figure 2 This is a schematic diagram illustrating the structural composition of the end-to-end multi-task stereo depth and semantic fusion visual perception system provided in this application embodiment, as shown below. Figure 2 As shown, this end-to-end multi-task stereo depth and semantic fusion visual perception system may specifically include the following modules:

[0033] The image acquisition module 201 is used to acquire synchronized image pairs under a unified time base through a binocular imaging unit and output image data carrying calibration parameter identifiers;

[0034] The fusion preprocessing module 202 is used to perform geometric correction, noise suppression and stereo registration on the synchronized image pairs according to the calibration parameters to obtain registered image pairs;

[0035] The multi-task network module 203 includes a shared feature extraction layer, a depth estimation branch, and a semantic segmentation branch; wherein, the shared feature extraction layer is used to generate a shared feature map from the registered image pair, the depth estimation branch is used to construct a cost representation based on the shared feature map and output a depth map, and the semantic segmentation branch is used to perform multi-scale semantic parsing based on the shared feature map and output a semantic mask.

[0036] The multi-task coupling unit 204 is connected to the shared feature extraction layer, the depth estimation branch, and the semantic segmentation branch. It is used to apply consistency constraints to the shared features and branch features, and to pass geometric priors and class priors between branches.

[0037] The output interface module 205 is used to encapsulate the depth map and semantic mask according to a preset data structure and output them to the real-time control interface of the external execution control system for use through the industrial real-time communication interface.

[0038] In some embodiments, the image acquisition module is used for:

[0039] Synchronization trigger signals are generated based on a unified time base and sent to the left and right imagers respectively.

[0040] Under the action of the synchronous trigger signal, the left image frame and the right image frame are acquired simultaneously, and a timestamp corresponding to the unified time base is written to the image frame;

[0041] During assembly or operation, parameter calibration is performed, and the calibration parameter identifiers are associated with image frames and output with each frame.

[0042] When the preset recalibration trigger condition is detected, recalibration is performed and the calibration parameters are updated. During the recalibration, the previous version of the calibration parameters are used for output. After the recalibration is completed, the updated calibration parameters are used to identify the output with each frame.

[0043] The binocular imaging unit is geometrically configured using a baseline adjustable mechanism and an optical axis alignment mechanism to meet the geometric relationships required for stereo registration.

[0044] Specifically, the image acquisition module provided in this embodiment is used to complete binocular synchronous acquisition under a unified time base, write timestamps and calibration parameter identifiers with each frame, perform recalibration and perform parameter atomic switching under trigger conditions, and complete geometric configuration through a baseline adjustable mechanism and an optical axis alignment mechanism.

[0045] The implementation methods of key technical features and the concepts of important terms involved in this embodiment are explained below:

[0046] I. Key Technical Features and Implementation Methods

[0047] Unified Time Base and Synchronous Triggering. Using the FPGA's internal time base as the unified time base, a synchronous trigger signal with a fixed pulse width is generated by a trigger generator and sent to the left and right imagers respectively via a 50Ω coaxial cable. The trigger signal pulse width is 10μs, and the channel delay is ≤10ns. The FPGA counter accumulates at a resolution of 1μs, serving as the end-to-end timestamp source.

[0048] The timestamp and calibration parameter identifier are associated with each frame. Each frame has an 8-byte timestamp and calibration parameter identifier written in the frame header. The identifier is formed by combining the current calibration parameter version number and the checksum, and is output with the frame and used as a parameter retrieval index downstream.

[0049] Parameter calibration and recalibration. Initial calibration is performed after assembly to obtain intrinsic, extrinsic, and distortion parameters, generating a calibration parameter identifier V0. During operation, when a vibration acceleration ≥5g or an ambient temperature change >10℃ is detected, the recalibration process is triggered. Ten sets of checkerboard images within a fixed field of view are acquired, and the new parameters are calculated using Zhang's calibration method, generating an identifier V1. During the recalibration calculation, V0 is used for frame-by-frame output; after completion, V0 is switched to V1 at the frame boundary. The entire recalibration process takes ≤30 seconds.

[0050] Geometric Configuration. The baseline adjustable mechanism consists of a rigid aluminum alloy base and a fine-tuning slide. The base flatness is ≤0.01mm, the minimum adjustment of the micrometer head is 0.01mm, the baseline adjustment range is 0~200mm, and the repeatability error after locking is ≤0.1mm. During assembly, the optical axis alignment mechanism uses a laser collimator (accuracy ±1″) to adjust the parallelism of the optical axes of the two imagers, with a parallelism error ≤0.05°. A cross-shaped level (accuracy 0.02mm / m) is installed at the bottom of the base to ensure a stable relationship between the camera's imaging plane and the working plane.

[0051] II. Explanation of Important Terms and Assembly Points

[0052] Unified time base: refers to a single clock reference provided by the FPGA for use in triggering, timing, and timestamp writing.

[0053] Calibration parameter identifier: A compact identifier indicating the current intrinsic, extrinsic, and distorted parameter versions, used for consistent parameter referencing and atomic switching in the processing chain.

[0054] Atomic parameter switching: refers to the replacement of the identifier at the boundary between the end of one frame and the beginning of the next frame, to avoid the mixing of parameters within the same frame.

[0055] Mechanical alignment sequence: first, coarsely adjust the baseline, then align the optical axis, and finally lock and record the target baseline value and attitude recording code as the initial guess for subsequent recalibration.

[0056] The implementation process of this embodiment will be explained in detail below with examples from specific scenarios. The specific content is as follows:

[0057] First, assembly and initial calibration. Two global shutter industrial cameras (1280×1024 resolution, 30fps frame rate) are mounted on a rigid base, and the baseline is set to 120mm and locked according to the target working condition. A laser collimator is used to correct the optical axis parallelism, and a level is used to adjust the perpendicular relationship with the normal of the work position. A fixed checkerboard calibration plate is placed within the field of view, 10 sets of calibration images are acquired, the intrinsic parameters, extrinsic parameters, and (k1k3, p1p2) distortion parameters are calculated, a calibration parameter identifier V0 is generated, and written to the FPGA register.

[0058] Next, synchronous acquisition and frame-by-frame writing are performed. The system enters the operating state, and the FPGA times the data at a resolution of 1μs and periodically outputs 10μs trigger pulses to the left and right imagers to achieve simultaneous exposure. After acquiring a pair of image frames, a timestamp T and the current calibration parameter identifier V0 are sequentially written into the frame header and then sent to the downstream fusion processing module. To control cross-camera jitter, the FPGA compares the timestamp difference after receiving the feedback from the two cameras and performs fine corrections within a range of ±0.2μs.

[0059] Then, recalibration and parameter switching occur. After long-term operation, when the accelerometer detects vibration ≥5g or the temperature sensor detects a temperature difference >10℃, the recalibration process is triggered. The system automatically acquires 10 sets of checkerboard images without stopping acquisition, calculates new parameters, and generates an identifier V1. After receiving the "new parameters ready" signal, the FPGA records the switching point. After the current frame output ends, the identifier changes from V0 to V1 along with the frame, and the old and new identifiers are retained for one cycle for downstream comparison.

[0060] Next, maintenance and reconfiguration. When the work station needs to be switched and the working distance needs to be adjusted, loosen the locking mechanism, adjust the micrometer head to the desired baseline value (e.g., from 120mm to 160mm), lock it again after completion, and repeat the above initial calibration process to generate a new identifier V2; the time base and trigger configuration remain unchanged, only the identifier is updated to drive the processing link to reference new parameters.

[0061] Through the above embodiments, the image acquisition module completes binocular synchronous acquisition under a unified time base, time stamp and calibration parameter identifiers are associated with each frame, recalibration and parameter atomic switching based on trigger conditions, and baseline and optical axis configuration that satisfies the geometric relationship of stereo registration, providing consistent input data and parameter references for subsequent fusion preprocessing and multi-task network inference.

[0062] In some embodiments, the fusion preprocessing module is used for:

[0063] The lookup table-based pixel mapping performs radial and tangential distortion compensation according to the calibration parameters and completes row alignment according to the epipolar constraint. Among them, geometric correction and parameter version identifier are associated with each frame and are atomically switched when the parameters are updated.

[0064] Spatial domain filtering is performed on the vector processing unit, and the filtering parameters are adaptively selected based on image statistics;

[0065] The relative displacement of the left and right views is estimated based on the correlation metric, and the synchronized image pairs are aligned to a uniform pixel grid through subpixel interpolation to output the registered image pairs.

[0066] A preprocessing pipeline with a fixed processing sequence is adopted, and the registered image pairs are associated with and stored with corresponding timestamps and parameter versions in a circular buffer structure for sequential reading by the multi-task network module.

[0067] Specifically, the fusion preprocessing module provided in this embodiment is used to complete the fixed-time processing of geometric correction, spatial domain filtering and stereo registration, and provides the registered image pairs and their timestamps and parameter version identifiers to the multi-task network module in a circular buffer manner.

[0068] The implementation methods of key technical features and the concepts of important terms involved in this embodiment are explained below:

[0069] I. Key Technical Features and Implementation Methods

[0070] Geometric correction and epipolar alignment. Based on the calibration parameter identifiers carried by the image acquisition module with each frame, the current intrinsic, extrinsic, and distortion parameters are retrieved from the parameter storage area. A multi-scale lookup table (LUT) is pre-generated, with 16×16 pixel blocks as index units, recording the mapping from distorted pixels to ideal pixel coordinates. During processing, the mapping is read block by block and bilinear resampling is performed to complete radial and tangential distortion compensation. Subsequently, row alignment is performed on the left and right views according to epipolar constraints, and the right view is translated by integer pixels based on the row offset calculated from the baseline and extrinsic parameters. During geometric correction, the timestamp and parameter version identifier are written to the frame header buffer. If a parameter update request is detected, the old identifier is replaced with the new identifier only at the frame boundary, ensuring atomic parameter switching.

[0071] Spatial domain filtering and adaptive parameter selection. The vector processing unit performs spatial domain filtering in parallel on the corrected left and right views. First, the histogram and local variance map of each frame are calculated, and the filtering strategy is selected accordingly: a 3×3 Gaussian kernel (σ=1.0) is used to smooth high-frequency noise in low-noise scenes; in high-reflectivity or salt-and-pepper noise scenes, the filtering switches between 3×3 and 5×5 adaptive median filtering according to the local variance threshold. Filtering is performed in a pipelined manner within blocks, and boundary pixels are expanded using mirroring.

[0072] Stereo registration and subpixel interpolation. To eliminate subtle view displacements caused by assembly tolerances and temperature drift, a phase correlation method is used to estimate the relative displacements (Δx, Δy) of the left and right views in the frequency domain. The initial displacement values ​​are provided by the row offset after epipolar alignment. The estimated displacement is decomposed into integer and fractional parts. The integer part is achieved by pixel shifting, while the fractional part is resampled using bilinear interpolation to align the right view to the uniform pixel grid of the left view, generating a registered image pair.

[0073] A fixed-time pipeline and a circular buffer are used. Internally, the module follows a three-stage pipeline: geometric correction → spatial domain filtering → stereo registration, with fixed inter-frame delays. A circular buffer is configured at the output, with each buffer entry containing {left registration frame, right registration frame, timestamp, parameter version identifier, and ready flag}. The write pointer advances frame by frame and sets the flag upon readiness. The read pointer is read sequentially by the multi-task network module. When a parameter version change is detected, a switching flag is recorded at the boundary between the first frame of the old and new versions, facilitating consistent referencing downstream during batch processing or fusion. To avoid write-read contention, the buffer employs single-write, single-read arbitration and minimum watermark control. When downstream processes are temporarily blocked, data is maintained in the ready queue order, preventing redundant computation within the pipeline.

[0074] II. Key Terms and Implementation Details

[0075] LUT pixel mapping: blocks are used as indices to reduce random access overhead, and coordinates are restored by pixel-level interpolation within the block; multi-scale pointers use hierarchical LUT sets for different focal lengths and distortion radii.

[0076] Parameter version identifier: Consists of version number and check code, used as a unique key for downstream parameter retrieval and processing; atomic switching is limited to inter-frame switching only.

[0077] Phase correlation estimation: The cross power spectrum is calculated in the frequency domain and the main peak is located to obtain the sub-pixel estimate of (Δx,Δy), which is used as the input for sub-pixel interpolation.

[0078] Circular cache depth and pointers: The cache depth is configured based on the frame rate and network inference time. For example, under the conditions of 30fps and downstream inference of 8ms, the cache depth is set to 8 to ensure sequential reading and version consistency.

[0079] The implementation process of this embodiment will be explained in detail below with examples from specific scenarios. The specific content is as follows:

[0080] Taking the robotic arm assembly station as an example, the input is a pair of synchronized images with a resolution of 1280×1024 and a frame rate of 30fps. The module first loads the corresponding LUT according to the calibration parameter identifier V1, performing distortion compensation and epipolar alignment. Then, based on local variance, a 3×3 Gaussian filter is selected to suppress surface noise on the metal parts. Next, phase correlation estimation is used to obtain the displacement of the right view relative to the left view, Δx=0.25 pixels and Δy=0.00 pixels, completing sub-pixel-level bilinear interpolation and resampling the right view to a unified pixel grid. The generated registered image pair, along with the timestamp Tn and parameter version identifier V1, is written to a circular buffer, and a ready flag is set for the multi-task network module to read. When recalibration is subsequently triggered to obtain a new identifier V2, the module completes the switch from V1 to V2 at the frame boundary and records the switch point in the cache metadata, ensuring that the downstream shared feature extraction layer references consistent parameters within the same batch.

[0081] Through the above embodiments, the fusion preprocessing module realizes stereo registration based on lookup table for geometric correction and epipolar alignment, vectorized adaptive spatial filtering, phase correlation and subpixel interpolation, and ordered data output with fixed-time pipeline and circular buffer as the core, providing the multi-task network module with input data that is consistent in registration, parameters and timestamps.

[0082] In some embodiments, the shared feature extraction layer is used for:

[0083] Weight-shared hierarchical convolutional feature encoding is performed on the left and right views of the registered image pair to obtain multi-scale feature representations;

[0084] Based on cross-layer connections, detailed features and semantic features at different scales are fused and a unified tensor format is used to form a shared feature map;

[0085] The shared feature map is provided to both the depth estimation branch and the semantic segmentation branch, and the feature index corresponding to the location of the shared feature map is output for use by the multi-task coupling unit.

[0086] Specifically, the shared feature extraction layer provided in this embodiment is used to perform weight-sharing hierarchical convolutional feature encoding, cross-layer fusion, and tensor format unification on the registered image pairs, generate a shared feature map, and simultaneously serve the depth estimation branch and the semantic segmentation branch; at the same time, it outputs the feature position index corresponding to the position of the shared feature map for the multi-task coupling unit to call.

[0087] First, the input consists of the registered image pair, its timestamp, and parameter version identifier output from the fusion preprocessing module. The shared feature extraction layer feeds into the isomorphic encoding branch according to the left and right views, using weight sharing to ensure consistency in the feature space of the two views. After encoding and fusion, a single-path shared feature map is generated, with a uniform tensor stride of 16. The number of channels is set to a fixed value during deployment to accommodate both downstream branches. The output includes the shared feature map and feature location index, which, along with the timestamp and parameter version identifier, are written to a read-only buffer.

[0088] Furthermore, the left and right views undergo the same convolutional layers and downsampling sequences, with convolutional kernel parameters fully shared to reduce domain differences introduced by disparity. The encoding layers extract edge, texture, and high semantic features sequentially from shallow to deep, using a combination of stride and pooling to progressively reduce resolution and increase receptive field. To maintain traceable geometric alignment, each layer employs symmetrical padding and deterministic downsampling, ensuring a one-to-one spatial index correspondence between left and right features within the same layer. The left and right encoded outputs are aligned and concatenated or averaged bitwise within the same layer to form a binocular fusion feature pyramid, which serves as input for subsequent cross-layer fusion.

[0089] Furthermore, to simultaneously preserve details and semantic information, high-resolution features from the shallow layers and high-semantic features from the deep layers in the pyramid are fused through cross-layer connections. Specifically, deep features are upsampled to the target resolution and then fused with corresponding shallow features along the channel dimension. During the fusion process, the channel dimension is standardized and the number of channels is unified to avoid downstream interface ambiguity caused by channel inconsistencies. The multi-scale fusion is performed progressively from deep to shallow, ultimately yielding intermediate features that balance spatial resolution and semantic abstraction.

[0090] Furthermore, after multi-scale fusion is completed, a shared feature map is generated according to a preset step size and number of channels. This tensor adopts a fixed memory layout and alignment strategy to facilitate parallel reading by the two downstream branches. Only one copy of the shared feature map is generated, and it is accessed simultaneously by the depth estimation branch and the semantic segmentation branch in read-only mode in the buffer to avoid redundant computation and extra copying.

[0091] Furthermore, to support consistency constraints in the boundary region for multi-task coupling units, a feature location index is constructed to characterize the mapping relationship between each position of the shared feature map and the coordinates of the original image. This index is calculated based on the registration mapping and the total step size, recording the center point of the original image coordinates and the effective sampling range corresponding to each feature position. It is encoded in a fixed-point format and stored in association with the parameter version identifier. The feature location index maintains the same spatial size and row and column order as the shared feature map, allowing the coupling unit to directly read it by position alignment.

[0092] Furthermore, after the shared feature map and feature location index are written to the read-only buffer, the depth estimation branch and the semantic segmentation branch consume data in parallel according to their sequential numbers. When a change in the parameter version identifier is detected, an atomic switch between the old and new versions is performed at the frame boundary to ensure that the two branches reference consistent features and location indices within the same frame. The buffer is equipped with a minimum watermark and overflow protection to maintain the reading order and avoid duplicate encoding when the downstream is briefly blocked.

[0093] The implementation process of this embodiment will be explained in detail below with examples from specific scenarios. The specific content is as follows:

[0094] At the robotic arm assembly station, with an input resolution of 1280×1024 and a uniform step size of 16, the spatial size of the shared feature map is 80×64. This shared feature map is used by the depth estimation branch to construct a disparity-related cost representation and by the semantic segmentation branch for multi-scale semantic parsing. The feature location index provides the spatial correspondence required for boundary alignment of the multi-task coupling units, thereby achieving unified referencing without changing the interface between the two branches.

[0095] Through the above embodiments, the shared feature extraction layer completes hierarchical encoding with weight sharing, cross-layer multi-scale fusion and tensor format unification, stably provides shared feature maps and feature location indexes to the downstream, and ensures consistent reference to the same frame data by the two branches and coupling units with the cooperation of frame-level parameter atomic switching and read-only buffer mechanism.

[0096] In some embodiments, the depth estimation branch is used for:

[0097] Cost representation is constructed based on shared feature maps within the adaptive disparity range, which is dynamically determined at the pixel or region level according to scene distance distribution or image statistics.

[0098] The class priors passed from the shared feature maps and multi-task coupling units are weighted and aggregated with the cost representations corresponding to the attention weights, and masking constraints are applied to unmatched regions.

[0099] Learnable subpixel thinning is performed on the aggregation result, inverse residual convolution is used to recover continuous disparity, and continuous disparity is converted into a depth map according to the imaging model;

[0100] During the inference phase, the shared feature map cache is reused and executed in parallel with the semantic segmentation branch to align the output depth map with the data structure of the output interface module.

[0101] Specifically, the depth estimation branch in this embodiment works on the shared feature map output by the shared feature extraction layer, and sequentially completes the adaptive disparity range determination, cost representation construction and weighted aggregation, unmatched region masking, learnable sub-pixel thinning and disparity to depth conversion. It is executed in parallel with the semantic segmentation branch during the inference stage, and outputs a depth map with the same field format as the output interface module.

[0102] First, the branch reuses the shared feature map and feature location index from the read-only buffer, while simultaneously reading the timestamp and calibration parameter identifier carried with the frame. The module establishes an internal context using the frame number as the key, obtaining the disparity statistical histogram and texture intensity map of the previous frame as priors. Based on local gradient magnitude, texture entropy, and the disparity distribution of the previous frame, it dynamically determines the adaptive disparity range at the pixel or region level, where region division uses the shared features... Figure 1 The mesh cells are uniform (e.g., 16×16). A larger upper limit is set for close-range cells with sufficient texture, and the range is shrunk for distant or low-texture cells. The upper and lower bounds are updated exponentially and truncated according to camera geometric constraints to obtain the search window [dmin, dmax] for each position.

[0103] Then, the branch aligns with the disparity candidates at each position among the shared features of the left and right views, and calculates a disparity-level similarity metric to form the cost representation. The similarity calculation uses a higher-level description of intra-channel correlation or cosine similarity. The output cost representation is encoded with a fixed length in the channel dimension and laid out in the order of consecutive disparity layers to facilitate subsequent sequential access and hardware cache prefetching. To ensure data alignment with the parallel semantic branches, the spatial step size of the cost representation is related to the shared features. Figure 1 In this case, the index relationship is provided by the feature location index.

[0104] Next, the branch receives the category prior tensor provided by the multi-task coupling unit and jointly generates an attention weight map with the shared feature map. The attention weights are weighted and aggregated on the cost representation in the spatial and disparity dimensions, so that different disparity candidates belonging to the same spatial location are superimposed according to their weights to obtain the aggregated cost response. To handle possible unmatched regions, the branch performs forward and backward checks on left-right consistency, and at the same time refers to occlusion indicators and texture confidence, applies masking constraints to positions that do not meet consistency or are judged to be occluded, resets the weights of the corresponding disparity terms to zero, and records the unmatched markers. The subsequent refinement stage uses this to bypass the masked candidates.

[0105] Subsequently, the branch performs learnable sub-pixel thinning on the aggregated discrete disparity responses. The thinning network employs a cascaded structure composed of inverse residual convolutional units with a stride of 1. If necessary, it incorporates interpolation branches to recover continuous disparities with a stride higher than the shared feature map. To improve geometric consistency at boundaries, the thinning network uses the boundary guiding tensor generated from the shared feature map as additional input, concatenating it with the backbone features while maintaining channel alignment, to participate in the thinning process. The network output is a continuous disparity map of the same size as the shared feature map, with an optional pixel-wise confidence plane.

[0106] After completing continuous disparity estimation, the branch converts the disparity into depth based on the imaging model. The conversion process reads the current baseline b and effective focal length f, indicated by the calibration parameters, and performs pixel-by-pixel calculations using D=bf / disp. To ensure compatibility with the output interface module fields, the calculation results are quantized in the fixed-point domain and limited to a preset depth range, outputting a 16-bit single-channel depth map. If a confidence plane exists, it is written to the payload buffer along with the depth map as an additional plane, with the field order consistent with the real-time object mapping of the output interface module.

[0107] During the inference phase, the depth estimation branch and the semantic segmentation branch execute in parallel with the same frame number. Shared feature maps and feature location indices are accessed simultaneously by both branches within a read-only buffer. When a change in the parameter version identifier is detected, switching occurs only at frame boundaries, ensuring that both branches reference the same set of internal and external distortion parameters within the same frame. After completing the above calculations, the depth estimation branch writes the depth map, timestamp, calibration parameter identifier, and model version identifier into a reserved area of ​​the payload buffer, awaiting the output interface module to read from the zero-copy address and send it according to a set period.

[0108] In one example scenario, the input image size at the robotic arm assembly station is 1280×1024, the shared feature map stride is 16, and the corresponding spatial size is 80×64. The system sets an adaptive disparity upper limit of 64 in the fixture region based on statistics from the previous frame, and sets it to 32 in the far-end background region. The cost is represented by stacking disparity layers and aggregating them with class prior weights; unmatched regions are jointly labeled by left-right consistency checks and occlusion detection. The continuous disparity output from the refined network is converted into a depth map using D=bf / disp and written to the payload buffer in a 16-bit fixed-point format. This map, along with a timestamp and calibration parameter identifier, is provided to the output interface module for real-time industrial communication. The above process constitutes the complete implementation path of the depth estimation branch in this embodiment, maintaining consistency with the aforementioned modules in terms of timing and data interface.

[0109] In some embodiments, semantic segmentation branches are used for:

[0110] Context modeling of shared feature maps is performed using multi-scale dilated convolutional arrays to form semantic feature maps;

[0111] Differential decoding and refinement are performed on the semantic boundary corresponding region based on the geometric priors provided by the multi-task coupling unit;

[0112] During the model training or update phase, adaptive reweighting is performed based on the class distribution, and pixels that meet the dynamic threshold are selected to participate in parameter updates based on online hard sample mining;

[0113] The refined semantic feature map is upsampled and aligned with pixels to generate a semantic mask, which is then output.

[0114] Specifically, the semantic segmentation branch in this embodiment works on the shared feature map output by the shared feature extraction layer, completing context modeling, differential decoding and boundary refinement based on geometric priors, adaptive reweighting and online hard sample mining during the training or update phases, and generating a semantic mask on the output side through upsampling and pixel alignment. The input consists of the shared feature map, feature location index, and geometric prior tensor provided by the multi-task coupling unit, and the output consists of a semantic mask and an optional confidence plane consistent with the data structure of the output interface module.

[0115] First, the branches perform context modeling on the shared feature maps using a multi-scale dilated convolutional array. The dilation rate is configured with 2, 4, 8, and 16 channels, while the kernel size and number of channels are fixed during deployment, allowing the network to obtain contextual information from different receptive fields without reducing spatial resolution. The outputs of each scale branch are aligned along the channel dimension and fused into a semantic feature map, preserving the shared features during the fusion process. Figure 1 Consistent step size and memory layout facilitate subsequent parallel reading and index consistency maintenance.

[0116] Secondly, the branch performs differentiated decoding and boundary refinement based on the geometric prior provided by the multi-task coupling unit. The geometric prior tensor consists of the planarity index, normal vector, and depth gradient calculated by the depth estimation branch. After size matching, it is concatenated with the semantic feature map in the channel dimension and then enters the decoding layer. The system enables the boundary decoding path within the boundary neighborhood calibrated by the feature location index, and enables the backbone decoding path in non-boundary regions. The two paths share parameter constraints and are selectively fused pixel by pixel at the output, thereby obtaining more refined classification results at boundary locations while maintaining stable semantic parsing in large areas. The boundary neighborhood is determined by a joint threshold of the depth gradient and the shared feature gradient. The threshold is stored as a configurable parameter after online calibration and is associated with the parameter version identifier.

[0117] Furthermore, the branch implements adaptive reweighting and online hard sample mining during model training or online update phases. Adaptive reweighting calculates class weights based on the number of valid samples in the class distribution, and these weights are refreshed according to statistics and written into the loss function each training epoch. Online hard sample mining sorts the current batch based on pixel-level loss, selecting pixels above a dynamic threshold for backpropagation. The dynamic threshold is determined by the batch loss quantile to adapt to varying difficulty levels in different scenarios. These strategies, along with the consistency constraints maintained by the multi-task coupling unit, ensure that the semantic branch shares a unified training step and cache queue with the depth branch when updating parameters.

[0118] Subsequently, the branch generates a semantic mask by upsampling and pixel alignment of the refined semantic feature map. The upsampling process preserves and shares features. Figure 1 The alignment strategy is consistent; pixel alignment maps feature coordinates back to the original image grid based on feature position indices, generating a category map with the same resolution as the input image. The category map uses 8-bit category encoding, and optionally outputs a pixel-wise confidence plane. Both the image and the confidence plane, along with the timestamp, calibration parameter identifier, and model version identifier, are written to the payload buffer, awaiting reading and transmission by the output interface module according to the real-time object mapping. If a change in the parameter version identifier is detected, the switching only occurs at frame boundaries, ensuring that semantic branches and depth branches within the same frame reference consistent calibration parameters and shared features.

[0119] For example, in a specific application at a robotic arm assembly station, the input image resolution is 1280×1024, and the shared feature map space size is 80×64. The semantic segmentation branch identifies 10 object categories, including bolts, nuts, washers, tooling plates, and conveyor belts. Hollow rates are extracted using parallel branches of 2, 4, 8, and 16 to extract multi-scale context. Planarity and normal vectors from the geometric prior are used to stabilize the main decoding path in the conveyor belt and tooling plate regions, while boundary decoding paths are enabled at the bolt and nut boundaries to enhance detail determination. During the training or update phase, class weights are calculated according to the class distribution, and online hard sample mining is used to select approximately 30% of high-loss pixels for parameter updates. The final output semantic mask and confidence plane are written into the payload buffer in a field order aligned with the output interface module and periodically sent to the external execution control system, where they, along with the parallel-generated depth map, are used for subsequent localization and planning. This process completes the entire semantic parsing process without changing the shared feature cache and frame-level atomic switching mechanism, maintaining consistency with the depth estimation branch in terms of timing and interface.

[0120] In some embodiments, the multi-task coupling unit is used for:

[0121] In the boundary neighborhood of the shared feature map, a consistency metric is calculated between the edge features of the depth estimation branch and the boundary features of the semantic segmentation branch, and the consistency metric is used as a training constraint to constrain the boundary alignment of the two branches.

[0122] The depth estimation branch generates a geometric prior tensor containing planarity and normal vector information. The geometric prior tensor is then input into the decoding layer of the semantic segmentation branch and fused with the feature tensor of the decoding layer to form geometrically constrained semantic features.

[0123] Generate a category prior tensor from the semantic segmentation branch, map the category prior tensor to the weights of pixel and disparity combinations, weight each item of the cost representation of the depth estimation branch, and set the weights of pixel and disparity combinations that are incompatible with the category prior to zero.

[0124] During the training phase, the training constraints, geometric prior fusion terms, and class prior weighting terms are jointly optimized. During the inference phase, fusion and weighting are performed based on the fixed parameters obtained during training.

[0125] Specifically, in this embodiment, the multi-task coupling unit is deployed between the shared feature extraction layer, the depth estimation branch, and the semantic segmentation branch. Its inputs include the shared feature map and feature location index, geometric features from the depth estimation branch, and category features from the semantic segmentation branch. The outputs are consistency constraints for the training phase, semantic features fused with geometric priors, and weighting coefficients for the cost representation. All tensors are associated with frame numbers and parameter version identifiers to ensure consistent references among the three parties within the same frame.

[0126] In constructing boundary consistency constraints, the coupling unit first locates the boundary neighborhood on the shared feature map based on the feature location index. The boundary neighborhood is determined by a joint threshold of the depth gradient magnitude and the shared feature gradient, with the neighborhood width taking 2-3 pixels on the feature map scale. The edge features output by the depth estimation branch are denoted as Ed, and the boundary features output by the semantic segmentation branch are denoted as Es. The coupling unit registers Ed and Es at the same spatial location and calculates a consistency metric. The consistency metric is formed into a single-channel constraint map using a combination of magnitude difference and orientation similarity. During the training phase, this constraint map is added to the loss function to constrain the alignment of the two branches in the boundary neighborhood. During the inference phase, only the consistency metric is retained as an internal quality reference and is not output to the external interface.

[0127] In terms of geometric prior generation and fusion, the coupling unit receives a geometric prior tensor containing planarity and normal vector information from the depth estimation branch. The geometric prior consists of local plane fitting confidence, three components of the unit normal, and depth gradient, and is aligned with the shared feature map in spatial size and channel order. After linear mapping, the geometric prior is concatenated with the feature tensor of the semantic segmentation branch's decoding layer in the channel dimension, thereby obtaining geometrically constrained semantic features. Higher geometric channel weights are retained in the boundary neighborhood to enhance the classification of small component boundaries; geometric channel weights are reduced in large near-planar regions to stabilize semantic parsing of weakly textured regions. The fusion of geometric prior and semantic features is completed in one step within the same frame to avoid prior leakage across frames.

[0128] Regarding the generation of category priors and cost weighting, the coupling unit receives the category probability map from the semantic segmentation branch and maps it to a category prior tensor. The category prior is aligned to the disparity search grid of the depth estimation branch through feature location indices. Based on the consistency relationship between category and disparity, the coupling unit maps the category prior to a weight table of pixel-disparity combinations, weighting each item in the cost representation of the depth estimation branch. For pixel-disparity combinations determined to be incompatible with the category prior, the weights are reset to zero and a masking marker is recorded to prevent the propagation of unreasonable matches in subsequent refinement stages. These mapping rules are configured according to the scenario during system deployment and can be upgraded with model version identification.

[0129] In the joint optimization phase during training, the coupling unit simultaneously feeds the boundary consistency constraint, geometric prior fusion, and class prior weighting into the training pipeline. The boundary consistency constraint has the pixel set of the boundary neighborhood as its scope, the geometric prior fusion has the semantic features output by the decoding layer as its scope, and the class prior weighting has the cost representation as its scope. All three share the same batch of feature location indices and parameter version identifiers to ensure consistent gradient registration. A fixed upper-level configuration of loss weights is used during optimization, supporting periodic updates of weight ratios based on the scene. Gradient backpropagation is stopped during the inference phase, retaining only the trained fusion and weighting parameters, which are atomically switched at frame boundaries along with the parameter version identifier.

[0130] For example, in a specific application example, the shared feature map space of the robotic arm assembly station has a size of 80×64 and a step size of 16. The coupling unit constrains the overlap between depth edges and semantic boundaries at the junction of bolts and tooling plates using a consistency metric, while simultaneously enhancing semantic decoding of the tooling plate region using a geometrically stable prior with high planarity. At bolt edge locations with high curvature, the category prior resets the weights of combination terms incompatible with large disparities to zero, thereby suppressing interference from background and mismatched disparities. This process is completed within a single frame and maintains temporal consistency with the parallel inference of the depth estimation branch and the semantic segmentation branch.

[0131] Through the coupling mechanism described in the above embodiments, the system achieves alignment constraints between depth and semantic features in the boundary neighborhood, introduces geometric priors derived from depth in semantic decoding, and uses category priors derived from semantics in disparity search, forming bidirectional information transmission under the same time base and parameter version. It can operate stably in both training and inference phases, and can complete prior fusion and weighted control while maintaining a unified data interface and frame-level atomic switching. This implementation enables the multi-task network to achieve consistent boundary registration, controlled cost search, and stable semantic decoding performance in complex industrial scenarios.

[0132] In some embodiments, the output interface module is used for:

[0133] The depth map and semantic mask are associated with the corresponding timestamp, calibration parameter identifier and model version identifier, and then framed and written into the payload buffer according to the preset data structure.

[0134] Based on the interface mapping table, the fields in the payload buffer are mapped to real-time cyclic data objects, the communication cycle and priority are configured, and the correspondence with the real-time control interface is established.

[0135] Associate the load buffer with the industrial real-time communication interface and send it according to the communication cycle when triggered by a hardware interrupt;

[0136] The sent frame carries a checksum and frame sequence number for order and integrity control, and when the parameter or model version is updated, the updated identifier is carried with the frame and an atomic switch is performed at the frame boundary.

[0137] When the main link is detected to meet the timeout threshold, the transmission path is switched to the backup physical link, and the payload buffer remains continuously readable during the switch.

[0138] Specifically, in this embodiment, the output interface module is located between the multi-task network module and the external execution control system. Its inputs include a depth map, semantic mask, and optional confidence plane, along with frame-specific timestamps, calibration parameter identifiers, and model version identifiers. The output is structured data frames periodically sent via the industrial real-time communication interface. The module operates using zero-copy DMA and interrupt-driven methods, and redundancy switching is implemented between primary and backup physical links.

[0139] First, the module associates the results from the multi-task network module with metadata fields. The associated content includes the timestamp T, calibration parameter identifier VID, model version identifier MID, depth map pointer and length, semantic mask pointer and length, and optional confidence plane pointer and length. The fields are arranged in a preset order to form a frame header. The frame header has a fixed-length layout and records the payload start offset and total length. Then, the frame header and payload pointer are written to the payload buffer. The payload buffer is located in the pre-mapped physical address range 0x40000000-0x40010000, and a double-buffered write strategy is used to ensure continuous readability during transmission.

[0140] Secondly, the module maps each field in the payload buffer to a real-time cyclic data object based on the interface mapping table, and configures the communication cycle and priority. For the EtherCAT scenario, the cycle is set to 1ms and jitter is guaranteed by the underlying hardware timing; for the PROFINET scenario, it operates at IRT level Class B and uses equal cycle parameters. After the mapping is established, the module registers the physical address and length of the payload buffer to the DMA descriptor queue of the communication stack, enabling the communication stack to directly retrieve data by object index in each cycle without CPU data transfer.

[0141] Next, the module establishes an association between the payload buffer and the industrial real-time communication interface and enters the interrupt-driven transmission process. A hardware interrupt is generated when the clock reaches the cycle boundary. The communication stack reads the data object from the currently ready buffer and assembles it into frames for transmission. Each transmitted frame includes a frame header, payload index, CRC checksum, and frame sequence number. The CRC is used for intra-frame integrity verification, and the frame sequence number is used for order and loss detection. If a calibration parameter or model version update is detected during the cycle, the module writes the updated VID and MID at the cycle boundary and switches to the new buffer, ensuring that atomic switching of parameters and models only occurs between frames.

[0142] Subsequently, the module continuously monitors the timeout count and error statistics of the main link. When the main link timeout exceeds a preset threshold (e.g., 500μs) or the error count reaches a threshold, the module triggers the link switching state machine, switching the transmission path to the backup physical link. During the switching process, the current ready buffer is kept in read-only state, and the frame resources being transmitted are not reclaimed, preventing cross-frame contention between the upper-layer read pointer and write pointer. After the switching is completed, the link identifier is included in the first frame for the host system to record.

[0143] Secondly, the module establishes signal linkage with the power supply and alarm modules. When a power failure alarm is received, the module prioritizes framing and sending the current frame, and attaches an abnormality identifier and the last valid frame sequence number to the end of the frame; if the power is restored within a single cycle, the transmission process automatically resets in the next cycle and continues to send according to the predetermined mapping, without the need for the upper-level system to reset the communication relationship.

[0144] For example, in an application example at a robotic arm assembly station, the system transmits data in EtherCAT 1ms cycles. The depth map in the payload buffer is in 16-bit single-channel format, and the semantic mask is in 8-bit category encoding with optional confidence plane. The module places T, VID, MID, frame sequence number, and CRC in a fixed area of ​​the frame header, and maps the depth map and semantic mask pointers to payload indices of two real-time objects. The communication stack directly retrieves data by DMA in object order and sends it when an interrupt occurs. After the model is updated online, the module writes a new MID at the boundary of the next frame cycle and switches the buffer. When the main link timeout exceeds the threshold, it switches to the backup link within ≤1ms. During the switching, the payload buffer remains continuously readable, and the frame sequence number received by the external execution control system is continuous and uninterrupted.

[0145] Through the above embodiments, the output interface module realizes framing based on field association and fixed-length frame header, real-time object mapping based on interface mapping table, zero-copy transmission based on DMA and interrupt, atomic switching of parameters and models based on frame boundaries, and link redundancy switching based on state machine. Thus, it provides the external execution control system with a unified format, detectable sequence, switchable link, and consistent with the upstream parameter version of the depth map and semantic mask data stream within a predetermined period.

[0146] In some embodiments, the system further includes a power supply and alarm module, used for:

[0147] It receives DC input power and generates multiple electrically isolated regulated outputs;

[0148] Monitor the voltage, current and temperature of DC input and each regulated output, determine overvoltage, overcurrent, undervoltage or overtemperature according to preset thresholds, and output alarm signals to the multi-task network module and output interface module.

[0149] When the voltage drop of the input power supply is detected to reach the threshold, the drive energy buffer unit maintains power supply for a short time, outputs an abnormality flag to the output interface module, and sends a controlled shutdown signal to the image acquisition module, the fusion preprocessing module and the multi-task network module.

[0150] Provide a power good signal and a reset control signal to establish the power-on and power-off sequence of the image acquisition module, fusion preprocessing module, multi-task network module and output interface module.

[0151] Specifically, in this embodiment, the power supply and alarm module is located between the system input power supply and each functional module. It is responsible for converting the 9-36V DC input into multiple electrically isolated regulated outputs and performing electrical parameter monitoring, anomaly detection, power-down buffering, and power-on / power-off timing control. The module output includes three isolated power supplies: 12V / 2A for the camera, 5V / 5A for the GPU, and 3.3V / 1A for the FPGA. The ripple is ≤50mV, and it supports overvoltage, overcurrent, and undervoltage protection. It is also linked with the multi-task network module and output interface module through hardware signals and status pins.

[0152] On the input side, the module employs a combination of EMI filtering and surge suppression. A common-mode inductor and π-type filter are placed at the input to reduce conducted interference, and a TVS diode is configured to suppress transient overvoltage. For reverse connection protection, an ideal diode controller and MOSFET are used to form a reverse connection protection path. The input undervoltage lockout threshold is set to ensure a stable operating voltage for the downstream DC-DC converter, while the undervoltage release threshold is relatively raised to create hysteresis, avoiding repeated start-stop cycles caused by edge jitter. The input current is measured by a sampling resistor and an operational amplifier, and the input temperature is monitored by a thermistor or onboard temperature sensor. The measured values ​​of these three parameters are used for alarm determination and event logging.

[0153] In terms of power conversion architecture, the module uses an isolated DC-DC converter as the primary stage to transform the input voltage to each intermediate bus, and then the synchronous buck regulator finely regulates it to the target level, thereby achieving low ripple and high dynamic response while meeting isolation requirements. Each channel is equipped with fast overcurrent limiting and short-circuit foldback. The overvoltage protection threshold is set to 38V, and the overcurrent threshold is set to 2.5A or calculated based on the safety factor of the rated current of that channel. Each output is equipped with a remote sampling line to compensate for the voltage drop of the interconnect. A small capacitor array and a low-ESR electrolytic capacitor are connected in parallel at the output to suppress voltage drops caused by load steps. The soft-start time of the FPGA and GPU power supplies are coordinated according to the order of the device datasheet to prevent power-on surges and incorrect power-on sequence.

[0154] In terms of monitoring and alarm logic, the module continuously samples the input voltage, current, three-channel output voltage, current, and onboard temperature. After comparing the sampled values ​​with preset thresholds, an alarm level is generated and simultaneously sent to the multi-task network module and output interface module via hardware interrupt lines and status pins. Thresholds are divided into two levels: warning and fault. Warnings indicate the boundary state that is about to trigger protection, while faults trigger a controlled shutdown process. For instantaneous spike signals, a time window filter is used; the fault flag is only set when the duration of exceeding the threshold exceeds a set value. The alarm logic has a hold and clear mechanism. Hold records the event, and clear is reset after upper-level confirmation to avoid duplicate responses.

[0155] For power-down handling, the module is equipped with an energy buffer unit to maintain short-term power supply when the input power supply voltage drops to a threshold. Power-down detection monitors the input voltage via a comparator and outputs a power-down edge when the voltage falls below the threshold. The module immediately outputs an anomaly flag to the output interface module, simultaneously latches the current frame sequence number and timestamp, and allows the output interface module to prioritize the transmission of the current frame's data. Simultaneously, controlled shutdown signals are sent to the image acquisition module, fusion preprocessing module, and multi-task network module, causing them to stop new calculations in a predetermined order and freeze parameter version identifiers and buffer read / write pointers to avoid inconsistent data during the voltage drop phase. The energy buffer unit consists of a high-capacity capacitor and a controlled discharge path. During the maintenance period, it does not accept new model or parameter switching requests, ensuring that the buffered energy is used to completely transmit the last frame.

[0156] Regarding power-on and power-off timing control, the module provides a good power signal and a reset control signal to each output. The power-on sequence is established in the order of "basic control layer → computing acceleration layer → peripheral layer". Specifically, the FPGA 3.3V is enabled first, and the FPGA reset is released after its PG stabilizes. Then, the GPU 5V is enabled, and the computing module reset is released after its PG stabilizes. Finally, the camera 12V is enabled, and the trigger logic and supplementary lighting control are turned on. The power-off sequence is executed in reverse order to ensure that the communication stack and inference tasks exit first. The PG signal is also sent to the output interface module as one of the conditions for allowing entry into the real-time transmission cycle.

[0157] For example, in an application example, taking the EtherCAT 1ms cycle operating condition of a robotic arm assembly station as an example, when the input power supply drops rapidly due to bus disturbance, the module's power-down threshold comparator first detects the voltage drop and outputs a power-down edge. The energy buffer unit then activates, and the output interface module, after receiving the anomaly flag, accelerates the current frame assembly and transmission. The multi-task network module, upon receiving the controlled shutdown signal, stops new inference batches and maintains a read-only shared feature map buffer. The image acquisition module stops sending new trigger pulses. If the input power supply recovers to above the undervoltage release threshold within a short time, the module restores each power supply sequentially according to the power-on sequence and clears the held fault flags. If the input power supply does not recover, the module completes the transmission of the last frame and locks all outputs before the buffer energy is exhausted, waiting for the upper layer to reset.

[0158] Through the above embodiments, the power supply and alarm module realizes unified power supply for wide voltage input and multi-channel isolated voltage regulation, real-time monitoring and hierarchical alarm of key input and output electrical parameters, rapid detection and short-term maintenance of power failure events, and power-on / power-off timing control consistent with other modules of the system. This enables the system to complete consistent data encapsulation and orderly exit under operating conditions such as power disturbances, load changes and link anomalies, and provides traceable event and status information for subsequent safe recovery.

[0159] The effectiveness of this solution in practical applications demonstrates that, compared to traditional solutions, the technical solution of this application has at least the following advantages:

[0160] 1. Stability and Reliability: Through baseline adjustable structure, synchronous trigger optimization and automatic recalibration, the system's mean time between failures (MTBF) reaches 10,000 hours, which is 60% better than the split solution; the depth error drift under vibration / temperature difference environment is ≤0.5%, which meets the industrial-grade stability requirements.

[0161] 2. Real-time breakthrough: End-to-end total latency (acquisition → output) ≤20ms (traditional solution is 50ms), including 3ms for preprocessing link, 8ms for network inference, and 5ms for communication, which can support 100Hz control closed loop.

[0162] 3. Space and cost optimization: Integrated design volume of 0.015m³ 3 (75% smaller than a separate type), power consumption 15W (60% lower), hardware cost reduced by 40% (eliminating the need for a separate industrial control computer).

[0163] 4. Accuracy and robustness: Depth estimation error ≤1% (15% for traditional solutions), semantic segmentation mIoU ≥92% (an 8% improvement); recognition rate remains above 85% under 60dB noise and ±30% illumination variation, adapting to complex industrial environments.

[0164] 5. Ease of maintenance: Supports online recalibration, hot model updates, and self-supervised optimization, reducing operation and maintenance costs by 50% and ensuring that the upgrade process does not interrupt production.

[0165] The above embodiments have described in detail the specific structure and function of the end-to-end multi-task stereo depth and semantic fusion visual perception system of this application. The implementation process of the end-to-end multi-task stereo depth and semantic fusion visual perception method of this application will be described in detail below with reference to specific embodiments. Figure 3 This is a flowchart illustrating the visual perception method of the end-to-end multi-task stereo depth and semantic fusion visual perception system provided in this application embodiment. Figure 3 As shown, the visual perception method of this end-to-end multi-task stereo depth and semantic fusion visual perception system may specifically include the following steps:

[0166] S301, acquires synchronized image pairs by binocular imaging under a unified time base, and associates timestamps and calibration parameter identifiers with synchronized image pairs;

[0167] S302, geometric correction and epipolar alignment are performed on the synchronized image pair according to the calibration parameters, noise suppression is performed, and stereo registration is completed based on correlation measurement and sub-pixel interpolation to obtain the registered image pair;

[0168] S303 performs weight-shared hierarchical convolutional encoding on the left and right views of the registered image pair and forms a shared feature map through cross-layer fusion;

[0169] S304 constructs a cost representation based on adaptive disparity range on the shared feature map, performs weighted aggregation on the cost representation and generates continuous disparity through learnable sub-pixel thinning, converts the continuous disparity into a depth map, generates semantic features based on multi-scale semantic parsing and decodes and outputs a semantic mask.

[0170] S305: Calculate the consistency measure between depth edge and semantic boundary in the boundary neighborhood of the shared feature map, input geometric prior to semantic decoding, and use category prior to perform item-by-item weighting and incompatible item masking on the cost representation in order to jointly correct the boundary region of the depth map and semantic mask.

[0171] S306 encapsulates the depth map and semantic mask along with the corresponding timestamp, calibration parameter identifier, and model version identifier according to a preset data structure, and outputs them to the real-time control interface of the external execution control system for use through the industrial real-time communication interface.

[0172] Specifically, for ease of understanding, the following will be combined with Figure 3The flowcharts S301 to S306 illustrate the implementation process of the end-to-end multi-task stereo depth and semantic fusion visual perception method of this application step by step. Each step corresponds to each functional module of the aforementioned system and runs collaboratively under the same time base and parameter version identifier.

[0173] S301 is a unified time-base synchronous acquisition and frame-based correlation system. Using the FPGA's internal clock as the unified time base, a trigger generator outputs a 10μs pulse width synchronous trigger signal at a set period, which is sent to the left and right imagers respectively, achieving simultaneous exposure. The acquired left and right image frames have an 8-byte timestamp and calibration parameter identifier written to their frame headers. The calibration parameter identifier consists of the current intrinsic parameters, extrinsic parameters, distortion parameter version number, and verification information. If recalibration is triggered during operation, the system performs an atomic switch between the old and new identifiers only at the frame boundary to ensure consistent parameter references within the same frame.

[0174] S302 is a preprocessing module for geometric correction, epipolar alignment, and registration based on calibration parameters. The fusion preprocessing module loads the corresponding intrinsic, extrinsic, and distortion parameters according to the calibration parameter identifiers carried by S301, and calls a multi-scale lookup table to complete radial and tangential distortion compensation. Subsequently, the left and right views are row-aligned according to epipolar constraints to obtain preliminary aligned image pairs. The vector processing unit calculates the full-frame histogram and local variance map, and adaptively selects spatial domain filtering parameters to suppress noise and reflection artifacts. Based on this, the phase correlation method is used to estimate the subtle relative displacements of the left and right views. Integer pixel displacements are achieved through shifting, and fractional pixel displacements are achieved through bilinear interpolation, allowing the right view to be resampled to match the left view. Figure 1 The pixel grid is matched to obtain registered image pairs. The above processing is organized into a pipeline with a fixed time sequence, and the registered image pairs, along with timestamps and parameter version identifiers, are written to a circular buffer frame by frame for subsequent reading.

[0175] S303 uses weight-sharing hierarchical convolutional encoding and cross-layer fusion to generate a shared feature map. The shared feature extraction layer performs isomorphic encoding on the left and right views of the registered image pair, with convolutional weights shared between the two branches to maintain feature space consistency. Encoding progressively extracts edge, texture, and high semantic features from shallow to deep layers, maintaining spatial index traceability through deterministic downsampling and symmetrical padding. To balance detail and semantics, the system upsamples deep features and merges them with corresponding shallow features at the same scale along the channel dimension, ultimately forming a shared feature map with a uniform stride and fixed number of channels. Simultaneously, feature location indices are generated to record the mapping relationship between the shared feature map positions and the original image coordinates. The shared feature map and location indices are written to a read-only buffer for parallel reading by the depth estimation and semantic segmentation branches.

[0176] S304, Parallel Inference of Depth and Semantics. The depth estimation branch first dynamically determines the adaptive disparity range [dmin, dmax] at the pixel or region level based on shared feature maps and statistics from the previous frame. It then aligns left and right shared features according to candidate disparities and calculates disparity-level similarity to construct a cost representation. Attention weights are generated by combining category priors from multi-task coupling units, and the cost representation is weighted and aggregated in the spatial and disparity dimensions. Simultaneously, masking constraints are applied to left-right inconsistencies or occlusion positions, and unmatchable markers are recorded. The aggregation result is fed into a learnable sub-pixel thinning network, which recovers continuous disparities using inverse residual convolution and interpolation branches. Based on D=bf / disp and the baseline b and focal length f corresponding to the parameter identifiers, it is converted pixel-by-pixel into a depth map and quantized to a 16-bit format. The semantic segmentation branch performs multi-scale dilated convolutional context modeling on the shared feature map to generate semantic features. Based on the geometric prior of the multi-task coupling unit input, it enables differentiated decoding paths in the boundary neighborhood and non-boundary regions, refining the features at the boundaries and maintaining stable parsing for large areas. Finally, the semantic features are upsampled and mapped back to the original pixel grid according to the feature position index to generate an 8-bit class-encoded semantic mask. The two branches run in parallel at the same frame number, and the shared feature map and parameter labels remain consistent within the frame.

[0177] S305, Bidirectional Prior Coupling and Joint Boundary Correction. The multi-task coupling unit determines the pixel set in the boundary neighborhood of the shared feature map based on the depth gradient and feature gradient. It calculates a consistency metric for edge features from the depth estimation branch and boundary features from the semantic segmentation branch, using this metric as a loss term during training to constrain the boundary alignment of the two branches. In the inference path, the coupling unit generates a geometric prior tensor containing planarity and normal vectors from the depth estimation branch. After size matching, this tensor is input to the semantic decoding layer and fused with its feature tensor, thus constraining the semantic boundary with geometric information. Simultaneously, it generates a category prior tensor from the semantic segmentation branch, mapping it to a weight table of pixel and disparity combinations. The cost representation of the depth branch is weighted item by item, and the weights of combinations incompatible with the category prior are reset to zero. After this bidirectional prior transfer, the system performs a consistent joint correction on the boundary regions of the depth map and the semantic mask, with the corrected result covering the corresponding region of the current frame.

[0178] S306, Result Encapsulation and Industrial Real-Time Output. The output interface module associates the depth map and semantic mask with their timestamps, calibration parameter identifiers, and model version identifiers, generates a fixed-length frame header, records the load start offset and length, and writes the frame header and load pointer to the load buffer. Based on the interface mapping table, each field is mapped to a real-time cyclic data object, the communication cycle and priority are configured, and the buffer physical address is registered to the communication stack DMA descriptor queue. When a hardware interrupt reaches the cycle boundary, the communication stack directly retrieves an array of frames from the buffer and sends them, carrying CRC and frame sequence numbers in the transmitted frames for integrity and order control. If a calibration parameter or model version update is detected, the module carries the updated identifier with the frame at the frame boundary and performs an atomic switch. If the main link times out beyond the threshold, it switches to the backup physical link. During the switch, the load buffer remains continuously readable, and the external execution control system can seamlessly receive data.

[0179] It should be noted that the above steps use a circular buffer and atomic switching strategy to ensure consistent reference of parameters, models and data under the same time base. S301 to S306 constitute a closed-loop process from acquisition, preprocessing, feature extraction, dual-branch parallel inference, prior coupling to industrial real-time output. It can complete online calibration and model version updates without stopping upstream acquisition and downstream communication, and maintain the consistency and time-series traceability of multi-task results within the frame.

[0180] It should be understood that the sequence number of each step in the above method embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this disclosure.

[0181] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although the technical solutions of this application have been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. An end-to-end multi-task stereo depth and semantic fusion visual perception system, characterized in that, include: The image acquisition module is used to acquire synchronized image pairs under a unified time base through a binocular imaging unit and output image data carrying calibration parameter identifiers; The fusion preprocessing module is used to perform geometric correction, noise suppression and stereo registration on the synchronized image pair according to the calibration parameters to obtain a registered image pair. The multi-task network module includes a shared feature extraction layer, a depth estimation branch, and a semantic segmentation branch; wherein, the shared feature extraction layer is used to generate a shared feature map from the registered image pair, the depth estimation branch is used to construct a cost representation based on the shared feature map and output a depth map, and the semantic segmentation branch is used to perform multi-scale semantic parsing based on the shared feature map and output a semantic mask; The multi-task coupling unit, connected to the shared feature extraction layer, the depth estimation branch, and the semantic segmentation branch, is used to apply consistency constraints to the shared features and branch features, and to pass geometric priors and class priors between branches; The output interface module is used to encapsulate the depth map and the semantic mask according to a preset data structure, and output them to the real-time control interface of an external execution control system for use through an industrial real-time communication interface. The shared feature extraction layer is used for: Weight-shared hierarchical convolutional feature encoding is performed on the left and right views of the registered image pair to obtain multi-scale feature representations; Based on cross-layer connections, detailed features and semantic features at different scales are fused and a unified tensor format is used to form the shared feature map; The shared feature map is provided to both the depth estimation branch and the semantic segmentation branch, and the feature index corresponding to the position of the shared feature map is output for the multi-task coupling unit to call. The multi-task coupling unit is used for: In the boundary neighborhood of the shared feature map, a consistency metric is calculated between the edge features of the depth estimation branch and the boundary features of the semantic segmentation branch, and the consistency metric is used as a training constraint to constrain the boundary alignment of the two branches. A geometric prior tensor containing planarity and normal vector information is generated from the depth estimation branch. The geometric prior tensor is input into the decoding layer of the semantic segmentation branch and fused with the feature tensor of the decoding layer to form geometrically constrained semantic features. A category prior tensor is generated from the semantic segmentation branch, the category prior tensor is mapped to the weights of pixel and disparity combinations, the cost representation of the depth estimation branch is weighted item by item, and the weights of pixel and disparity combinations that are incompatible with the category prior are set to zero. During the training phase, the training constraints, geometric prior fusion terms, and category prior weighting terms are jointly optimized, and during the inference phase, fusion and weighting are performed based on the fixed parameters obtained during training.

2. The system according to claim 1, characterized in that, The image acquisition module is used for: Synchronization trigger signals are generated based on a unified time base and sent to the left and right imagers respectively. Under the action of the synchronization trigger signal, the left image frame and the right image frame are acquired simultaneously, and a timestamp corresponding to the unified time base is written to the image frame; During assembly or operation, parameter calibration is performed, and the calibration parameter identifier is associated with the image frame and output with the frame; When the preset recalibration trigger condition is detected, recalibration is performed and the calibration parameters are updated. During the recalibration, the previous version of the calibration parameters are used for output. After the recalibration is completed, the updated calibration parameters are used to identify the output with each frame. The binocular imaging unit is geometrically configured using a baseline adjustable mechanism and an optical axis alignment mechanism to meet the geometric relationships required for stereo registration.

3. The system according to claim 1, characterized in that, The fusion preprocessing module is used for: The pixel mapping based on the lookup table performs radial and tangential distortion compensation according to the calibration parameters and completes row alignment according to the epipolar constraint. The geometric correction and parameter version identifier are associated with each frame and are atomically switched when the parameters are updated. Spatial domain filtering is performed on the vector processing unit, and the filtering parameters are adaptively selected based on image statistics; The relative displacement of the left and right views is estimated based on the correlation metric, and the synchronized image pair is aligned to a uniform pixel grid through subpixel interpolation, and the registered image pair is output. A preprocessing pipeline with a fixed processing sequence is adopted, and the registered image pairs are associated with and stored with the corresponding timestamps and parameter versions in a circular buffer structure for sequential reading by the multi-task network module.

4. The system according to claim 1, characterized in that, The depth estimation branch is used for: A cost representation is constructed based on the shared feature map within the adaptive disparity range, which is dynamically determined at the pixel or region level according to scene distance distribution or image statistics. The category priors passed from the shared feature map and the multi-task coupling unit are weighted and aggregated with the cost representations corresponding to the attention weights, and masking constraints are applied to unmatched regions. Learnable subpixel thinning is performed on the aggregation result, inverse residual convolution is used to recover continuous disparity, and the continuous disparity is converted into the depth map according to the imaging model; During the inference phase, the shared feature map cache is reused and executed in parallel with the semantic segmentation branch to output the depth map in alignment with the data structure of the output interface module.

5. The system according to claim 1, characterized in that, The semantic segmentation branch is used for: The shared feature map is used to perform context modeling based on a multi-scale dilated convolutional array to form a semantic feature map; Differential decoding and refinement are performed on the semantic boundary corresponding region based on the geometric priors provided by the multi-task coupling unit; During the model training or update phase, adaptive reweighting is performed based on the class distribution, and pixels that meet the dynamic threshold are selected to participate in parameter updates based on online hard sample mining; The refined semantic feature map is upsampled and aligned with pixels to generate the semantic mask, which is then output.

6. The system according to claim 1, characterized in that, The output interface module is used for: The depth map and the semantic mask are associated with the corresponding timestamp, calibration parameter identifier and model version identifier, and then framed and written into the payload buffer according to the preset data structure. The fields in the load buffer are mapped to real-time cyclic data objects according to the interface mapping table, the communication cycle and priority are configured, and a correspondence with the real-time control interface is established. The load buffer is associated with the industrial real-time communication interface and sent according to the communication cycle under hardware interrupt triggering; The sent frame carries a checksum and frame sequence number for order and integrity control, and when the parameter or model version is updated, the updated identifier is carried with the frame and an atomic switch is performed at the frame boundary. When the main link is detected to meet the timeout threshold, the transmission path is switched to the backup physical link, and the payload buffer remains continuously readable during the switch.

7. The system according to claim 1, characterized in that, The system also includes a power supply and alarm module, used for: It receives DC input power and generates multiple electrically isolated regulated outputs; Monitor the voltage, current and temperature of the DC input and each regulated output, determine overvoltage, overcurrent, undervoltage or overtemperature according to preset thresholds, and output alarm signals to the multi-task network module and the output interface module; When the voltage drop of the input power supply is detected to reach a threshold, the drive energy buffer unit maintains power supply for a short time, outputs an abnormality flag to the output interface module, and sends a controlled shutdown signal to the image acquisition module, the fusion preprocessing module, and the multi-task network module. Provide a power good signal and a reset control signal to establish the power-on and power-off timing of the image acquisition module, the fusion preprocessing module, the multi-task network module, and the output interface module.

8. A visual perception method based on an end-to-end multi-task stereo depth and semantic fusion visual perception system as described in any one of claims 1 to 7, characterized in that, include: Synchronous image pairs are acquired by binocular imaging under a unified time base, and timestamps and calibration parameter identifiers are associated with the synchronous image pairs. Geometric correction and epipolar alignment are performed on the synchronized image pair according to the calibration parameters, noise suppression is performed, and stereo registration is completed based on correlation metric and subpixel interpolation to obtain a registered image pair. The left and right views of the registered image pair are subjected to weight-shared hierarchical convolutional encoding, and a shared feature map is formed by cross-layer fusion. A cost representation is constructed on the shared feature map based on an adaptive disparity range. The cost representation is weighted and aggregated and then thinned by learnable sub-pixel refinement to generate continuous disparity. The continuous disparity is then converted into a depth map. Semantic features are generated based on multi-scale semantic parsing and a semantic mask is decoded and output. The consistency measure between depth edges and semantic boundaries is calculated in the boundary neighborhood of the shared feature map. Geometric priors are input into semantic decoding, and category priors are used to perform item-by-item weighting and incompatible item masking on the cost representation in order to jointly correct the boundary region of the depth map and the semantic mask. The depth map, the semantic mask, and the corresponding timestamp, calibration parameter identifier, and model version identifier are encapsulated according to a preset data structure and output to the real-time control interface of an external execution control system for use via an industrial real-time communication interface.

Images