Binocular vision system with integrated optical code on acquisition end and system building method

Abstract

The application discloses a binocular vision system with integrated optical coding at the acquisition end and a system building method, relates to the technical field of binocular cameras, and adds an optical coding assembly to the binocular camera module in the binocular vision system on the basis of a conventional structure. After the optical field entering the binocular camera module is phase-modulated and diffractively propagated by the optical coding assembly to complete optical coding operation, optical coding feature maps are formed on the left and right eye image sensor planes, respectively. A rear-end processing module directly perceives and reasons the optical coding feature maps by using an image perception model and outputs a perception result to complete a binocular vision perception task. The system prepositions the optical coding operation originally completed by the rear-end processing module into the binocular camera module, and the optical coding assembly realizes feature screening and spatial coding at the physical layer, thereby reducing the computation, storage and transmission power consumption of the rear-end processing module, and reducing the delay and heat dissipation, and having better performance.

Description

Technical Field

[0001] This application relates to the field of binocular camera technology, and in particular to a binocular vision system with integrated optical coding at the acquisition end and a method for building the system. Background Technology

[0002] In applications such as industrial inspection, AGV (Automated Guided Vehicle) / AMR (Autonomous Mobile Robot), drone inspection, collaborative robots, warehousing and logistics, and embodied intelligent terminals, equipment needs to rely on binocular vision systems for extended periods to perform binocular vision perception tasks such as target classification, target detection, depth estimation, and defect identification.

[0003] Currently, classic binocular vision systems mainly consist of a binocular camera module and a back-end processing module. In the binocular camera module, the left and right lenses image the light field, forming raw images on the left and right image sensor planes respectively. The back-end processing module first performs optical encoding operations on the raw images, including image enhancement, feature representation, and information filtering, before performing perceptual reasoning and outputting the perception results to complete binocular vision perception tasks such as depth estimation, defect detection, target localization, grasping guidance, obstacle recognition, person detection, and action recognition. This architecture places high performance demands on the back-end processing module, leading to problems such as high power consumption, high computational overhead, high heat generation, and high latency in binocular vision systems, limiting their battery life and deployment flexibility on terminal devices. Summary of the Invention

[0004] This application addresses the aforementioned problems and technical requirements by proposing a binocular vision system with integrated optical coding at the acquisition end and a system construction method. The technical solution of this application is as follows: In a first aspect, a binocular vision system with integrated optical coding at the acquisition end is provided, the binocular vision system comprising a binocular camera module and a back-end processing module; An optical coding component is integrated into the internal optical path of the binocular camera module. The light field entering the binocular camera module undergoes phase modulation and diffraction propagation by the optical coding component to complete the optical coding operation. Optical coding feature maps are then formed on the left and right eye image sensor planes and output to the back-end processing module. The optical coding operation performed by the optical coding component matches the business scenario parameters when the binocular vision system performs binocular vision perception tasks. The back-end processing module uses an image perception model that matches the task type of the binocular vision perception task to perform perceptual reasoning on the optically encoded feature map and outputs the perception results to complete the binocular vision perception task.

[0005] The further technical solution is that the optical coding components integrated in the binocular camera module are left and right eye DOE optical films. The optical coding operation performed by the left and right eye DOE optical films is determined by their physical parameters. The physical parameters of the left and right eye DOE optical films include the film thickness, effective aperture, material parameters, and design parameters of the surface two-dimensional micro-nano phase structure. The design parameters of the surface two-dimensional micro-nano phase structure include the arrangement of each microstructure unit in the surface two-dimensional micro-nano phase structure and the size of each microstructure unit.

[0006] The further technical solution is that the business scenario parameters when the binocular vision system performs binocular vision perception tasks include: the relative positions of each component in the binocular camera module, the sensor parameters of the left and right eye image sensors, and the light source wavelength of the light field entering the binocular camera module.

[0007] A further technical solution is that the left and right eye DOE optical films can be detachably installed in the optical path of the binocular camera module. When the binocular vision system performs different binocular vision perception tasks, the binocular camera module integrates left and right eye DOE optical films with different physical parameters.

[0008] A further technical solution is that the effective aperture center of the optical coding component integrated in the binocular camera module is aligned with the center of the left and right eye image sensors in the binocular camera module; when the binocular camera module includes left and right eye lenses, the optical coding component integrated in the binocular camera module is located on the optical path channel between the left and right eye lenses and the left and right eye image sensors and is located at the back focal plane of the left and right eye lenses. The effective aperture center of the optical coding component is coaxial with the optical axis of the left and right eye lenses in the binocular camera module, respectively. After the light field entering the binocular camera module is imaged by the left and right eye lenses, it is optically encoded by the optical coding component and forms an optical coding feature map on the plane of the left and right eye image sensors.

[0009] A further technical solution is that multiple layers of left and right eye DOE optical films are stacked sequentially along the light propagation path in the internal optical path of the binocular camera module.

[0010] Secondly, a method for building a binocular vision system with integrated optical coding at the acquisition end, the method comprising: Obtain a sample dataset, which includes multiple sample input images, and each sample input image carries a label of its perception result under the binocular vision perception task required by the binocular vision system. Based on the business scenario parameters when the binocular vision system performs binocular vision perception tasks, the fixed hyperparameters in the all-optical diffraction network model are set, and the optical coding operations performed by the optical coding component are represented by the optimizable model parameters of the hidden diffraction layer in the all-optical diffraction network model. Thus, an all-optical diffraction network model containing an input layer, a hidden diffraction layer, and an output layer is established. The process of light field entering the binocular camera module and propagating to the left and right eye image sensor planes is simulated by using an all-optical diffraction network model. Physical simulation is performed on each sample input image to obtain the corresponding encoded light intensity map. The image perception model that matches the model structure with the task type of the binocular vision perception task is used to perform perception inference on the encoded light intensity map and output the perception result prediction value. The loss function is calculated using the predicted value of the perception result and the label of the perception result for each sample input image. Based on the loss function, the optimizable model parameters of the all-optical diffraction network model and the image perception model are jointly optimized. The target optical coding operation performed by the optical coding component is determined based on the optimizable model parameters of the hidden diffraction layer in the trained all-optical diffraction network model. An optical coding component that performs the target optical coding operation is integrated into the internal optical path of the binocular camera module in a binocular vision system, and the trained image perception model is deployed to the back-end processing module of the binocular vision system, thereby constructing a binocular vision system with integrated optical coding at the acquisition end as described in any one of claims 1-6.

[0011] A further technical solution is that the optical encoding components integrated in the binocular camera module are left and right eye DOE optical films, and the optical encoding operations performed by the left and right eye DOE optical films are determined by their physical parameters: The optical coding operations performed by the optical coding component are represented by the optimizable model parameters of the hidden diffraction layer in the all-optical diffraction network model. These operations include: the effective aperture of the left and right eye DOE optical films is represented by the spatial region range of the optical modulation region in the hidden diffraction layer; the arrangement of each neuron in the hidden diffraction layer is represented by the arrangement of each microstructure unit in the two-dimensional micro-nano phase structure on the surface of the left and right eye DOE optical films; and the phase modulation amount of each neuron in the hidden diffraction layer is represented by the phase modulation amount of each microstructure unit in the two-dimensional micro-nano phase structure on the surface of the left and right eye DOE optical films. The phase modulation amount of each microstructure unit has an inherent relationship with the film thickness, material parameters, and microstructure unit size. Integrating the optical coding component that performs the target optical coding operation into the internal optical path of the binocular camera module in a binocular vision system includes: determining the effective aperture of the left and right eye DOE optical films, the arrangement of each microstructure unit in the two-dimensional micro / nano phase structure on the surface, and the phase modulation amount achieved by each microstructure unit based on the trained all-optical diffraction network model; and determining the film thickness, material parameters, and microstructure unit size of each microstructure unit when achieving the required phase modulation amount by combining the inherent relationship between the phase modulation amount of each microstructure unit and the film thickness, material parameters, and microstructure unit size, thus obtaining the target physical parameters when the left and right eye DOE optical films perform the target optical coding operation, including at least the effective aperture, film thickness, material parameters, arrangement of each microstructure unit, and size of each microstructure unit; fabricating the optical coding component that performs the target optical coding operation according to the target physical parameters, and integrating the optical coding component into the internal optical path of the binocular camera module in a binocular vision system.

[0012] A further technical solution involves setting fixed hyperparameters in the all-optical diffraction network model based on the business scenario parameters when the binocular vision system performs binocular vision perception tasks. The wavelength parameters in the all-optical diffraction network model are determined based on the wavelength of the light source band to which the light field enters the binocular camera module; the free space propagation distance between each network layer in the all-optical diffraction network model is determined based on the relative positions of each component in the binocular camera module; and the spatial sampling mechanism used to obtain the coded light intensity map is determined based on the sensor parameters of the left and right eye image sensors.

[0013] Its further technical solution is a loss function. ,in, It is based on the perception loss and is calculated according to the deviation between the predicted value of the perception result and the label of the perception result; It is a phase smoothing loss regularization term and is calculated based on the difference in phase modulation of adjacent microstructure units; It is the phase quantization error regularization term, and is calculated based on the difference between the discrete phase levels corresponding to the phase modulation amounts of different microstructure units; It is the diffraction efficiency loss regularization term, and is calculated based on the proportion of effective light energy in the target area of ​​the left and right eye image sensors to the total light field energy; , , These are the coefficients of three regularization terms.

[0014] The beneficial technical effects of this application are: This application discloses a binocular vision system with integrated optical coding at the acquisition end and a system construction method. In this binocular vision system, the binocular camera module adds an optical coding component to a conventional structure. The left and right eye image sensors acquire the output light field modulated by the optical coding component and output optically coded feature maps. The back-end processing module directly performs perceptual inference on the optically coded feature maps. This system moves the optical coding operation, which would normally be performed by the back-end processing module, to the binocular camera module. The optical coding component performs feature selection and spatial coding at the physical layer, reducing the computational, storage, and transmission power consumption of the back-end processing module, while also reducing latency and heat dissipation, resulting in superior performance.

[0015] This binocular vision system reduces the back-end processing module's reliance on high-performance processors, allowing the back-end processing module to use lightweight models. Furthermore, the light field after optical encoding significantly reduces the identifiable information of the original scene. While meeting the accuracy requirements of the target task, it can reduce the back-end model's processing needs for high-dimensional original image features and reduce reliance on high-resolution image sensors, thereby reducing system power consumption, cost, and deployment difficulty.

[0016] The optical coding components integrated in the binocular camera module employ static, passive left and right eye DOE optical films, requiring no continuous power supply, further reducing power consumption and heat dissipation. Furthermore, the system construction method provided in this application jointly optimizes an all-optical diffraction network model and an image perception model, thereby determining the physical parameters of the left and right eye DOE optical films and the image perception model that match the binocular visual perception task. This allows both to achieve superior binocular visual perception results through hardware and software collaboration. Attached Figure Description

[0017] Figure 1 This is a system architecture diagram of a binocular vision system in one embodiment of this application.

[0018] Figure 2 This is a flowchart of a system construction method in one embodiment of this application. Detailed Implementation

[0019] The specific embodiments of this application will be further described below with reference to the accompanying drawings.

[0020] This application discloses a binocular vision system with integrated optical coding at the acquisition end. Please refer to... Figure 1 The system architecture diagram shown illustrates that the binocular vision system includes a binocular camera module and a back-end processing module, wherein: The binocular camera module in this application is an optimization based on the traditional module structure. A traditional binocular camera module includes left and right eye lenses 110 and 120, and left and right eye image sensors 210 and 220. The light field entering the binocular camera module is imaged by the left and right eye lenses 110 and 120 respectively, forming the original images of the left and right eyes on the left and right eye image sensor planes 210 and 220. The original images of the left and right eyes satisfy the parallax relationship and epipolar constraint. Some binocular camera modules do not include left and right eye lenses, achieving smaller size and higher integration.

[0021] The binocular camera module in this application, based on the aforementioned traditional structure, adds an optical encoding component 300 without disrupting the binocular parallax relationship, epipolar constraints, and target spatial geometry of the binocular camera module. This optical encoding component 300 is integrated into the optical path transmitting data from the binocular camera module to the left and right eye image sensors 210 and 220. This optical encoding component 300 has optical encoding capabilities, ensuring that the light field entering the binocular camera module undergoes phase modulation and diffraction propagation via the optical encoding component 300 to complete the optical encoding operation. The integrated optical encoding component 300 in the binocular camera module utilizes the principle of diffraction optics to phase modulate the incident light field. After passing through the optical encoding component 300, the incident light field undergoes diffraction propagation, and different spatial positions and frequency components are redistributed during propagation. This physically disperses the image data acquired by the left and right eye image sensors 210 and 220, resulting in the formation of optically encoded feature maps on the planes of the left and right eye image sensors 210 and 220, rather than the unprocessed original images.

[0022] Based on the inherent structure of the binocular camera module, the optical encoding component 300 is located in the optical path channel before reaching the left and right eye image sensors 210 and 220, and the effective aperture center of the optical encoding component 300 is aligned with the center of the left and right eye image sensors 210 and 220 to ensure coaxiality.

[0023] When the binocular camera module includes left and right eye lenses 110 and 120, the optical encoding component 300 integrated in the binocular camera module is located on the optical path channel between the left and right eye lenses 110 and 120 and the left and right eye image sensors 210 and 220. The effective aperture center of the optical encoding component 300 is coaxial with the optical axis of the left and right eye lenses 110 and 120 in the binocular camera module, and is installed at the back focal plane of the left and right eye lenses 110 and 120. After the light field entering the binocular camera module is imaged by the left and right eye lenses 110 and 120, the optical encoding operation is completed by the optical encoding component 300 and optical encoding feature maps are formed on the planes of the left and right eye image sensors 210 and 220.

[0024] The optical coding assembly 300 is installed using a snap-fit ​​structure, limiting groove, pressure ring, bracket, or other mechanical fasteners to ensure its stability under equipment vibration, temperature changes, or long-term operating conditions. The optical coding assembly can be fixedly installed or detachably installed.

[0025] The left and right eye image sensors 210 and 220 output optically encoded feature maps to the back-end processing module. The back-end processing module is equipped with an image perception model that matches the task type of the binocular vision perception task. After receiving the optically encoded feature maps, the back-end processing module directly uses the image perception model to perform perceptual inference on the optically encoded feature maps and outputs the perception results to complete the binocular vision perception task.

[0026] This allows the optical encoding operations, which would normally be performed by the back-end processing module using electronic convolution, filtering, and feature extraction, to be moved forward to the binocular camera module. The optical encoding component performs spatial feature extraction and dimensionality reduction at the physical layer, while the back-end processing module only needs to perform classification and temporal inference of specific behaviors based on the optically encoded feature maps. This reduces the computational load on the back-end processing module, improves inference speed, reduces power consumption and heat generation, and also reduces reliance on high-performance processors. In addition, since the light field modulated by the optical encoding component significantly reduces the identifiable information of the original scene, the resolution requirements for the left and right eye image sensors can be reduced.

[0027] This binocular vision system is suitable for binocular vision perception tasks in various application scenarios. Common binocular vision perception tasks include depth estimation, defect detection, target localization, grasping guidance, obstacle recognition, person detection, and action recognition. Different binocular vision perception tasks have different perception requirements, task types and content vary, and business scenario parameters also differ. Furthermore, the performance of this binocular vision system in performing binocular vision perception tasks depends not only on the optically encoded feature maps output from the left and right eye image sensors but also on the perception inference performance of the image perception model. Therefore: On the one hand, the optical encoding operations performed by the optical encoding component integrated in the binocular camera module in this application need to match the business scenario parameters when the binocular vision system performs binocular visual perception tasks. The business scenario parameters when the binocular vision system performs binocular visual perception tasks include: the relative positions of each component in the binocular camera module, the sensor parameters of the left and right eye image sensors, and the light source wavelength of the light field entering the binocular camera module. The relative positions of each component in the binocular camera module include the relative distance between the optical encoding component and the left and right eye image sensors; when there are left and right eye lenses, it also includes the relative position and relative distance between the optical encoding component and the left and right eye lenses. The sensor parameters of the left and right eye image sensors include the resolution and pixel size of the left and right eye image sensors.

[0028] On the other hand, the image perception model deployed in the backend processing module must also be matched with the binocular vision perception task. This includes ensuring that the model structure used matches the task type, and that the image perception model is pre-trained based on the task content. Typical task types include image classification, object detection, semantic segmentation, face recognition, OCR text recognition, video action recognition, and video object tracking. (a) The model structure of the image perception model deployed in the backend processing module can use a convolutional neural network (CNN) architecture for spatial feature extraction to meet the binocular vision perception task of classifying / detecting / segmenting a single image. (b) The model structure of the image perception model deployed in the backend processing module can use a long short-term memory (LSTM) network architecture for temporal behavior modeling to meet the binocular vision perception task of classifying behavior based on feature sequences. (c) The model structure of the image perception model deployed in the backend processing module can use a CNN+LSTM combined architecture to first extract spatial features and then model temporal relationships to meet video-level spatiotemporal perception tasks such as video action recognition, video anomaly detection, and video object tracking. (d) The image perception model deployed in the backend processing module can adopt the Transformer architecture to complete large-scale, high-precision, multimodal perception tasks. It captures global contextual dependencies through a self-attention mechanism, has a receptive field much larger than CNNs, and naturally supports temporal and cross-modal information fusion. (e) The image perception model deployed in the backend processing module can adopt lightweight networks such as MobileNet and ResNet18. Through depthwise separable convolutions (MobileNet) or simplified residual blocks (ResNet18), it significantly reduces FLOPs and model size while maintaining controllable accuracy loss, meeting low power consumption and low latency requirements. The image perception model deployed in the backend processing module can also be constructed by combining the above architectures.

[0029] In the binocular vision system of this application, the optical coding component integrated in the binocular camera module adopts either a passive static optical coding component or an active dynamic optical coding component. Each of these two types has its own advantages and disadvantages. A common passive static optical coding component is the DOE optical film. Passive static optical coding components do not require continuous power during use, resulting in a simpler structure and lower power consumption. However, the optical coding operations performed by passive static optical coding components remain fixed and cannot be adjusted in real time according to the parameters of the business scenario. Therefore, they are only suitable for single binocular vision perception tasks, exhibiting low flexibility.

[0030] Common active dynamic optical coding (AVC) components include liquid crystal phase modulators, spatial light modulators, tunable metasurfaces, or other tunable optical elements. The optical coding operations performed by AVC components can be dynamically modulated in real time according to the parameters of the application scenario, thus offering greater flexibility and enabling the binocular vision system to perform a variety of binocular vision perception tasks. However, AVC components require continuous power during use, which increases the complexity of the driving circuitry, control, and power consumption.

[0031] Considering the application requirements of low-power visual sensing devices, in one embodiment, the optical coding component 300 integrated in the binocular camera module adopts a passive static optical coding component, specifically a left and right eye DOE optical film. The optical coding operation performed by the left and right eye DOE optical film is determined by its physical parameters. The surface two-dimensional micro-nano phase structure of the left and right eye DOE optical film includes an array formed by multiple microstructure units. Each microstructure unit generates a predetermined phase delay for light passing through that position to achieve optical coding. The phase delay effect varies depending on the size, arrangement, effective aperture, and material of the microstructure units. Therefore, the physical parameters that determine the optical coding operation performed by the left and right eye DOE optical film include the film thickness, effective aperture, material parameters, and design parameters of the surface two-dimensional micro-nano phase structure. The design parameters of the surface two-dimensional micro-nano phase structure include the arrangement of each microstructure unit in the surface two-dimensional micro-nano phase structure and the size of each microstructure unit.

[0032] To compensate for the low flexibility of passive static optical coding components, in another embodiment, the left and right eye DOE optical films are detachably installed in the optical path of the binocular camera module. When the binocular vision system performs different binocular vision perception tasks, the binocular camera module integrates left and right eye DOE optical films with different physical parameters. Therefore, by replacing the left and right eye DOE optical films with different physical parameters, the binocular vision system can adapt to the business scenario parameters of different binocular vision perception tasks. Although this method of offline replacement of left and right eye DOE optical films cannot continuously and in real-time change the phase distribution during operation like active dynamic optical coding components, it still improves the system's flexibility under different business scenarios.

[0033] In addition, multiple layers of left and right eye DOE optical films are stacked sequentially along the light propagation path in the internal optical path of the binocular camera module. This increases the feature selection depth and light field dispersion effect, but it also affects the overall system size. In practical applications, the number of layers of left and right eye DOE optical films can be determined by balancing optical efficiency and system size.

[0034] As mentioned above, the optical encoding operations performed by the optical encoding component and the image perception model deployed in the back-end processing module both need to be matched with the binocular vision perception task, and both directly affect the performance of the binocular vision system for the binocular vision perception task. Therefore, in order to optimize the performance of the binocular vision system for the binocular vision perception task, this application also discloses a system construction method for a binocular vision system with integrated optical encoding at the acquisition end. This system construction method is mainly used to specifically determine the optical encoding operations performed by the optical encoding component and the image perception model deployed in the back-end processing module based on the hardware structure of the binocular vision system of this application, including the following steps, please refer to... Figure 2 The flowchart shown is as follows: Step 210: Obtain the sample dataset.

[0035] This sample dataset includes multiple input images, each carrying a perception result label for the binocular vision perception task required by the binocular vision system. The content of the perception result label differs depending on the binocular vision perception task. For example, when the binocular vision perception task is size measurement, the perception result label is a size label. When the binocular vision perception task is defect detection, the perception result label is a defect classification result. Perception result labels for different binocular vision perception tasks can be annotated using existing methods.

[0036] Step 220: Set the fixed hyperparameters in the all-optical diffraction network model according to the business scenario parameters when the binocular vision system performs binocular vision perception tasks. Use the optimizable model parameters of the hidden diffraction layer in the all-optical diffraction network model to represent the optical coding operation performed by the optical coding component, and establish an all-optical diffraction network model containing an input layer, a hidden diffraction layer and an output layer.

[0037] The all-optical diffraction network model designs a spatial structure of multiple diffractive optical elements, allowing light to diffract and interfere as it passes through these layers, thus automatically performing specific function operations at the output. The input layer loads the optical image or data to be processed as the initial field distribution of the light wave. The hidden diffraction layer is the core computational part of the all-optical diffraction network model. The hidden diffraction layer consists of an array of neurons, each of which can independently control the phase of the transmitted light. The hidden diffraction layer simulates the optical encoding operations performed by the optical encoding components. The output layer receives the light field and converts the light intensity distribution into the final encoded light intensity map.

[0038] In establishing the all-optical diffraction network model, this application sets the hyperparameters in the all-optical diffraction network model according to the business scenario parameters, and uses the optimizable model parameters to be optimized in the all-optical diffraction network model to represent the optical coding operations performed by the optical coding component, thereby performing iterative optimization, wherein: As mentioned above, the business scenario parameters include the relative positions of each component in the binocular camera module, the sensor parameters of the left and right eye image sensors, and the light source band to which the light field entering the binocular camera module belongs. Therefore, setting the fixed hyperparameters in the all-optical diffraction network model according to the business scenario parameters includes: (1) determining the wavelength parameters in the all-optical diffraction network model based on the wavelength of the light source band to which the light field entering the binocular camera module belongs. (2) determining the free space propagation distance between each network layer in the all-optical diffraction network model based on the relative positions of each component in the binocular camera module. The relative positions of the left and right eye lenses, the optical coding component 300, and the left and right eye image sensors 210 and 220 in the binocular camera module can be referred to the description in the above embodiments. The optical coding component 300 is located at the back focal plane of the left and right eye lenses; therefore, the axial distance between the optical coding component 300 and the left and right eye lenses is determined based on the back focal length of the left and right eye lenses. The axial distance between the optical coding component 300 and the left and right eye image sensors 210 and 220 is determined in advance according to the imaging characteristics. Specifically, it is determined according to the wavelength of the light source band, the phase structure of the optical coding component 300, the sensor parameters of the left and right eye image sensors and the target coding light intensity distribution, and can be fine-tuned through simulation and experimental calibration. (3) The sensor parameters of the left and right eye image sensors include the resolution and pixel size of the left and right eye image sensors. The spatial sampling mechanism used to obtain the coding light intensity map is determined according to the sensor parameters of the left and right eye image sensors. The spatial sampling mechanism here can be oversampling or undersampling configuration, which is determined according to the actual situation.

[0039] When the optical coding component uses left and right eye DOE optical films, the optical coding operations performed by the left and right eye DOE optical films are directly determined by their physical parameters. Therefore, when representing the optical coding operations performed by the optical coding component using the optimizable model parameters of the hidden diffraction layer, it is directly related to its physical parameters. This includes: representing the effective aperture of the left and right eye DOE optical films by the spatial range of the optical modulation region in the hidden diffraction layer; representing the arrangement of each neuron in the hidden diffraction layer by the arrangement of each microstructure unit in the two-dimensional micro-nano phase structure on the surface of the left and right eye DOE optical films; and representing the phase modulation amount of each neuron in the hidden diffraction layer by the phase modulation amount of each microstructure unit in the two-dimensional micro-nano phase structure on the surface of the left and right eye DOE optical films. The phase modulation amount of each microstructure unit has an inherent relationship with the film thickness, material parameters, and microstructure unit size.

[0040] Step 230: Use an all-optical diffraction network model to simulate the process of light field entering the binocular camera module and propagating to the left and right eye image sensor planes, and perform physical simulation on each sample input image to obtain the corresponding coded light intensity map.

[0041] Step 240: Use an image perception model that matches the model structure with the task type of the binocular vision perception task to perform perception inference on the encoded light intensity map and output the predicted value of the perception result.

[0042] Step 250: Calculate the loss function using the predicted value of the perception result and the label of the perception result for each sample input image. Based on the loss function, jointly optimize the optimizable model parameters in the all-optical diffraction network model and the image perception model until the all-optical diffraction network model and the image perception model are trained.

[0043] The loss function used varies depending on the task type of binocular vision perception. For example, cross-entropy loss can be used for classification tasks; classification loss, bounding box regression loss, and confidence loss can be used for object detection tasks; disparity error or depth error loss can be used for depth estimation tasks; and for multi-task systems, the classification, detection, depth estimation, and localization losses can be weighted and combined. The appropriate loss function should be set according to the specific situation.

[0044] To improve the manufacturability and system stability of the left and right eye DOE optical films, in another embodiment, a loss function is employed. ,in, , , These are three regularization coefficients, and each loss term includes: The basic perception loss is calculated based on the deviation between the predicted value and the label of the perceived result. As mentioned above, depending on the perception task, cross-entropy loss, classification loss, bounding box regression loss, and confidence loss can be used to calculate the basic perception loss. .

[0045] This is the phase smoothing loss regularization term, calculated based on the difference in phase modulation between adjacent microstructure units. This is used to constrain the phase changes of adjacent microstructural units, thereby constraining the height difference and surface slope of adjacent microstructural units.

[0046] This is the phase quantization error regularization term. After obtaining the phase modulation amount of each microstructure unit, the phase modulation amount is mapped to 0~2π, and then projected onto multiple preset discrete phase levels. This determines the discrete phase level of each microstructure unit. Then, the phase quantization error regularization term is calculated based on the difference between the discrete phase levels corresponding to the phase modulation amounts of different microstructure units. The phase quantization error regularization term Used to constrain the discrete height levels and minimum height differences of different microstructural units in the left and right eye DOE optical films.

[0047] This is a diffraction efficiency loss regularization term, calculated based on the proportion of effective light energy within the target area of ​​the left and right eye image sensors to the total light field energy. The phase distribution used to constrain the left and right eye DOE optical films allows the light field to be better concentrated on the target area of ​​the left and right eye image sensors.

[0048] By using the aforementioned loss function to jointly optimize the optimizable model parameters in the all-optical diffraction network model and the image perception model, the distribution of the two-dimensional micro-nano phase structure on the surface of the left and right eye DOE optical films obtained from the training is made more suitable for subsequent photolithography, nanoimprinting, etching or other micro-nano fabrication processes.

[0049] Furthermore, under the constraint of model complexity, the aforementioned loss function is used to jointly optimize the optimizable model parameters in the all-optical diffraction network model and the image perception model. The model complexity constraint is used to constrain at least one of the following in the image perception model: number of layers, number of channels, number of parameters, and number of floating-point operations. The model complexity constraint is a structural constraint on the image perception model and is not necessarily implemented through the regularization term in the loss function.

[0050] Step 260: Determine the target optical coding operation to be performed by the optical coding component based on the optimizable model parameters of the hidden diffraction layer in the trained all-optical diffraction network model.

[0051] Step 270: The optical coding component that performs the target optical coding operation is integrated into the internal optical path of the binocular camera module in the binocular vision system. Additionally, the trained image perception model is deployed to the back-end processing module of the binocular vision system, together with the binocular camera module, to build the binocular vision system.

[0052] When an active dynamic optical coding component is used, it can be configured according to the target optical coding operation. However, when a passive static optical coding component is used, it needs to be designed and fabricated according to the target optical coding operation. For example, when the optical coding component is a left- or right-eye DOE optical film: the effective aperture, the arrangement of each microstructure unit in the two-dimensional micro / nano phase structure on the surface, and the phase modulation achieved by each microstructure unit can be determined based on the trained all-optical diffraction network model. Then, based on the inherent relationship between the phase modulation of each microstructure unit and the film thickness, material parameters, and microstructure unit size, the combination of film thickness, material parameters, and microstructure unit size for each microstructure unit to achieve the required phase modulation is determined. Finally, the target physical parameters for the left- or right-eye DOE optical film to perform the target optical coding operation include at least the effective aperture, film thickness, material parameters, arrangement of each microstructure unit, and size of each microstructure unit.

[0053] Finally, left and right eye DOE optical films for performing the target optical coding operation are fabricated according to the target physical parameters. These films are then integrated into the internal optical path of the binocular camera module in the binocular vision system to build the required binocular vision system. Existing mature optical manufacturing processes, including 3D printing and photolithography, can be used to manufacture the optical coding components.

[0054] When integrating the fabricated left and right eye DOE optical films into the internal optical path of the binocular camera module, in addition to conforming to the relative positions of the components in the binocular camera module, it is also necessary to align the effective aperture center of the left and right eye DOE optical films with the centers of the left and right eye image sensors 210 and 220, and to align the effective aperture center of the left and right eye DOE optical films with the optical axes of the left and right eye lenses 110 and 120 in the binocular camera module. This ensures the coaxiality between the components in the binocular camera module, which can be controlled by positioning steps, radial limits, and coded image center calibration. Furthermore, it is necessary to ensure that the surface normals of the left and right eye DOE optical films are parallel to the optical axes of the left and right eye lenses 110 and 120, and that the horizontal and vertical directions of the phase matrix of the left and right eye DOE optical films maintain the same directional relationship as the pixel coordinate directions of the left and right eye image sensors 210 and 220 as the training model. This can be controlled by anti-rotation keys, positioning notches, or directional markings. Positioning steps, spacers, shims, or adjustable mounting brackets can be used to control the position of the left and right eye DOE optical films, which are then locked after adjustment. The fasteners used to secure the left and right eye DOE optical films are located outside the effective aperture of the films, with mechanical fastening being preferred. When adhesive fastening is used, the adhesive should be placed outside the light-transmitting area. After installation, the system can be calibrated using a calibration board, standard light source, or standard test target. Calibration includes the coaxial relationship between the DOE and the lens, the relative position of the DOE and the sensor, the intrinsic and extrinsic parameters of the left and right eye cameras, epipolar correction parameters, and the mapping relationship between the DOE-encoded image and the back-end algorithm input.

[0055] The left and right DOE optical films employ a collaborative optimization design. In practical applications, the physical parameters of the left and right DOE optical films are identical, which helps maintain consistency in left and right eye imaging and facilitates stereo matching and depth estimation by the back-end processing module based on similar coded images. Alternatively, the physical parameters of the left and right DOE optical films can differ, allowing them to enhance information in different directions, spatial frequencies, or local structures, thus creating complementary features in the left and right coded images. Regardless of whether an identical or complementary design is used, the left and right DOE optical films must be designed to maintain binocular parallax, epipolar constraints, and target spatial geometry.

[0056] When multiple layers of left and right eye DOE optical films are stacked sequentially along the light propagation path within the optical path of a binocular camera module, the resulting all-optical diffraction network model includes multiple layers of hidden diffraction layers connected in sequence. Each hidden diffraction layer corresponds to one left or right eye DOE optical film. The free-space propagation distance between the corresponding hidden diffraction layers is determined based on the stacking distance between the left and right eye DOE optical films. The optical coding operation performed by the corresponding left and right eye DOE optical films is represented by the optimizable model parameters of each hidden diffraction layer. The model is trained using the above method. Based on the optimizable model parameters of the hidden diffraction layers in the trained all-optical diffraction network model, the target optical coding operation performed by each layer of left and right eye DOE optical films is determined, and the target physical parameters of each layer of left and right eye DOE optical films are obtained. Finally, each layer of left and right eye DOE optical films can be fabricated and integrated into the binocular camera module.

[0057] The above descriptions are merely preferred embodiments of this application, and this application is not limited to the above embodiments. It is understood that other improvements and variations that can be directly derived or conceived by those skilled in the art without departing from the spirit and concept of this application should be considered to be included within the protection scope of this application.

Claims

1. A binocular vision system with integrated optical coding at the acquisition end, characterized in that, The binocular vision system includes a binocular camera module and a back-end processing module; The binocular camera module integrates an optical coding component in its internal optical path. The light field entering the binocular camera module undergoes phase modulation and diffraction propagation by the optical coding component to complete the optical coding operation. After that, optical coding feature maps are formed on the left and right eye image sensor planes respectively and output to the back-end processing module. The optical coding operation performed by the optical coding component is matched with the business scenario parameters when the binocular vision system performs binocular vision perception tasks. The back-end processing module uses an image perception model that matches the task type of the binocular vision perception task to perform perceptual reasoning on the optically encoded feature map and outputs the perception results to complete the binocular vision perception task.

2. The binocular vision system according to claim 1, characterized in that, The optical coding components integrated in the binocular camera module are left and right eye DOE optical films. The optical coding operation performed by the left and right eye DOE optical films is determined by their physical parameters. The physical parameters of the left and right eye DOE optical films include the film thickness, effective aperture, material parameters, and design parameters of the surface two-dimensional micro-nano phase structure. The design parameters of the surface two-dimensional micro-nano phase structure include the arrangement of each microstructure unit in the surface two-dimensional micro-nano phase structure and the size of each microstructure unit.

3. The binocular vision system according to claim 1, characterized in that, The service scenario parameters of the binocular vision system when performing binocular vision perception tasks include: the relative positions of each component in the binocular camera module, the sensor parameters of the left and right eye image sensors, and the light source wavelength of the light field entering the binocular camera module.

4. The binocular vision system according to claim 2, characterized in that, The left and right eye DOE optical films can be detachably installed in the optical path of the binocular camera module. When the binocular vision system performs different binocular vision perception tasks, the binocular camera module integrates left and right eye DOE optical films with different physical parameters.

5. The binocular vision system according to claim 1, characterized in that, The effective aperture center of the optical coding component integrated in the binocular camera module is aligned with the center of the left and right eye image sensors in the binocular camera module. When the binocular camera module includes left and right eye lenses, the optical coding component integrated in the binocular camera module is located on the optical path channel between the left and right eye lenses and the left and right eye image sensors and is located at the back focal plane of the left and right eye lenses. The effective aperture center of the optical coding component is coaxial with the optical axis of the left and right eye lenses in the binocular camera module. After the light field entering the binocular camera module is imaged by the left and right eye lenses, it is optically encoded by the optical coding component and forms an optical coding feature map on the plane of the left and right eye image sensors.

6. The binocular vision system according to claim 2, characterized in that, The binocular camera module contains multiple layers of left and right eye DOE optical films stacked sequentially along the light propagation path.

7. A method for constructing a binocular vision system with integrated optical coding at the acquisition end, characterized in that, The system setup method includes: Obtain a sample dataset, which includes multiple sample input images, and each sample input image carries a label of its perception result under the binocular vision perception task required by the binocular vision system; Based on the business scenario parameters when the binocular vision system performs binocular vision perception tasks, the fixed hyperparameters in the all-optical diffraction network model are set, and the optical coding operations performed by the optical coding component are represented by the optimizable model parameters of the hidden diffraction layer in the all-optical diffraction network model. Thus, an all-optical diffraction network model containing an input layer, a hidden diffraction layer, and an output layer is established. The process of light field entering the binocular camera module and propagating to the left and right eye image sensor planes is simulated using the all-optical diffraction network model. Physical simulation is performed on each sample input image to obtain the corresponding encoded light intensity map. The image perception model that matches the model structure with the task type of the binocular vision perception task is used to perform perception inference on the encoded light intensity map and output the perception result prediction value. The loss function is calculated using the predicted value of the perception result and the label of the perception result for each sample input image. Based on the loss function, the optimizable model parameters of the all-optical diffraction network model and the image perception model are jointly optimized. The target optical coding operation performed by the optical coding component is determined based on the optimizable model parameters of the hidden diffraction layer in the trained all-optical diffraction network model. The optical coding component that performs the target optical coding operation is integrated into the internal optical path of the binocular camera module in the binocular vision system, and the trained image perception model is deployed to the back-end processing module of the binocular vision system to build a binocular vision system with integrated optical coding at the acquisition end as described in any one of claims 1-6.

8. The system construction method according to claim 7, characterized in that, The optical encoding components integrated in the binocular camera module are left and right eye DOE optical films. The optical encoding operations performed by the left and right eye DOE optical films are determined by their physical parameters. The optical coding operations performed by the optical coding component are represented by the optimizable model parameters of the hidden diffraction layer in the all-optical diffraction network model. These operations include: the effective aperture of the left and right eye DOE optical films is represented by the spatial region range of the optical modulation region in the hidden diffraction layer; the arrangement of each neuron in the hidden diffraction layer is represented by the arrangement of each microstructure unit in the two-dimensional micro-nano phase structure on the surface of the left and right eye DOE optical films; and the phase modulation amount of each neuron in the hidden diffraction layer is represented by the phase modulation amount of each microstructure unit in the two-dimensional micro-nano phase structure on the surface of the left and right eye DOE optical films. The phase modulation amount of each microstructure unit has an inherent relationship with the film thickness, material parameters, and microstructure unit size. Integrating the optical coding component that performs the target optical coding operation into the internal optical path of the binocular camera module in a binocular vision system includes: determining the effective aperture of the left and right eye DOE optical films, the arrangement of each microstructure unit in the two-dimensional micro / nano phase structure on the surface, and the phase modulation amount achieved by each microstructure unit based on the trained all-optical diffraction network model; and determining the film thickness, material parameters, and microstructure unit size of each microstructure unit when achieving the required phase modulation amount by combining the inherent relationship between the phase modulation amount of each microstructure unit and the film thickness, material parameters, and microstructure unit size, thus obtaining the target physical parameters of the left and right eye DOE optical films when performing the target optical coding operation, including at least the effective aperture, film thickness, material parameters, arrangement of each microstructure unit, and size of each microstructure unit; fabricating the optical coding component that performs the target optical coding operation according to the target physical parameters; and integrating the optical coding component into the internal optical path of the binocular camera module in a binocular vision system.

9. The system construction method according to claim 8, characterized in that, The fixed hyperparameters in the all-optical diffraction network model are set according to the business scenario parameters when the binocular vision system performs binocular vision perception tasks, including: The wavelength parameters in the all-optical diffraction network model are determined based on the wavelength of the light source band to which the light field enters the binocular camera module; the free space propagation distance between each network layer in the all-optical diffraction network model is determined based on the relative positions of each component in the binocular camera module; and the spatial sampling mechanism used to obtain the coded light intensity map is determined based on the sensor parameters of the left and right eye image sensors.

10. The system construction method according to claim 8, characterized in that, The loss function ,in, It is based on the perception loss and is calculated according to the deviation between the predicted value of the perception result and the label of the perception result; It is a phase smoothing loss regularization term and is calculated based on the difference in phase modulation of adjacent microstructure units; It is the phase quantization error regularization term, and is calculated based on the difference between the discrete phase levels corresponding to the phase modulation amounts of different microstructure units; It is the diffraction efficiency loss regularization term, and is calculated based on the proportion of effective light energy in the target area of ​​the left and right eye image sensors to the total light field energy; , , These are the coefficients of three regularization terms.

Images