Multi-camera image processing method, system, equipment and medium

Abstract

The invention discloses a multi-camera image processing method, system and device and a medium, and is applied to the technical field of vehicle-mounted visual perception. Comprising the following steps: acquiring RAW data, exposure parameters, ambient light intensity and exposure delay of each camera; based on the exposure parameter, the ambient light intensity and the exposure delay of each camera, predicting the exposure parameter compensation amount of each camera through a Transform dynamic exposure compensation model; based on the camera topological graph, performing cross-camera noise reduction and dynamic range alignment processing on the RAW data of each camera through a graph neural network to obtain optimized RAW data; and on the basis of the exposure parameter compensation amount of each camera and the optimized RAW data, the optimal ISP parameter combination of each camera is predicted through a Transform dynamic parameter prediction model to generate a vehicle-mounted visual image, so that the space-time synchronization precision, the heterogeneous data processing efficiency and the dynamic environment adaptability are improved.

Description

Technical Field

[0001] This application relates to the field of vehicle-mounted visual perception technology, and in particular to a multi-camera image processing method, system, device and medium. Background Technology

[0002] Current in-vehicle multi-camera systems primarily rely on individual image signal processors (ISPs) to process image sensor data separately, and achieve frame-level time synchronization through hardware synchronization mechanisms (such as VSYNC signals). A typical architecture involves each camera (i.e., image sensor) connecting to its corresponding ISP module via a MIPI interface. Each ISP module processes the RAW data acquired by its respective camera in parallel and outputs YUV / RGB format images to the main control module. The main control module then uses a fusion algorithm to complete tasks such as target detection and 3D reconstruction. However, traditional solutions suffer from the following core pain points: (1) Insufficient spatiotemporal synchronization accuracy: Hardware synchronization is easily affected by differences in sensor exposure delay, MIPI interface transmission jitter and ISP pipeline fluctuations, and the inter-frame synchronization error often exceeds 200μs. In high-speed dynamic scenes (such as driving at 120km / h), this error will cause spatial misalignment of more than 6.7cm in multi-view images, significantly reducing the accuracy of target tracking and 3D reconstruction.

[0003] (2) Low efficiency of heterogeneous data processing: Each ISP module competes for memory bandwidth and computing units, and high-resolution data streams are prone to pipeline blockage, with end-to-end latency exceeding 50ms, making it difficult to meet real-time decision-making requirements. At the same time, the noise reduction and HDR parameters independently optimized by each ISP module result in inconsistent color / brightness of the output image, increasing the complexity of the fusion algorithm.

[0004] (3) Poor adaptability to dynamic environments: Each ISP module adopts a fixed pipeline processing (such as the 3A algorithm), which cannot dynamically adjust the collaborative strategy according to sudden changes in lighting (such as entering and exiting tunnels). For example, in a strong backlight scene, the front-view camera needs to activate the 150dB HDR mode, while the surround-view camera is in normal exposure. The dynamic range of the two is mismatched, resulting in local overexposure of the fused image. In addition, in low light environment, due to the difference in signal-to-noise ratio, the various cameras have different levels of detail retention, which affects the efficiency and accuracy of obstacle detection at night. Summary of the Invention

[0005] This application provides a multi-camera image processing method, system, device, and medium to address the problems of insufficient spatiotemporal synchronization accuracy, low efficiency in processing heterogeneous data, and poor adaptability to dynamic environments in existing technologies when processing RAW data from multiple cameras. The technical solution provided by this application is as follows: On the one hand, this application provides a multi-camera image processing method, including: Acquire RAW data, exposure parameters, ambient light intensity, and exposure delay for each camera in the current time period, as well as exposure parameters, ambient light intensity, and exposure delay in historical time periods prior to the current time period; Based on the exposure parameters, ambient light intensity, and exposure delay of each camera in the current and historical time periods, the Transformer dynamic exposure compensation model is used to predict the amount of exposure parameter compensation for each camera in future time periods after the current time period. A camera topology graph is constructed with each camera as a node. In the camera topology graph, the node features of each node include the spatial coordinates of the camera and the RAW data, exposure parameters and ambient light intensity of the camera in the current time period. The edge features between each node include the spatial distance between cameras, the percentage of overlapping viewpoints and the difference in optical distortion parameters. Based on the camera topology map, the RAW data of each camera in the current time period is processed by a graph neural network (GNN) to perform cross-camera noise reduction and dynamic range alignment to obtain the optimized RAW data of each camera in the current time period. Based on the exposure parameter compensation amount of each camera in the future time period and the optimized RAW data in the current time period, the optimal ISP parameter combination of each camera in the current time period is predicted by the Transformer dynamic parameter prediction model. The optimal ISP parameter combination and optimized RAW data for each camera in the current time period are sent to the ISP modules of each camera respectively, so that the ISP modules of each camera can process the optimized RAW data into vehicle vision images based on the optimal ISP parameter combination.

[0006] Optionally, before acquiring the RAW data, exposure parameters, ambient light intensity, and exposure delay of each camera in the current time period, as well as the exposure parameters, ambient light intensity, and exposure delay in historical time periods prior to the current time period, the method further includes: The system receives PPS-level pulses via the Global Positioning System (GPS), uses these PPS-level pulses as a global time reference, generates a clock synchronization signal, and distributes it to each camera via the GMSL3 link. This ensures that the exposure clock of each camera is phase-locked with the global time reference, and embeds a nanosecond-level timestamp in the header of the RAW data during synchronous acquisition.

[0007] Optionally, based on the exposure parameters, ambient light intensity, and exposure delay of each camera in the current and historical time periods, after predicting the exposure parameter compensation amount for each camera in future time periods after the current time period using the Transformer dynamic exposure compensation model, the method further includes: Based on the exposure parameter compensation amount of each camera in the future time period, the trigger phase of the clock synchronization signal is adjusted in the future time period.

[0008] Optionally, the Transformer dynamic exposure compensation model includes a first Transformer encoder and a first Transformer decoder; based on the exposure parameters, ambient light intensity, and exposure delay of each camera in the current and historical time periods, the Transformer dynamic exposure compensation model predicts the exposure parameter compensation amount for each camera in future time periods after the current time period, including: The first Transformer encoder is used to positionally encode the exposure parameters, ambient light intensity and exposure delay of each camera in the current time period and historical time period to obtain high-dimensional temporal features. The first multi-head self-attention mechanism is used to capture the temporal dependence features of the exposure delay of each camera based on the high-dimensional temporal features. The first Transformer decoder predicts the amount of exposure parameter compensation for each camera in future time periods based on the exposure delay time-series dependency features.

[0009] Optionally, based on the camera topology map, a graph neural network is used to perform cross-camera noise reduction and dynamic range alignment processing on the RAW data of each camera in the current time period to obtain optimized RAW data of each camera in the current time period, including: Using a graph neural network, edge weights between nodes are calculated based on edge features in the camera topology graph. Then, RAW noise features and dynamic range metrics for each node are calculated based on node features in the topology graph. For each node in the topology graph, the RAW noise features of that node are fused with the weighted RAW noise features of its neighboring nodes to obtain a cross-camera noise estimate. The dynamic range metric of that node is fused with the weighted dynamic range metric of its neighboring nodes to obtain a global dynamic range reference value. Based on the cross-camera noise estimate and the global dynamic range reference value, cross-camera noise reduction and dynamic range alignment are performed on the node's RAW data to obtain optimized RAW data. Here, the weighted RAW noise feature is the weighted result of the RAW noise feature and edge weights, and the weighted dynamic range metric is the weighted result of the dynamic range metric and edge weights.

[0010] Optionally, the graph neural network also outputs the optimal exposure parameter set for each camera; based on the camera topology graph, after performing cross-camera noise reduction and dynamic range alignment processing on the RAW data of each camera in the current time period using the graph neural network to obtain the optimized RAW data of each camera in the current time period, it also includes: For backlit or overexposed scenes, the exposure parameters of each camera are adjusted based on the optimal exposure parameter set of each camera.

[0011] Optionally, the Transformer dynamic parameter prediction model includes a second Transformer encoder and a second Transformer decoder; based on the exposure parameter compensation amount of each camera in future time periods and the optimized RAW data in the current time period, the Transformer dynamic parameter prediction model predicts the optimal ISP parameter combination for each camera in the current time period, including: The second Transformer encoder extracts the exposure parameter compensation amount of each camera in the future time period and the optimized RAW data in the current time period to obtain the compensation features and RAW features corresponding to each camera. The second multi-head self-attention mechanism is used to capture the global dependency features between the compensation features and RAW features corresponding to each camera. The second Transformer decoder predicts the optimal ISP parameter combination for each camera in the current time period based on the global dependency features between the compensation features and RAW features corresponding to each camera.

[0012] On the other hand, this application provides a multi-camera image processing system, including: The data acquisition unit is used to acquire RAW data, exposure parameters, ambient light intensity and exposure delay of each camera in the current time period, as well as exposure parameters, ambient light intensity and exposure delay in historical time periods before the current time period. The offset prediction unit is used to predict the amount of exposure parameter compensation for each camera in future time periods after the current time period based on the exposure parameters, ambient light intensity and exposure delay of each camera in the current time period and historical time periods, using the Transformer dynamic exposure compensation model. The topology graph construction unit is used to construct a camera topology graph with each camera as a node. In the camera topology graph, the node features of each node include the spatial coordinates of the camera and the RAW data, exposure parameters and ambient light intensity of the camera in the current time period. The edge features between each node include the spatial distance between cameras, the percentage of overlapping views and the difference in optical distortion parameters. The RAW optimization unit is used to perform cross-camera noise reduction and dynamic range alignment processing on the RAW data of each camera in the current time period based on the camera topology map and through graph neural network to obtain the optimized RAW data of each camera in the current time period. The parameter optimization unit is used to predict the optimal ISP parameter combination for each camera in the current time period based on the exposure parameter compensation amount for each camera in the future time period and the optimized RAW data in the current time period, using the Transformer dynamic parameter prediction model. The image generation unit is used to send the optimal ISP parameter combination and optimized RAW data of each camera in the current time period to the ISP module of each camera, so that the ISP module of each camera can process the optimized RAW data into an in-vehicle visual image based on the optimal ISP parameter combination.

[0013] On the other hand, this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-described multi-camera image processing method.

[0014] On the other hand, this application also provides a computer-readable storage medium storing computer instructions, which, when executed by a processor, implement the above-described multi-camera image processing method.

[0015] The beneficial effects of this application are as follows: This application utilizes a dynamic compensation algorithm and an end-to-end latency prediction model to compress frame synchronization errors, reduce multi-view target localization errors, and achieve millisecond-level synchronization accuracy. Furthermore, by employing a heterogeneous computing task dynamic migration mechanism, memory bandwidth is saved, reducing intermediate format conversions for RAW data in subsequent AI collaborative processing and improving resource utilization and processing efficiency. In addition, unified optimization of ISP parameters is achieved through neural networks, ensuring output consistency and significantly reducing image quality differences between multiple cameras.

[0016] Other features and advantages of this application will be set forth in the following description, and will be apparent in part from the description, or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings. Attached Figure Description

[0017] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 This is a schematic diagram outlining the multi-camera image processing method in the embodiments of this application; Figure 2 This is a functional structure diagram of the multi-camera image processing system in the embodiments of this application; Figure 3 This is a schematic diagram of the hardware structure of the electronic device in the embodiments of this application. Detailed Implementation

[0018] To make the objectives, technical solutions, and beneficial effects of this application clearer, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0019] This application provides a multi-camera image processing method, applied to electronic devices such as main control modules. (See attached document.) Figure 1 As shown, the general flow of the multi-camera image processing method provided in this application embodiment is as follows: Step 101: Obtain the RAW data, exposure parameters, ambient light intensity, and exposure delay of each camera in the current time period, as well as the exposure parameters, ambient light intensity, and exposure delay in historical time periods before the current time period.

[0020] In this embodiment, to achieve global clock synchronization, it is necessary to receive PPS-level pulses via the Global Positioning System (GPS). Using these PPS-level pulses as the global time reference, a clock synchronization signal is generated and distributed to each camera via the GMSL3 link. This ensures that the exposure clock of each camera remains phase-locked with the global time reference, and a nanosecond-level timestamp is embedded in the RAW data header during synchronous RAW data acquisition. Subsequently, the RAW data, exposure parameters, ambient light intensity, and exposure delay of each camera within the current time period, as well as the exposure parameters, ambient light intensity, and exposure delay of historical time periods preceding the current time period, are acquired. The current time period and historical time periods can be adjacent or non-adjacent.

[0021] Step 102: Based on the exposure parameters, ambient light intensity, and exposure delay of each camera in the current and historical time periods, predict the amount of exposure parameter compensation for each camera in future time periods after the current time period using the Transformer dynamic exposure compensation model.

[0022] In this embodiment, the Transformer dynamic exposure compensation model includes a first Transformer encoder and a first Transformer decoder. Based on this, when predicting the exposure parameter compensation amount for each camera in future time periods after the current time period using the Transformer dynamic exposure compensation model, based on the exposure parameters, ambient light intensity, and exposure delay of each camera in the current and historical time periods, the following methods can be used, but are not limited to: The first Transformer encoder is used to positionally encode the exposure parameters, ambient light intensity and exposure delay of each camera in the current time period and historical time period to obtain high-dimensional temporal features. The first multi-head self-attention mechanism is used to capture the temporal dependence features of the exposure delay of each camera based on the high-dimensional temporal features. The first Transformer decoder predicts the amount of exposure parameter compensation for each camera in future time periods based on the exposure delay time-series dependency features.

[0023] Furthermore, based on the exposure parameters, ambient light intensity, and exposure delay of each camera in the current and historical time periods, the Transformer dynamic exposure compensation model predicts the exposure parameter compensation amount of each camera in future time periods after the current time period. Based on the exposure parameter compensation amount of each camera in future time periods, the trigger phase of the clock synchronization signal in future time periods can be adjusted, thereby effectively reducing the exposure delay of each camera in future time periods.

[0024] Step 103: Construct a camera topology graph with each camera as a node; wherein, in the camera topology graph, the node features of each node include the spatial coordinates of the camera and the RAW data, exposure parameters and ambient light intensity of the camera in the current time period; the edge features between each node include the spatial distance between cameras, the percentage of overlapping views and the difference in optical distortion parameters.

[0025] In this embodiment of the application, when constructing a camera topology map with each camera as a node, the following methods may be used, but are not limited to: Node establishment: Each camera in the vehicle multi-camera system is treated as an independent node. A unique identifier is assigned to each node, and the spatial coordinates of the node are written into the node characteristics.

[0026] Node feature filling: Within the current time period, write the following node features for each node: RAW data of the camera, exposure parameters, and ambient light intensity; the above features constitute the complete spatiotemporal exposure features of each node.

[0027] Edge candidate generation: For any two nodes, calculate their spatial distance, viewpoint overlap ratio, and optical distortion parameter difference to form edge features; when the spatial distance, viewpoint overlap ratio, and optical distortion parameter difference all meet the preset gating conditions, construct the edge between the two nodes and assign a unique edge identifier.

[0028] Edge feature assignment: For each constructed edge, the spatial location distance, the percentage of view overlap, and the difference in optical distortion parameters are written into the edge features to complete the construction of the camera topology map.

[0029] Step 104: Based on the camera topology map, perform cross-camera noise reduction and dynamic range alignment processing on the RAW data of each camera in the current time period using a graph neural network to obtain the optimized RAW data of each camera in the current time period.

[0030] In this embodiment of the application, when performing cross-camera noise reduction and dynamic range alignment processing on the RAW data of each camera in the current time period based on the camera topology map and using a graph neural network to obtain the optimized RAW data of each camera in the current time period, the following methods can be used, but are not limited to: By employing a graph neural network, edge weights between nodes are calculated based on their edge features in the camera topology graph. Then, RAW noise features and dynamic range metrics for each node are calculated based on their node features. For each node in the topology graph, the node's RAW noise features are fused with the weighted RAW noise features of its neighbors to obtain a cross-camera noise estimate. The node's dynamic range metric is then fused with the weighted dynamic range metric of its neighbors to obtain a global dynamic range reference value. Based on the cross-camera noise estimate and the global dynamic range reference value, cross-camera noise reduction and dynamic range alignment are performed on the node's RAW data to obtain optimized RAW data. Here, the weighted RAW noise feature is the weighted result of the RAW noise feature and edge weights, and the weighted dynamic range metric is the weighted result of the dynamic range metric and edge weights. In this way, through graph neural network processing, the RAW data processing can skip the YUV conversion process, thereby significantly reducing memory bandwidth and effectively narrowing the dynamic range differences between multiple cameras.

[0031] Furthermore, in this embodiment, the graph neural network also outputs the optimal exposure parameter set for each camera. Based on this, and using the camera topology graph, after performing cross-camera noise reduction and dynamic range alignment processing on the RAW data of each camera in the current time period to obtain the optimized RAW data of each camera in the current time period, the exposure parameters of each camera can be adjusted based on the optimal exposure parameter set for each camera in backlight or overexposure scenarios, thereby ensuring the real-time performance and accuracy of the exposure parameters of each camera in backlight or overexposure scenarios.

[0032] Step 105: Based on the exposure parameter compensation amount of each camera in the future time period and the optimized RAW data in the current time period, predict the optimal ISP parameter combination of each camera in the current time period using the Transformer dynamic parameter prediction model.

[0033] In this embodiment, the second multi-head self-attention mechanism built into the Transformer dynamic parameter prediction model has a global perception advantage and can better adapt to the color consistency requirements of multiple cameras. The Transformer dynamic parameter prediction model includes a second Transformer encoder and a second Transformer decoder; based on this, when predicting the optimal ISP parameter combination of each camera in the current time period based on the exposure parameter compensation amount of each camera in the future time period and the optimized RAW data in the current time period, the Transformer dynamic parameter prediction model can adopt, but is not limited to, the following methods: The second Transformer encoder extracts the exposure parameter compensation amount of each camera in the future time period and the optimized RAW data in the current time period to obtain the compensation features and RAW features corresponding to each camera. The second multi-head self-attention mechanism is used to capture the global dependency features between the compensation features and RAW features corresponding to each camera. The second Transformer decoder predicts the optimal ISP parameter combination for each camera in the current time period based on the global dependency features between the compensation features and RAW features corresponding to each camera.

[0034] Step 106: Send the optimal ISP parameter combination and optimized RAW data of each camera in the current time period to the ISP module of each camera, so that the ISP module of each camera can process the optimized RAW data into vehicle vision images based on the optimal ISP parameter combination.

[0035] In this embodiment, by distributing the optimal ISP parameter combination and optimized RAW data of each camera within the current time period to the ISP modules of each camera, the ISP modules of each camera can process the optimized RAW data into vehicle-mounted visual images based on the optimal ISP parameter combination. This, through dynamic compensation algorithms and end-to-end latency prediction models, can compress frame synchronization errors, reduce multi-view target positioning errors, and achieve millisecond-level synchronization accuracy. Furthermore, by employing a heterogeneous computing task dynamic migration mechanism, memory bandwidth can be saved, reducing intermediate format conversions in subsequent AI collaborative processing of RAW data and improving resource utilization and processing efficiency. In addition, unified ISP parameter optimization is achieved through neural networks, ensuring output consistency and significantly reducing image quality differences between multiple cameras.

[0036] The multi-camera image processing method provided in this application embodiment will be further described in detail below. The complete technical flow of the multi-camera image processing method provided in this application embodiment is as follows: First, at the hardware layer, multiple cameras acquire RAW data and embed timestamps to confirm time synchronization. This part requires two processes: first, triggering synchronization on the hardware side. The FPGA generates a 40MHz global clock signal, which is distributed to each camera via the GMSL3 link to control the CMOS exposure start time. Second, embedding timestamps: a nanosecond-level timestamp (based on GPS time aligned with the PTP protocol) is embedded in the header of the RAW data stream to provide a reference for subsequent timing alignment.

[0037] Secondly, timing modeling is performed based on the Transformer dynamic exposure compensation model: each camera uploads exposure parameters (such as shutter speed, ISO), ambient light intensity, and exposure delay in real time to obtain timing data. This timing data is input into the Transformer dynamic exposure compensation model. In the encoder layer, the timing data is position-encoded into high-dimensional features, and a self-attention mechanism is used to capture the timing-dependent features of exposure delay for each camera. In the decoder layer, the exposure parameter compensation amount (i.e., phase offset ±5ns) for the next frame of each camera is output based on the timing-dependent features of exposure delay for each camera. Furthermore, the FPGA trigger signal phase is dynamically adjusted based on the exposure parameter compensation amount (i.e., phase offset ±5ns) for the next frame of each camera. This process significantly reduces exposure timing drift error and improves the accuracy of multi-frame camera exposure.

[0038] Secondly, collaborative processing based on GNN is performed in the RAW domain: each camera is treated as a node, with RAW data blocks, spatial coordinates, and exposure parameters as node features, and spatial distances between nodes, viewpoint overlap ratios, and optical distortion parameters as edge features, constructing a camera topology graph. Based on this topology graph, GNN performs cross-camera noise reduction and dynamic range alignment on the RAW data of each camera within the current time period to obtain optimized RAW data for each camera within the current time period. The GNN message passing method is as follows: GNN achieves two effects. Firstly, cross-camera noise reduction. Nodes aggregate the RAW noise patterns of neighboring nodes, dynamically adjusting the noise reduction intensity through edge weights, and sharing weights to reduce redundant computation. Secondly, dynamic range alignment. For backlit scenes, GNN outputs the optimal exposure parameters for each camera, which are sent in real-time via the I2C bus. After GNN processing, RAW data processing can skip the YUV conversion process, reducing memory bandwidth by 10% and minimizing the dynamic range differences among multiple cameras.

[0039] Finally, ISP parameters are uniformly optimized: a lightweight Transformer dynamic parameter prediction model is used to integrate the optimized RAW data from each camera output by the GNN and the exposure parameter compensation amount output by the Transformer dynamic exposure compensation model, and output the optimal parameter combination of the ISP module of each camera, so that each ISP module can convert the optimized RAW data into in-vehicle vision images based on the optimal parameter combination.

[0040] The following example illustrates a scenario of multiple cameras working together to monitor a vehicle entering a tunnel under alternating light and dark conditions: First, the hardware layer performs system initialization and clock synchronization. The domain controller receives PPS-level pulses from GPS as a global time source and distributes a 40MHz synchronization clock signal to all cameras via the GMSL3 link. Then, a dynamic compensation mechanism is triggered, using the infrared camera (camera ID = 0) as a reference to calculate the relative exposure delay of other cameras. The Transformer dynamic exposure compensation model takes into account the current frame's RAW data, exposure parameters (such as shutter speed, ISO, etc.), ambient light intensity, and exposure delay of each camera, as well as the exposure parameters, ambient light intensity, and exposure delay of the past 100 frames. It then predicts the phase shift (i.e., the exposure parameter compensation amount) through a multi-head self-attention mechanism.

[0041] Then, when the vehicle enters the tunnel, the forward-facing wide-angle camera encounters strong backlighting, while the surround-view camera is in shadow. At this point, the GNN network needs to perform dynamic range completion: the GNN uses the camera as a node, the camera's RAW data, exposure parameters, and ambient light intensity as node features, and the spatial distance between cameras, the percentage of overlapping views, and the differences in optical distortion parameters as edge features to construct a camera topology graph; after completing the camera topology graph construction, the GNN aggregates the node features and fuses the camera data from neighboring nodes.

[0042] Finally, the RAW domain performs noise reduction based on the camera topology map of the GNN. After the wide-angle and panoramic RAW data are input into the GNN, the telephoto nodes aggregate the high-frequency features of the wide-angle view and share weights for noise reduction, reducing redundant calculations and lowering memory bandwidth usage. This also allows skipping YUV conversion, with RAW data directly transmitted to the ISP module, reducing memory bandwidth to 1.2GB / s.

[0043] Based on the above embodiments, this application provides a multi-camera image processing system, see below. Figure 2 As shown, the multi-camera image processing system 200 provided in this application embodiment includes at least: The data acquisition unit 201 is used to acquire the RAW data, exposure parameters, ambient light intensity and exposure delay of each camera in the current time period, as well as the exposure parameters, ambient light intensity and exposure delay in historical time periods before the current time period. The offset prediction unit 202 is used to predict the amount of exposure parameter compensation for each camera in future time periods after the current time period based on the exposure parameters, ambient light intensity and exposure delay of each camera in the current time period and historical time periods, using the Transformer dynamic exposure compensation model. The topology graph construction unit 203 is used to construct a camera topology graph with each camera as a node; wherein, in the camera topology graph, the node features of each node include the spatial position coordinates of the camera and the RAW data, exposure parameters and ambient light intensity of the camera in the current time period; the edge features between each node include the spatial position distance between cameras, the percentage of overlapping viewpoints and the difference in optical distortion parameters. RAW optimization unit 204 is used to perform cross-camera noise reduction and dynamic range alignment processing on the RAW data of each camera in the current time period based on the camera topology map and through graph neural network to obtain optimized RAW data of each camera in the current time period. The parameter optimization unit 205 is used to predict the optimal ISP parameter combination of each camera in the current time period based on the exposure parameter compensation amount of each camera in the future time period and the optimized RAW data in the current time period through the Transformer dynamic parameter prediction model. The image generation unit 206 is used to send the optimal ISP parameter combination and optimized RAW data of each camera in the current time period to the ISP module of each camera, so that the ISP module of each camera can process the optimized RAW data into an in-vehicle visual image based on the optimal ISP parameter combination.

[0044] In one possible implementation, the multi-camera image processing system 200 provided in this application embodiment further includes: The clock synchronization unit 207 is used to receive PPS second-level pulses through the global positioning system, use the PPS second-level pulses as the global time reference, generate clock synchronization signals and distribute them to each camera through the GMSL3 link, so that the exposure clock of each camera is phase-locked with the global time reference, and embed nanosecond-level timestamps in the header of the RAW data when synchronously acquiring RAW data.

[0045] In one possible implementation, the offset prediction unit 202 is also used to adjust the trigger phase of the clock synchronization signal in the future time period based on the exposure parameter compensation amount of each camera in the future time period.

[0046] In one possible implementation, the Transformer dynamic exposure compensation model includes a first Transformer encoder and a first Transformer decoder; the offset prediction unit 202 is used to obtain high-dimensional temporal features by position encoding the exposure parameters, ambient light intensity and exposure delay of each camera in the current time period and historical time periods through the first Transformer encoder, and to capture the exposure delay temporal dependence features of each camera based on the high-dimensional temporal features by using a first multi-head self-attention mechanism; and to predict the exposure parameter compensation amount of each camera in future time periods based on the exposure delay temporal dependence features by the first Transformer decoder.

[0047] In one possible implementation, the RAW optimization unit 204 is configured to calculate the edge weights between nodes based on the edge features between nodes in the camera topology graph using a graph neural network, and calculate the RAW noise features and dynamic range metric of each node based on the node features of each node in the camera topology graph. For each node in the camera topology graph, the RAW noise features of the node are fused with the weighted RAW noise features of the node's neighboring nodes to obtain a cross-camera noise estimate. The dynamic range metric of the node is fused with the weighted dynamic range metric of the node's neighboring nodes to obtain a global dynamic range reference value. Based on the cross-camera noise estimate and the global dynamic range reference value, cross-camera noise reduction and dynamic range alignment processing are performed on the RAW data of the node to obtain optimized RAW data. The weighted RAW noise features are the weighted result of the RAW noise features and edge weights, and the weighted dynamic range metric is the weighted result of the dynamic range metric and edge weights.

[0048] In one possible implementation, the multi-camera image processing system 200 provided in this application embodiment further includes: The exposure adjustment unit 208 is used to adjust the exposure parameters of each camera based on the optimal exposure parameter set of each camera for backlit or overexposed scenes.

[0049] In one possible implementation, the Transformer dynamic parameter prediction model includes a second Transformer encoder and a second Transformer decoder; the parameter optimization unit 205 is used to extract features of the exposure parameter compensation amount of each camera in the future time period and the optimized RAW data in the current time period through the second Transformer encoder to obtain the compensation features and RAW features corresponding to each camera, and to capture the global dependency features between the compensation features and RAW features corresponding to each camera by using a second multi-head self-attention mechanism; and to predict the optimal ISP parameter combination of each camera in the current time period by the second Transformer decoder based on the global dependency features between the compensation features and RAW features corresponding to each camera.

[0050] It should be noted that the principle of the multi-camera image processing system 200 provided in the embodiments of this application to solve the technical problem is similar to that of the multi-camera image processing method provided in the embodiments of this application. Therefore, the implementation of the multi-camera image processing system 200 provided in the embodiments of this application can refer to the implementation of the multi-camera image processing method provided in the embodiments of this application, and the repeated parts will not be described again.

[0051] After introducing the multi-camera image processing method and system provided in the embodiments of this application, the electronic device provided in the embodiments of this application will be briefly introduced next.

[0052] The electronic device provided in this application embodiment may be, but is not limited to, a main control module, etc., see reference. Figure 3 As shown, the electronic device 300 provided in this application embodiment includes at least a processor 301, a memory 302, and a computer program stored in the memory 302 and executable on the processor 301. When the processor 301 executes the computer program, it implements the multi-camera image processing method provided in this application embodiment.

[0053] In one possible implementation, processor 301 can be a single processing element or a collective term for multiple processing elements. For example, processor 301 can be a central processing unit (CPU), or one or more integrated circuits configured to implement the multi-camera image processing method provided in the embodiments of this application. Specifically, processor 301 can be a general-purpose processor, including but not limited to CPUs, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0054] In one possible implementation, memory 302 may include a readable medium in the form of volatile memory, such as random access memory (RAM) 3021 and / or cache memory 3022, and may further include read-only memory (ROM) 3023; memory 302 may also include a program tool 3025 having a set (at least one) of program modules 3024, including but not limited to: operating subsystem, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0055] In one possible implementation, the electronic device 300 provided in this application embodiment may further include a bus 303 connecting different components (including processor 301 and memory 302). The bus 303 represents one or more types of bus structures, including memory bus, peripheral bus, local area bus, etc.

[0056] In one possible implementation, the electronic device 300 can also communicate with one or more devices that enable a user to interact with the electronic device 300 (e.g., mobile phones, computers, etc.), and / or with external devices 304 such as devices that enable the electronic device 300 to communicate with one or more other electronic devices 300 (e.g., routers, modems, etc.). This communication can be performed via an input / output (I / O) interface 305. Furthermore, the electronic device 300 can also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 306. Figure 3 As shown, network adapter 306 communicates with other modules of electronic device 300 via bus 303. It should be understood that, although... Figure 3 As not shown, other hardware and / or software modules may be used in conjunction with electronic device 300, including but not limited to microcode, device drivers, redundant processors, external disk drive arrays, Redundant Arrays of Independent Disks (RAID) subsystems, tape drives, and data backup storage subsystems.

[0057] It should be noted that, Figure 3 The electronic device 300 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0058] Furthermore, this application embodiment also provides a computer-readable storage medium storing computer instructions that, when executed by a processor, implement the multi-camera image processing method provided in this application embodiment. Specifically, the computer instructions may be built into or installed in a processor, so that the processor implements the multi-camera image processing method provided in this application embodiment by executing the built-in or installed computer instructions.

[0059] Of course, the multi-camera image processing method provided in the embodiments of this application can also be implemented as a program product, which includes program code. When the program code is executed by a processor, it implements the multi-camera image processing method provided in the embodiments of this application.

[0060] The program product provided in this application embodiment can be any combination of one or more readable media, wherein the readable media can be a readable signal medium or a readable storage medium, and the readable storage medium can be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination thereof. Specifically, more specific examples of readable storage media (a non-exhaustive list) include: electrical connections with one or more wires, portable disks, hard disks, RAM, ROM, erasable programmable read-only memory (EPROM), optical fibers, portable compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0061] The program product provided in this application embodiment can be a CD-ROM and include program code, and can also run on an electronic device such as a main control module. However, the program product provided in this application embodiment is not limited to this. In this application embodiment, the readable storage medium can be any tangible medium that contains or stores a program, which can be used by or in conjunction with an instruction execution system, device, or apparatus.

[0062] It should be noted that although several units or sub-units of the device have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of this application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided and embodied by multiple units.

[0063] Furthermore, although the operations of the method of this application are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0064] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0065] Obviously, those skilled in the art can make various modifications and variations to the embodiments of this application without departing from the spirit and scope of the embodiments of this application. Therefore, if these modifications and variations to the embodiments of this application fall within the scope of the claims of this application and their equivalents, this application also intends to include these modifications and variations.

Claims

1. A multi-camera image processing method, characterized in that, include: Acquire RAW data, exposure parameters, ambient light intensity, and exposure delay for each camera in the current time period, as well as exposure parameters, ambient light intensity, and exposure delay in historical time periods prior to the current time period; Based on the exposure parameters, ambient light intensity, and exposure delay of each camera in the current time period and the historical time period, the exposure parameter compensation amount of each camera in the future time period after the current time period is predicted by the Transformer dynamic exposure compensation model. A camera topology graph is constructed using each camera as a node; wherein, in the camera topology graph, the node features of each node include the spatial coordinates of the camera and the RAW data, exposure parameters and ambient light intensity of the camera in the current time period; the edge features between each node include the spatial distance between cameras, the percentage of overlapping viewpoints and the difference in optical distortion parameters; Based on the camera topology, the RAW data of each camera in the current time period is processed by a graph neural network to perform cross-camera noise reduction and dynamic range alignment to obtain the optimized RAW data of each camera in the current time period. Based on the exposure parameter compensation amount of each camera in the future time period and the optimized RAW data in the current time period, the optimal ISP parameter combination of each camera in the current time period is predicted by the Transformer dynamic parameter prediction model. The optimal ISP parameter combination and optimized RAW data of each camera in the current time period are respectively sent to the ISP module of each camera, so that the ISP module of each camera can process the optimized RAW data into vehicle vision images based on the optimal ISP parameter combination.

2. The multi-camera image processing method as described in claim 1, characterized in that, Before acquiring the RAW data, exposure parameters, ambient light intensity, and exposure delay of each camera within the current time period, as well as the exposure parameters, ambient light intensity, and exposure delay within historical time periods prior to the current time period, the process also includes: The system receives PPS-level pulses via the Global Positioning System (GPS), uses these PPS-level pulses as a global time reference, generates a clock synchronization signal, and distributes it to each camera via the GMSL3 link. This ensures that the exposure clocks of each camera are phase-locked with the global time reference, and embeds nanosecond-level timestamps in the header of the RAW data during synchronous acquisition.

3. The multi-camera image processing method as described in claim 2, characterized in that, Based on the exposure parameters, ambient light intensity, and exposure delay of each camera in the current time period and the historical time period, after predicting the exposure parameter compensation amount of each camera in future time periods after the current time period using the Transformer dynamic exposure compensation model, the method further includes: Based on the exposure parameter compensation amount of each camera in the future time period, the trigger phase of the clock synchronization signal in the future time period is adjusted.

4. The multi-camera image processing method as described in claim 1, characterized in that, The Transformer dynamic exposure compensation model includes a first Transformer encoder and a first Transformer decoder; based on the exposure parameters, ambient light intensity, and exposure delay of each camera in the current time period and the historical time period, the Transformer dynamic exposure compensation model predicts the exposure parameter compensation amount for each camera in future time periods after the current time period, including: The first Transformer encoder performs positional encoding on the exposure parameters, ambient light intensity, and exposure delay of each camera in the current time period and the historical time period to obtain high-dimensional temporal features. The first multi-head self-attention mechanism is then used to capture the exposure delay temporal dependency features of each camera based on the high-dimensional temporal features. The first Transformer decoder predicts the exposure parameter compensation amount for each camera in a future time period based on the exposure delay time-series dependency features.

5. The multi-camera image processing method as described in claim 1, characterized in that, Based on the camera topology, a graph neural network is used to perform cross-camera noise reduction and dynamic range alignment processing on the RAW data of each camera within the current time period to obtain optimized RAW data of each camera within the current time period, including: Using the graph neural network, edge weights between nodes are calculated based on edge features in the camera topology graph. Then, RAW noise features and dynamic range metrics for each node are calculated based on node features in the camera topology graph. For each node in the camera topology graph, the RAW noise features of that node are fused with the weighted RAW noise features of its neighboring nodes to obtain a cross-camera noise estimate. The dynamic range metric of that node is fused with the weighted dynamic range metric of its neighboring nodes to obtain a global dynamic range reference value. Based on the cross-camera noise estimate and the global dynamic range reference value, cross-camera noise reduction and dynamic range alignment are performed on the RAW data of that node to obtain optimized RAW data. Here, the weighted RAW noise features are the weighted result of RAW noise features and edge weights, and the weighted dynamic range metric is the weighted result of dynamic range metric and edge weights.

6. The multi-camera image processing method as described in claim 1, characterized in that, The graph neural network also outputs the optimal exposure parameter set for each camera; based on the camera topology, after performing cross-camera noise reduction and dynamic range alignment processing on the RAW data of each camera within the current time period using the graph neural network to obtain the optimized RAW data of each camera within the current time period, the process further includes: For backlit or overexposed scenes, the exposure parameters of each camera are adjusted based on the optimal exposure parameter set of each camera.

7. The multi-camera image processing method according to any one of claims 1-6, characterized in that, The Transformer dynamic parameter prediction model includes a second Transformer encoder and a second Transformer decoder; based on the exposure parameter compensation of each camera in the future time period and the optimized RAW data in the current time period, the Transformer dynamic parameter prediction model predicts the optimal ISP parameter combination of each camera in the current time period, including: The second Transformer encoder extracts the exposure parameter compensation amount of each camera in the future time period and the optimized RAW data in the current time period to obtain the compensation features and RAW features corresponding to each camera. The second multi-head self-attention mechanism is used to capture the global dependency features between the compensation features and RAW features corresponding to each camera. The second Transformer decoder predicts the optimal ISP parameter combination for each camera in the current time period based on the global dependency features between the compensation features and RAW features corresponding to each camera.

8. A multi-camera image processing system, characterized in that, include: The data acquisition unit is used to acquire RAW data, exposure parameters, ambient light intensity and exposure delay of each camera in the current time period, as well as exposure parameters, ambient light intensity and exposure delay in historical time periods before the current time period; The offset prediction unit is used to predict the amount of exposure parameter compensation for each camera in a future time period after the current time period based on the exposure parameters, ambient light intensity and exposure delay of each camera in the current time period and the historical time period, using the Transformer dynamic exposure compensation model. The topology graph construction unit is used to construct a camera topology graph with each camera as a node; wherein, in the camera topology graph, the node features of each node include the spatial position coordinates of the camera and the RAW data, exposure parameters and ambient light intensity of the camera in the current time period; the edge features between each node include the spatial position distance between cameras, the percentage of overlapping viewpoints and the difference in optical distortion parameters. The RAW optimization unit is used to perform cross-camera noise reduction and dynamic range alignment processing on the RAW data of each camera in the current time period based on the camera topology map and through a graph neural network to obtain the optimized RAW data of each camera in the current time period. The parameter optimization unit is used to predict the optimal ISP parameter combination of each camera in the current time period based on the exposure parameter compensation amount of each camera in the future time period and the optimized RAW data in the current time period, using the Transformer dynamic parameter prediction model. The image generation unit is used to send the optimal ISP parameter combination and optimized RAW data of each camera in the current time period to the ISP module of each camera, so that the ISP module of each camera can process the optimized RAW data into an in-vehicle visual image based on the optimal ISP parameter combination.

9. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the multi-camera image processing method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed by a processor, implement the multi-camera image processing method as described in any one of claims 1-7.

Images