Image data processing method, electronic device, and storage medium

Abstract

The application provides an image data processing method, an electronic device and a storage medium. The method comprises the following steps: determining the frame index of each original image to be read and the starting address of a target image segment to be read in each original image according to a data reading request; reading the image data of the target image segment in each original image from a DDR SDRAM according to the frame index of each original image and the starting address of the target image segment to be read in each original image, and respectively caching the image data in a buffer corresponding to each exposure degree in the FPGA; and when the data amount in the buffer corresponding to each exposure degree meets a preset fusion threshold, reading at least part of the image data in the cached target image segment from the buffer corresponding to each exposure degree to perform image fusion, and obtaining a current fusion image of a target scene. The scheme can realize the data acquisition and fusion processing of multi-exposure images by using limited FPGAs, improve the processing efficiency, and reduce the cost.

Description

Technical Field

[0001] This application relates to the field of computer technology, and more specifically, to an image data processing method, an electronic device, and a storage medium. Background Technology

[0002] In machine vision, especially high dynamic range imaging and 3D structured light measurement, systems need to acquire multiple images of the same scene with different exposure times and then fuse these images into a single, richly detailed final image. This process generates massive amounts of raw image data, posing significant challenges to data caching and real-time processing.

[0003] In existing technologies, all raw image data is first cached in the external high-capacity Double Data Rate Synchronous Dynamic Random-Access Memory (DDR SDRAM) of the Field-Programmable Gate Array (FPGA). Then, after all raw image data has been cached, all raw image data stored in the DDR SDRAM is read out at once and loaded into the FPGA for fusion calculation.

[0004] The limited storage resources inside the FPGA prevent it from accommodating multiple raw image data simultaneously, thus limiting processing efficiency. Summary of the Invention

[0005] The purpose of this application is to address the shortcomings of the prior art by providing an image data processing method, electronic device, and storage medium, so as to effectively utilize limited FPGA resources to realize the acquisition and fusion pipeline processing of multi-exposure sequence images through a segmented data reading mechanism, thereby reducing end-to-end latency and improving processing efficiency.

[0006] To achieve the above objectives, the technical solutions adopted in the embodiments of this application are as follows: In a first aspect, embodiments of this application provide an image data processing method, including: A data read request is obtained, which includes multiple sub-requests. Each sub-request is used to request the reading of image data of a target image fragment in the original image of the target scene at a certain exposure. Based on the data reading request, determine the frame index of each original image to be read and the starting address of the target image segment to be read in each original image; Based on the frame index of each original image and the starting address of the target image segment to be read in each original image, the image data of the target image segment in each original image is read from the external high-capacity double data rate synchronous dynamic random access memory DDR SDRAM, and cached into the buffer corresponding to each exposure inside the field programmable gate array FPGA respectively. When the amount of data in the buffer corresponding to each exposure level meets the preset fusion threshold, at least a portion of the image data in the cached target image fragments is read from the buffer corresponding to each exposure level, and image fusion is performed on at least a portion of the image data in the target image fragments of each original image to obtain the current fused image of the target scene.

[0007] Optionally, before reading the image data of the target image segment from each original image from an external high-capacity double data rate synchronous dynamic random access memory (DDR SDRAM), the method further includes: The image segment length is determined based on the buffer capacity of the FPGA's internal buffer, the bus burst length of the DDR SDRAM, and the upper limit of the data processing volume for fusion processing. Based on the image segment length and along the preset image direction, the original images of the target scene acquired in real time at various exposures are divided into image segments to obtain the segment index and segment data of each image segment of each original image. Assign address ranges and base addresses to each original image; Based on the base address of each original image, the preset address offset, the priority of each image segment in each original image, and the segment index of each image segment, the segment data of each image segment of each original image are sequentially written into the address range corresponding to each original image in the DDRSDRAM.

[0008] Optionally, the step of sequentially writing the fragment data of each image segment of each original image into the address range corresponding to each original image in the DDR SDRAM according to the base address of each original image, a preset address offset, and the priority of each image segment in each original image includes: The image segment to be written is determined based on the base address of the original image and the priority of each image segment in the original image; The starting address of the image segment to be written is determined based on the base address of the original image, the segment index of the current image segment to be written, the image segment length, and the width of the original image. Starting from the beginning address of the current image segment to be written, the data of each pixel in the current image segment to be written is sequentially and continuously written into the address range corresponding to the original image according to the preset address offset.

[0009] Optionally, determining the frame index of each original image to be read and the starting address of the target image segment to be read in each original image according to the data reading request includes: Determine the frame index of the original image to be read and the segment index of the target image segment to be read in the original image corresponding to each sub-request in the data reading request; Based on the frame index of the original image to be read, query and obtain the mapping relationship between the segment index of each image segment in the original image to be read and the starting address of each image segment; Based on the segment index of the target image segment to be read in the original image and the mapping relationship, the starting address of the target image segment to be read in the original image is determined.

[0010] Optionally, the step of reading image data of the target image segment from an external high-capacity double data rate synchronous dynamic random access memory (DDR SDRAM) based on the frame index of each original image and the starting address of the target image segment to be read in each original image includes: The data length of the target image segment is determined based on the image height and image width of the target image segment in the original image; The address range corresponding to the original image is determined based on the frame index of the original image; Starting from the starting address in the address range, read data of the specified length and use the read data as the image data of the target image segment in the original image.

[0011] Optionally, determining the data length of the target image segment based on its image height and image width in the original image includes: If the target image segment is the last image segment in the original image, then the image height of the target image segment is determined based on the height of the original image, the image segment length, and the total number of image segments corresponding to the original image; The data length of the target image segment is determined based on the image height and image width of the target image segment in the original image.

[0012] Optionally, the step of reading at least a portion of the image data from the currently cached target image segment from the buffer corresponding to each exposure when the amount of data in the buffer corresponding to each exposure meets the preset fusion threshold includes: When the amount of data in the buffer corresponding to each exposure reaches the amount of data corresponding to at least one row of pixels in the target image segment, at least a portion of the image data in the currently cached target image segment is synchronously read from the buffer corresponding to each exposure.

[0013] Optionally, it also includes: If the free capacity in the buffer corresponding to each exposure is greater than or equal to the complete data volume of an image segment, a new data read request is triggered. The new data read request is used to indicate the reading of image data of the next image segment of the target image segment in each original image of the target scene.

[0014] Secondly, embodiments of this application also provide an image data processing apparatus, including: an acquisition module, a determination module, a read cache module, and a fusion processing module; The acquisition module is used to acquire a data reading request, which includes multiple sub-requests. Each sub-request is used to request the reading of image data of a target image fragment in the original image of the target scene at a certain exposure. The determining module is used to determine the frame index of each original image to be read and the starting address of the target image segment to be read in each original image according to the data reading request. The read cache module is used to read the image data of the target image segment in each original image from the external high-capacity double data rate synchronous dynamic random access memory (DDR SDRAM) according to the frame index of each original image and the starting address of the target image segment to be read in each original image, and cache them respectively in the buffer corresponding to each exposure inside the field programmable gate array (FPGA). The fusion processing module is used to read at least a portion of the image data from the buffer corresponding to each exposure when the amount of data in the buffer corresponding to each exposure meets the preset fusion threshold, and to perform image fusion on at least a portion of the image data in the target image fragments of each original image to obtain the current fused image of the target scene.

[0015] Optionally, it may also include: a storage module; The storage module is used to determine the image segment length based on the cache capacity of the buffer inside the FPGA, the bus burst length of the DDR SDRAM, and the upper limit of the data processing volume of the fusion processing. Based on the image segment length and along the preset image direction, the original images of the target scene acquired in real time at various exposures are divided into image segments to obtain the segment index and segment data of each image segment of each original image. Assign address ranges and base addresses to each original image; Based on the base address of each original image, the preset address offset, the priority of each image segment in each original image, and the segment index of each image segment, the segment data of each image segment of each original image are sequentially written into the address range corresponding to each original image in the DDRSDRAM.

[0016] Optionally, the storage module is specifically used to determine the image segment to be written based on the base address of the original image and the priority of each image segment in the original image; The starting address of the image segment to be written is determined based on the base address of the original image, the segment index of the current image segment to be written, the image segment length, and the width of the original image. Starting from the beginning address of the current image segment to be written, the data of each pixel in the current image segment to be written is sequentially and continuously written into the address range corresponding to the original image according to the preset address offset.

[0017] Optionally, the determining module is specifically used to determine the frame index of the original image to be read corresponding to each sub-request in the data reading request and the segment index of the target image segment to be read in the original image; Based on the frame index of the original image to be read, query and obtain the mapping relationship between the segment index of each image segment in the original image to be read and the starting address of each image segment; Based on the segment index of the target image segment to be read in the original image and the mapping relationship, the starting address of the target image segment to be read in the original image is determined.

[0018] Optionally, the read cache module is specifically used to determine the data length of the target image segment based on the image height and image width of the target image segment in the original image; The address range corresponding to the original image is determined based on the frame index of the original image; Starting from the starting address in the address range, read data of the specified length and use the read data as the image data of the target image segment in the original image.

[0019] Optionally, the read cache module is specifically used to determine the image height of the target image segment based on the height of the original image, the image segment length, and the total number of image segments corresponding to the original image if the target image segment is the last image segment in the original image; The data length of the target image segment is determined based on the image height and image width of the target image segment in the original image.

[0020] Optionally, the fusion processing module is specifically used to synchronously read at least a portion of the image data in the currently cached target image segment from the buffer corresponding to each exposure when the amount of data in the buffer corresponding to each exposure reaches the amount of data corresponding to at least one row of pixels in the target image segment.

[0021] Optionally, the acquisition module is further configured to trigger a new data read request if the free capacity in the buffer corresponding to each exposure is greater than or equal to the complete data volume of an image segment. The new data read request is used to indicate the reading of image data of the next image segment of the target image segment in each original image of the target scene.

[0022] Thirdly, embodiments of this application provide an electronic device, including: a processor, a storage medium, and a bus. The storage medium stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the storage medium via the bus, and the processor executes the machine-readable instructions to implement the image data processing method provided in the first aspect.

[0023] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, performs the image data processing method as provided in the first aspect.

[0024] The beneficial effects of this application are: This application provides an image data processing method, electronic device, and storage medium, comprising: acquiring a data reading request, the data reading request including multiple sub-requests, each sub-request being used to request the reading of image data of a target image segment in an original image of a target scene at a certain exposure; determining, according to the data reading request, the frame index of each original image to be read and the starting address of the target image segment to be read in each original image; reading the image data of the target image segment in each original image from an external high-capacity double data rate synchronous dynamic random access memory (DDR SDRAM) according to the frame index of each original image and the starting address of the target image segment to be read in each original image, and buffering them respectively in a buffer corresponding to each exposure within a field-programmable gate array (FPGA); when the amount of data in the buffer corresponding to each exposure meets a preset fusion threshold, reading at least a portion of the image data in the buffer corresponding to each exposure from the buffer, and performing image fusion on the at least a portion of the image data in the target image segment of each original image to obtain a current fused image of the target scene. This method employs a segmented data reading mechanism, transforming large-scale complete data reading into continuous segmented data reading. Each data read retrieves only data from a specific image segment within each original image at each exposure level, enabling the reading and processing of multi-exposure images using limited FPGA resources. Specifically, a data read request quickly determines the index of each original image to be read and the starting address of the image segment within that original image. Based on this, a burst read request can be submitted to the DDR SDRAM, leveraging the high efficiency of burst reads from consecutive addresses within the DDR SDRAM to quickly retrieve the image data of the target image segment from each original image. Furthermore, by setting independent buffers for the exposure channels at each exposure level, out-of-order storage of image segment data from each original image can be avoided. Simultaneously, the storage of data read from the DDR SDRAM in the buffer and the fusion processing of data read from the buffer can be performed synchronously, improving processing efficiency.

[0025] Secondly, by decomposing each original image into image segments and writing them segment by segment, the foundation is laid for subsequent segment-by-segment reading. This eliminates the need for the system to read all the complete data of each original image into the FPGA for image fusion processing, making it possible to process ultra-high resolution, multi-exposure sequence images using the limited resources of the FPGA, which greatly reduces system cost and design difficulty.

[0026] In addition, the fusion processing of this solution does not require waiting for all data of the target image segments of each original image to be read before execution is triggered. This allows for highly parallel execution of DDR SDRAM access and back-end image fusion calculations, maximizing the utilization of memory bandwidth and computing units, achieving a processing speed close to the theoretical limit, and significantly reducing the end-to-end latency from data acquisition to result output.

[0027] By reading data blocks at the same location in different frames according to the fragment index, the access mode has good spatial locality, which is conducive to making full use of the burst transmission characteristics of DDR SDRAM, reducing the addressing overhead of the memory controller, and thus improving the overall memory access efficiency.

[0028] Finally, this solution architecture is easily expandable. Simply increasing the DDR SDRAM capacity and adjusting the number of cache areas corresponding to the exposure level can support data processing under more exposure levels. Simply adjusting the data size of image segments and the row cache depth for image fusion processing can support higher resolution images, making the system highly adaptable. Attached Figure Description

[0029] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0030] Figure 1 This is a schematic diagram of the architecture of an image data processing system provided in an embodiment of this application; Figure 2 A flowchart illustrating the image data processing method provided in the embodiments of this application. Figure 1 ; Figure 3 A flowchart illustrating the image data processing method provided in the embodiments of this application. Figure 2 ; Figure 4 A schematic diagram illustrating the segmentation of an original image as provided in an embodiment of this application; Figure 5 A flowchart illustrating the image data processing method provided in the embodiments of this application. Figure 3 ; Figure 6 This is a schematic diagram of data storage in DDR SDRAM provided in an embodiment of this application; Figure 7 A flowchart illustrating the image data processing method provided in the embodiments of this application. Figure 4 ; Figure 8A flowchart illustrating the image data processing method provided in the embodiments of this application. Figure 5 ; Figure 9 A flowchart illustrating the image data processing method provided in the embodiments of this application. Figure 6 ; Figure 10 This is a schematic diagram of an image data processing apparatus provided in an embodiment of this application; Figure 11 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the accompanying drawings in this application are for illustrative and descriptive purposes only and are not intended to limit the scope of protection of this application. Furthermore, it should be understood that the schematic drawings are not drawn to scale. The flowcharts used in this application illustrate operations implemented according to some embodiments of this application. It should be understood that the operations in the flowcharts may not be implemented in sequence, and steps without logical contextual relationships may be reversed or implemented simultaneously. In addition, those skilled in the art, guided by the content of this application, may add one or more other operations to the flowcharts, or remove one or more operations from the flowcharts.

[0032] Furthermore, the described embodiments are merely some, not all, of the embodiments of this application. The components of the embodiments of this application described and illustrated herein can typically be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0033] It should be noted that the term "comprising" will be used in the embodiments of this application to indicate the presence of the features declared thereafter, but does not exclude the addition of other features.

[0034] In machine vision, particularly high dynamic range (HDR) imaging and 3D structured light measurement, systems need to acquire multiple images of the same scene with different exposure times and then fuse these images into a single, richly detailed final image. This process generates massive amounts of raw image data. For example, a system with 8 exposures, 10 frames per second, and 5 megapixels generates approximately 4GB (8 5M 10 (8 bytes), which poses a huge challenge to data caching and real-time processing.

[0035] Existing technologies typically process this data in two ways: 1. All data is transmitted to the host computer for processing: All raw image data is transmitted to the host computer (industrial control computer) via a high-speed interface (such as Camera Link or CoaXPress), where it is fused by the Central Processing Unit (CPU) and software. This method consumes a large amount of transmission bandwidth and places a heavy burden on the host computer's CPU, resulting in low system frame rate, high latency, and difficulty in achieving real-time processing.

[0036] 2. Simple Field-Programmable Gate Array (FPGA) Buffering and Processing: Multiple images are first stored in a large-capacity Double Data Rate Synchronous Dynamic Random-Access Memory (DDR SDRAM) outside the FPGA, and then fused internally within the FPGA. However, the traditional approach is to read all the original images stored in the DDR SDRAM at once and load them into the FPGA's internal Block RAM (BRAM) at the start of fusion, and then perform the fusion calculation. This method has a fatal flaw: the BRAM resources inside the FPGA are very limited and expensive, and cannot possibly accommodate multiple high-resolution images simultaneously (for example, three 5-megapixel 8-bit images require approximately 120Mb of BRAM, which far exceeds the available capacity of most FPGAs).

[0037] Therefore, how to achieve caching and efficient processing of massive multi-exposure image data without relying on huge on-chip caches has become a key bottleneck restricting system performance and integration.

[0038] The image data processing method provided in this application segments the original images acquired at various exposure levels in a uniform manner. An address region is allocated in DDR SDRAM for each original image acquired at each exposure level. The pixel information of each original image is then sequentially stored in the corresponding address region according to the priority of the segmented image fragments and a preset address offset value. Based on this, during image fusion processing, a burst read request can be submitted to read identical image fragments from the DDR SDRAM segment by segment, and image fusion processing can be performed segment by segment. This eliminates the need to cache each original image in the FPGA's internal BRAM before starting fusion processing, effectively utilizing the limited resources of the FPGA to achieve multi-image fusion processing at multiple exposure levels, reducing processing latency, and improving processing efficiency.

[0039] Figure 1 This is a schematic diagram of the architecture of an image data processing system provided in an embodiment of this application, such as... Figure 1 As shown, the system may include: an image sensor, an optomechanical system, an FPGA chip, external DDR SDRAM, a host computer, and a processing device. The host computer and the processing device can be the same device or two separate devices. The host computer is primarily responsible for sending control commands to the FPGA to control image acquisition; the processing device is primarily responsible for further back-end processing of the fused image transmitted from the FPGA.

[0040] The FPGA serves as the control center of the entire system, primarily integrating a collaborative control module, an image acquisition module, a segmented control management module, and an image fusion processing module. The collaborative control module controls the optical engine to project light in a specific pattern onto the object being photographed. The image sensor acquires an image of the object with the projected pattern. During image acquisition, the collaborative control module simultaneously controls the image sensor's exposure to ensure that it acquires raw images at different exposure levels.

[0041] The image acquisition module is responsible for receiving raw images at various exposure levels sent by the image sensor and sending each raw image to the segmented control and management module. The segmented control and management module stores each image segment of each raw image sequentially into the DDR SDRAM according to the image segmented storage method proposed in this scheme. The segmented control and management module is also responsible for submitting burst read requests to the DDR SDRAM to read out the image segments of each raw image segment by segment. The data of the same image segment of each raw image requested by each burst read request is finally extracted by the image fusion processing module and image fusion is performed to obtain a fused image. After all the raw images at various exposure levels in the required scene are fused and processed, a detailed final image of the required scene is obtained. The image fusion processing module sends the final image to the processing device for subsequent back-end processing.

[0042] The core of this solution lies in the various data interaction operations between the FPGA and the DDR SDRAM.

[0043] Figure 2 A flowchart illustrating the image data processing method provided in the embodiments of this application. Figure 1 The execution entity of this method is the FPGA in the aforementioned system, such as... Figure 2 As shown, the method includes: S101. Obtain a data read request. The data read request includes multiple sub-requests. Each sub-request is used to request the read of image data of the target image fragment in the original image of the target scene at a certain exposure.

[0044] The FPGA can maintain a local scheduling queue, which records the current data read requests. After the data read by the current data read request is processed, the next data read request will be written to the tail of the queue to keep the queue saturated and ensure that new data read requests can be continuously obtained.

[0045] A data read request can include multiple sub-requests, each of which requests to read image data at a certain exposure level; the target scene can refer to a static scene, which is the scene you want to photograph, such as a building.

[0046] In this embodiment, each sub-request requests image data of a specified image segment from original images taken at different exposures of the same target scene.

[0047] For example: Original image 1 at exposure 1, original image 2 at exposure 2, and original image 3 at exposure 3 are stored in advance. These three original images were all taken at different exposures of the same target scene. Original image 1, original image 2, and original image 3 are divided into the same multiple image segments. Image segment 1 in original image 1 is exactly the same as the image segment 1 in original image 2 and the image segment 1 in original image 3.

[0048] Data read requests may include: Sub-request 1 - read image fragment 1 of original image 1, Sub-request 2 - read image fragment 1 of original image 2, Sub-request 3 - read image fragment 1 of original image 3.

[0049] S102. Based on the data reading request, determine the frame index of each original image to be read and the starting address of the target image segment to be read in each original image.

[0050] Each exposure channel corresponds to a unique index n (n=0,1…N-1), thus the frame index of the original image at each exposure level can be determined. In other words, the index of the exposure channel is used as the frame index of the original image at the exposure level corresponding to the exposure channel. For example, if there are 8 exposure channels, each exposure channel corresponds to an exposure level, and the original image at the corresponding exposure level is stored under each exposure channel, and the indexes of the exposure channels are 0-7, then the frame index of the original image at each exposure level is 0-7.

[0051] The same target image segments in each original image point to the same segment. However, since the storage addresses of original images at different exposures are different during data storage, the starting addresses of the target image segments to be read in each original image are different, and it is necessary to calculate them based on the base address of each original image.

[0052] S103. Based on the frame index of each original image and the starting address of the target image segment to be read in each original image, read the image data of the target image segment in each original image from the external high-capacity double data rate synchronous dynamic random access memory (DDR SDRAM), and cache them into the buffer corresponding to each exposure inside the field programmable gate array (FPGA).

[0053] Once the frame indices of each original image are known, it can be determined which original images need to be read in this data read operation. The starting address of the target image segment to be read within each original image is also known, thus determining from which address to begin reading the data sequentially to retrieve the target image segment's data. Based on this information, a burst read request can be sent to the DDR SDRAM to leverage the high efficiency of burst reads for consecutive addresses within the DDR SDRAM, quickly retrieving the image data of the target image segment from each original image from the DDR SDRAM.

[0054] Since data from multiple raw images are read simultaneously, the read latency of different channels may vary, potentially causing the image data of the target image segments returned by the DDR SDRAM to be out of order. In this embodiment, an independent First-In-First-Out (FIFO) buffer can be set up inside the FPGA for each exposure channel to store the image data of the raw images at each exposure level read from the DDR SDRAM.

[0055] This embodiment reads data in segments. Therefore, for each data read request, the image data of the target image segment in the original image at each exposure level is cached in the corresponding FIFO buffer.

[0056] S104. When the amount of data in the buffer corresponding to each exposure meets the preset fusion threshold, at least part of the image data in the cached target image fragment is read from the buffer corresponding to each exposure, and image fusion is performed on at least part of the image data in the target image fragment of each original image to obtain the current fused image of the target scene.

[0057] Optionally, when the amount of data in the buffer corresponding to each exposure is sufficient for the amount of data to be fused, that is, when the basic amount of data to be fused is reached, at least a portion of the image data in the cached target image fragment can be read from the buffer corresponding to each exposure. The reason for reading at least a portion of the image data in the target image fragment is that reading data out of the buffer for fusion and writing data into the buffer are performed simultaneously. When reading, the buffer may not have completed caching of all the data of the target image fragment of the original image, so the currently stored part of the image data is read first for fusion processing.

[0058] For example, if the third row of pixel data from the target image segment of the original image is currently being written to the buffer, then when reading, only the first and second rows of pixel data from the target image segment of the original image can be read.

[0059] With this setting, it is not necessary to wait for all the data of the target image segment of the original image to be written before performing data reading and fusion operations. At the same time as writing the data of the target image segment of the original image to the buffer, the stored data is pulled from the buffer for synchronous fusion processing, which improves processing efficiency.

[0060] In summary, the image data processing method provided in this embodiment includes: obtaining a data reading request, which includes multiple sub-requests, each sub-request being used to request the reading of image data of a target image segment in an original image of a target scene at a certain exposure; determining the frame index of each original image to be read and the starting address of the target image segment to be read in each original image according to the data reading request; reading the image data of the target image segment in each original image from an external high-capacity double data rate synchronous dynamic random access memory (DDRSDRAM) according to the frame index of each original image and the starting address of the target image segment to be read in each original image, and caching them respectively in the buffers corresponding to each exposure within the field-programmable gate array (FPGA); when the amount of data in the buffers corresponding to each exposure meets a preset fusion threshold, reading at least a portion of the image data in the cached target image segment from the buffers corresponding to each exposure, and performing image fusion on the at least a portion of the image data in the target image segment of each original image to obtain the current fused image of the target scene. This method employs a segmented data reading mechanism, transforming large-scale complete data reading into continuous segmented data reading. Each data read retrieves only data from a specific image segment within each original image at each exposure level, enabling the reading and processing of multi-exposure images using limited FPGA resources. Specifically, a data read request quickly determines the index of each original image to be read and the starting address of the image segment within that original image. Based on this, a burst read request can be submitted to the DDR SDRAM, leveraging the high efficiency of burst reads from consecutive addresses within the DDR SDRAM to quickly retrieve the image data of the target image segment from each original image. Furthermore, by setting independent buffers for the exposure channels at each exposure level, out-of-order storage of image segment data from each original image can be avoided. Simultaneously, the storage of data read from the DDR SDRAM in the buffer and the fusion processing of data read from the buffer can be performed synchronously, improving processing efficiency.

[0061] Figure 3 A flowchart illustrating the image data processing method provided in the embodiments of this application. Figure 2 Optionally, in step S103, before reading the image data of the target image segment from each original image from the external high-capacity double data rate synchronous dynamic random access memory (DDR SDRAM), the method further includes: S201. Determine the image segment length based on the buffer capacity of the FPGA's internal buffer, the bus burst length of the DDR SDRAM, and the upper limit of the data processing volume for fusion processing.

[0062] This embodiment illustrates how to write the acquired raw images into DDR SDRAM in segments.

[0063] To achieve the aforementioned efficient segmented reading, the acquired raw images must be arranged in a regular manner when writing them into the DDR SDRAM.

[0064] The cache capacity of the FIFO in the FPGA, the bus burst length of the DDR SDRAM (e.g., 128 or 256), and the fusion throughput of the image fusion processing module (i.e., the upper limit of the amount of data that can be processed in one fusion process) can be read from the configuration register. The optimal image segment length S can be adaptively calculated by combining the above parameters.

[0065] S202. Based on the image segment length and along the preset image direction, divide the original images of the target scene acquired in real time at various exposure levels into image segments to obtain the segment index and segment data of each image segment of each original image.

[0066] In this embodiment, in order to ensure the address continuity and readability of each image segment after segmentation, each original image can be divided into multiple image segments along the vertical direction of the image. The total number of image segments obtained by segmentation is K = ceil(H / S), where H represents the height of the original image.

[0067] The actual height of all image segments except the last one is S. The height of the last image segment may not reach S. The actual height of the last image segment can be calculated.

[0068] Figure 4 This is a schematic diagram illustrating the segmentation of an original image according to an embodiment of this application. The original image at each exposure level is segmented using the same method, so the same image segment in each original image corresponds to the same region of the image.

[0069] Of course, in practical applications, the image can also be segmented along the horizontal direction, and the subsequent address allocation can be adjusted accordingly.

[0070] Based on the segmentation results, the index of each image segment in the original image can be generated, and the image data contained in each image segment, i.e., segment data, can be determined.

[0071] In this system, identical image segments in different original images have the same index, which facilitates the simultaneous reading of data from the same image segments in different original images.

[0072] S203. Assign address ranges and base addresses to each original image.

[0073] In DDR SDRAM, an address range is allocated for each raw image. This address range can be contiguous. A base address is also allocated for each raw image, which can refer to the starting address where the raw image is stored. Assuming the raw image is stored line by line starting from the top left corner, the base address can refer to the address of the top-left pixel of the raw image.

[0074] S204. Based on the base address of each original image, the preset address offset, the priority of each image segment in each original image, and the segment index of each image segment, the segment data of each image segment of each original image are sequentially written into the address range corresponding to each original image in the DDR SDRAM.

[0075] Each original image has a different base address, but the calculation rule for the address offset within the address range of each original image can be the same, thus ensuring that address offset within the address range of different original images can find the data of the same pixel.

[0076] In some embodiments, the address offset in the address range of each original image can be set to the same value, for example, set to 1, so that each address stores the data of one pixel.

[0077] When the base address of the original image is known, the data of each pixel in the original image can be written into the address range sequentially according to the address offset.

[0078] Optionally, when writing data, the data can be stored according to the priority of each image segment. Assuming the original image is divided into K-1 image segments, starting from the base address, all pixels of the 0th image segment are stored continuously (in row-major order, i.e., all pixels of the first row, then all pixels of the second row, and so on), followed by all pixels of the 1st segment, and so on, until the K-1th segment.

[0079] Figure 5 A flowchart illustrating the image data processing method provided in the embodiments of this application. Figure 3 Optionally, in step S204, according to the base address of each original image, the preset address offset, and the priority of each image segment in each original image, the segment data of each image segment of each original image are sequentially written into the address range corresponding to each original image in the DDR SDRAM, including: S301. Determine the image segment to be written based on the base address of the original image and the priority of each image segment in the original image.

[0080] Continuing with the example above, initially, the image fragment to be written is fragment 0. After the data of image fragment 0 is written, image fragment 1 is taken as the new image fragment to be written, and so on, until image fragment K-1.

[0081] S302. Determine the starting address of the image segment to be written based on the base address of the original image, the segment index of the current image segment to be written, the image segment length, and the width of the original image.

[0082] Let the base address of the original image n (frame index n, i.e. the nth exposure channel) be BaseAddr_n, the image segment length be S, and the width of the original image be W (number of pixels). Then the starting address of the kth image segment in the original image n is: Addr(n, k) = BaseAddr_n + k × (S × W), where k∈[0, K-1].

[0083] S303. Starting from the starting address of the current image segment to be written, the data of each pixel in the current image segment to be written is sequentially and continuously written into the address range corresponding to the original image according to the preset address offset.

[0084] The starting address of any image segment to be written can be calculated using the above method. When the starting address of the image segment to be written is known, the data of each pixel in the image segment to be written can be written sequentially line by line into the address range corresponding to the original image, starting from the starting address and in row-major order, with a preset address offset.

[0085] In this embodiment, by decomposing each original image into image segments and writing them segment by segment, the foundation for subsequent segment-by-segment reading is laid. This eliminates the need for the system to read all the complete data of each original image into the FPGA for image fusion processing, making it possible to process ultra-high resolution, multi-exposure sequence images using the limited resources of the FPGA, which greatly reduces system cost and design difficulty.

[0086] Figure 6 This is a schematic diagram of data storage in DDR SDRAM provided in an embodiment of this application. Figure 6The data storage of the original images at three exposure levels is only schematically shown. Frame 0 corresponds to the original image 1 under short exposure, frame 1 corresponds to the original image 2 under medium exposure, and frame 2 corresponds to the original image 3 under long exposure. The base address of the original image 1 is BaseAddr[0], the base address of the original image 2 is BaseAddr[1], and the base address of the original image 3 is BaseAddr[2]. Each original image is divided into the same number of image segments. The index of the image segments ranges from 0 to K-1. The same image segments in different original images have the same index. For example, the segment index of segment 0 in original image 1, segment 0 in original image 2, and segment 0 in original image 3 are all k=0, and these three segments all correspond to the same region in the original image. The starting address of segment 0 in original image 1 is SegAddr[0][0], the starting address of segment 1 is SegAddr[0][1], etc.

[0087] Figure 7 A flowchart illustrating the image data processing method provided in the embodiments of this application. Figure 4 Optionally, in step S102, based on the data reading request, the frame index of each original image to be read and the starting address of the target image segment to be read in each original image are determined, including: S401. Determine the frame index of the original image to be read and the segment index of the target image segment to be read in the original image corresponding to each sub-request in the data reading request.

[0088] The data read requests currently recorded in the scheduling queue are updated in real time. After the current data read request is completed, the next data read request is written to the tail of the queue. The data read request may contain the frame index of the original image, the segment index k of the target image segment to be read, and information such as the target buffer number (Ping or Pong). The target buffer refers to DDR SDRAM, which is typically divided into two large areas: a Ping area and a Pong area. These two areas store data alternately. The target buffer number in the data read request indicates which area of ​​the DDR SDRAM the current data read request should read from.

[0089] Each exposure corresponds to a sub-request for the original image, so a sub-request can include: the frame index of an original image and the fragment index of the target image fragment.

[0090] During initialization, the fragment index k=0. Assuming the exposure is n (n=0,1,…,N-1), the initial data reading requests can be: sub-request 1 (frame index 0, fragment index 0), sub-request 2 (frame index 1, fragment index 0), sub-request 3 (frame index 3, fragment index 0)…. sub-request n (frame index N-1, fragment index 0).

[0091] After the image fusion processing module finishes processing the image fusion of image segment 0, the next data read request is appended to the tail of the scheduling queue. Thus, the next data read request can be determined as: sub-request 1 (frame index 1, segment index 1), sub-request 2 (frame index 1, segment index 1), sub-request 3 (frame index 3, segment index 1)... sub-request n (frame index N-1, segment index 1).

[0092] Therefore, based on the current data reading request, the frame index of each original image to be read and the segment index of the target image segment to be read can be determined.

[0093] S402. Based on the frame index of the original image to be read, query and obtain the mapping relationship between the segment index of each image segment in the original image to be read and the starting address of each image segment.

[0094] During the process of writing the data of the original image into the DDR SDRAM, the mapping relationship between the segment index of each image segment of the original image and the starting address of each image segment is recorded synchronously. This allows the corresponding original image to be quickly queried based on the frame index of the original image to be read, and the mapping relationship between the segment index of each image segment and the starting address of each image segment in the original image to be obtained.

[0095] S403. Determine the starting address of the target image segment to be read in the original image based on the segment index and mapping relationship of the target image segment to be read in the original image.

[0096] The target image segment can be determined by the segment index of the target image segment to be read, and the mapping relationship records the starting address corresponding to different image segments. Therefore, the starting address of the target image segment can be extracted according to the mapping relationship.

[0097] It is worth noting that each original image maintains a separate mapping relationship, thereby ensuring that the starting address of the target image fragment in the corresponding original image can be accurately queried.

[0098] Figure 8 A flowchart illustrating the image data processing method provided in the embodiments of this application. Figure 5Optionally, in step S103, image data of the target image segment in each original image is read from an external high-capacity double data rate synchronous dynamic random access memory (DDR SDRAM) according to the frame index of each original image and the starting address of the target image segment to be read in each original image, including: S501. Determine the data length of the target image segment based on the image height and image width of the target image segment in the original image.

[0099] In some embodiments, the data length of the target image segment, i.e. the total number of pixels contained in the target image segment, can be calculated based on the actual height and width of the target image segment. Of course, if necessary, the total number of bytes contained in the target image segment can also be calculated based on the total number of pixels and the number of bytes occupied by each pixel.

[0100] Specifically, when writing data to or reading data from DDR SDRAM according to image segments, the data length of the image segment can be used as the transmission length to guide whether the data reading or writing is complete.

[0101] S502. Determine the address range corresponding to the original image based on the frame index of the original image.

[0102] After the data is written to the DDR SDRAM, a mapping relationship between the frame index of the original image and the corresponding address range can be established synchronously. Based on this mapping relationship, the address range corresponding to the original image can be quickly determined.

[0103] S503. Starting from the starting address in the address range, read data of the specified length and use the read data as the image data of the target image segment in the original image.

[0104] When reading data, the starting address of the target image segment in the address range corresponding to the original image can be used as the starting address. Data that meets the data length of the target image segment can be read and used as the image data of the target image segment in the original image.

[0105] Assuming the data length of the target image segment is 100, data is read continuously starting from the beginning address of the target image segment, and reading stops when the length of the read data reaches 100.

[0106] Figure 9 A flowchart illustrating the image data processing method provided in the embodiments of this application. Figure 6 Optionally, in step S501, determining the data length of the target image segment based on its image height and width in the original image includes: S601. If the target image segment is the last image segment in the original image, then the image height of the target image segment is determined based on the height of the original image, the image segment length, and the total number of image segments corresponding to the original image.

[0107] When the target image segment is the last image segment in the original image, the height of the target image segment may be less than the image segment length S. In this case, the height of the target image segment cannot be divided evenly, so the actual height of the target image segment is not S. The actual height of the target image segment can be calculated as follows: S_last = H - (K-1) × S, where H is the height of the original image, K is the total number of image segments, K = ceil(H / S), and S is the image segment length.

[0108] The actual height of all image segments except the last one is S.

[0109] Of course, in some cases, the height of the last image segment may be exactly equal to S, in which case the actual height of all image segments is S.

[0110] S602. Determine the data length of the target image segment based on the image height and image width of the target image segment in the original image.

[0111] The data length of the target image fragment can be obtained by multiplying its actual image height and actual image width (i.e., the width of the original image).

[0112] Optionally, in step S104, when the amount of data in the buffer corresponding to each exposure meets the preset fusion threshold, at least part of the image data in the currently cached target image segment is read from the buffer corresponding to each exposure, including: when the amount of data in the buffer corresponding to each exposure reaches the amount of data corresponding to at least one row of pixels in the target image segment, at least part of the image data in the currently cached target image segment is read synchronously from the buffer corresponding to each exposure.

[0113] Typically, image fusion processing is performed pixel by pixel. For example, pixel 1 in the original image 1 is fused with the corresponding pixel 1 in the original image 2 to obtain the fused result of pixel 1.

[0114] Based on this, the fusion process of this solution does not need to wait for all the data of the target image fragments of the original images to be fully cached in each buffer before it is triggered. During the data reading process, the data status in each buffer can be continuously monitored. When the amount of cached data in each buffer reaches the amount of data corresponding to at least one row of pixels in the target image fragment, it is considered that the data of the target image fragment is ready and fusion can begin.

[0115] At this point, the cached data can be read synchronously from each buffer, ensuring that data from different exposures and the same pixel coordinates are sent to the image fusion processing module in the same clock cycle, thus guaranteeing the accuracy of the image fusion processing results.

[0116] This solution can be implemented in this way because reading data from DDR SDRAM is faster than the back-end fusion processing speed, ensuring that there is always data to be read in the buffer and preventing the fusion processing from stalling.

[0117] Of course, in practical applications, the preset fusion threshold does not necessarily have to be set to at least one row of pixels. It can also be 1 / 2 row, or 2 rows, etc., as long as the buffer is not empty during the synchronous execution of data writing and reading, so as to ensure the continuous and uninterrupted execution of data reading and data fusion and improve data processing efficiency.

[0118] Optionally, the method further includes: if the free capacity in the buffer corresponding to each exposure is greater than or equal to the complete data volume of an image segment, a new data read request is triggered, the new data read request being used to indicate the reading of image data of the next image segment of the target image segment in each original image of the target scene.

[0119] In some embodiments, in order to maximize throughput, this scheme adopts a double-buffered pipeline management combined with a pre-read mechanism to realize inter-segment pipeline. When the current image segment begins to merge, if the free capacity in the buffer corresponding to each exposure is greater than or equal to the data size of a complete image segment, the data reading request for the next image segment is immediately initiated. This means that the data reading of the next image segment and the image fusion calculation of the current image segment completely overlap in time.

[0120] This embodiment enables highly parallel execution of DDR SDRAM access and back-end image fusion calculations, maximizing the utilization of memory bandwidth and computing units, achieving a processing speed close to the theoretical limit, and significantly reducing end-to-end latency from data acquisition to result output.

[0121] In some embodiments, when the image fusion processing module finishes processing the last pixel of the current image segment, it generates a segment end marker. At this time, the segment index of the current image segment is incremented by 1, and the next data read request is regenerated and added to the tail of the scheduling queue.

[0122] If the fragment index of the current image fragment is less than K-1, it is considered not to be the last image fragment, and the next image fragment is processed according to the above method. If the fragment index of the current image fragment is equal to K-1, it means that all image fragments of each original image have been processed and the final fused image of the target scene has been obtained. Then, wait for the acquisition of the original images of each exposure of the next new target scene (i.e., the new optical-mechanical pattern) to be completed or wait for external instructions.

[0123] In scenarios where the system continuously runs multiple optomechanical patterns, the segmented reading process and synchronous acquisition process of this solution form a deeper pipeline. When the segmented control and management module is processing the multi-exposure image of the j-th group of optomechanical patterns, the cooperative control module can acquire the multi-exposure image of the (j+1)-th group of optomechanical patterns in parallel and write it to another address area of ​​the DDR SDRAM (for example, if the multi-exposure image of the j-th group of optomechanical patterns is written to the ping area, then the multi-exposure image of the (j+1)-th group of optomechanical patterns is written to the pong area). The two sets of operations use different DDR SDRAM address spaces and do not interfere with each other. This design enables data acquisition, DDR SDRAM writing, segmented reading, and fusion to form a three-stage pipeline.

[0124] In summary, the image data processing method provided in this embodiment includes: obtaining a data reading request, which includes multiple sub-requests, each sub-request being used to request the reading of image data of a target image segment in an original image of a target scene at a certain exposure; determining the frame index of each original image to be read and the starting address of the target image segment to be read in each original image according to the data reading request; reading the image data of the target image segment in each original image from an external high-capacity double data rate synchronous dynamic random access memory (DDRSDRAM) according to the frame index of each original image and the starting address of the target image segment to be read in each original image, and caching them respectively in the buffers corresponding to each exposure within the field-programmable gate array (FPGA); when the amount of data in the buffers corresponding to each exposure meets a preset fusion threshold, reading at least a portion of the image data in the cached target image segment from the buffers corresponding to each exposure, and performing image fusion on the at least a portion of the image data in the target image segment of each original image to obtain the current fused image of the target scene. This method employs a segmented data reading mechanism, transforming large-scale complete data reading into continuous segmented data reading. Each data read retrieves only data from a specific image segment within each original image at each exposure level, enabling the reading and processing of multi-exposure images using limited FPGA resources. Specifically, a data read request quickly determines the index of each original image to be read and the starting address of the image segment within that original image. Based on this, a burst read request can be submitted to the DDR SDRAM, leveraging the high efficiency of burst reads from consecutive addresses within the DDR SDRAM to quickly retrieve the image data of the target image segment from each original image. Furthermore, by setting independent buffers for the exposure channels at each exposure level, out-of-order storage of image segment data from each original image can be avoided. Simultaneously, the storage of data read from the DDR SDRAM in the buffer and the fusion processing of data read from the buffer can be performed synchronously, improving processing efficiency.

[0125] Secondly, by decomposing each original image into image segments and writing them segment by segment, the foundation is laid for subsequent segment-by-segment reading. This eliminates the need for the system to read all the complete data of each original image into the FPGA for image fusion processing, making it possible to process ultra-high resolution, multi-exposure sequence images using the limited resources of the FPGA, which greatly reduces system cost and design difficulty.

[0126] In addition, the fusion processing of this solution does not require waiting for all data of the target image segments of each original image to be read before execution is triggered. This allows for highly parallel execution of DDR SDRAM access and back-end image fusion calculations, maximizing the utilization of memory bandwidth and computing units, achieving a processing speed close to the theoretical limit, and significantly reducing the end-to-end latency from data acquisition to result output.

[0127] By reading data blocks at the same location in different frames according to the fragment index, the access mode has good spatial locality, which is conducive to making full use of the burst transmission characteristics of DDR SDRAM, reducing the addressing overhead of the memory controller, and thus improving the overall memory access efficiency.

[0128] Finally, this solution architecture is easily expandable. Simply increasing the DDR SDRAM capacity and adjusting the number of cache areas corresponding to the exposure level can support data processing under more exposure levels. Simply adjusting the data size of image segments and the row cache depth for image fusion processing can support higher resolution images, making the system highly adaptable.

[0129] The following describes the apparatus, device, and storage medium used to execute the image data processing method provided in this application. The specific implementation process and technical effects are described above and will not be repeated below.

[0130] Figure 10 This is a schematic diagram of an image data processing apparatus provided in an embodiment of this application. The functions implemented by this image data processing apparatus correspond to the steps performed by the method described above. This apparatus can be understood as the aforementioned server, or a server's processor, or as a component that implements the functions of this application under the control of the server, independent of the aforementioned server or processor. Figure 10 As shown, the device includes: an acquisition module 100, a determination module 200, a read cache module 300, and a fusion processing module 400; The acquisition module 100 is used to acquire data reading requests. The data reading request includes multiple sub-requests, each of which is used to request the reading of image data of a target image fragment in the original image of the target scene at a certain exposure. The determination module 200 is used to determine the frame index of each original image to be read and the starting address of the target image segment to be read in each original image according to the data reading request. The read cache module 300 is used to read the image data of the target image segment in each original image from the external large-capacity double data rate synchronous dynamic random access memory DDR SDRAM according to the frame index of each original image and the starting address of the target image segment to be read in each original image, and cache them respectively into the buffer corresponding to each exposure inside the field programmable gate array FPGA. The fusion processing module 400 is used to read at least a portion of the image data in the cached target image fragments from the buffers corresponding to each exposure when the amount of data in the buffers corresponding to each exposure meets the preset fusion threshold, and to perform image fusion on at least a portion of the image data in the target image fragments of each original image to obtain the current fused image of the target scene.

[0131] Optionally, it may also include: a storage module; The storage module is used to determine the image segment length based on the buffer capacity of the FPGA's internal buffer, the bus burst length of the DDR SDRAM, and the upper limit of the data processing volume of the fusion process. Based on the image segment length and along the preset image direction, the original images of the target scene acquired in real time at various exposures are divided into image segments to obtain the segment index and segment data of each image segment of each original image. Assign address ranges and base addresses to each original image; Based on the base address of each original image, the preset address offset, the priority of each image segment in each original image, and the segment index of each image segment, the segment data of each image segment of each original image are sequentially written into the address range corresponding to each original image in the DDR SDRAM.

[0132] Optionally, the storage module is specifically used to determine the image segment to be written based on the base address of the original image and the priority of each image segment in the original image; The starting address of the image segment to be written is determined based on the base address of the original image, the segment index of the current image segment to be written, the image segment length, and the width of the original image. Starting from the beginning address of the current image segment to be written, the data of each pixel in the current image segment to be written is sequentially and continuously written into the address range corresponding to the original image according to the preset address offset.

[0133] Optionally, the determining module 200 is specifically used to determine the frame index of the original image to be read and the segment index of the target image segment to be read in the original image corresponding to each sub-request in the data reading request; Based on the frame index of the original image to be read, query and obtain the mapping relationship between the segment index of each image segment in the original image to be read and the starting address of each image segment; Based on the segment index and mapping relationship of the target image segment to be read in the original image, determine the starting address of the target image segment to be read in the original image.

[0134] Optionally, the read cache module 300 is specifically used to determine the data length of the target image segment based on the image height and image width of the target image segment in the original image; Determine the address range corresponding to the original image based on the frame index of the original image; Starting from the address in the address range, read data of the specified length and use the read data as the image data of the target image segment in the original image.

[0135] Optionally, the read cache module 300 is specifically used to determine the image height of the target image segment based on the height of the original image, the image segment length, and the total number of image segments corresponding to the original image if the target image segment is the last image segment in the original image. The data length of the target image segment is determined based on the image height and image width of the target image segment in the original image.

[0136] Optionally, the fusion processing module 400 is specifically used to synchronously read at least a portion of the image data in the currently cached target image segment from the buffer corresponding to each exposure when the amount of data in the buffer corresponding to each exposure reaches the amount of data corresponding to at least one row of pixels in the target image segment.

[0137] Optionally, the acquisition module 100 is further configured to trigger a new data read request if the free capacity in the buffer corresponding to each exposure is greater than or equal to the complete data volume of an image segment. The new data read request is used to indicate the reading of image data of the next image segment of the target image segment in each original image of the target scene.

[0138] The above-described device is used to execute the method provided in the foregoing embodiments, and its implementation principle and technical effect are similar, so they will not be described again here.

[0139] These modules can be one or more integrated circuits configured to implement the above methods, such as one or more Application Specific Integrated Circuits (ASICs), one or more digital signal processors (DSPs), or one or more Field Programmable Gate Arrays (FPGAs). Alternatively, when a module is implemented using processing element scheduler code, the processing element can be a general-purpose processor, such as a Central Processing Unit (CPU) or other processor capable of calling program code. Furthermore, these modules can be integrated together as a system-on-a-chip (SOC).

[0140] The modules described above can be connected or communicate with each other via wired or wireless connections. Wired connections may include metal cables, optical fibers, hybrid cables, or any combination thereof. Wireless connections may include connections via LAN, WAN, Bluetooth, ZigBee, or NFC, or any combination thereof. Two or more units can be combined into a single module, and any module can be divided into two or more units. Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems and devices described above can be referred to the corresponding processes in the method embodiments, and will not be repeated here.

[0141] Figure 11 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The device can be a computing device with data processing function, and the electronic device integrates the aforementioned FPGA chip.

[0142] The device includes a processor 801, a storage medium 802, and a bus 803. The storage medium 802 stores program instructions that can be executed by the processor 801. When the electronic device is running, the processor 801 communicates with the storage medium 802 through the bus 803. The processor 801 executes the program instructions to implement the image data processing method as described in the embodiment.

[0143] The storage medium 802 stores program code, which, when executed by the processor 801, causes the processor 801 to perform various steps in the image data processing methods according to various exemplary embodiments of this application as described in the "Exemplary Methods" section above.

[0144] The processor 801 can be a general-purpose processor, such as a central processing unit (CPU), digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor.

[0145] Storage medium 802, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. The storage medium can include at least one type of storage medium, such as flash memory, hard disk, multimedia card, card-type storage medium, random access memory (RAM), static random access memory (SRAM), programmable read-only memory (PROM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), magnetic storage medium, magnetic disk, optical disk, etc. The storage medium is any other medium capable of carrying or storing desired program code in the form of instructions or data structures that can be accessed by a computer, but is not limited thereto. In the embodiments of this application, storage medium 802 can also be a circuit or any other device capable of implementing storage functions for storing program instructions and / or data.

[0146] Optionally, this application also provides a program product, such as a computer-readable storage medium, including a program that, when executed by a processor, performs the above-described method embodiments.

[0147] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0148] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0149] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in a combination of hardware and software functional units.

[0150] The integrated units implemented as software functional units described above can be stored in a computer-readable storage medium. These software functional units, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute some steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. An image data processing method, characterized in that, include: A data read request is obtained, which includes multiple sub-requests. Each sub-request is used to request the reading of image data of a target image fragment in the original image of the target scene at a certain exposure. Based on the data reading request, determine the frame index of each original image to be read and the starting address of the target image segment to be read in each original image; Based on the frame index of each original image and the starting address of the target image segment to be read in each original image, the image data of the target image segment in each original image is read from the external high-capacity double data rate synchronous dynamic random access memory DDR SDRAM, and cached into the buffer corresponding to each exposure inside the field programmable gate array FPGA respectively. When the amount of data in the buffer corresponding to each exposure level meets the preset fusion threshold, at least a portion of the image data in the cached target image fragments is read from the buffer corresponding to each exposure level, and image fusion is performed on at least a portion of the image data in the target image fragments of each original image to obtain the current fused image of the target scene.

2. The method according to claim 1, characterized in that, Before reading the image data of the target image segment from each original image from the external high-capacity double data rate synchronous dynamic random access memory (DDR SDRAM), the method further includes: The image segment length is determined based on the buffer capacity of the FPGA's internal buffer, the bus burst length of the DDR SDRAM, and the upper limit of the data processing volume for fusion processing. Based on the image segment length and along the preset image direction, the original images of the target scene acquired in real time at various exposures are divided into image segments to obtain the segment index and segment data of each image segment of each original image. Assign address ranges and base addresses to each original image; Based on the base address of each original image, the preset address offset, the priority of each image segment in each original image, and the segment index of each image segment, the segment data of each image segment of each original image are sequentially written into the address range corresponding to each original image in the DDRSDRAM.

3. The method according to claim 2, characterized in that, The step of writing the fragment data of each image segment of each original image into the address range corresponding to each original image in the DDR SDRAM according to the base address of each original image, the preset address offset, and the priority of each image segment in each original image includes: The image segment to be written is determined based on the base address of the original image and the priority of each image segment in the original image; The starting address of the image segment to be written is determined based on the base address of the original image, the segment index of the current image segment to be written, the image segment length, and the width of the original image. Starting from the beginning address of the current image segment to be written, the data of each pixel in the current image segment to be written is sequentially and continuously written into the address range corresponding to the original image according to the preset address offset.

4. The method according to claim 1, characterized in that, The step of determining the frame index of each original image to be read and the starting address of the target image segment to be read in each original image according to the data reading request includes: Determine the frame index of the original image to be read and the segment index of the target image segment to be read in the original image corresponding to each sub-request in the data reading request; Based on the frame index of the original image to be read, query and obtain the mapping relationship between the segment index of each image segment in the original image to be read and the starting address of each image segment; Based on the segment index of the target image segment to be read in the original image and the mapping relationship, the starting address of the target image segment to be read in the original image is determined.

5. The method according to claim 1, characterized in that, The step of reading image data of the target image segment from each original image based on the frame index of each original image and the starting address of the target image segment to be read in each original image includes: The data length of the target image segment is determined based on the image height and image width of the target image segment in the original image; The address range corresponding to the original image is determined based on the frame index of the original image; Starting from the starting address in the address range, read data of the specified length and use the read data as the image data of the target image segment in the original image.

6. The method according to claim 5, characterized in that, Determining the data length of the target image segment based on its image height and width in the original image includes: If the target image segment is the last image segment in the original image, then the image height of the target image segment is determined based on the height of the original image, the image segment length, and the total number of image segments corresponding to the original image. The data length of the target image segment is determined based on the image height and image width of the target image segment in the original image.

7. The method according to claim 1, characterized in that, When the amount of data in the buffer corresponding to each exposure level meets the preset fusion threshold, reading at least a portion of the image data from the currently cached target image segment from the buffer corresponding to each exposure level includes: When the amount of data in the buffer corresponding to each exposure reaches the amount of data corresponding to at least one row of pixels in the target image segment, at least a portion of the image data in the currently cached target image segment is synchronously read from the buffer corresponding to each exposure.

8. The method according to claim 1, characterized in that, Also includes: If the free capacity in the buffer corresponding to each exposure is greater than or equal to the complete data volume of an image segment, a new data read request is triggered. The new data read request is used to indicate the reading of image data of the next image segment of the target image segment in each original image of the target scene.

9. An electronic device, characterized in that, include: The device includes a processor, a storage medium, and a bus, wherein the storage medium stores program instructions executable by the processor, and when the electronic device is running, the processor communicates with the storage medium via the bus, and the processor executes the program instructions to implement the method as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, The storage medium stores a computer program that is executed by a processor to implement the method as described in any one of claims 1 to 8.

Images